Writing study: AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Jonathan Bragg, Mike D'Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi Mishra, Sergey Feldman, Dany Haddad, Jena Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefanus Candra, Jason Dunkelberger, Daniel Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Ngoc-Uyen Nguyen, Smita Rao, Amber Tanaka, Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal, Daniel Weld · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

88 physical pages

Before references / appendix

≈ 11.11 page-equivalents

Main figures / tables

2 / 3

Attached appendix figures / tables

8 / 20

Attached appendix span

≈ 72.49 page-equivalents

Before references / appendix: ≈ 12.63%References / intervening material: ≈ 5%Attached appendix: ≈ 82.38%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2.39 pp.	2.72%
2 RELATED WORK	4	≈ 1.01 pp.	1.15%
3 ASTABENCH: A HOLISTIC SCIENTIFIC RESEARCH BENCHMARK SUITE	5	≈ 0.44 pp.	0.5%
4 ASTA ENVIRONMENT	5	≈ 1.75 pp.	1.99%
5 EXPERIMENTS	7	≈ 2.95 pp.	3.35%
6 CONCLUSION AND FUTURE WORK	10	≈ 0.41 pp.	0.47%
ETHICS STATEMENT	10	≈ 0.3 pp.	0.34%
REPRODUCIBILITY STATEMENT	11	≈ 0.13 pp.	0.15%
AUTHOR CONTRIBUTIONS	11	≈ 0.58 pp.	0.66%
ACKNOWLEDGMENTS	11	≈ 0.29 pp.	0.33%
REFERENCES	12	≈ 4.4 pp.	5%
A PRINCIPLES FOR BENCHMARKING AGENTS	16	≈ 0.8 pp.	0.91%
B EVALUATION TOOLKIT: OPENNESS AND TOOLING	17	≈ 1.27 pp.	1.44%
C SUPPORTING EXPERIMENTAL RESULTS	18	≈ 2.92 pp.	3.32%
D FULL EXPERIMENTAL RESULTS	21	≈ 10.61 pp.	12.06%
E EVALUATIONS	32	≈ 10.1 pp.	11.48%
F AGENTS	42	≈ 11.54 pp.	13.11%
G ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS	53	≈ 1.64 pp.	1.86%
H EVALUATION TASK SAMPLES AND PROMPTS	55	≈ 33.61 pp.	38.19%

All detected figure and table locations

Main text

• Table 1 · p. 3
• Figure 1 · p. 3
• Table 2 · p. 6
• Table 3 · p. 8
• Figure 2 · p. 8

Attached appendix

• Table 4 · p. 18
• Table 5 · p. 18
• Table 6 · p. 19
• Table 7 · p. 19
• Table 8 · p. 20
• Table 9 · p. 20
• Table 10 · p. 20
• Table 11 · p. 21
• Figure 3 · p. 22
• Figure 4 · p. 23
• Table 12 · p. 24
• Figure 5 · p. 24
• Table 13 · p. 25
• Table 14 · p. 26
• Table 15 · p. 27
• Figure 6 · p. 28
• Table 16 · p. 29
• Figure 7 · p. 29
• Table 17 · p. 30
• Figure 8 · p. 31
• Table 18 · p. 38
• Table 19 · p. 41
• Table 20 · p. 41
• Figure 9 · p. 47
• Figure 10 · p. 53
• Table 21 · p. 53
• Table 22 · p. 54
• Table 23 · p. 55

Argument and organization

1. Evaluation gaps become design requirements

   Five motivating gaps map to the benchmark suite, shared tools, agents and cost-aware leaderboard.

2. Results with a layered audit trail

   Main cost–quality plots lead to complete score tables, evaluator validation and executable-interface examples.

Problem, claim and contribution

The introduction links each evaluation deficiency to a concrete contribution; Table 1 includes partial-coverage labels instead of treating all comparator suites as uniformly inadequate.

Pages 1–3, introduction and Table 1. · Main paper

Argument and section order

Step completion is explicitly distinguished from whole-task completion: Table 20 reports at most 0.05 complete success despite much higher step scores. That denominator change prevents an inflated autonomy claim.

Page 41, Appendix E.9 and Table 20; Table 10. · Main paper

Figures, captions and visual explanation

Figure 1 depicts the agent–tools–task–judge loop. Figure 2 uses logarithmic cost axes and Pareto lines across overall and category panels; its crowded legend is a readability tradeoff.

Page 3, Figure 1; page 8, Figure 2. · Main paper

Tables, comparisons and evidence

Table 4 pairs quality and cost, with bold identifying Pareto-optimal pairs rather than accuracy winners alone. Asta v0 scores 53.0 at $3.40 versus ReAct GPT-5's 44.0 at $0.31 under the paper's pricing snapshot.

Page 18, Table 4; Section 4 and Appendix D. · Main paper

Appendix and reproducibility

Task definitions, model identifiers, source-code revisions, routing rules and prompts are discoverable through separate appendices. Figure 10 and Table 21 expose report/code/artifact disagreement rather than trusting polished reports.

Appendices E–H, pages 32–88; page 53, Figure 10/Table 21. · Main paper

Clarity and writing conventions

The evaluator discussion reports rubric-source bias and only moderate overall human agreement. Citation scoring explicitly does not verify that quoted snippets occur in real sources—an essential qualification for the word rigorous.

Pages 36–37, Appendix E.3; pages 52–53, Appendix F.9. · Main paper

What to reuse—and what not to infer

Transferable lesson: Make an evaluation paper auditable in layers: motivation-to-design mapping, cost–quality comparisons, exact aggregation rules, evaluator stress tests, then reproducible task and prompt specifications.

Boundary: This is a CS-weighted scientific-agent suite, not proof of autonomous discovery. Custom tools, cached agents, historical API prices, model drift and LLM-judge weaknesses limit like-for-like comparisons; no benchmark runs were replicated here.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–6 and ethics/reproducibility statements; references on pages 12–16.

Appendix / supplement

Pages 16–88, Appendices A–H: design principles, tools/openness, full results/statistical aggregation, all eleven task protocols, agent implementations/routing, evaluator validation, and complete task/prompt/rubric examples.

Visually inspected PDF pages

3, 8, 18, 53

• ICLR 2026 final proceedings · 88 pages