Writing study: Benchmarking at the Edge of Comprehension

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Samuele Marro, Jialin Yu, Emanuele La Malfa, Oishi Deb, Jiawei Li, Yibo Yang, Ebey Abraham, Sunando Sengupta, Eric Sommerlade, Michael J. Wooldridge, Philip Torr · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

33 physical pages

Before references / appendix

≈ 9.39 page-equivalents

Main figures / tables

3 / 2

Attached appendix figures / tables

15 / 10

Attached appendix span

≈ 20.91 page-equivalents

Before references / appendix: ≈ 28.45%References / intervening material: ≈ 8.18%Attached appendix: ≈ 63.36%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.42 pp.	4.3%
2. Benchmarking Beyond Human Capability	3	≈ 1.22 pp.	3.7%
3. Critique-Resilient Correctness	4	≈ 0.92 pp.	2.79%
4. The Protocol	5	≈ 0.53 pp.	1.61%
5. From Outcomes to Rankings	5	≈ 0.98 pp.	2.97%
6. Experiments	6	≈ 1.13 pp.	3.42%
7. Results	7	≈ 0.99 pp.	3%
8. Related Work	8	≈ 0.6 pp.	1.82%
9. Conclusion	9	≈ 0.47 pp.	1.42%
Impact Statement	9	≈ 0.44 pp.	1.33%
Acknowledgments	10	≈ 0.07 pp.	0.21%
References	10	≈ 2.7 pp.	8.18%
A. Why Benchmarking Breaks in the Post-Comprehension Regime	13	≈ 1 pp.	3.03%
B. Unipartite vs. Bipartite vs. Tripartite Decomposition	14	≈ 0.63 pp.	1.91%
C. Sample Questions	14	≈ 1.93 pp.	5.85%
D. Full Experimental Setup	16	≈ 2.06 pp.	6.24%
E. Additional Results	18	≈ 3.96 pp.	12%
F. Additional Analyses	22	≈ 1.84 pp.	5.58%
G. Hyperparameter Selection via Empirical Bayes	24	≈ 3.3 pp.	10%
H. Uncertainty via Bootstrap	27	≈ 6.21 pp.	18.82%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 7
• Table 2 · p. 7
• Figure 2 · p. 8
• Figure 3 · p. 8

Attached appendix

• Figure 4 · p. 19
• Figure 5 · p. 20
• Figure 6 · p. 20
• Figure 7 · p. 21
• Figure 8 · p. 21
• Table 3 · p. 22
• Table 4 · p. 23
• Table 5 · p. 23
• Table 6 · p. 24
• Table 7 · p. 24
• Table 8 · p. 25
• Table 9 · p. 25
• Figure 9 · p. 26
• Table 10 · p. 26
• Table 11 · p. 27
• Table 12 · p. 27
• Figure 10 · p. 28
• Figure 11 · p. 29
• Figure 12 · p. 29
• Figure 13 · p. 30
• Figure 14 · p. 30
• Figure 15 · p. 31
• Figure 16 · p. 32
• Figure 17 · p. 32
• Figure 18 · p. 33

Argument and organization

1. Problem and contribution

   Defines surviving bounded critique as an operational replacement for unavailable ground truth, explicitly acknowledging that failure to find an error is not proof of global correctness.

2. Argument sequence

   The argument proceeds from domain assumptions and admissible claims to protocol outcomes, eligible episodes and the rating model before presenting research questions.

3. Evidence and verification

   Validation asks separately whether scores are stable/externally aligned and whether weaker adjudicators preserve rankings; human expertise and adjudication effort are reported.

Problem, claim and contribution

Defines surviving bounded critique as an operational replacement for unavailable ground truth, explicitly acknowledging that failure to find an error is not proof of global correctness.

Introduction, p. 2; limitations, p. 9 · Main paper

Argument and section order

The argument proceeds from domain assumptions and admissible claims to protocol outcomes, eligible episodes and the rating model before presenting research questions.

Sections 2-6, pp. 3-7 · Main paper

Figures, captions and visual explanation

The protocol diagram separates dropped items from both win outcomes and routes only wins/losses into the score; the results page places correlation intervals beside protocol details.

Figure 1, p. 2; Tables 1-2, p. 7; both PNGs inspected · Main paper

Tables, comparisons and evidence

Validation asks separately whether scores are stable/externally aligned and whether weaker adjudicators preserve rankings; human expertise and adjudication effort are reported.

Sections 6.1-7, pp. 6-8 · Main paper

Appendix and reproducibility

The uncertainty recipe resamples complete question clusters, not independent edges, preserving dependence from shared item difficulty; Elo conversion is identified as presentation only.

Appendix H, pp. 27-28 · Main paper

Clarity and writing conventions

Interpretation limits accompany the latent scores: benchmarker strength is conditional on this interaction protocol, not an intrinsic general capability.

Section 5.3, p. 6 · Main paper

What to reuse—and what not to infer

Transferable lesson: Define exactly what a novel metric means, what is excluded from its denominator, and how dependencies enter uncertainty estimates.

Boundary: Critique resilience remains relative to critic capability and budget; it must not be relabeled verified mathematical correctness.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-9, pp. 1-9: operational correctness, protocol, rating model, experiment questions, validation and limitations; impact scope continues p. 10.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-H, pp. 13-33: assumptions, model identifiability, example tasks, setup, additional analyses, empirical Bayes and bootstrap specification, plus prompt figures.

Visually inspected PDF pages

2, 7

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 33 pages