Writing study: How Reliable is Language Model Micro-Benchmarking?

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Gregory Yauney, Shahzaib Warraich, Swabha Swayamdipta · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Datasets, benchmarks & measurement resources.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

24 physical pages

Before references / appendix

≈ 10.24 page-equivalents

Main figures / tables

6 / 1

Attached appendix figures / tables

12 / 4

Attached appendix span

≈ 10.89 page-equivalents

Before references / appendix: ≈ 42.67%References / intervening material: ≈ 11.96%Attached appendix: ≈ 45.38%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.48 pp.	6.17%
2 MICRO-BENCHMARKING PRELIMINARIES	3	≈ 1 pp.	4.17%
3 MDAD: A META-EVALUATION FOR MICRO-BENCHMARK RELIABILITY	4	≈ 1.27 pp.	5.29%
4 EXPERIMENTAL DESIGN	5	≈ 1.36 pp.	5.67%
5 RESULTS	6	≈ 3.72 pp.	15.5%
6 DISCUSSION AND CONCLUSION	10	≈ 0.36 pp.	1.5%
REPRODUCIBILITY STATEMENT	10	≈ 0.28 pp.	1.17%
ACKNOWLEDGMENTS	11	≈ 0.13 pp.	0.54%
REFERENCES	11	≈ 2.87 pp.	11.96%
A ADDITIONAL RELATED WORK	14	≈ 0.61 pp.	2.54%
B DETAILS OF MICRO-BENCHMARKING METHODS	14	≈ 0.6 pp.	2.5%
C DIFFERENT AGREEMENT THRESHOLDS FOR MDAD	15	≈ 0.8 pp.	3.33%
D FULL EXPERIMENTAL SETUP AND COMPUTING INFRASTRUCTURE	16	≈ 0.41 pp.	1.71%
E QUANTITATIVE ERROR ANALYSIS OF MDAD	16	≈ 0.59 pp.	2.46%
F DIFFERENT BUCKET RESOLUTIONS FOR MDAD	17	≈ 1 pp.	4.17%
G FULL RESULTS	18	≈ 1 pp.	4.17%
H MDAD CAN ANALYZE METHODS THAT SELECT ENTIRE SUBTASKS	19	≈ 0.26 pp.	1.08%
I GENERALIZING TO NEW TASK DRAWS WHEN SELECTING FROM ENTIRE	19	≈ 0.74 pp.	3.08%
J GENERALIZING TO NEW TASK DRAWS WHEN SELECTING FROM SUBTASKS	20	≈ 1 pp.	4.17%
K INCREASING NUMBER OF SOURCE MODELS HAS MODEST EFFECT	21	≈ 1 pp.	4.17%
L COMPARING SPECIFIC SIZES OF MODELS	22	≈ 2.89 pp.	12.04%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Table 1 · p. 5
• Figure 3 · p. 6
• Figure 4 · p. 8
• Figure 5 · p. 9
• Figure 6 · p. 10

Attached appendix

• Figure 7 · p. 15
• Table 2 · p. 16
• Table 3 · p. 16
• Figure 8 · p. 17
• Figure 9 · p. 18
• Figure 10 · p. 18
• Table 4 · p. 19
• Figure 11 · p. 19
• Figure 12 · p. 20
• Figure 6 · p. 20
• Table 5 · p. 20
• Figure 13 · p. 21
• Figure 14 · p. 22
• Figure 15 · p. 22
• Figure 16 · p. 23
• Figure 17 · p. 24

Argument and organization

1. A practical ranking question before a metric

   The introduction motivates pairwise ranking reliability; preliminaries separate pointwise, aggregate and conditional comparisons.

2. Metric, controlled comparisons, practitioner decisions

   MDAD is illustrated before benchmark sweeps, same-size-model examples and held-out-set checks.

Problem, claim and contribution

The opening asks whether a tiny subset can distinguish two models, rather than merely reconstruct a leaderboard. This gives the proposed metric a concrete decision target.

Pages 1–2, Section 1. · Main paper

Argument and section order

Table 1 compares metrics by measured quantity, comparison unit and aggregation. Subsequent subsections answer distinct questions instead of repeating one overall superiority claim.

Page 5, Table 1; Sections 5.1–5.4. · Main paper

Figures, captions and visual explanation

Figure 1 contrasts aggregate rank correlation above conditional agreement curves, with a shaded difficult region. Figure 2 reuses lettered threshold crossings in its summary panel, making the curve-to-MDAD reduction inspectable.

Page 2, Figure 1; page 4, Figure 2. · Main paper

Tables, comparisons and evidence

The 32-model, 8B MMLU-Pro case ties the distribution of accuracy gaps to reliability: 51% of pairs lie within five points. That is a statement about comparisons below the threshold, not a measured 51% reversal rate.

Page 9, Figure 5 and Section 5.3. · Main paper

Appendix and reproducibility

Thresholds, bucket widths, trial counts and held-out draws receive separate sensitivity checks. Table 2 includes selection runtime; Table 3 exposes uncertainty rather than hiding it behind the 50-trial choice.

Appendices C–F and I–L; page 16, Tables 2–3. · Main paper

Clarity and writing conventions

The discussion preserves a legitimate coarse-screening use for tiny benchmarks. It distinguishes evaluating existing selectors from releasing an allegedly universal best subset.

Page 6, experimental-design scope; page 10, Section 6. · Main paper

What to reuse—and what not to infer

Transferable lesson: When introducing a metric, visualize the decision it supports, compare its unit of analysis with existing metrics, and test the parameters that could change its conclusion.

Boundary: MDAD is an empirical, binned agreement threshold for the sampled model population, not a significance test or guarantee for every pair. Classification-only cached predictions and random splits do not establish robustness to future model families; 250 examples is not a universal rule.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–6 and reproducibility statement; references on pages 11–13.

Appendix / supplement

Pages 14–24, Appendices A–L: related work/method implementations, agreement thresholds, infrastructure, trial and bin sensitivity, full results, held-out task draws and model-size strata.

Visually inspected PDF pages

2, 4, 16

• ICLR 2026 final proceedings · 24 pages