Writing study: Train-before-Test Harmonizes Language Model Rankings

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Guanhua Zhang, Ricardo Dominguez-Olmedo, Moritz Hardt · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Datasets, benchmarks & measurement resources.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

33 physical pages

Before references / appendix

≈ 9.56 page-equivalents

Main figures / tables

8 / 2

Attached appendix figures / tables

7 / 6

Attached appendix span

≈ 15.89 page-equivalents

Before references / appendix: ≈ 28.97%References / intervening material: ≈ 22.88%Attached appendix: ≈ 48.15%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.71 pp.	5.18%
2 RELATED WORK	3	≈ 1.03 pp.	3.12%
3 EXPERIMENTS	4	≈ 5.28 pp.	16%
4 DISCUSSION, LIMITATIONS, AND CONCLUSION	9	≈ 0.92 pp.	2.79%
REFERENCES	10	≈ 7.55 pp.	22.88%
A ADDITIONAL EXPERIMENT SETTING	18	≈ 2.46 pp.	7.45%
B ADDITIONAL EXPERIMENT RESULTS	20	≈ 7.54 pp.	22.85%
C ACCOUNTING FOR STATISTICAL SIGNIFICANCE	28	≈ 5 pp.	15.15%
D BROADER IMPACTS	33	≈ 0.11 pp.	0.33%
E REPRODUCIBILITY STATEMENT	33	≈ 0.08 pp.	0.24%
F THE USE OF LARGE LANGUAGE MODELS	33	≈ 0.71 pp.	2.15%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 4
• Table 2 · p. 5
• Figure 2 · p. 5
• Figure 3 · p. 6
• Figure 4 · p. 7
• Figure 5 · p. 7
• Figure 6 · p. 8
• Figure 7 · p. 8
• Figure 8 · p. 9

Attached appendix

• Figure 9 · p. 20
• Figure 10 · p. 21
• Table 3 · p. 22
• Figure 11 · p. 23
• Table 4 · p. 24
• Table 5 · p. 25
• Table 6 · p. 26
• Table 7 · p. 27
• Table 8 · p. 27
• Figure 12 · p. 30
• Figure 13 · p. 30
• Figure 14 · p. 31
• Figure 15 · p. 32

Argument and organization

1. Performance versus potential

   Conflicting rankings motivate a controlled adaptation-based evaluation rather than a new model.

2. Three consequences

   Cross-task agreement leads to perplexity agreement and then score-matrix PCA, followed by scope and access limitations.

Problem, claim and contribution

The performance-versus-potential distinction gives the study a clear estimand: adaptation under a common recipe, not deployment-ready capability. The conclusion explicitly keeps direct evaluation as a complementary tool.

Sections 1 and 4. · Main paper

Argument and section order

The contributions are mirrored by downstream ranking, perplexity and PCA subsections. This creates a simple three-claim empirical narrative without introducing a complicated framework.

Section 1.1 and Sections 3.2–3.4. · Main paper

Figures, captions and visual explanation

Figure 1 uses aligned ranking flows to make disagreement visible. Its caption discloses greedy reordering within confidence constraints; the apparent straightening is not an unprocessed ranking plot.

Page 2, Figure 1; Appendix C.1 and Algorithms 1–3. · Main paper

Matched PCA bar charts show how explained variance shifts between evaluation protocols. Their common scale and printed values are effective, but 86% explained variance is not an exactly rank-one matrix or proof of one universal capability.

Page 8, Figure 6; page 9, Figure 8. · Main paper

Tables, comparisons and evidence

The headline average Kendall correlation rises from 0.52 to 0.76, but same-size bins have substantially weaker agreement. The appendix preserves this important model-selection qualification rather than reporting only the pooled trend.

Page 5, Figure 2; page 26, Table 6. · Main paper

Appendix and reproducibility

The setup discloses benchmark exclusions, a validation-selected three-learning-rate LoRA search, template choice and fallback to pre-finetuning scores. These choices define the actual standardized protocol and its selection effects.

Appendix A.1–A.2, pages 18–19. · Main paper

What to reuse—and what not to infer

Transferable lesson: Define what a comparison estimates, map each claim to a repeated visual grammar, and report stratified checks that reveal where a pooled trend weakens.

Boundary: The 61-model, mainly multiple-choice, up-to-14B study does not prove rankings transfer to every new task or larger model. Benchmark exclusions depend partly on adaptation behavior; equal recipe is not equal compute or optimal adaptation. The main text says four Gemma exclusions while Appendix Table 3 lists eight; cite the explicit experimental subset carefully.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–4; references through page 17.

Appendix / supplement

Pages 18–33, A training and selection details, B additional ranking/PCA experiments, C significance-aware rankings and algorithms, D–F impact, reproducibility and LLM use.

Visually inspected PDF pages

2, 8, 26

• ICLR 2026 final proceedings · 33 pages