Writing study: Pre-training under infinite compute

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Learning & optimization algorithms.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

41 physical pages

Before references / appendix

≈ 10.11 page-equivalents

Main figures / tables

9 / 0

Attached appendix figures / tables

16 / 12

Attached appendix span

≈ 19.84 page-equivalents

Before references / appendix: ≈ 24.66%References / intervening material: ≈ 26.95%Attached appendix: ≈ 48.39%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.57 pp.	3.83%
2 STANDARD PRE-TRAINING	3	≈ 1.09 pp.	2.66%
3 REGULARIZED PARAMETER SCALING	4	≈ 0.56 pp.	1.37%
4 ENSEMBLE SCALING	4	≈ 1.58 pp.	3.85%
5 SCALING THE SEED TOKEN COUNT UNDER INFINITE COMPUTE	6	≈ 1.33 pp.	3.24%
6 DATA EFFICIENCY UNDER PARAMETER CONSTRAINTS	7	≈ 0.91 pp.	2.22%
7 DOWNSTREAM TASKS	8	≈ 0.67 pp.	1.63%
8 RELATED WORK	9	≈ 0.57 pp.	1.39%
9 DISCUSSION	9	≈ 0.28 pp.	0.68%
10 ACKNOWLEDGEMENTS	10	≈ 0.18 pp.	0.44%
11 ETHICS	10	≈ 0.09 pp.	0.22%
12 REPRODUCIBILITY	10	≈ 0.72 pp.	1.76%
REFERENCES	11	≈ 11.06 pp.	26.98%
A CONTINUED PRE-TRAINING	22	≈ 0.5 pp.	1.22%
B PROBLEM SETTING	22	≈ 0.85 pp.	2.07%
C STANDARD PRE-TRAINING DETAILS	23	≈ 4.06 pp.	9.9%
D ENSEMBLING DETAILS	27	≈ 4.65 pp.	11.34%
E DATA SCALING	32	≈ 1.07 pp.	2.61%
F DISTILLATION DETAILS	33	≈ 1.43 pp.	3.49%
G DOWNSTREAM TASK DETAILS	34	≈ 0.97 pp.	2.37%
H CONTINUED PRE-TRAINING	35	≈ 0.71 pp.	1.73%
I POWER LAWS	36	≈ 0.45 pp.	1.1%
J ADDITIONAL RELATED WORK	36	≈ 1.25 pp.	3.05%
K UPDATES DURING REBUTTALS	38	≈ 3.89 pp.	9.49%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 3
• Figure 3 · p. 4
• Figure 4 · p. 5
• Figure 5 · p. 6
• Figure 6 · p. 6
• Figure 7 · p. 7
• Figure 8 · p. 8
• Figure 9 · p. 8

Attached appendix

• Table 1 · p. 22
• Table 2 · p. 23
• Figure 10 · p. 24
• Figure 11 · p. 25
• Figure 12 · p. 26
• Figure 13 · p. 26
• Figure 14 · p. 27
• Figure 15 · p. 28
• Figure 16 · p. 29
• Figure 17 · p. 30
• Figure 18 · p. 33
• Table 3 · p. 33
• Figure 19 · p. 34
• Table 4 · p. 34
• Table 5 · p. 35
• Table 6 · p. 36
• Table 7 · p. 36
• Figure 20 · p. 37
• Table 8 · p. 38
• Table 9 · p. 38
• Table 10 · p. 38
• Figure 3 · p. 39
• Figure 4 · p. 39
• Figure 5 · p. 39
• Table 11 · p. 39
• Figure 21 · p. 39
• Figure 22 · p. 40
• Table 12 · p. 41

Argument and organization

1. Introduction / Data-constrained pre-training

   Changes the resource constraint before defining experiments and failure cases.

2. Scaling parameters / ensembles / data

   Builds progressively nested scaling estimates from observed runs.

3. Distillation / Downstream tasks / Discussion

   Tests compression and capability implications of the data-efficiency findings.

4. Appendices A–K

   Separates practical transfer, optimization details, sensitivity tests and rebuttal additions.

Problem, claim and contribution

The title names an idealized resource regime, while the introduction contrasts fixed-data optimization with compute-optimal training. This makes additional training cost an intentional experimental tradeoff rather than an omitted efficiency metric.

§§1–2, pp.1–3 · Main paper

Argument and section order

The argument moves from failed naive repetition/scaling to regularization, independent ensembles and finally distillation. Each intervention answers a limitation established immediately before it, giving the long empirical study a causal narrative without claiming a new universal theorem.

§§2–6, pp.3–8 · Main paper

Figures, captions and visual explanation

Figure 8 distinguishes finite measured models from fitted dashed asymptotes, using arrows to show teacher-to-student compression. Its parameter-count axis makes deployment size visible while leaving training compute a separate question.

Figure 8, p.8 · Main paper

Tables, comparisons and evidence

The matched-token self-distillation ablation isolates whether retaining real data matters. Its explicit no-mixing comparison supports a narrower and stronger conclusion than a general statement that synthetic training avoids collapse.

Appendix F.3 and Table 4, p.34 · Main paper

Appendix and reproducibility

Appendix F reveals sampling from individual teachers, batch mixing ratios and repeated synthetic data caused by generation limits. These details materially qualify the simplified main-text ensemble-distillation description and are essential for a faithful method summary.

Appendices F.1–F.2, p.33 · Main paper

Clarity and writing conventions

The sensitivity section calls asymptotes rough estimates and admits that nested fits were not stress-tested. Rebuttal additions then test larger, aspect-ratio-corrected models, improving credibility without converting extrapolation into certainty.

Appendix I.1, p.36; Appendices K.1–K.3, pp.38–39 · Main paper

What to reuse—and what not to infer

Transferable lesson: Distinguish observed points, fitted curves and asymptotic predictions visually; connect each methodological addition to a demonstrated failure; use a matched-budget ablation for the key mechanism.

Boundary: The headline infinite-compute/data-efficiency factors are fitted limits, not achieved training runs. Local grid search is not certified global optimality. Final Appendix K re-evaluations differ slightly from Table 5; quote one named table consistently. Eight-times-smaller inference parameter count is not an eight-times reduction in training cost.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

PDF pp.1–10, §§1–9, ethics and reproducibility; references pp.11–19

Appendix / supplement

PDF pp.20–41, appendix contents and Appendices A–K, including final rebuttal updates and the landscape seven-task results table

Visually inspected PDF pages

8, 33, 34, 35

• official ICLR 2026 proceedings PDF · 41 pages