Writing study: Bridging Compute- and Data-Optimal Pretraining

NeurIPS overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← NeurIPS 2026 oral writing corpus

Tian Qin, Kimia Hamidieh, David Alvarez-Melis · Official NeurIPS 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

19 physical pages

Before references / appendix

≈ 11.09 page-equivalents

Main figures / tables

8 / 1

Attached appendix figures / tables

4 / 6

Attached appendix span

≈ 4.91 page-equivalents

Before references / appendix: ≈ 58.37%References / intervening material: ≈ 15.79%Attached appendix: ≈ 25.84%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Top-level heading and reference/appendix boundaries checked against source PDF text. Bibliography titles/author initials, numbered contribution/property bullets, diagram steps, equations and TOC entries are not substantive headings. Section spans are page-coordinate estimates, not ink area or prose-only length. This is an author manuscript mirror, not a verified accepted-version template. Source layout uses physical single-column y/page-height, including equation-heavy pages.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 Introduction	1	≈ 1.64 pp.	8.63%
2 Related Work	3	≈ 0.88 pp.	4.63%
3 Methodology and Experiment Setup	4	≈ 0.74 pp.	3.89%
4 Parametric Form of CD-Scaling Law	4	≈ 3.83 pp.	20.16%
5 CD-Scaling Laws Bridge Compute- and Data-Optimal Training	8	≈ 2.14 pp.	11.26%
6 Conclusion & Discussion	10	≈ 0.95 pp.	5%
Acknowledgments	11	≈ 0.34 pp.	1.79%
References	12	≈ 3 pp.	15.79%
A Implementation Details and Ablations for the CD-Scaling Fit	15	≈ 2 pp.	10.53%
B Additional Experiment Details and Results	17	≈ 2.91 pp.	15.32%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 5
• Table 1 · p. 6
• Figure 3 · p. 6
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 9
• Figure 7 · p. 10
• Figure 8 · p. 11

Attached appendix

• Table 2 · p. 15
• Table 3 · p. 16
• Table 4 · p. 16
• Table 5 · p. 16
• Table 6 · p. 17
• Figure 9 · p. 17
• Figure 10 · p. 18
• Figure 11 · p. 18
• Figure 12 · p. 19
• Table 7 · p. 19

Argument and organization

1. 1 Introduction

   Identifies a data-supply limitation of compute-optimal scaling.

2. 2 Related Work

   Positions fresh-token and repeated-data scaling work.

3. 3 Methodology and Experiment Setup

   Defines effective derived tokens and controlled pretraining setup.

4. 4 Parametric Form of CD-Scaling Law

   Moves from empirical regularities to fitting, ablations and held-out validation.

5. 5 CD-Scaling Laws Bridge Compute- and Data-Optimal Training

   Derives resource-allocation regimes and practical strategy maps.

6. 6 Conclusion & Discussion

   Summarizes the fit and its extrapolation/data-strategy limitations.

Problem, claim and contribution

The introduction identifies the fresh-data assumption in classic scaling laws and makes derived-token effectiveness the missing quantity. Figure 1 previews both the resource trade-off and the new explanatory variable before the fitting procedure.

PDF pp.1–2, Introduction and Figure 1 · Main paper

Argument and section order

The paper separates empirical behavior of effectiveness, a motivated functional form, parameter fitting, held-out scale validation, downstream transfer, and practical consequences. This order helps distinguish a measured regularity from the modeling assumptions used to extrapolate it.

PDF pp.4–11, Sections 3–6 · Main paper

Figures, captions and visual explanation

Table 1 reports fitted constants with confidence intervals and source-specific fit error. Figure 3 compares fresh-token and effective-token axes, while Figure 8 converts the fitted surface into strategy-region and recommended-compute heatmaps.

PDF p.6, Table 1 and Figure 3; p.11, Figure 8 · Main paper

Tables, comparisons and evidence

The study separates in-fit residuals, held-out larger-model prediction, functional-form ablations, and downstream evaluations. Its limitations explain that effectiveness can absorb baseline misspecification and that only repetition and paraphrase were fitted at small-to-medium model sizes.

PDF pp.7–8, Sections 4.4–4.5; p.11, Limitations · Main paper

Appendix and reproducibility

The appendix exposes fitting details, outlier trimming, alternative forms, term ablations, held-out-scale errors, per-task downstream results, and training settings. Readers can inspect how robust the practical recommendation is to a modeling choice rather than seeing only the final heatmap.

PDF pp.15–19, Appendices A–B and Tables 2–7 · Main paper

Clarity and writing conventions

Captions distinguish fitted lines, confidence bands, and overlapping intervals. The practical heatmap is a derived recommendation, whereas the text explicitly identifies the measured model-size range and untested data-expansion strategies.

PDF p.7, Figure 4; p.11, Figure 8 and Limitations · Main paper

What to reuse—and what not to infer

Transferable lesson: For scaling-law writing, show the raw explanatory trend, fitted transformation, held-out test, and decision map as separate evidential steps.

Boundary: Figure 8 includes model sizes beyond the stated 14M–600M training range; those cells are fitted extrapolations, not direct training measurements. Confidence intervals do not remove model misspecification, and the author manuscript is not confirmed identical to the accepted revision.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-02

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented source scan of framing, argument roadmap, selected substantive passages, method/theorem statements, main figure/table captions, evidence and limits across PDF pp.1–11. Two selected main-body pages were actually inspected as rendered images. This is not exhaustive line-by-line technical verification or experiment reproduction. Title and all author names compared against official program metadata and manuscript first page; matching authors, with capitalization, middle initials or expanded given names where applicable.

Appendix / supplement

Fitting/validation section openings, trimming and ablation table captions, downstream-result captions, and training-detail headings were inspected.

Visually inspected PDF pages

6, 11

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Public author manuscript arXiv:2607.25271v1; not verified identical to the accepted OpenReview revision · 19 pages