Writing study: Midtraining Bridges Pretraining and Posttraining Distributions

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Emmy Liu, Graham Neubig, Chenyan Xiong · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

26 physical pages

Before references / appendix

≈ 8.57 page-equivalents

Main figures / tables

6 / 4

Attached appendix figures / tables

7 / 5

Attached appendix span

≈ 12.91 page-equivalents

Before references / appendix: ≈ 32.96%References / intervening material: ≈ 17.38%Attached appendix: ≈ 49.65%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Wrapped main heading labels restored from the immediately following same-size source line in the physical PDF. Heading start positions are unchanged. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.89 pp.	3.42%
2. Preliminaries	2	≈ 1.14 pp.	4.38%
3. Experimental Setting	3	≈ 0.98 pp.	3.77%
4. Which downstream tasks benefit most from midtraining?	4	≈ 0.41 pp.	1.58%
5. What data is most effective for midtraining?	4	≈ 2.05 pp.	7.88%
6. When and how much midtraining data should be introduced?	6	≈ 0.82 pp.	3.15%
7. How does midtraining change model representations?	7	≈ 0.49 pp.	1.88%
8. Related Work	8	≈ 0.97 pp.	3.73%
9. Conclusion	9	≈ 0.19 pp.	0.73%
Impact Statement	9	≈ 0.11 pp.	0.42%
Acknowledgements	9	≈ 0.17 pp.	0.65%
References	9	≈ 4.52 pp.	17.38%
A. Theoretical Analysis	14	≈ 3.21 pp.	12.35%
B. Pretraining Settings	17	≈ 0.29 pp.	1.12%
C. Posttraining Settings	17	≈ 0.27 pp.	1.04%
D. Dataset Similarity Matrix	17	≈ 0.6 pp.	2.31%
E. SFT in-domain loss and C4 Losses after Finetuning for 70m and 160m models	18	≈ 2.3 pp.	8.85%
F. Representative training loss curves for midtrained vs. base models	20	≈ 0.06 pp.	0.23%
G. Additional CKA results on APPS	20	≈ 0.06 pp.	0.23%
H. CKA results on C4	20	≈ 5.22 pp.	20.08%
I. Statement on LLM Usage	26	≈ 0.91 pp.	3.5%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Table 1 · p. 3
• Table 2 · p. 5
• Table 3 · p. 5
• Figure 2 · p. 5
• Figure 3 · p. 6
• Table 4 · p. 6
• Figure 4 · p. 6
• Figure 5 · p. 7
• Figure 6 · p. 8

Attached appendix

• Table 5 · p. 17
• Table 6 · p. 17
• Table 7 · p. 18
• Table 8 · p. 18
• Table 9 · p. 19
• Figure 7 · p. 19
• Figure 8 · p. 21
• Figure 9 · p. 21
• Figure 10 · p. 22
• Figure 11 · p. 23
• Figure 12 · p. 24
• Figure 13 · p. 25

Argument and organization

1. Problem and claim

   Defines midtraining as a mixture-based bridge between general pretraining and task-specific SFT, then asks which domains, data and start times benefit. The comparison target is a training schedule, not an unspecified stronger model.

2. Argument sequence

   Research-question headings lead from domain specificity to proximity, mixture timing and representation change; boxed findings close each argument. Related work follows the diagnostic sequence rather than interrupting its definitions.

3. Evidence and verification

   Table 4 places adaptation and C4 retention side by side and states five-seed averaging. The pure-pretraining baseline still has the lowest C4 loss for 70M GSM8K, so the mixed-versus-specialist comparison must not become a claim of winning every cell.

Problem, claim and contribution

Defines midtraining as a mixture-based bridge between general pretraining and task-specific SFT, then asks which domains, data and start times benefit. The comparison target is a training schedule, not an unspecified stronger model.

Sections 1–3, pp. 1–3 · Main paper

Argument and section order

Research-question headings lead from domain specificity to proximity, mixture timing and representation change; boxed findings close each argument. Related work follows the diagnostic sequence rather than interrupting its definitions.

Sections 4–9, pp. 4–9 · Main paper

Figures, captions and visual explanation

Figure 3 uses model-size small multiples, color for target data and shape for midtraining data. The caption says near-zero pairs are greyed for display but retained in correlations; Figure 4 then separates start time from mixture weight.

Figures 3–4, physical p. 6; raster inspected · Main paper

Tables, comparisons and evidence

Table 4 places adaptation and C4 retention side by side and states five-seed averaging. The pure-pretraining baseline still has the lowest C4 loss for 70M GSM8K, so the mixed-versus-specialist comparison must not become a claim of winning every cell.

Table 4 and Section 5.2, physical p. 6; raster inspected · Main paper

Appendix and reproducibility

The appendix separates local theoretical assumptions, optimizer/SFT grids and full numerical results from CKA panels. Main-text pointers distinguish descriptive representation probes from the optimization argument.

Appendices A–I, pp. 14–26; Section 7, p. 7 · Main paper

Clarity and writing conventions

The timing sweep is explicitly small-scale, and the CKA discussion declines to infer fundamentally different final representations. This keeps interpretation narrower than a suggestive diagnostic plot.

Sections 6–7, p. 7 · Main paper

What to reuse—and what not to infer

Transferable lesson: Use question-led headings and compact findings, but carry comparison scope and diagnostic limitations into the interpretation.

Boundary: Correlation with token proximity and qualitative CKA do not establish a causal mechanism or behavior at much larger model scales.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused scan of Sections 1–9, physical pp. 1–9; definitions, question-led experiments, captions and qualification of representation claims.

Appendix / supplement

Structural and selected-detail scan of Appendix A–I, pp. 14–26: local assumptions, training settings, similarity definition, full losses, training curves, CKA panels and LLM-use disclosure. No independent proof audit or experiment reproduction.

Visually inspected PDF pages

4, 6

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 26 pages