Writing study: Procedural Pretraining: Warming Up Language Models with Abstract Data

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Liangze Jiang, Zachary Shinnick, Anton Van Den Hengel, Hemanth Saratchandran, Damien Teney · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

30 physical pages

Before references / appendix

≈ 8.7 page-equivalents

Main figures / tables

8 / 1

Attached appendix figures / tables

15 / 13

Attached appendix span

≈ 17.91 page-equivalents

Before references / appendix: ≈ 29%References / intervening material: ≈ 11.3%Attached appendix: ≈ 59.7%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Source-checked physical-page text restores wrapped top-level heading labels on pp. 3, 5, 8. Heading starts and measured positions are unchanged; this is label completeness, not a new section. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.12 pp.	3.73%
2. Related Work	2	≈ 0.38 pp.	1.27%
3. Preliminaries	3	≈ 0.76 pp.	2.53%
4. Probing Procedural Pretraining with Algorithmic Reasoning	3	≈ 1.51 pp.	5.03%
5. Can Procedural Data Complement or Replace Standard Data?	5	≈ 3.02 pp.	10.07%
6. Combining Multiple Types of Procedural Data	8	≈ 0.82 pp.	2.73%
7. Discussion	9	≈ 0.43 pp.	1.43%
Impact Statement	9	≈ 0.05 pp.	0.17%
References	9	≈ 3.39 pp.	11.3%
Appendix	13	≈ 0.32 pp.	1.07%
A. Extended Literature Review	13	≈ 1 pp.	3.33%
B. Procedural Pretraining	14	≈ 1.24 pp.	4.13%
C. Model details	15	≈ 0.11 pp.	0.37%
D. Algorithmic Task Descriptions	15	≈ 0.73 pp.	2.43%
E. Experimental Details	16	≈ 1.87 pp.	6.23%
F. Testing Simpler Explanations	18	≈ 2.73 pp.	9.1%
G. Procedural Data Hyperparameter Grid Search	21	≈ 0.63 pp.	2.1%
H. Longer Sequences for Procedural Pretraining	21	≈ 0.09 pp.	0.3%
I. Transferability Analysis	21	≈ 0.71 pp.	2.37%
J. The Effect of Vocabulary Size	22	≈ 0.6 pp.	2%
K. Weight Decay Ablation	23	≈ 0.35 pp.	1.17%
L. Scaling Procedural Pretraining	23	≈ 0.91 pp.	3.03%
M. Results on downstream tasks	24	≈ 1.08 pp.	3.6%
N. Additional Results	25	≈ 5.54 pp.	18.47%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Figure 3 · p. 5
• Figure 4 · p. 5
• Figure 5 · p. 6
• Figure 6 · p. 7
• Figure 7 · p. 7
• Figure 8 · p. 8
• Table 1 · p. 9

Attached appendix

• Figure 9 · p. 14
• Table 2 · p. 16
• Figure 10 · p. 18
• Figure 11 · p. 19
• Figure 12 · p. 20
• Figure 13 · p. 20
• Figure 14 · p. 20
• Figure 15 · p. 21
• Figure 16 · p. 21
• Figure 17 · p. 22
• Figure 18 · p. 22
• Figure 19 · p. 22
• Figure 20 · p. 23
• Table 3 · p. 23
• Table 4 · p. 24
• Table 5 · p. 24
• Table 6 · p. 24
• Table 7 · p. 25
• Table 8 · p. 25
• Table 9 · p. 26
• Table 10 · p. 26
• Table 11 · p. 26
• Table 12 · p. 27
• Table 13 · p. 27
• Figure 21 · p. 27
• Figure 22 · p. 28
• Figure 23 · p. 29
• Table 14 · p. 30

Argument and organization

1. Problem and contribution

   A terminology block separates procedural training data, algorithmic diagnostic tasks and semantic data; additive and substitutive settings ask different benefit and resource questions.

2. Argument sequence

   Skill probes and component transfer precede semantic-domain tests; simple alternative explanations are tested before proceeding to downstream persistence and data/weight mixtures.

3. Evidence and verification

   Explicit attention sharpening, layerwise shuffling and noise test alternatives to transferred structure; selective transfer reinitializes unselected components rather than freezing them throughout subsequent training.

Problem, claim and contribution

A terminology block separates procedural training data, algorithmic diagnostic tasks and semantic data; additive and substitutive settings ask different benefit and resource questions.

Sections 3-3.1, p. 3 · Main paper

Argument and section order

Skill probes and component transfer precede semantic-domain tests; simple alternative explanations are tested before proceeding to downstream persistence and data/weight mixtures.

Sections 4-6, pp. 4-8 · Main paper

Figures, captions and visual explanation

Shared component colors and baseline lines connect algorithmic and domain results; mixture plots encode ratios with small pie markers, while main captions point readers from mean-only plots to appendix variance.

Figure 3, p. 5; Figure 8, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

Explicit attention sharpening, layerwise shuffling and noise test alternatives to transferred structure; selective transfer reinitializes unselected components rather than freezing them throughout subsequent training.

Sections 3.1 and 4.3, pp. 3, 5; Appendix F, pp. 18-20 · Main paper

Appendix and reproducibility

Generator examples identify loss-bearing output tokens, and tables retain seed variability, scale settings and cases where individual procedural sources transfer poorly.

Appendix B, pp. 14-15; L, pp. 23-24; N.3/Table 14, p. 30 · Main paper

Clarity and writing conventions

Repeated Setup/Results/Take-away blocks make experimental questions easy to follow; discussion treats mixture composition as a proof of concept and recognizes smaller-than-frontier training scales.

Sections 5-7, pp. 5-9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Name separate data-budget contracts, use component transfer to localize a benefit, and test simpler explanations before assigning an abstract mechanism.

Boundary: Token savings at a matched loss are not measured wall-clock savings; component-level transfer is not a complete mechanistic account, and results up to 1.3B parameters do not establish frontier-scale behavior.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-7, pp. 1-9: procedural-data motivation and terminology, additive/substitutive contracts, skill and component probes, alternative explanations, semantic/downstream transfer and mixture proof of concept.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-N, pp. 13-30: generators, architecture and training protocols, controls, hyperparameter/length/vocabulary/decay checks, larger models, downstream evaluations and variance-bearing numerical tables.

Visually inspected PDF pages

5, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 30 pages