Writing study: How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, Wenguang Chen · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Learning & optimization algorithms.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

32 physical pages

Before references / appendix

≈ 10.21 page-equivalents

Main figures / tables

6 / 2

Attached appendix figures / tables

3 / 10

Attached appendix span

≈ 14.89 page-equivalents

Before references / appendix: ≈ 31.91%References / intervening material: ≈ 21.56%Attached appendix: ≈ 46.53%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2.1 pp.	6.56%
2 LEARNING RATE DECAY COUNTERACTS DATA CURRICULUM	3	≈ 1.93 pp.	6.03%
3 UNLOCKING DATA CURRICULA POTENTIAL VIA MODERATE LR DECAY	5	≈ 3.88 pp.	12.13%
4 A THEORETICAL DEMONSTRATION	9	≈ 1.33 pp.	4.16%
5 CONCLUSION	10	≈ 0.38 pp.	1.19%
REFERENCES	11	≈ 6.89 pp.	21.53%
A RELATED WORK	18	≈ 0.63 pp.	1.97%
B PRELIMINARY	18	≈ 1.37 pp.	4.28%
C EXPERIMENTS	20	≈ 3.7 pp.	11.56%
D LARGE-SCALE CONTINUAL PRETRAINING WITH MULTI-DOMAIN	23	≈ 3.59 pp.	11.22%
E DISCUSSION	27	≈ 1.7 pp.	5.31%
F PROOFS IN SECTION 4	29	≈ 3.89 pp.	12.16%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Figure 3 · p. 5
• Table 1 · p. 6
• Figure 4 · p. 7
• Table 2 · p. 8
• Figure 5 · p. 9
• Figure 6 · p. 10

Attached appendix

• Table 3 · p. 20
• Figure 7 · p. 21
• Table 4 · p. 21
• Table 5 · p. 22
• Figure 8 · p. 23
• Table 6 · p. 24
• Table 7 · p. 24
• Table 8 · p. 25
• Table 9 · p. 25
• Figure 9 · p. 26
• Table 10 · p. 26
• Table 11 · p. 27
• Table 12 · p. 28

Argument and organization

1. An interaction explains disappointing curricula

   Reframes data ordering as coupled to learning-rate decay.

2. Moderate decay and checkpoint averaging

   Tests individual remedies before their combination.

3. Illustrative theory and larger-scale appendix

   Separates mechanism intuition from empirical validation and deployment recipes.

Problem, claim and contribution

The title poses a concrete mismatch, and the introduction turns earlier weak results into a testable interaction rather than dismissing curriculum learning.

Section 1, pp. 1–3 · Main paper

Argument and section order

The analysis changes decay length and final rate separately, then evaluates averaging; this makes the combined recipe easier to understand causally.

Sections 2–3.4, pp. 3–9 · Main paper

Figures, captions and visual explanation

The landscape diagram labels itself an intuition and color-codes update trajectories and the average; it does not pretend to be a measured LLM landscape.

Figure 4, p. 7 · Main paper

Tables, comparisons and evidence

Tables name the exact baseline for delta subscripts and retain task-level regressions, preventing the overall mean from hiding tradeoffs.

Table 1, p. 6; Table 10, p. 26 · Main paper

Appendix and reproducibility

The supplement includes both a simple three-stage algorithm and domain-specific ranking/interleaving for a larger continual-pretraining experiment.

Algorithms 1–2, pp. 23–25; Appendix D, pp. 23–27 · Main paper

Clarity and writing conventions

The averaging notation needs reconciliation: the recursive EMA coefficient in Appendix B and the normalized powers in Algorithm 1 are not the same parameter convention.

Appendix B, p. 19; Algorithm 1, p. 23 · Main paper

What to reuse—and what not to infer

Transferable lesson: When a familiar technique appears ineffective, organize the paper around a confounding interaction and vary its components independently before presenting a combined recipe.

Boundary: Quality is scorer- and task-dependent; a high-quality validation split shares the DCLM scoring preference. The toy quadratic model illustrates a mechanism rather than proving the LLM claim. Report gains as percentage points and reconcile EMA conventions and headline 1.64/1.68 figures.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–5, including all schedule comparisons and the illustrative quadratic model

Appendix / supplement

Pages 18–32, Appendices A–F: schedules, averaging, experimental details, 3.2B continual pretraining, discussion and proofs

Visually inspected PDF pages

7, 6

• ICLR 2026 final proceedings · 32 pages