Writing study: WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Xin Zhao, Zhiqiang Zhang, JUN ZHOU · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Learning & optimization algorithms (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

29 physical pages

Before references / appendix

≈ 9.72 page-equivalents

Main figures / tables

5 / 5

Attached appendix figures / tables

6 / 5

Attached appendix span

≈ 11.89 page-equivalents

Before references / appendix: ≈ 33.52%References / intervening material: ≈ 25.48%Attached appendix: ≈ 41%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.26 pp.	4.34%
2 PRELIMINARY	2	≈ 0.56 pp.	1.93%
3 THE PROPOSED METHODOLOGY	3	≈ 1.85 pp.	6.38%
4 EXPERIMENT	5	≈ 4.23 pp.	14.59%
5 RELATED WORK	9	≈ 0.83 pp.	2.86%
6 CONCLUSION	10	≈ 0.3 pp.	1.03%
ACKNOWLEDGMENTS	10	≈ 0.1 pp.	0.34%
REFERENCES	10	≈ 7.38 pp.	25.45%
A EXPERIMENTAL SETTINGS	18	≈ 0.67 pp.	2.31%
B DETAILS OF THEORETICAL CONNECTION BETWEEN DECAY AND	18	≈ 1.85 pp.	6.38%
C DISCUSSIONS	20	≈ 1.14 pp.	3.93%
D ADDITIONAL EXPERIMENTS ON MUON OPTIMIZER	21	≈ 0.49 pp.	1.69%
E ALGORITHM	22	≈ 0.57 pp.	1.97%
F ADDITIONAL EXPERIMENTS ON WSD	22	≈ 0.74 pp.	2.55%
G HOW CHECKPOINT MERGING INFLUENCES NETWORK BEHAVIOR:	23	≈ 0.98 pp.	3.38%
H PARAMETER TRAJECTORIES AND MODEL PERFORMANCE: CONSTANT	24	≈ 0.25 pp.	0.86%
I COMPARISON WITH A LONG-COOLDOWN WSD BASELINE	24	≈ 1.32 pp.	4.55%
J DETAILED EVALUATION RESULTS	26	≈ 3.89 pp.	13.41%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 4
• Figure 3 · p. 6
• Table 1 · p. 6
• Table 2 · p. 6
• Figure 4 · p. 7
• Figure 5 · p. 8
• Table 3 · p. 8
• Table 4 · p. 8
• Table 5 · p. 9

Attached appendix

• Table 6 · p. 18
• Table 7 · p. 22
• Figure 6 · p. 23
• Figure 7 · p. 24
• Figure 8 · p. 24
• Figure 9 · p. 25
• Figure 10 · p. 25
• Figure 11 · p. 26
• Table 8 · p. 27
• Table 9 · p. 28
• Table 10 · p. 29

Argument and organization

1. Scheduling problem and algebraic bridge

   Motivates flexible training duration and derives checkpoint weights from desired cumulative update weights.

2. Main comparison and factor analysis

   Separates base/post-training results from merge algorithm, duration, granularity and decay-compatibility experiments.

3. Implementation and broader controls

   Supplies pseudocode, proof details, a Muon experiment, longer cooldown and task-level results.

Problem, claim and contribution

The initial comparison diagram makes the operational difference visible: keep the live training rate stable and form evaluation models from stored checkpoints. Scheduling flexibility is distinguished from downstream performance.

Figure 1, page 1; Sections 1 and 3.2, pages 2 and 4–5. · Main paper

Argument and section order

The method moves from a weighted average through a rearranged update sum to a telescoping weight formula. The short theorem and corresponding visual make the algebra easy to locate.

Equations 1–5 and Theorem 3.1, pages 3–4. · Main paper

Figures, captions and visual explanation

Figure 2 vertically aligns each desired decay curve with its checkpoint-weight bars and explains that smooth curves illustrate discrete operations. Figure 3 retains the unmerged control and shades continuation beyond the decay horizon.

Figures 2 and 3, pages 4 and 6. · Main paper

Tables, comparisons and evidence

Tables 1–2 explicitly select the best average-score checkpoint or epoch and distinguish relative improvement percentages from raw scores. The instruction-model code score decreases, preserving an exception to the overall gain.

Tables 1–2, page 6. · Main paper

Appendix and reproducibility

Pseudocode separates live checkpoints from asynchronous merged outputs; the appendix adds a different optimizer, a longer cooldown baseline and full task-level results. Storage overhead is discussed rather than hidden.

Appendices C–E and I–J, pages 20–22 and 24–29. · Main paper

Clarity and writing conventions

Experimental constants need one authoritative record: the main setup gives peak LR 4.78e-4, whereas Appendix A gives 3.74e-4. Broad generalization language around spectral diagnostics should remain a hypothesis, not a proven consequence.

Section 4.1, page 5; Appendices A and G, pages 18 and 23–24. · Main paper

What to reuse—and what not to infer

Transferable lesson: Make an implementation change tangible with a before/after workflow, derive its central mapping in a few traceable steps, and separate the main comparison from sensitivity and compatibility controls.

Boundary: The algebra reweights updates from an existing trajectory; it does not prove equivalence to retraining with a different LR, gradients and optimizer states. Results depend on selected checkpoints, merge windows and curated annealing data, and do not establish universal gains across architectures.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–6, LLM-use and acknowledgement statements; references on pages 10–17.

Appendix / supplement

Pages 18–29, Appendices A–J, including the weight derivation, algorithm, optimizer test, spectral diagnostics, long-cooldown control and every detailed results table.

Visually inspected PDF pages

1, 4, 6, 27

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 29 pages