Writing study: Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, Tieniu Tan · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Learning & optimization algorithms (primary) · Inference, algorithms & systems efficiency.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

24 physical pages

Before references / appendix

≈ 9.47 page-equivalents

Main figures / tables

3 / 3

Attached appendix figures / tables

0 / 1

Attached appendix span

≈ 11.64 page-equivalents

Before references / appendix: ≈ 39.46%References / intervening material: ≈ 12.04%Attached appendix: ≈ 48.5%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.16 pp.	4.83%
2 RELATED WORKS	2	≈ 0.91 pp.	3.79%
3 METHOD	3	≈ 2.61 pp.	10.88%
4 EXPERIMENTAL RESULTS	6	≈ 3.81 pp.	15.88%
5 CONCLUSION	10	≈ 0.31 pp.	1.29%
REFERENCES	10	≈ 2.89 pp.	12.04%
A PROOF OF THEORETICAL RESULTS	13	≈ 1.14 pp.	4.75%
B DETAILED ALGORITHMS OF LORA-PRE FOR ADAM AND MUON	14	≈ 3.05 pp.	12.71%
C THEORETICAL ANALYSIS OF APPROXIMATION ERROR AND	17	≈ 5.63 pp.	23.46%
D ADDITIONAL EXPERIMENTAL RESULTS	23	≈ 0.93 pp.	3.88%
E STATEMENT OF THE USE OF LARGE LANGUAGE MODELS	24	≈ 0.89 pp.	3.71%

All detected figure and table locations

Main text

• Figure 1 · p. 4
• Table 1 · p. 7
• Table 2 · p. 8
• Figure 2 · p. 8
• Figure 3 · p. 9
• Table 3 · p. 9

Attached appendix

• Table 4 · p. 23

Argument and organization

1. EMA as online regression

   Uses a simple algebraic reinterpretation to motivate state compression.

2. Factorized moments

   Derives first-moment updates, then handles nonnegative second-moment reconstruction.

3. Pretraining, fine-tuning and rank ablations

   Tests two optimizer families and distinguishes model scales and adaptation tasks.

Problem, claim and contribution

The central equivalence is stated as one short equation before the full method. This creates an accessible entry point into optimizer-state compression without beginning with several pages of matrix calculus.

Equation 1, page 2; Section 3.2. · Main paper

Argument and section order

The second-moment section first explains why a naive factorization is unsuitable for a square-root denominator, then introduces the magnitude-based surrogate. This obstacle-first ordering is effective.

Section 3.3, pages 5–6. · Main paper

Figures, captions and visual explanation

The conceptual figure explicitly contrasts an unfactorized online learner with its low-rank form, retaining the incoming gradient sequence on both sides. It is an explanatory equation diagram, not a model-architecture cartoon.

Figure 1, page 4. · Main paper

Tables, comparisons and evidence

Tables separate full and low-rank optimizers and identify reproduced baselines. The prose should preserve the fact that full Muon wins the pretraining comparison and that fine-tuning averages hide task-specific losses.

Tables 1–3, pages 7–9. · Main paper

Appendix and reproducibility

Color-differentiated comparison algorithms show exactly which steps change; the later theoretical appendix clarifies that effective moments include the current exact gradient rather than only the stored factors.

Algorithms 1–2, pages 15–17; Appendix C.1. · Main paper

Clarity and writing conventions

Broken section references remain in the experiment roadmap. More importantly, the appendix introduces regularized inverses and a smoothed denominator absent from the basic pseudocode, so the implementation and theorem variants need explicit reconciliation.

Section 4, page 6; Appendix C, pages 17–22. · Main paper

What to reuse—and what not to infer

Transferable lesson: Explain a compression method by first rewriting the familiar operation, then show the precise algorithmic differences and the accuracy–resource tradeoff.

Boundary: Compressing optimizer history is not the same as restricting model weights to LoRA. Squared reconstructed gradient magnitudes are not exact Adam second moments, and the convergence result assumes reference-optimizer descent and bounded approximation errors.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–5; references through page 12.

Appendix / supplement

Pages 13–24, Appendices A–E: factor updates, full algorithms, approximation/convergence analysis, sensitivity and disclosure.

Visually inspected PDF pages

4, 8, 21

• ICLR 2026 final proceedings · 24 pages