Writing study: SVRG and Beyond via Posterior Correction

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Nico Daheim, Thomas Möllenhoff, Ming Liang Ang, Mohammad Emtiyaz Khan · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

24 physical pages

Before references / appendix

≈ 9.04 page-equivalents

Main figures / tables

5 / 0

Attached appendix figures / tables

4 / 1

Attached appendix span

≈ 8.91 page-equivalents

Before references / appendix: ≈ 37.67%References / intervening material: ≈ 25.21%Attached appendix: ≈ 37.13%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.09 pp.	4.54%
2. SVRG and Bayesian Methods	2	≈ 2.04 pp.	8.5%
3. Beyond SVRG via Posterior Correction	4	≈ 3.34 pp.	13.92%
4. Experiments & Results	7	≈ 1.84 pp.	7.67%
5. Conclusion	9	≈ 0.11 pp.	0.46%
Acknowledgements	9	≈ 0.08 pp.	0.33%
Impact Statement	9	≈ 0.26 pp.	1.08%
References	10	≈ 6.04 pp.	25.17%
A. Derivation of Eq. 17	16	≈ 0.47 pp.	1.96%
B. Proof of Thm. 1	16	≈ 0.89 pp.	3.71%
C. Derivation of the Newton-like SVRG Extension	17	≈ 1.93 pp.	8.04%
D. Derivation of the Adam-like SVRG Extension	19	≈ 1.41 pp.	5.88%
E. Additional Results	20	≈ 2.05 pp.	8.54%
F. Hyperparameters and Additional Details	22	≈ 2.17 pp.	9.04%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 2
• Figure 3 · p. 7
• Figure 4 · p. 8
• Figure 5 · p. 8

Attached appendix

• Figure 6 · p. 20
• Figure 7 · p. 21
• Figure 8 · p. 21
• Figure 9 · p. 22
• Table 1 · p. 22

Argument and organization

1. 1 Introduction; 2 SVRG and Bayesian Methods

   Frames a conceptual connection and establishes both optimizer vocabularies.

2. 3 Posterior Correction

   Recovers a known algorithm, derives richer-posterior extensions and accounts for their overhead.

3. 4 Experiments; 5 Conclusion

   Progresses from convex problems to deep learning and reports where optimization gains do not become speed gains.

Problem, claim and contribution

The main novelty is a derivational connection rather than an unconditional faster-training claim.

Abstract and Introduction, p.1 · Main paper

Argument and section order

Theorems separate the Gaussian posterior update from the approximation that recovers ordinary SVRG, then motivate Newton-like and diagonal variants.

Sections 3.2-3.3, p.5 · Main paper

Figures, captions and visual explanation

Side-by-side pseudocode highlights changed operations; paired plots later change the horizontal axis from optimizer steps to gradient evaluations or examples seen.

Figure 1, p.2; Figures 4-5, p.8 · Main paper

Tables, comparisons and evidence

The language-model comparison includes a larger-batch alternative with similar compute cost and explicitly reports no tangible training speed-up.

Section 4.2, p.9 · Main paper

Appendix and reproducibility

The first derivation flags its constant-base-measure assumption before extending it for isotropic Gaussians; ablations distinguish correction strength and refresh frequency.

Appendices A-B and E.4, pp.16,22 · Main paper

Clarity and writing conventions

The conclusion retains the negative deep-learning speed result instead of equating improved final perplexity with improved efficiency.

Conclusion, p.9 · Main paper

What to reuse—and what not to infer

Transferable lesson: When proposing an optimizer, show both the algebraic connection and comparisons under the resource accounting that could overturn a speed headline.

Boundary: Step efficiency and compute efficiency differ; the new Bayesian perspective does not establish a universal runtime benefit, and proofs were not independently checked.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 5, 7, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 16, 22. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

2, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 24 pages