Writing study: Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods

NeurIPS overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← NeurIPS 2026 oral writing corpus

Grigory Begunov, Alexander Tyurin · Official NeurIPS 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal analysis (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

29 physical pages

Before references / appendix

≈ 8.04 page-equivalents

Main figures / tables

6 / 0

Attached appendix figures / tables

11 / 1

Attached appendix span

≈ 18.91 page-equivalents

Before references / appendix: ≈ 27.72%References / intervening material: ≈ 7.07%Attached appendix: ≈ 65.21%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Source-checked 2026-10-02: replace footnote-derived false Sections 2-4 with the real section headings, restore Section 8, and use the visibly two-column reading coordinate for main-text and reference headings. Full-width figure captions had caused the adaptive parser to misorder Sections 7 and 8. Restore the side-by-side Figure 2 caption and remove the prose cross-reference Figure 8a wrongly counted as a figure.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 2.37 pp.	8.17%
2. Synchronous Methods Are Nearly Optimal under Fixed Computation Model	3	≈ 0.73 pp.	2.52%
3. Random Computation Model	4	≈ 0.93 pp.	3.21%
4. On the Optimal Selection of Active Workers	5	≈ 0.52 pp.	1.79%
5. Universal Computation Model	5	≈ 1.58 pp.	5.45%
6. When Asynchronicity Might Be Needed	7	≈ 0.97 pp.	3.34%
7. Experiments	8	≈ 0.39 pp.	1.34%
8. Conclusion	8	≈ 0.28 pp.	0.97%
References	9	≈ 2.04 pp.	7.03%
A. Notations	11	≈ 0.23 pp.	0.79%
B. Proof of Theorem 2.3	11	≈ 1.42 pp.	4.9%
C. Proof of Theorem 3.2	12	≈ 1.05 pp.	3.62%
D. Proof of Theorem 3.3	13	≈ 2.59 pp.	8.93%
E. Proof of Proposition 4.1	16	≈ 0.26 pp.	0.9%
F. Proof of Theorems 5.3	16	≈ 0.59 pp.	2.03%
G. Proof of Theorem 5.5	17	≈ 0.86 pp.	2.97%
H. Proof of Proposition 4.2	18	≈ 0.43 pp.	1.48%
I. A Theoretical Example Where Asynchronicity Is Needed	18	≈ 0.89 pp.	3.07%
J. NanoGPT Benchmark	19	≈ 0.68 pp.	2.34%
K. Extra Experiments and Details	20	≈ 9.91 pp.	34.17%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 2
• Figure 3 · p. 7
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 8

Attached appendix

• Table 1 · p. 11
• Figure 7 · p. 19
• Figure 8 · p. 21
• Figure 9 · p. 22
• Figure 10 · p. 23
• Figure 11 · p. 24
• Figure 12 · p. 24
• Figure 13 · p. 25
• Figure 14 · p. 26
• Figure 15 · p. 27
• Figure 16 · p. 28
• Figure 17 · p. 29

Argument and organization

1. Problem and contribution

   A practical-theoretical tension motivates the paper: synchronization is widely used despite apparent idle-time waste. The response is a regime-qualified near-optimality analysis, not a claim that asynchronous optimization is always unnecessary.

2. Argument sequence

   Worker-time assumptions and algorithm definitions precede fixed, random and time-varying computation regimes. Partial participation and heterogeneous objectives are treated as separate cases. The conclusion lists the conditions under which its positive statement holds.

3. Evidence and verification

   The experimental page pairs a heterogeneous-worker quadratic example with a small neural-network example using loss against time, while the prose explicitly distinguishes simulated computation assumptions from real NanoGPT timing in the appendix. The numerical figures illustrate rather than prove the bounds.

Problem, claim and contribution

A practical-theoretical tension motivates the paper: synchronization is widely used despite apparent idle-time waste. The response is a regime-qualified near-optimality analysis, not a claim that asynchronous optimization is always unnecessary.

Introduction and Section 1.1, pp. 1–3 · Main paper

Argument and section order

Worker-time assumptions and algorithm definitions precede fixed, random and time-varying computation regimes. Partial participation and heterogeneous objectives are treated as separate cases. The conclusion lists the conditions under which its positive statement holds.

Sections 1–6 and conclusion, pp. 1–8 · Main paper

Figures, captions and visual explanation

Side-by-side worker timelines visualize idle synchronization bubbles and asynchronous updates on the same time axis. The nearby m-synchronous pseudocode makes the alternative concrete: aggregate the first m workers and discard late gradients from that iteration.

Figures 1–2 and Algorithm 3, p. 2; rendered page inspected · Main paper

Tables, comparisons and evidence

The experimental page pairs a heterogeneous-worker quadratic example with a small neural-network example using loss against time, while the prose explicitly distinguishes simulated computation assumptions from real NanoGPT timing in the appendix. The numerical figures illustrate rather than prove the bounds.

Figures 5–6 and Section 7, p. 8; rendered page inspected · Main paper

Appendix and reproducibility

Separate proof sections track the different computation models. An explicit regime where asynchronicity is needed preserves a counterexample to an overbroad reading. The experiment appendix records synthetic timing setups, step sizes and a four-worker NanoGPT wall-clock test.

Appendices B–I and K, pp. 11–29 · Main paper

Clarity and writing conventions

The question-form title is followed by precise assumptions, constants and exceptions. The conclusion acknowledges possible constant- or logarithmic-factor asynchronous advantages, a useful model for keeping a provocative framing narrower than a universal systems claim.

Sections 1.2 and 8, pp. 3 and 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: For a counterintuitive theory claim, give readers a shared visual model first, then organize guarantees, exceptions and experiments by the exact assumptions.

Boundary: The 1,000-worker quadratic setup uses a specified simulated timing model and is not a measured 1,000-machine deployment. This scan does not verify proofs or establish that the two-column preprint follows NeurIPS submission formatting.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-02

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented distributed manuscript scan of Sections 1–8, physical pp. 1–8, in a two-column author manuscript: section sequence, substantive prose windows, captions, comparison contracts and conclusions. Not exhaustive technical verification.

Appendix / supplement

Writing-oriented distributed scan of Appendices A–K, physical pp. 11–29, including proofs, counterexamples, simulated-time protocols and a measured NanoGPT experiment. Not independent proof checking; code and experiments were not executed.

Visually inspected PDF pages

2, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Public author arXiv manuscript 2602.03802v1; not verified identical to the accepted or camera-ready OpenReview revision · 29 pages