Writing study: On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
← AAAI 2026 oral writing corpus
Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang, Andi Han, Wei Huang, Taiji Suzuki, Junchi Yan · Official AAAI 2026 oral record
Figure, table and section ledger
Editorial contribution type: Theory & formal analysis (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 9 physical pages
- Before references / appendix
- ≈ 7.04 page-equivalents
- Main figures / tables
- 5 / 1
- Attached appendix figures / tables
- 0 / 0
- Attached appendix span
- No boundary detected; not assumed zero
Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.
Source-bound correction: Checked 2026-10-25 against main-paper reading, physical-page rasters 4,5,7 and Poppler boxes. Restore the two-line theoretical-analysis heading at its first-line position. Phase I, simulation, Phase II and experiment headings are subsections within that analysis, not separate top-level sections. Large embedded chart labels are not headings. Figures 1–5 and Table 1 are genuine distinct labels; Algorithm 1 is outside this figure/table count. No attached appendix.
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
| Top-level heading | Starts on PDF page | Estimated span | Share of entire PDF |
|---|---|---|---|
| Introduction | 1 | ≈ 0.86 pp. | 9.56% |
| Related Work | 2 | ≈ 0.71 pp. | 7.89% |
| Preliminaries | 2 | ≈ 0.51 pp. | 5.67% |
| Theoretical Analysis: The Learning Dynamics of Label Noise SGD | 3 | ≈ 3.45 pp. | 38.33% |
| Extension: From Label Noise SGD to SAM | 6 | ≈ 1 pp. | 11.11% |
| Conclusion and Outlook | 7 | ≈ 0.19 pp. | 2.11% |
| References | 8 | ≈ 1.96 pp. | 21.78% |
All detected figure and table locations
Attached appendix
No recognized caption labels.
Argument and organization
- Introduction; Related Work; Preliminaries
Starts from the counterintuitive empirical benefit of label noise, positions implicit-regularization explanations, and defines the optimizer and linear-network setting before theoretical claims.
- Theoretical Analysis: The Learning Dynamics of Label Noise SGD
Provides a roadmap, six conditions and a two-phase summary table, then develops diminishing norms, a controlled oscillation simulation, and alignment/convergence under separately stated updates.
- Extension: From Label Noise SGD to SAM
Reuses diagnostic plots to ask whether the proposed dynamical picture also appears under SAM, presenting an empirical extension rather than a new SAM theorem.
- Conclusion and Outlook; References
Returns to the oscillation-to-feature-learning explanation and explicitly limits the theory to two-layer linear regression, leaving nonlinear and classification analysis open.
Problem, claim and contribution
Begins with an empirical puzzle—adding label noise can improve generalization and pruning behavior—then narrows the question to learning dynamics rather than claiming an immediately general training prescription. The transition from ResNet/CIFAR motivation to a two-layer linear regression analysis is visible in the setup, so a reader can distinguish the motivating observation from the system actually analyzed.
Introduction, pp.1–2; Figure 1 and Roadmap/Setup, p.3 · Main paper
Argument and section order
Table 1 is a compact theory roadmap: Phase I maps diminishing weight norms and departure from the lazy regime to Lemma 3/Theorem 2; Phase II maps alignment and convergence to Lemmas 5–6. Condition 1 and its explanatory remarks appear before these claims. Local proof sketches and an oscillation-isolating Markov simulation break a long theoretical chain into interpretable steps instead of leaving every explanation to a supplement.
Table 1, Condition 1 and Phase I, p.4; simulation and Phase II opening, p.5; Lemmas 5–6, p.6 · Main paper
Figures, captions and visual explanation
Figure 2 combines train/test loss, mean norm/alignment, a single-neuron phase portrait with iteration color, and four population snapshots. These panels explain change over training, not merely the final score. Figure 4 repeats the visual grammar for SAM, allowing a direct conceptual comparison; Figure 5 separates train loss, test loss and first-layer norm. The caption must explain the dual axes and color encoding because they encode different quantities, not uncertainty.
Figure 2, p.5; Figures 4–5, p.7 · Main paper
Tables, comparisons and evidence
Separates introductory CIFAR experiments, synthetic two-layer diagnostics, a deliberately simplified Markov-process intervention and NTK-based comparisons on only 64 randomly chosen CIFAR-10 images. The 64-image restriction is explicitly tied to NTK computational cost. A strong presentation lesson is to label these roles and denominators separately: a controlled mechanism test and a small diagnostic cannot be pooled into a full-scale performance claim, and a smoothed curve is not a confidence interval.
Figure 1 and setup, p.3; simulation, p.5; Experiments and Figure 3, p.6; Figure 5 caption, p.7 · Main paper
Appendix and reproducibility
Main-text proof sketches explain the source of weight-norm decrease, while references to Appendix A route constants and longer arguments elsewhere. Appendix D is cited for the pruning protocol. Keeping the intuition and assumptions in the body is useful, but the separate material was not inspected here; the proceedings pointers cannot be counted as a completed proof or reproducibility audit.
Pruning discussion, p.3; Condition 1 footnotes and proof sketch, p.4; further theoretical pointers, pp.5–6 · Main paper
Clarity and writing conventions
Uses explicit questions and phase names to make a technical analysis navigable, and ends with a direct limitations paragraph. Crucially, the Phase II lemmas analyze gradient descent for tractability after entering a small-weight regime, while the SAM section offers empirical analogies. Those update-rule and epistemic changes must remain visible in a summary; the two-phase table alone could otherwise suggest a stronger continuous noisy-SGD guarantee than the body establishes.
Phase II and Lemmas 5–6, pp.5–6; Extension, pp.6–7; Limitations, p.7 · Main paper
What to reuse—and what not to infer
Transferable lesson: A theoretical paper can teach a dynamical mechanism through a condition-to-result roadmap, local proof intuition and a coordinated family of diagnostic plots. Repeat the same visual encodings for an extension, but label the assumptions and update rule for each theorem beside the claim it supports.
Boundary: Do not present the piecewise analysis as an end-to-end theorem for noisy-SGD training of deep classifiers: Phase II changes the optimizer to gradient descent, Lemma 6 adds an alignment assumption, and the SAM extension is empirical. The NTK comparisons use 64 images, not the full CIFAR-10 benchmark. This study did not verify proofs, implementation or statistical repeatability.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-10-25
- Reading status
- Writing-focused manuscript scan + selected visuals inspected
- Main text
- Writing-oriented reading of physical pp.1–7, including introductory experiments, setup, conditions, proof sketches, simulation, Phase II analysis, SAM extension and limitations; reference-only pp.8–9 checked. Full-page rasters of pp.4, 5 and 7 inspected. No complete proof audit, code execution or experimental replication was performed.
- Appendix / supplement
- The proceedings PDF contains no attached appendix. Pointers to Appendix A for theoretical details and Appendix D for pruning details were inspected. The separate arXiv:2603.10397 manuscript and linked code were identified but not read; external proofs and settings remain outside this study.
- Visually inspected PDF pages
- 4, 5, 7
This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.