Writing study: To Grok Grokking: Provable Grokking in Ridge Regression

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Mingyue Xu, Gal Vardi, Itay Safran · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal analysis (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

23 physical pages

Before references / appendix

≈ 8.75 page-equivalents

Main figures / tables

4 / 0

Attached appendix figures / tables

3 / 0

Attached appendix span

≈ 11.91 page-equivalents

Before references / appendix: ≈ 38.04%References / intervening material: ≈ 10.17%Attached appendix: ≈ 51.78%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 2.58 pp.	11.22%
2. Preliminaries	3	≈ 0.74 pp.	3.22%
3. Informal Results	4	≈ 0.27 pp.	1.17%
4. Grokking in Ridge Regression	4	≈ 2.31 pp.	10.04%
5. Experiments	7	≈ 1.54 pp.	6.7%
6. Discussion and Future Directions	8	≈ 0.83 pp.	3.61%
Acknowledgements	9	≈ 0.09 pp.	0.39%
Impact Statement	9	≈ 0.05 pp.	0.22%
References	9	≈ 2.34 pp.	10.17%
A. Missing Proofs	12	≈ 10.4 pp.	45.22%
B. Additional Experiments	22	≈ 1.28 pp.	5.57%
C. Technical Lemmas	23	≈ 0.23 pp.	1%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 7
• Figure 3 · p. 8
• Figure 4 · p. 8

Attached appendix

• Figure 5 · p. 22
• Figure 6 · p. 23
• Figure 7 · p. 23

Argument and organization

1. 1 Introduction; 2 Preliminaries; 3 Informal Results

   Defines the delayed-generalization target and supplies an accessible theorem preview.

2. 4 Grokking in Ridge Regression

   Builds from a zero-teacher warmup to realizable teachers and separate fitting/generalization results.

3. 5 Experiments; 6 Discussion

   Tests hyperparameter effects in linear, random-feature and nonlinear settings, then states unresolved extensions.

Problem, claim and contribution

Grokking is decomposed into early fitting, persistent poor generalization and eventual small error rather than reduced to a visually abrupt accuracy jump.

Abstract, p.1; main result, p.5 · Main paper

Argument and section order

The zero-teacher warmup exposes different training and population decay rates before the general theorem adds sample, dimension and regularization conditions.

Sections 4.1-4.2, p.5 · Main paper

Figures, captions and visual explanation

Matched four-panel layouts vary one hyperparameter at a time; the linear experiment overlays dashed bounds and solid observations, while later loss plots separate train/test line styles.

Figure 2, p.7; Figures 3-4, p.8 · Main paper

Tables, comparisons and evidence

The random-feature experiment is acknowledged as non-realizable, and nonlinear-network results are described as qualitative agreement rather than a proved extension.

Sections 5.2-5.3, p.8 · Main paper

Appendix and reproducibility

The proof appendix separates sample-span and orthogonal dynamics, while the additional experiments test surrogate accuracy and noisy labels beyond the proved setting.

Appendix A, p.13; Appendix B, p.22 · Main paper

Clarity and writing conventions

The discussion explains why thresholded accuracy can show a plateau when regression loss decays smoothly, and warns that tiny weight decay can impede generalization in agnostic settings.

Discussion and Future Directions, p.9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Define the phenomenon operationally before plotting it, and distinguish sufficient-condition bounds from observed transition times and extensions beyond the theorem.

Boundary: The realizable ridge-regression guarantee is not a universal account of deep-network grokking, and a threshold-induced plateau should not be confused with a discontinuity in the underlying loss.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 4, 5, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 13, 20, 22, 23. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

7, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 23 pages