Writing study: Learning Reveals Invisible Structure in Low-Rank RNNs
← NeurIPS 2026 oral writing corpus
Yoav Ger, Omri Barak · Official NeurIPS 2026 oral record
Figure, table and section ledger
Editorial contribution type: Theory & formal analysis (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 30 physical pages
- Before references / appendix
- ≈ 9.26 page-equivalents
- Main figures / tables
- 4 / 0
- Attached appendix figures / tables
- 8 / 0
- Attached appendix span
- ≈ 16.91 page-equivalents
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
| Top-level heading | Starts on PDF page | Estimated span | Share of entire PDF |
|---|---|---|---|
| 1 Introduction | 1 | ≈ 0.97 pp. | 3.23% |
| 2 Preliminaries | 2 | ≈ 1.02 pp. | 3.4% |
| 3 Low-rank linear RNN | 3 | ≈ 1.16 pp. | 3.87% |
| 4 Implications of visible and invisible overlaps | 4 | ≈ 1.94 pp. | 6.47% |
| 5 Low-rank nonlinear RNN | 6 | ≈ 1.85 pp. | 6.17% |
| 6 Related work | 8 | ≈ 0.86 pp. | 2.87% |
| 7 Discussion | 9 | ≈ 0.72 pp. | 2.4% |
| Acknowledgments | 10 | ≈ 0.17 pp. | 0.57% |
| References | 10 | ≈ 3.83 pp. | 12.77% |
| Appendix | 14 | ≈ 1 pp. | 3.33% |
| A Linear rank-1 RNN | 15 | ≈ 6 pp. | 20% |
| B Nonlinear rank-1 RNN | 21 | ≈ 6 pp. | 20% |
| C Derivation of the augmented Gram matrix Ḡ | 27 | ≈ 2 pp. | 6.67% |
| D Rank-2 linear RNN | 29 | ≈ 1.91 pp. | 6.37% |
All detected figure and table locations
Argument and organization
- Problem and contribution
The gap is between explaining a network's current input-output behavior and explaining how that network will learn next. The distinction between loss-visible and loss-invisible overlaps provides the vocabulary for expressing why functionally similar networks may retain different training histories.
- Argument sequence
The argument first defines the RNN and overlaps, then derives a tractable linear rank-one case before discussing hidden structure and history dependence. The nonlinear case follows with its Gaussian and large-network qualifications, rather than being presented as the same exact result.
- Evidence and verification
The nonlinear multi-panel figure aligns target and predicted trajectories, loss curves, Gaussian diagnostics and a history-decoding protocol. Visible and invisible overlaps are compared with and without noise, and the caption identifies error bars as one standard deviation rather than leaving uncertainty unspecified.
Problem, claim and contribution
The gap is between explaining a network's current input-output behavior and explaining how that network will learn next. The distinction between loss-visible and loss-invisible overlaps provides the vocabulary for expressing why functionally similar networks may retain different training histories.
Introduction and Sections 2–4, pp. 1–6 · Main paper
Argument and section order
The argument first defines the RNN and overlaps, then derives a tractable linear rank-one case before discussing hidden structure and history dependence. The nonlinear case follows with its Gaussian and large-network qualifications, rather than being presented as the same exact result.
Sections 2–5, pp. 2–8 · Main paper
Figures, captions and visual explanation
The opening explanatory figure links a network sketch, a parameter-to-overlap landscape projection and corresponding loss curves. It distinguishes the dynamics induced by the original parameterization from directly optimizing in overlap space, making a subtle mathematical non-equivalence visible.
Figure 1, p. 3; rendered page inspected · Main paper
Tables, comparisons and evidence
The nonlinear multi-panel figure aligns target and predicted trajectories, loss curves, Gaussian diagnostics and a history-decoding protocol. Visible and invisible overlaps are compared with and without noise, and the caption identifies error bars as one standard deviation rather than leaving uncertainty unspecified.
Figure 4, p. 7; rendered page inspected · Main paper
Appendix and reproducibility
Linear and nonlinear derivations are separated from training details and additional history experiments. Optimizer and distribution checks expose where the Gaussian or gradient-flow picture can fail. A higher-rank example is presented as an extension, not silently substituted for the rank-one derivation.
Appendices A–D, pp. 15–30; Figures 7–12 · Main paper
Clarity and writing conventions
The prose repeatedly separates behavior from learning and exact identities from closure assumptions. This consistent terminology lets the reader follow the result without treating numerical agreement as a proof of the nonlinear approximation.
Sections 3, 5 and discussion, pp. 3–9 · Main paper
What to reuse—and what not to infer
Transferable lesson: Introduce two easily conflated mathematical objects with one aligned schematic, then use diagnostics that directly test the assumptions needed to connect them.
Boundary: Rank-one and Gaussian-limit qualifications matter. The history-decoding demonstrations are not a universal claim about all RNNs or optimizers, and the manuscript scan is not independent proof verification.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-10-02
- Reading status
- Writing-focused manuscript scan + selected visuals inspected
- Main text
- Writing-oriented distributed manuscript scan of Sections 1–7, physical pp. 1–9: section sequence, substantive prose windows, captions, comparison contracts and conclusions. Not exhaustive technical verification.
- Appendix / supplement
- Writing-oriented distributed scan of Linear and nonlinear derivations, extra experiments and higher-rank discussion, physical pp. 15–30. Not independent proof checking; code and experiments were not executed.
- Visually inspected PDF pages
- 3, 7
This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.