Writing study: Sequences of Logits Reveal the Low Rank Structure of Language Models

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Noah Golowich, Allen Liu, Abhishek Shetty · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Theory & formal foundations.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

37 physical pages

Before references / appendix

≈ 10.3 page-equivalents

Main figures / tables

6 / 0

Attached appendix figures / tables

6 / 3

Attached appendix span

≈ 22.35 page-equivalents

Before references / appendix: ≈ 27.84%References / intervening material: ≈ 11.76%Attached appendix: ≈ 60.41%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2.69 pp.	7.27%
2 SETUP AND NOTATION	4	≈ 0.66 pp.	1.78%
3 EXPERIMENTS	4	≈ 4.61 pp.	12.46%
4 THEORETICAL RESULTS	9	≈ 1.25 pp.	3.38%
5 CONCLUSIONS	10	≈ 0.16 pp.	0.43%
ACKNOWLEDGEMENTS	10	≈ 0.24 pp.	0.65%
ETHICS STATEMENT	11	≈ 0.11 pp.	0.3%
REPRODUCIBILITY STATEMENT	11	≈ 0.08 pp.	0.22%
REFERENCES	11	≈ 4.35 pp.	11.76%
A RELATED WORK	15	≈ 1.01 pp.	2.73%
B EXPERIMENTAL DETAILS	16	≈ 11.45 pp.	30.95%
C COMPARISON TO NATURAL BASELINES	28	≈ 0.49 pp.	1.32%
D DEFERRED PROOFS OF THEORETICAL RESULTS	28	≈ 8.82 pp.	23.84%
E FUTURE DIRECTIONS	37	≈ 0.32 pp.	0.86%
F STATEMENT ON USE OF LLMS	37	≈ 0.25 pp.	0.68%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 5
• Figure 3 · p. 6
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 8

Attached appendix

• Figure 7 · p. 18
• Figure 8 · p. 19
• Figure 9 · p. 20
• Figure 10 · p. 20
• Figure 11 · p. 22
• Figure 12 · p. 23
• Table 1 · p. 24
• Table 2 · p. 26
• Table 3 · p. 27

Argument and organization

1. Intuitive compression viewpoint and formal logit matrix

   Motivates a model-agnostic object before specifying mean-centered next-token blocks.

2. Empirical low rank, transfer and generation

   Builds from a structural observation to a consequence that can be tested.

3. ISAN theory and future directions

   Separates exact-rank guarantees from approximate empirical behavior.

Problem, claim and contribution

The opening asks what a low-dimensional representation must preserve—future generation—and then contrasts multi-token structure with the familiar one-token softmax bottleneck.

Section 1.1, pp. 2–3 · Main paper

Argument and section order

Three empirical subsections form an argument chain: low rank exists, row relations transfer, and those relations enable a generation procedure.

Sections 3.1–3.3 · Main paper

Figures, captions and visual explanation

Figure 1 juxtaposes rank–error scaling and token-position generation error. Figure 5's repeated gray history and green continuation blocks make the shared-prefix query operation easy to follow.

Figures 1 and 5, pp. 2 and 8 · Main paper

Tables, comparisons and evidence

The single-token variant succeeds at the first token but deteriorates later, providing a diagnostic control for why extending the logit matrix matters.

Figure 6 and Section 3.3, pp. 8–9 · Main paper

Appendix and reproducibility

Appendix B discloses token-column subsampling, power-law fitting weights, sequence lengths and non-cherry-picked examples; Appendix D separates equivalence, expressivity, query learning and approximate generation.

Appendices B and D, pp. 16–36 · Main paper

Clarity and writing conventions

The approachable introductory matrix is explicitly distinguished from the formal mean-centered definition, but numerous references label definitions as theorems and should be cleaned up.

Definition 2.2 and its explanatory paragraph, p. 4; Sections 3–4 · Main paper

What to reuse—and what not to infer

Transferable lesson: Make the key mathematical object answer an intuitive question, then demonstrate one concrete consequence with a control that would fail if the proposed distinction did not matter.

Boundary: Finite sampled submatrices do not establish global approximate rank. LINGEN coefficient fitting still queries target-containing sequences, so generation-time unrelated prompts do not imply a demonstrated end-to-end target-free jailbreak. Exact-rank query-learning guarantees are not guarantees for approximate practical models.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–5 and ethics/reproducibility

Appendix / supplement

Pages 15–37, Appendices A–F, including complete experimental specification and deferred proofs

Visually inspected PDF pages

2, 8

• ICLR 2026 final proceedings · 37 pages