Writing study: Equivalence of Context and Parameter Updates in Modern Transformer Blocks

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Adrian Goldwaser, Michael Munn, Javier Gonzalvo, Benoit Dherin · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal analysis (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

25 physical pages

Before references / appendix

≈ 9.08 page-equivalents

Main figures / tables

5 / 1

Attached appendix figures / tables

7 / 0

Attached appendix span

≈ 13.91 page-equivalents

Before references / appendix: ≈ 36.32%References / intervening material: ≈ 8.04%Attached appendix: ≈ 55.64%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Extraction notes: Figure numbering discontinuity; see source note. Zero means no recognized caption, not independently verified absence.

Source-bound correction: Join the wrapped section3 title. Figure-numbering gap10 remains unresolved intentionally: physical p19 contains five stacked plot panels overflowing the bottom edge, with no visible or extractable parent Figure10 caption. Do not infer a caption count from sequence alone; p19 rendered and visually checked.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.83 pp.	3.32%
2. Background	2	≈ 0.65 pp.	2.6%
3. Results: Implicit Updates in Modern Transformers	2	≈ 2.1 pp.	8.4%
4. Experimental Validation	4	≈ 2 pp.	8%
5. A General Framework for Implicit Updates	6	≈ 2.3 pp.	9.2%
6. Conclusion	9	≈ 0.64 pp.	2.56%
Acknowledgments	9	≈ 0.05 pp.	0.2%
Impact Statement	9	≈ 0.18 pp.	0.72%
References	10	≈ 2 pp.	8%
A. General Proofs	12	≈ 1.13 pp.	4.52%
B. Appendix: Numerically Stable Update and RMSNorm Inversion	13	≈ 2.48 pp.	9.92%
C. Extended Experimental Results	15	≈ 3.14 pp.	12.56%
D. Impossibility of Controllability for RMS Post-Norm Without a Trainable Scaling Vector	18	≈ 1.76 pp.	7.04%
E. Deriving the Implicit Update as a Gradient Step	20	≈ 5.41 pp.	21.64%

All detected figure and table locations

Main text

• Figure 1 · p. 3
• Figure 2 · p. 4
• Figure 3 · p. 5
• Figure 4 · p. 7
• Figure 5 · p. 7
• Table 1 · p. 8

Attached appendix

• Figure 6 · p. 16
• Figure 7 · p. 16
• Figure 8 · p. 17
• Figure 9 · p. 18
• Figure 11 · p. 23
• Figure 12 · p. 24
• Figure 13 · p. 25

Argument and organization

1. 1 Introduction; 2 Background; 3 Implicit Updates in Modern Transformers

   Extends a vanilla-block equivalence through a concrete Gemma construction and layerwise induction.

2. 4 Experimental Validation

   Compares matched-history logits across numerical precisions and stabilization variants.

3. 5 A General Framework; 6 Conclusion

   Abstracts input/output controllability and states token-dependence and architectural limits.

Problem, claim and contribution

The gap is existence and form of context-equivalent updates in modern gated and normalized blocks, not an efficient reusable replacement for prompting.

Introduction and Background, pp.1-2 · Main paper

Argument and section order

A concrete block theorem and constructive algorithm precede the general controllability abstraction, allowing the reader to see the mechanism before its taxonomy.

Sections 3-5, pp.3-8 · Main paper

Figures, captions and visual explanation

The block diagram uses the same colored parameters as the equations; precision plots show distribution distances and token mismatches, including an image-context example.

Figure 1, p.3; Figures 4-5, p.7 · Main paper

Tables, comparisons and evidence

Updates are recomputed for each token, and diverging predictions are followed using the baseline token to keep histories identical during the comparison.

Experimental setup, p.5 · Main paper

Appendix and reproducibility

The stable variant separates approximate RMSNorm inversion from a residual scale correction, while an impossibility proof explains why a fixed post-normalization scale fails controllability.

Appendix B, p.13; Appendix D, p.20 · Main paper

Clarity and writing conventions

The limitations explicitly distinguish an algebraic descriptive equivalence from evidence that a transformer executes that learning algorithm in ordinary inference.

Section 6.1, p.9 · Main paper

What to reuse—and what not to infer

Transferable lesson: State the unit of equivalence and comparison protocol precisely, and use an architectural counterexample to expose the assumptions behind a broad theorem.

Boundary: These are history- and token-dependent patches under nonzero-activation/controllability conditions, not a global reusable context compression or inference-speed guarantee; finite precision creates measurable deviations.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 2, 3, 4, 5, 6, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 13, 17, 20. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

3, 7

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 25 pages