Writing study: From Markov to Laplace: How Mamba In-Context Learns Markov Chains

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Marco Bondaschi, Nived Rajaraman, Xiuying Wei, Razvan Pascanu, Caglar Gulcehre, Michael Gastpar, Ashok Makkuva · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal foundations (primary) · Empirical analysis, diagnosis & auditing.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

31 physical pages

Before references / appendix

≈ 9.47 page-equivalents

Main figures / tables

3 / 1

Attached appendix figures / tables

10 / 8

Attached appendix span

≈ 17.89 page-equivalents

Before references / appendix: ≈ 30.55%References / intervening material: ≈ 11.74%Attached appendix: ≈ 57.71%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.74 pp.	5.61%
2 PROBLEM SETUP	3	≈ 2.32 pp.	7.48%
3 DOES MAMBA LEARN IN-CONTEXT ESTIMATORS?	5	≈ 1.09 pp.	3.52%
4 HOW MAMBA IMPLEMENTS THE LAPLACIAN ESTIMATOR	6	≈ 2.66 pp.	8.58%
5 BEYOND MARKOV	9	≈ 0.8 pp.	2.58%
6 CONCLUSION	10	≈ 0.19 pp.	0.61%
ACKNOWLEDGMENTS	10	≈ 0.07 pp.	0.23%
REFERENCES	10	≈ 3.64 pp.	11.74%
A PRELIMINARIES ON LAPLACIAN SMOOTHING	14	≈ 2 pp.	6.45%
B MAMBA-BASED LANGUAGE MODELING ARCHITECTURE	16	≈ 1 pp.	3.23%
C PRELIMINARIES AND PROOF OF THM. 1 AND THM. 4	17	≈ 5 pp.	16.13%
D PROOF OF THM. 2	22	≈ 3 pp.	9.68%
E ADDITIONAL RESULTS	25	≈ 5 pp.	16.13%
F MODEL ARCHITECTURES AND HYPER-PARAMETERS	30	≈ 1.89 pp.	6.1%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Figure 3 · p. 6
• Table 1 · p. 9

Attached appendix

• Figure 4 · p. 17
• Figure 5 · p. 18
• Figure 6 · p. 24
• Figure 7 · p. 25
• Figure 8 · p. 25
• Figure 9 · p. 26
• Figure 10 · p. 26
• Table 2 · p. 27
• Figure 11 · p. 27
• Figure 12 · p. 27
• Table 3 · p. 28
• Figure 13 · p. 28
• Table 4 · p. 28
• Table 5 · p. 29
• Table 6 · p. 30
• Table 7 · p. 30
• Table 8 · p. 30
• Table 9 · p. 31

Argument and organization

1. Observable phenomenon → architectural ablation

   Finds a simple explanation target before formal analysis.

2. Simplified model → construction → lower bound

   Pairs an existence result with a capacity limitation.

3. Beyond Markov

   Tests which lessons survive a change of data distribution and model depth.

Problem, claim and contribution

Randomly drawing a new transition matrix for each sequence gives an identifiable in-context task: memorizing one training chain cannot solve the held-out chain.

Section 2.1, p. 3 · Main paper

Argument and section order

The narrative moves from a convolution-removal failure to MambaZero, then to a constructive counting mechanism. Naming the simplified model makes its modified normalization visible.

Sections 3–4.2, pp. 5–8; Figure 3, p. 6 · Main paper

Figures, captions and visual explanation

The opening pairs one-sequence probability traces with aggregate deviation across Markov orders; its caption identifies standard deviations over five runs rather than leaving error bars unexplained.

Figure 1, p. 2 · Main paper

Tables, comparisons and evidence

Representation existence is distinguished from learning dynamics. The exact construction covers finite-state first-order chains, while higher-order successes are empirical and finite-precision recurrence incurs an exponential memory lower bound.

Theorems 1–2 and Section 4.3, pp. 7–9 · Main paper

Appendix and reproducibility

Supplementary tests add convolution to transformers, remove it from deeper Mamba, vary width and hold out parts of the transition simplex. This tests the proposed component rather than only the winning architecture.

Appendix E.5–E.8, pp. 26–29 · Main paper

Clarity and writing conventions

The natural-language follow-up qualifies the original lesson: gating also matters, and convolution's relative benefit shrinks in deeper models. Architecture-shape and training-setting tables separately support reproduction.

Section 5.2, pp. 9–10; Appendices E.8 and F, pp. 28–31 · Main paper

What to reuse—and what not to infer

Transferable lesson: Build a mechanistic paper around a tractable task with a known optimal estimator, then distinguish observed learning, representational construction and capacity limits.

Boundary: The MambaZero proof uses L1-normalized nonnegative logits, not standard softmax Mamba training. First-order representability and small language-model ablations do not prove universal in-context optimality or broad superiority over transformers.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–6, all setup, empirical observations, constructive result, lower bound and natural-language extension.

Appendix / supplement

Pages 14–31, Appendices A–F, complete estimator and representation proofs, all supplementary experiments and hyperparameter tables.

Visually inspected PDF pages

2, 6, 30

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 31 pages