Writing study: Mamba-3: Improved Sequence Modeling using State Space Principles

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Aakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, Albert Gu · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Architecture & representation design (primary) · Inference, algorithms & systems efficiency.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

30 physical pages

Before references / appendix

≈ 10.19 page-equivalents

Main figures / tables

2 / 4

Attached appendix figures / tables

4 / 8

Attached appendix span

≈ 12.86 page-equivalents

Before references / appendix: ≈ 33.97%References / intervening material: ≈ 23.17%Attached appendix: ≈ 42.87%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.51 pp.	5.03%
2 PRELIMINARIES	3	≈ 0.64 pp.	2.13%
3 METHODOLOGY	3	≈ 4.1 pp.	13.67%
4 EMPIRICAL VALIDATION	7	≈ 2.92 pp.	9.73%
5 CONCLUSION AND FUTURE WORK	10	≈ 0.42 pp.	1.4%
REFERENCES	11	≈ 6.95 pp.	23.17%
A RELATED WORK	18	≈ 1.3 pp.	4.33%
B EXPONENTIAL-TRAPEZOIDAL DISCRETIZATION	19	≈ 3 pp.	10%
C COMPLEX SSM PROOFS	22	≈ 3.19 pp.	10.63%
D MIMO FOR MAMBA-3	25	≈ 1.65 pp.	5.5%
E EXPERIMENTAL DETAILS	27	≈ 0.34 pp.	1.13%
F ADDITIONAL EXPERIMENTAL RESULTS	27	≈ 1.49 pp.	4.97%
G ARCHITECTURE ABLATIONS	29	≈ 0.41 pp.	1.37%
H INFERENCE KERNEL LATENCY ANALYSIS	29	≈ 1.48 pp.	4.93%

All detected figure and table locations

Main text

• Figure 1 · p. 4
• Table 1 · p. 8
• Table 2 · p. 9
• Table 3 · p. 9
• Table 4 · p. 10
• Figure 2 · p. 10

Attached appendix

• Table 5 · p. 19
• Table 6 · p. 22
• Figure 3 · p. 25
• Table 7 · p. 25
• Figure 4 · p. 27
• Table 8 · p. 28
• Figure 5 · p. 28
• Figure 6 · p. 28
• Table 9 · p. 29
• Table 10 · p. 29
• Table 11 · p. 30
• Table 12 · p. 30

Argument and organization

1. Three bottlenecks, three changes

   Introduces quality, state-tracking capability and hardware utilization as separate motivations for the three model changes.

2. Principle to recurrence to implementation

   Derives discretization and complex-state updates before discussing MIMO and the final block architecture.

3. Distinct evaluation axes

   Separates language quality, retrieval, synthetic state tracking, state-size trade-offs and measured kernel latency.

Problem, claim and contribution

The introduction repeatedly maps each limitation to one mechanism and one empirical claim: discretization to quality, complex states to tracking, and MIMO to memory-bound decoding. This repeated structure makes a technically dense three-part contribution easier to follow.

Sections 1, 3.1–3.3 and 5 · Main paper

Argument and section order

Numbered propositions supply the main mathematical bridge, while short remarks identify special cases and qualifications. In particular, the error-rate remark states a condition on the trapezoidal gate and immediately says the practical model does not enforce it.

Propositions 1–4; Remarks 2–4; Appendix B.4 · Main paper

Figures, captions and visual explanation

The appendix architecture comparison keeps the Mamba-2 and Mamba-3 blocks aligned and uses a shared legend for projections, transformations, optional MIMO operations and nonlinearities. The differences become visible without requiring the reader to compare two unrelated layouts.

Figure 3, page 25 · Main paper

Tables, comparisons and evidence

The retrieval table groups pure and hybrid models, names training-context and extrapolation lengths, and admits that fixed-state models still lose information-extraction capability. The adjacent ablations distinguish language-modeling perplexity from synthetic state-tracking accuracy rather than presenting them as the same capability.

Tables 2–3, page 9 · Main paper

Appendix and reproducibility

The appendix records the MLP-width changes used to parameter-match MIMO models and the implementation languages/fusion boundaries used in speed tests. Those details reveal what is actually controlled: parameter matching does not make training FLOPs equal, and different kernels are part of a wall-clock comparison.

Appendix D, page 27; Appendix H, Tables 10–12 · Main paper

Clarity and writing conventions

The paper usefully names state size and perplexity as proxies before showing timing measurements. Nevertheless, the statement that SISO is lowest-latency in every configuration exceeds Table 4: Mamba-2 is faster in its FP32, state-size-64 column. Quantifiers such as every should be checked against all cells.

Section 4.3–4.4; Table 4, page 10 · Main paper

What to reuse—and what not to infer

Transferable lesson: Organize a multi-component architecture around separate failure modes and preserve that mapping through derivation, ablation and measured outcomes. Compare old/new block diagrams in the same visual coordinate system, and report exactly how model size and runtime protocols are matched.

Boundary: The inspected experiments concern language and synthetic sequences, not CV, pathology or a pretrained WSI encoder. MIMO adds training/prefill cost despite efficient decoding. The learned gate does not enforce the condition used for the stated higher-order approximation, and hybrid retrieval results depend on normalization and attention choices.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

All Sections 1–5, pages 1–10, including recurrence derivations, state-space/rotary equivalences, architecture and evaluations.

Appendix / supplement

All Appendices A–H, pages 18–30: related work, discretization/complex-SSM proofs, MIMO parameterization, experimental settings, further results and kernel measurements.

Visually inspected PDF pages

9, 25

• Official ICLR 2026 proceedings manuscript with appendices A–H · 30 pages