Writing study: Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

John Cooper, Ilias Diakonikolas, Mingchen Ma, Frederic Sala · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal analysis (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

25 physical pages

Before references / appendix

≈ 8.18 page-equivalents

Main figures / tables

8 / 1

Attached appendix figures / tables

8 / 1

Attached appendix span

≈ 14.81 page-equivalents

Before references / appendix: ≈ 32.72%References / intervening material: ≈ 8.04%Attached appendix: ≈ 59.24%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Source-checked physical-page text restores wrapped top-level heading labels on pp. 3. Heading starts and measured positions are unchanged; this is label completeness, not a new section. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 Introduction	1	≈ 1.02 pp.	4.08%
2 Preliminaries and Notations	2	≈ 0.71 pp.	2.84%
3 Function Compositions and Limitations of Pure Models	3	≈ 1.23 pp.	4.92%
4 Hybrid Model for S2FC	4	≈ 1.58 pp.	6.32%
5 Experiments	5	≈ 2.89 pp.	11.56%
6 Conclusion	8	≈ 0.25 pp.	1%
7 Acknowledgments	9	≈ 0.08 pp.	0.32%
Impact Statement	9	≈ 0.05 pp.	0.2%
References	9	≈ 2.01 pp.	8.04%
A Related Works	11	≈ 0.56 pp.	2.24%
B Complete Preliminaries and Notations	11	≈ 1.77 pp.	7.08%
C Omitted Proofs in Section 3	13	≈ 1.78 pp.	7.12%
D Omitted Proofs in Section 4	15	≈ 6.49 pp.	25.96%
E Experiment Details	21	≈ 4.21 pp.	16.84%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Figure 3 · p. 5
• Figure 4 · p. 6
• Figure 5 · p. 7
• Figure 6 · p. 7
• Figure 7 · p. 8
• Figure 8 · p. 8
• Table 1 · p. 8

Attached appendix

• Table 2 · p. 12
• Figure 9 · p. 21
• Figure 10 · p. 21
• Figure 11 · p. 22
• Figure 12 · p. 23
• Figure 13 · p. 24
• Figure 14 · p. 24
• Figure 15 · p. 24
• Figure 16 · p. 25

Argument and organization

1. Problem and contribution

   Distinguishes input-dependent working memory from input-independent parameters, then defines a compositional task family that requires both control extraction and retrieval.

2. Argument sequence

   Pure SSM and sliding-window attention limitations precede hybrid constructions; experiments explicitly split construction validation, learned realizations and broader-distribution tests into C1-C3.

3. Evidence and verification

   The implementation tests hand-constructed models separately from training, and reverse hybrids test layer order. The appendix reports identical learning-rate sweeps and 11 runs with 10th/90th percentile bars.

Problem, claim and contribution

Distinguishes input-dependent working memory from input-independent parameters, then defines a compositional task family that requires both control extraction and retrieval.

Sections 2-3, pp. 2-3 · Main paper

Argument and section order

Pure SSM and sliding-window attention limitations precede hybrid constructions; experiments explicitly split construction validation, learned realizations and broader-distribution tests into C1-C3.

Sections 3-5, pp. 3-6 · Main paper

Figures, captions and visual explanation

A token-level construction diagram explains the division of labor between SSM and attention; parameter-count and length plots share readable model legends while the OOD table retains a setting where the pure SSM wins.

Figure 3, p. 5; Figures 7-8 and Table 1, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

The implementation tests hand-constructed models separately from training, and reverse hybrids test layer order. The appendix reports identical learning-rate sweeps and 11 runs with 10th/90th percentile bars.

Sections 5.1-5.2, p. 6; Appendix E.1, p. 21 · Main paper

Appendix and reproducibility

Proof headings mirror theorem numbers, while additional window/state experiments explicitly distinguish optimization failures from expressivity predictions.

Appendices C-D, pp. 13-20; E.2, pp. 22-25 · Main paper

Clarity and writing conventions

The conclusion limits the results to synthetic tasks and restricted attention; the source's p. 8 table caption says consistently although its last row favors SSM, so prose must retain the exception.

Table 1 and conclusion, p. 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Separate existence, implemented construction and learned performance into different evidence obligations; report when optimization breaks the trend expected from expressivity.

Boundary: Theorems depend on task distributions and attention restrictions; memory bounds, parameter counts and practical inference cost are not interchangeable, and synthetic separations do not establish universal hybrid superiority.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-6, pp. 1-8: memory distinctions, compositional task family, pure-model lower bounds, explicit hybrid constructions, learnability and transfer experiments.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-E, pp. 11-25: related work, full definitions, indexed proofs, implementation of constructions, shared training sweeps and extended experiments.

Visually inspected PDF pages

5, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 25 pages