Writing study: TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu, Wanru Zhao, Marco Ciccone, Colin Raffel · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

46 physical pages

Before references / appendix

≈ 9.07 page-equivalents

Main figures / tables

2 / 2

Attached appendix figures / tables

11 / 30

Attached appendix span

≈ 30.21 page-equivalents

Before references / appendix: ≈ 19.72%References / intervening material: ≈ 14.61%Attached appendix: ≈ 65.67%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.55 pp.	3.37%
2. Background	2	≈ 1.15 pp.	2.5%
3. The TokSuite Models	4	≈ 1.12 pp.	2.43%
4. The TokSuite Benchmarks	5	≈ 1.4 pp.	3.04%
5. Findings	6	≈ 1.5 pp.	3.26%
6. Related Work	8	≈ 0.63 pp.	1.37%
7. Discussion	8	≈ 0.51 pp.	1.11%
8. Future Work & Limitations	9	≈ 0.21 pp.	0.46%
9. Conclusion	9	≈ 0.26 pp.	0.57%
Acknowledgements	9	≈ 0.16 pp.	0.35%
Impact Statement	9	≈ 0.23 pp.	0.5%
References	10	≈ 6.73 pp.	14.63%
A. Tokenizer Processing Glossary	16	≈ 1.9 pp.	4.13%
B. Intrinsic Tokenization Efficiency Metrics	18	≈ 1.13 pp.	2.46%
C. Model Training	19	≈ 3.5 pp.	7.61%
D. TokSuite Benchmark Details	23	≈ 1.54 pp.	3.35%
E. Detailed Benchmark Results	24	≈ 13.22 pp.	28.74%
F. Decoding Algorithms	38	≈ 1.15 pp.	2.5%
G. Additional Ablations	39	≈ 4.12 pp.	8.96%
H. Statistical Significance	43	≈ 3.65 pp.	7.93%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 4
• Table 1 · p. 7
• Table 2 · p. 16

Attached appendix

• Table 3 · p. 17
• Table 4 · p. 19
• Figure 3 · p. 20
• Figure 4 · p. 20
• Figure 5 · p. 21
• Figure 6 · p. 21
• Figure 7 · p. 22
• Table 5 · p. 22
• Figure 8 · p. 25
• Table 6 · p. 25
• Table 7 · p. 25
• Table 8 · p. 26
• Table 9 · p. 26
• Table 10 · p. 27
• Table 11 · p. 28
• Table 12 · p. 29
• Table 13 · p. 32
• Table 14 · p. 32
• Table 15 · p. 32
• Table 16 · p. 34
• Table 17 · p. 34
• Table 18 · p. 36
• Table 19 · p. 37
• Figure 9 · p. 37
• Table 20 · p. 38
• Table 21 · p. 38
• Table 22 · p. 39
• Table 23 · p. 39
• Table 24 · p. 40
• Table 25 · p. 40
• Table 26 · p. 40
• Table 27 · p. 41
• Table 28 · p. 41
• Table 29 · p. 41
• Table 30 · p. 42
• Table 31 · p. 42
• Table 32 · p. 44
• Figure 10 · p. 45
• Figure 11 · p. 45
• Figure 12 · p. 46
• Figure 13 · p. 46

Argument and organization

1. Problem and claim

   Explains why comparing existing language models cannot isolate tokenizer effects, then introduces both controlled models and language-specific perturbations as the missing experimental instrument.

2. Argument sequence

   Defines tokenizer design choices before model construction, then describes the benchmark before presenting findings. Related work, practical interpretation and explicit future-work limitations are separated.

3. Evidence and verification

   The main protocol fixes token budget rather than raw text volume and openly states that byte exposure differs. Appendix G.2 tests a fixed-text alternative; reporting both makes the practical comparison and its confound inspectable.

Problem, claim and contribution

Explains why comparing existing language models cannot isolate tokenizer effects, then introduces both controlled models and language-specific perturbations as the missing experimental instrument.

Introduction and contribution framing, pp. 1–2 · Main paper

Argument and section order

Defines tokenizer design choices before model construction, then describes the benchmark before presenting findings. Related work, practical interpretation and explicit future-work limitations are separated.

Sections 2–9, pp. 2–9 · Main paper

Figures, captions and visual explanation

Figure 2 compresses shared training data, fourteen tokenizer choices and a common model architecture into one design diagram. Figure 10 uses perturbation-specific horizontal interval plots, with bootstrap endpoints explained in the caption.

Figure 2, physical p. 4; Figures 10–11, physical p. 45; rasters inspected · Main paper

Tables, comparisons and evidence

The main protocol fixes token budget rather than raw text volume and openly states that byte exposure differs. Appendix G.2 tests a fixed-text alternative; reporting both makes the practical comparison and its confound inspectable.

Section 3.3, p. 4; Appendix C.3, pp. 22–23; Appendix G.2 and Tables 24–26, p. 40 · Main paper

Appendix and reproducibility

An appendix contents list routes readers to tokenizer definitions, complete questions, perturbation cases and statistical diagnostics. Granular examples explain what a category means rather than supplying only a leaderboard.

Appendix contents, p. 16; Appendices D–H, pp. 23–46 · Main paper

Clarity and writing conventions

The paper distinguishes an off-the-shelf design comparison from a controlled intervention on every tokenizer component. Its limitations acknowledge incomplete isolation of tokenization decisions despite shared model controls.

Section 8, p. 9; model construction, pp. 4–5 · Main paper

What to reuse—and what not to infer

Transferable lesson: For a benchmark paper, make the unit of comparison, fixed budget and perturbation taxonomy explicit before reporting rankings.

Boundary: Figure 10 colors are based on distance from a cross-model mean, whereas bootstrap intervals and the paired tests in Appendix H answer different statistical questions; do not label all three as one significance test.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused scan of Sections 1–9, pp. 1–9, including budget controls, perturbation construction, measurement definitions and limitations; impact statement continues onto p. 10.

Appendix / supplement

Structural and selected-detail scan of Appendices A–H, pp. 16–46: glossary, intrinsic metrics, model training, benchmark examples, granular results, decoding and scale/text-budget/seed ablations, significance tables and interval plots. Not a statistical reanalysis.

Visually inspected PDF pages

4, 45

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 46 pages