Writing study: Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura, Okamoto, Daisuke Nohara, Jun Suzuki, Rio Yokota · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

25 physical pages

Before references / appendix

≈ 10.21 page-equivalents

Main figures / tables

8 / 1

Attached appendix figures / tables

17 / 3

Attached appendix span

≈ 9.45 page-equivalents

Before references / appendix: ≈ 40.84%References / intervening material: ≈ 21.36%Attached appendix: ≈ 37.8%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.23 pp.	4.92%
2 BACKGROUND AND RELATED WORK	2	≈ 1.49 pp.	5.96%
3 EXPERIMENTS	4	≈ 5.36 pp.	21.44%
4 DISCUSSION AND LIMITATIONS	9	≈ 0.67 pp.	2.68%
5 CONCLUSION	10	≈ 0.23 pp.	0.92%
6 REPRODUCIBILITY STATEMENT	10	≈ 0.17 pp.	0.68%
ACKNOWLEDGEMENTS	10	≈ 0.15 pp.	0.6%
AUTHOR CONTRIBUTIONS	10	≈ 0.35 pp.	1.4%
REFERENCES	11	≈ 5.34 pp.	21.36%
A TRAINING SETUP	16	≈ 1.69 pp.	6.76%
B EVALUATION SETUP	18	≈ 0.3 pp.	1.2%
C ADDITIONAL EXPERIMENTS	18	≈ 5.29 pp.	21.16%
D THE USE OF LARGE LANGUAGE MODELS	23	≈ 2.18 pp.	8.72%

All detected figure and table locations

Main text

• Figure 1 · p. 5
• Figure 2 · p. 5
• Figure 3 · p. 6
• Figure 4 · p. 6
• Figure 5 · p. 7
• Figure 6 · p. 7
• Figure 7 · p. 8
• Figure 8 · p. 9
• Table 1 · p. 16

Attached appendix

• Table 2 · p. 17
• Figure 9 · p. 17
• Table 3 · p. 18
• Figure 10 · p. 19
• Figure 11 · p. 19
• Figure 12 · p. 20
• Figure 13 · p. 20
• Figure 14 · p. 21
• Figure 15 · p. 21
• Figure 16 · p. 22
• Figure 17 · p. 22
• Figure 18 · p. 23
• Figure 19 · p. 23
• Figure 20 · p. 24
• Figure 21 · p. 24
• Figure 22 · p. 24
• Figure 23 · p. 25
• Figure 24 · p. 25
• Figure 25 · p. 25
• Table 4 · p. 25

Argument and organization

1. Loss is not downstream reasoning

   Introduces a counterexample to monotonic scaling intuition.

2. Active compute and total TPP

   Separates two experimental axes rather than calling all parameters capacity.

3. Stress tests and limitations

   Adds coding, post-training, decoding, depth and data-budget checks.

Problem, claim and contribution

The paper leads with a disagreement between improving pre-training loss and worsening mathematical accuracy. This concrete contradiction is stronger than a generic efficiency motivation and directly determines the experimental sweep.

§1 and Figure1, pp.1–2 · Main paper

Argument and section order

The results move from loss-to-loss and loss-to-accuracy plots to matched active-parameter comparisons, then total tokens per parameter. Defining total versus active parameters prevents the sparsity argument from silently changing its compute denominator.

§§3.2–3.4, pp.4–8 · Main paper

Figures, captions and visual explanation

Figure5 repeats task columns, aligns loss and accuracy rows, and uses color for active capacity. Dashed segments mark reversals; this visual grammar makes the exception to monotonic scaling inspectable despite a crowded panel grid.

Figure5, p.7 · Main paper

Tables, comparisons and evidence

The near-20 total-TPP reasoning optimum is an empirical result for the selected corpus and architecture. Table4 changes both token volume and mixture emphasis in its 1T extension, so it cannot isolate a universal effect of additional data.

§3.4, p.8; AppendixC.8 and Table4, pp.22–25 · Main paper

Appendix and reproducibility

Supplementary checks remove GSM8K-derived training sets, expand self-consistency budgets, increase depth and vary optimization. The appendix honestly notes that larger sampling mitigates the decline for top-8 but amplifies it for top-2, qualifying the simpler main-text narrative.

AppendicesC.2–C.8, pp.18–25; Figure13, p.20 · Main paper

Clarity and writing conventions

Calling TriviaQA and HellaSwag memorization benchmarks is a useful operational grouping, not a pure decomposition of cognition. The reproducibility statement still contains submission-time anonymity explanations, which should be updated in a final version.

§§4–6, pp.9–10; AppendixB, p.18 · Main paper

What to reuse—and what not to infer

Transferable lesson: Build a scaling paper around a falsifiable mismatch, hold comparison axes explicit, and use supplementary stress tests to narrow rather than inflate the headline claim.

Boundary: This is a Mixtral-style fixed-data study, not proof that sparse models cannot reason. Approximate FLOP matching does not establish equal latency, memory or communication cost; task labels and the TPP optimum remain protocol dependent.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

pp.1–10, §§1–6; contributions and references pp.10–15

Appendix / supplement

pp.16–25, A–D: training, evaluation, supplementary experiments and LLM-use disclosure

Visually inspected PDF pages

7, 25

• official ICLR 2026 proceedings PDF · 25 pages