Writing study: Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Houyi Li, Ka Man Lo, Shijie Xuyang, Ziqi Wang, Wenzhen Zheng, Haocheng Zhang, Zhao Li, Shuigeng Zhou, Xiangyu Zhang, Daxin Jiang · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

22 physical pages

Before references / appendix

≈ 10.11 page-equivalents

Main figures / tables

3 / 2

Attached appendix figures / tables

2 / 15

Attached appendix span

≈ 7.18 page-equivalents

Before references / appendix: ≈ 45.95%References / intervening material: ≈ 21.41%Attached appendix: ≈ 32.64%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.07 pp.	4.86%
2 RELATED WORK	2	≈ 0.99 pp.	4.5%
3 EXPERIMENTAL METHODOLOGY	3	≈ 1.89 pp.	8.59%
4 OPTIMIZED MOE ARCHITECTURE	5	≈ 1 pp.	4.55%
5 OPTIMAL ACTIVATION RATE	6	≈ 1.98 pp.	9%
6 DATA REUSE STRATEGY	8	≈ 1.45 pp.	6.59%
7 ANALYSIS OF DOWNSTREAM PERFORMANCE	10	≈ 0.51 pp.	2.32%
8 CONCLUSION AND FUTURE WORKS	10	≈ 0.43 pp.	1.95%
REFERENCES	11	≈ 4.71 pp.	21.41%
APPENDIX	15	≈ 0.04 pp.	0.18%
A BACKGROUND: MIXTURE-OF-EXPERTS	15	≈ 0.68 pp.	3.09%
B EXTENDED ANALYSIS ON RELATED WORK	16	≈ 0.89 pp.	4.05%
C PRETRAINING DATA RECIPE	17	≈ 0.09 pp.	0.41%
D COMPREHENSIVE LIST OF BENCHMARKS	17	≈ 0.39 pp.	1.77%
E MORE EXPERIMENTAL RESULTS	17	≈ 4.21 pp.	19.14%
F THE USE OF LARGE LANGUAGE MODELS: AN EXPLANATION	22	≈ 0.89 pp.	4.05%

All detected figure and table locations

Main text

• Table 1 · p. 3
• Figure 1 · p. 6
• Figure 2 · p. 7
• Figure 3 · p. 8
• Table 2 · p. 9

Attached appendix

• Table 3 · p. 17
• Figure 4 · p. 18
• Table 4 · p. 18
• Table 5 · p. 18
• Table 6 · p. 18
• Table 7 · p. 18
• Table 8 · p. 19
• Table 9 · p. 19
• Table 10 · p. 19
• Table 11 · p. 20
• Table 12 · p. 20
• Table 13 · p. 20
• Table 14 · p. 20
• Table 15 · p. 20
• Figure 5 · p. 21
• Table 16 · p. 21
• Table 17 · p. 21

Separate source files—excluded from canonical-PDF totals

A separate author version may repeat main text and references. Whole-file lengths and counts are not extra appendix material and are never added to the canonical PDF totals. Read the study's source scope for which sections received substantive reading versus layout inspection.

Additional source 1

Separate PDF (or ZIP containing it): whole-file layout of 20 physical pages; 5 detected figures and 16 tables. These may include overlapping main-text material.

Located appendix only: starts on physical p.14; ≈ 6.71 page-equivalents, 2 figure labels and 14 table labels. This interval is distinct from the whole file.

Detected heading	PDF page	Estimated span
1 INTRODUCTION	1	≈ 1.05 pp.
2 RELATED WORK	2	≈ 0.8 pp.
3 EXPERIMENTAL METHODOLOGY	3	≈ 2.11 pp.
4 OPTIMIZED MOE ARCHITECTURE	5	≈ 0.96 pp.
5 OPTIMAL ACTIVATION RATE	6	≈ 1.96 pp.
6 DATA REUSE STRATEGY	8	≈ 0.76 pp.
7 ANALYSIS OF DOWNSTREAM PERFORMANCE	9	≈ 0.5 pp.
8 CONCLUSION AND FUTURE WORKS	9	≈ 0.43 pp.
REFERENCES	10	≈ 4.18 pp.
APPENDIX	14	≈ 0.04 pp.
A BACKGROUND: MIXTURE-OF-EXPERTS	14	≈ 0.23 pp.
B EXTENDED ANALYSIS ON RELATED WORK	14	≈ 0.56 pp.
C COMPREHENSIVE LIST OF BENCHMARKS	15	≈ 0.42 pp.
D MORE EXPERIMENTAL RESULTS	15	≈ 0.91 pp.
E THE USE OF LARGE LANGUAGE MODELS: AN EXPLANATION	16	≈ 4.56 pp.

• Table 1 · p. 3
• Figure 1 · p. 5
• Figure 2 · p. 6
• Figure 3 · p. 7
• Table 2 · p. 8
• Figure 4 · p. 15
• Table 3 · p. 15
• Table 4 · p. 16
• Table 5 · p. 16
• Table 6 · p. 17
• Table 7 · p. 17
• Table 8 · p. 17
• Table 9 · p. 18
• Table 10 · p. 18
• Table 11 · p. 18
• Table 12 · p. 18
• Table 13 · p. 19
• Figure 5 · p. 19
• Table 14 · p. 19
• Table 15 · p. 20
• Table 16 · p. 20

Argument and organization

1. A controlled-comparison question

   Defines parameter, compute and data budgets, then orders experiments as architecture tuning, activation-rate sweeps and unique-data reuse.

2. Upstream, downstream and reproducibility detail

   Contrasts pretraining loss curves with task-level evaluation and provides complete configuration tables.

Problem, claim and contribution

An explicit question about three resource constraints is more informative than an unqualified claim that sparse models are more efficient.

Introduction, p. 2. · Main paper · Additional source 2

Argument and section order

The three-step experimental plan is announced before results, so each later section answers one identified confound.

Section 3.3, p. 5; Sections 4–6. · Main paper · Additional source 2

Figures, captions and visual explanation

Paired panels switch between fixed-data and fixed-compute views; horizontal dense baselines make the useful activation-rate region visually interpretable.

Figure 1, p. 6. · Main paper · Additional source 2

Tables, comparisons and evidence

A compact task table exposes important exceptions hidden by domain averages: strict reuse improves some tasks but lowers MMLU, MMLU-Redux and several mathematics/code scores.

Table 2, p. 9. · Main paper · Additional source 2

Appendix and reproducibility

Architecture, tokens, epochs, FLOPs and optimization settings are tabulated per run, while the corpus recipe names source families rather than releasing the internal dataset itself.

Appendices C–E, pp. 17–21. · Main paper · Additional source 2

Clarity and writing conventions

The reuse discussion distinguishes unique from consumed tokens, but the headline word 'strictly' is stronger than approximate parameter matching and theoretical compute accounting justify.

Sections 3.1–3.3 and 6; Tables 12–15. · Main paper · Additional source 2

What to reuse—and what not to infer

Transferable lesson: For efficiency papers, organize the narrative around controlled resources and expose the accounting in per-run tables rather than relying on a single performance-versus-size chart.

Boundary: The approximately 20% activation optimum is empirical within the studied sizes and recipe, not a universal scaling law. Equal FLOPs are not equal wall-clock cost. Check algebra independently: Section 3.2's token-multiplier wording and Equation 6 contain inconsistencies, and downstream gains are not uniform.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Canonical conference paper pp. 1–10, Sections 1–8 and limitations.

Appendix / supplement

Canonical paper pp. 15–22, Appendices A–F. Separately linked supplement identified from pp. 1–6 as an older anonymous full-paper draft; not used as the canonical evidence version.

Visually inspected PDF pages

6, 9

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 22 pages
• Supplemental-Conference file: older anonymous review draft, not an additional appendix; only opening pages inspected for version identification. · 20 pages