Writing study: Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Ang Lv, Jin Ma, Yiyuan Ma, Siyuan Qiao · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Learning & optimization algorithms (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

21 physical pages

Before references / appendix

≈ 10.11 page-equivalents

Main figures / tables

7 / 1

Attached appendix figures / tables

3 / 2

Attached appendix span

≈ 6.54 page-equivalents

Before references / appendix: ≈ 48.14%References / intervening material: ≈ 20.71%Attached appendix: ≈ 31.14%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.28 pp.	6.1%
2 BACKGROUND	2	≈ 0.85 pp.	4.05%
3 METHOD	3	≈ 1.89 pp.	9%
4 EXPERIMENTS	5	≈ 4.22 pp.	20.1%
5 RELATED WORKS	9	≈ 0.66 pp.	3.14%
6 CONCLUSIONS	10	≈ 0.21 pp.	1%
ACKNOWLEDGMENTS	10	≈ 0.42 pp.	2%
REFERENCES	11	≈ 4.35 pp.	20.71%
A FREQUENTLY ASKED QUESTIONS	15	≈ 1.23 pp.	5.86%
B DETERMINING THE MAXIMUM MULTIPLICATIVE NOISE LEVEL	16	≈ 1.96 pp.	9.33%
C EFFICIENCY ANALYSIS	18	≈ 3.36 pp.	16%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 3
• Figure 9 · p. 6
• Figure 3 · p. 6
• Table 1 · p. 6
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 9

Attached appendix

• Table 2 · p. 16
• Figure 7 · p. 16
• Table 3 · p. 17
• Figure 8 · p. 20
• Figure 9 · p. 21

Argument and organization

1. Three principles and three computation steps

   Connects the efficiency constraint to proxy tokens and the two-sided activation-margin objective.

2. Performance, scale and specialization

   Tests the method before using its control parameter as an analysis instrument.

Problem, claim and contribution

The problem is defined as a mismatch between router decisions and expert responses, not simply a demand for more expert diversity. Three design principles constrain the proposed fix.

Sections 1 and 3, pp. 1–4 · Main paper

Argument and section order

Performance, larger-scale validation, specialization analysis and component ablations have separate subsections, preventing interpretability observations from replacing task evidence.

Sections 4.2–4.5, pp. 6–9 · Main paper

Figures, captions and visual explanation

The three-panel first figure maps cluster perturbation to expert projections to row/column margin penalties. Matching mathematical symbols bridge the geometric intuition and objective.

Figure 1, p. 2 · Main paper

Tables, comparisons and evidence

Accuracy and load-balance curves are paired on the same page, then a compact eight-task table tests the larger model. The missing expensive baseline at 15B is explicitly explained.

Figure 3 and Table 1, p. 6 · Main paper

Appendix and reproducibility

The FAQ anticipates the alternative explanation that parameter norms game the loss. Complexity derivations are followed by measured distributed throughput and readable implementation pseudocode.

Appendices A–C and Figure 8, pp. 15–20 · Main paper

Clarity and writing conventions

The specialization discussion resists a simplistic more-is-better conclusion and gives a practical starting value while admitting that the optimal value remains configuration-dependent.

Section 4.4, pp. 7–8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Pair each design principle with an observable mechanism, then use ablations to distinguish that mechanism from plausible cheaper explanations; report both analytic complexity and actual distributed throughput.

Boundary: Activation norms and t-SNE separation are specialization proxies, not proofs of semantic capability or optimal token routing. The Euclidean cluster interpretation assumes comparable router norms. The printed Section 3.3 reference to K being millions appears to mean token count T; pseudocode is illustrative rather than a complete training release.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–6 including proxy loss, perturbation bound, scaling, specialization and ablations.

Appendix / supplement

Pages 15–21, Appendices A–C, norm diagnostics, bound derivation, distributed efficiency, pseudocode and per-task figures.

Visually inspected PDF pages

2, 6

• Official ICLR 2026 proceedings conference PDF; PDF creation metadata: Tue May 12 11:35:51 2026 EDT · 21 pages