Writing study: Pruning and Distilling Mixture-of-Experts into Dense Language Models

NeurIPS overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← NeurIPS 2026 oral writing corpus

Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho · Official NeurIPS 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Efficiency, inference & systems (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

40 physical pages

Before references / appendix

≈ 13.34 page-equivalents

Main figures / tables

8 / 7

Attached appendix figures / tables

0 / 12

Attached appendix span

≈ 23.91 page-equivalents

Before references / appendix: ≈ 33.35%References / intervening material: ≈ 6.88%Attached appendix: ≈ 59.77%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Top-level heading and reference/appendix boundaries checked against source PDF text. Bibliography titles/author initials, numbered contribution/property bullets, diagram steps, equations and TOC entries are not substantive headings. Section spans are page-coordinate estimates, not ink area or prose-only length. This is an author manuscript mirror, not a verified accepted-version template. Source layout uses physical single-column y/page-height, including equation-heavy pages. Complete numbered-caption inventory cross-checked against source text; restored real punctuation-free captions and excluded inline references. Selected pages visually inspected, not every image/table.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.62 pp.	4.05%
2. Related work	3	≈ 0.66 pp.	1.65%
3. Method	3	≈ 3.78 pp.	9.45%
4. Experiments	7	≈ 6.27 pp.	15.68%
5. Conclusion	13	≈ 0.41 pp.	1.02%
6. Limitations	14	≈ 0.13 pp.	0.33%
References	14	≈ 2.75 pp.	6.88%
A. Block concatenation preserves representative activations	17	≈ 1 pp.	2.5%
B. Algorithms	18	≈ 1 pp.	2.5%
C. Scoring baseline definitions	19	≈ 0.23 pp.	0.57%
D. Excluded scoring and grouping methods	19	≈ 0.22 pp.	0.55%
E. Proofs for D-Optimal expert selection	19	≈ 2.92 pp.	7.3%
F. Additional theoretical results	22	≈ 8.84 pp.	22.1%
G. Grouping strategy definitions	31	≈ 0.41 pp.	1.02%
H. Down-projection scaling equations	31	≈ 0.39 pp.	0.97%
I. Base vs. post-trained teacher comparison	32	≈ 1.13 pp.	2.82%
J. Full distillation hyperparameters	33	≈ 0.87 pp.	2.17%
K. Full pre-distill PPL grid	34	≈ 2 pp.	5%
L. Full distillation results	36	≈ 1 pp.	2.5%
M. Dense-to-Dense (D2D) pruning baseline	37	≈ 1 pp.	2.5%
N. Error taxonomy and examples	38	≈ 1 pp.	2.5%
O. Cross-model full results	39	≈ 1 pp.	2.5%
P. Model architecture details	40	≈ 0.91 pp.	2.27%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Table 1 · p. 2
• Table 2 · p. 4
• Figure 2 · p. 5
• Figure 3 · p. 5
• Figure 4 · p. 8
• Table 3 · p. 9
• Table 4 · p. 10
• Table 5 · p. 10
• Figure 5 · p. 10
• Table 6 · p. 11
• Figure 6 · p. 11
• Figure 7 · p. 12
• Table 7 · p. 13
• Figure 8 · p. 13

Attached appendix

• Table 8 · p. 32
• Table 9 · p. 32
• Table 10 · p. 33
• Table 11 · p. 34
• Table 12 · p. 35
• Table 13 · p. 36
• Table 14 · p. 37
• Table 15 · p. 38
• Table 16 · p. 38
• Table 17 · p. 39
• Table 18 · p. 39
• Table 19 · p. 40

Argument and organization

1. Problem and contribution

   Distinguishes reducing active computation from eliminating the memory burden of storing all experts; related methods are classified by their output architecture.

2. Argument sequence

   Names scoring, grouping, scaling and distillation as separate design axes before short-run sweeps, extended training and cross-model tests.

3. Evidence and verification

   The text distinguishes the 350 pre-distillation configurations from 35 post-distillation scoring/grouping combinations and later long-run validation.

Problem, claim and contribution

Distinguishes reducing active computation from eliminating the memory burden of storing all experts; related methods are classified by their output architecture.

Introduction; Table 1, p. 2 · Main paper

Argument and section order

Names scoring, grouping, scaling and distillation as separate design axes before short-run sweeps, extended training and cross-model tests.

Sections 3–4 · Main paper

Figures, captions and visual explanation

Paired bar panels compare marginal effects of scoring and grouping on the same accuracy scale, supporting a relative-importance argument.

Figure 4, p. 8 · Main paper

Tables, comparisons and evidence

The text distinguishes the 350 pre-distillation configurations from 35 post-distillation scoring/grouping combinations and later long-run validation.

Section 4.2, p. 8; Section 4.4 · Main paper

Appendix and reproducibility

Full configuration grids, KL/teacher-routing details, proofs and training hyperparameters support the condensed main comparisons.

Appendices B, E–F, J–L · Main paper

Clarity and writing conventions

Finding-style lead sentences explain why a design axis matters before referring to individual values.

Sections 4.2–4.6 · Main paper

What to reuse—and what not to infer

Transferable lesson: Expose the design space and distinguish screening budget from full training; report memory, active computation and accuracy as different quantities.

Boundary: The displayed marginal-effect bars are not automatically seed confidence intervals. The long public author version is not an initial NeurIPS page-budget model.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-02

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented distributed manuscript scan of Sections 1–6, pp. 1–14; references through p. 16: abstract/framing, section sequence, substantive text windows, captions, evidence organization and conclusion; not exhaustive technical verification.

Appendix / supplement

Writing-oriented scan of Appendices A onward, pp. 17–40; proofs, algorithms, full grids and training specifications; code and experiments were not executed.

Visually inspected PDF pages

2, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Public author arXiv manuscript 2605.28207v2; not verified identical to the accepted or camera-ready OpenReview revision · 40 pages