Writing study: Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Zhongzhi Li, Xuansheng Wu, Yijiang Li, Lijie Hu, Ninghao Liu · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

54 physical pages

Before references / appendix

≈ 8.87 page-equivalents

Main figures / tables

7 / 4

Attached appendix figures / tables

11 / 15

Attached appendix span

≈ 39.92 page-equivalents

Before references / appendix: ≈ 16.43%References / intervening material: ≈ 9.65%Attached appendix: ≈ 73.93%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.73 pp.	1.35%
2. Related Work	2	≈ 0.17 pp.	0.31%
3. Preliminaries	2	≈ 0.13 pp.	0.24%
4. Quantify Generalization of Synthetic Data	2	≈ 0.5 pp.	0.93%
5. Reduce Distribution Gap in Feature Space	3	≈ 0.95 pp.	1.76%
6. Reduce the Sampling Error under Dgen	4	≈ 0.96 pp.	1.78%
7. Experiments	5	≈ 4.4 pp.	8.15%
8. Conclusion	9	≈ 0.11 pp.	0.2%
Impact Statement	9	≈ 0.11 pp.	0.2%
References	9	≈ 5.22 pp.	9.67%
Appendix	15	≈ 1 pp.	1.85%
A. Definition of Notations	16	≈ 1 pp.	1.85%
B. Proof of Theorem 4.1 (Generalization Error Upper Bound)	17	≈ 0.53 pp.	0.98%
C. Analysing the Residual Term εcond in Equation (2)	17	≈ 0.72 pp.	1.33%
D. Proof of Minimizing the Distribution Gap between PZ and QZ	18	≈ 2.24 pp.	4.15%
E. Proof of Lemma 6.1 (Upper Bound of Sampling Error)	20	≈ 1.2 pp.	2.22%
F. Analysing the Feature Alignment and Uncertainty Reduction	21	≈ 1.31 pp.	2.43%
G. SAE provides an exact characterization of feature coverage	23	≈ 1.65 pp.	3.06%
H. Related Work	24	≈ 0.64 pp.	1.19%
I. Experimental Setup	25	≈ 1.1 pp.	2.04%
J. Training Details	26	≈ 1.19 pp.	2.2%
K. Details of Identifying Task-relevant Features	27	≈ 1.15 pp.	2.13%
L. Additional Experimental Results	28	≈ 26.19 pp.	48.5%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 3
• Table 1 · p. 6
• Figure 3 · p. 6
• Figure 4 · p. 7
• Figure 5 · p. 7
• Table 2 · p. 8
• Table 3 · p. 8
• Table 4 · p. 9
• Figure 6 · p. 9
• Figure 7 · p. 9

Attached appendix

• Table 5 · p. 29
• Table 6 · p. 29
• Figure 8 · p. 30
• Table 7 · p. 30
• Table 8 · p. 31
• Table 9 · p. 31
• Table 10 · p. 31
• Table 11 · p. 32
• Table 12 · p. 32
• Figure 9 · p. 33
• Table 13 · p. 33
• Table 14 · p. 34
• Figure 10 · p. 34
• Figure 11 · p. 35
• Figure 12 · p. 35
• Figure 13 · p. 36
• Figure 14 · p. 37
• Figure 15 · p. 38
• Figure 16 · p. 38
• Figure 17 · p. 39
• Figure 18 · p. 40
• Table 15 · p. 40
• Table 16 · p. 42
• Table 17 · p. 43
• Table 18 · p. 44
• Table 19 · p. 45

Argument and organization

1. Problem and contribution

   Replaces surface linguistic diversity with task-relevant feature coverage, then separates distribution mismatch from finite-sample error rather than claiming all diversity is useful.

2. Argument sequence

   Formalization and implementation subsections recur for the two error terms; evaluation questions revisit coverage, transfer, interpretation and sensitivity.

3. Evidence and verification

   A fixed-total-200-sample condition helps separate feature coverage from dataset size; human verification and injected annotation noise challenge machine-generated feature descriptions.

Problem, claim and contribution

Replaces surface linguistic diversity with task-relevant feature coverage, then separates distribution mismatch from finite-sample error rather than claiming all diversity is useful.

Sections 1 and 4-6, pp. 1-4 · Main paper

Argument and section order

Formalization and implementation subsections recur for the two error terms; evaluation questions revisit coverage, transfer, interpretation and sensitivity.

Sections 5-7, pp. 3-9 · Main paper

Figures, captions and visual explanation

The overview numbers three stages and uses set differences to connect uncovered features to generation. The results page separates task-specific units under grouped column headers.

Figure 2, p. 3; Table 1 and Figure 3, p. 6; both PNGs inspected · Main paper

Tables, comparisons and evidence

A fixed-total-200-sample condition helps separate feature coverage from dataset size; human verification and injected annotation noise challenge machine-generated feature descriptions.

Section 7.3, p. 6; Section 7.5, p. 8; Appendix L.1-L.2, pp. 28-29 · Main paper

Appendix and reproducibility

Training details distinguish LoRA and head-only protocols, with corresponding table destinations; full task-specific synthesis templates make the data intervention inspectable.

Appendices J-K, pp. 26-27; prompt documentation, pp. 40-54 · Main paper

Clarity and writing conventions

Numbered research questions and repeated setup-to-finding paragraphs provide navigation across heterogeneous tasks, but correlation language sometimes becomes mechanistic interpretation.

Sections 7.2-7.6, pp. 5-9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Map each term in a motivating error decomposition to an implemented intervention and a controlled evaluation question.

Boundary: A fitted coverage-performance relationship is not itself causal proof. Table 1 and head-only appendix results use different adaptation protocols.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-8, pp. 1-9: diversity framing, generalization decomposition, feature-coverage synthesis and research-question-led experiments.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendix overview p. 15, notation/proofs A-G pp. 16-23, related work and setup H-K pp. 24-27, additional checks L pp. 28-39 and full prompt/example documentation through p. 54.

Visually inspected PDF pages

3, 6

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 54 pages