Writing study: OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang, Junhao Zheng, Kexin Yang, Xingzhang Ren, Dayiheng Liu, Linfeng Zhang · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

42 physical pages

Before references / appendix

≈ 9.18 page-equivalents

Main figures / tables

5 / 5

Attached appendix figures / tables

2 / 6

Attached appendix span

≈ 28.45 page-equivalents

Before references / appendix: ≈ 21.86%References / intervening material: ≈ 10.4%Attached appendix: ≈ 67.74%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.31 pp.	3.12%
2. Background	3	≈ 0.57 pp.	1.36%
3. Methodology: OPUS	3	≈ 2.04 pp.	4.86%
4. Experiments	5	≈ 4.12 pp.	9.81%
5. Conclusion and Future Direction	9	≈ 0.17 pp.	0.4%
Acknowledgements	10	≈ 0.05 pp.	0.12%
Impact Statement	10	≈ 0.09 pp.	0.21%
References	10	≈ 4.37 pp.	10.4%
A. Pseudo Code for OPUS Algorithm	14	≈ 0.15 pp.	0.36%
B. Related Work	14	≈ 0.88 pp.	2.1%
C. Optimizer-Induced Preconditioners	15	≈ 2.5 pp.	5.95%
D. Bench-Proxy Construction	18	≈ 0.3 pp.	0.71%
E. Experimental Details	18	≈ 2.21 pp.	5.26%
F. Evaluation Details	20	≈ 1.26 pp.	3%
G. Additional Ablation Results	21	≈ 0.86 pp.	2.05%
H. Additional Results	22	≈ 20.28 pp.	48.29%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 2
• Table 1 · p. 6
• Table 2 · p. 6
• Table 3 · p. 7
• Table 4 · p. 8
• Figure 3 · p. 8
• Figure 4 · p. 8
• Table 5 · p. 8
• Figure 5 · p. 9

Attached appendix

• Table 6 · p. 19
• Table 7 · p. 20
• Table 8 · p. 21
• Table 9 · p. 22
• Table 10 · p. 22
• Table 11 · p. 23
• Figure 6 · p. 23
• Figure 7 · p. 24

Argument and organization

1. Problem and contribution

   Frames data choice as part of each optimizer step rather than static curation, identifying the mismatch between raw-gradient scoring and adaptive-optimizer geometry.

2. Argument sequence

   Names an objective, estimator and selection rule before expanding them mathematically; method equations disclose successive utility approximations instead of hiding them in implementation.

3. Evidence and verification

   Ablations separate greedy selection from soft sampling and standard from benchmark-matched proxies; timing and compute are reported alongside token efficiency.

Problem, claim and contribution

Frames data choice as part of each optimizer step rather than static curation, identifying the mismatch between raw-gradient scoring and adaptive-optimizer geometry.

Introduction, pp. 1-2 · Main paper

Argument and section order

Names an objective, estimator and selection rule before expanding them mathematically; method equations disclose successive utility approximations instead of hiding them in implementation.

Introduction and Sections 3.1-3.3, pp. 2-5 · Main paper

Figures, captions and visual explanation

The pipeline marks preconditioned candidate gradients versus raw proxy gradients and carries notation into projection and sampling. The evaluation page colocates cost analysis with a control table containing non-winning task cells.

Figure 2, p. 2; Figure 4 and Table 5, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

Ablations separate greedy selection from soft sampling and standard from benchmark-matched proxies; timing and compute are reported alongside token efficiency.

Sections 4.3-4.5, p. 8 · Main paper

Appendix and reproducibility

Optimizer-specific derivations and proxy construction have named destinations; qualitative examples preserve candidate IDs, scores and selection decisions for one common buffer.

Appendices C-D, pp. 15-18; H, pp. 22-42 · Main paper

Clarity and writing conventions

The paper repeatedly distinguishes token efficiency, extra online overhead and total computation; those are complementary, not interchangeable, savings claims.

Sections 4.3-4.4, p. 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Expose the mathematical approximation, actual execution path and measured overhead together when proposing a scalable training intervention.

Boundary: Benchmark-aligned proxy access is a consequential protocol choice; preserve that condition when reusing the comparison or efficiency claim.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-5, pp. 1-9: optimizer-dependent selection motivation, utility approximations, scalable implementation, controlled comparisons, cost and ablations.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-H, pp. 14-42: algorithm, optimizer preconditioners, proxy construction, experimental/evaluation settings, ablations and selected/not-selected document examples.

Visually inspected PDF pages

2, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 42 pages