Writing study: The Signal is in the Steps: Local Scoring for Reasoning Data Selection

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Hoang Anh Just, Myeongseob Ko, Ruoxi Jia · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

31 physical pages

Before references / appendix

≈ 9.36 page-equivalents

Main figures / tables

6 / 4

Attached appendix figures / tables

4 / 18

Attached appendix span

≈ 16.91 page-equivalents

Before references / appendix: ≈ 30.19%References / intervening material: ≈ 15.26%Attached appendix: ≈ 54.55%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.95 pp.	3.06%
2. Related Work	2	≈ 0.55 pp.	1.77%
3. Problem Setting and Baseline	2	≈ 0.4 pp.	1.29%
4. When Global Selection Fails	3	≈ 0.8 pp.	2.58%
5. Why GALP Fails	4	≈ 0.52 pp.	1.68%
6. Method: Local Average Log Probability	4	≈ 1.63 pp.	5.26%
7. Evaluation	6	≈ 3.62 pp.	11.68%
8. Conclusion	9	≈ 0.27 pp.	0.87%
Impact Statement	10	≈ 0.27 pp.	0.87%
References	10	≈ 4.73 pp.	15.26%
A. Toy theoretical model: locality induces a reasoning gap	15	≈ 5 pp.	16.13%
B. Dataset and evaluation details	20	≈ 2.72 pp.	8.77%
C. Additional Results	22	≈ 9.19 pp.	29.65%

All detected figure and table locations

Main text

• Table 1 · p. 3
• Figure 1 · p. 3
• Figure 2 · p. 5
• Figure 3 · p. 5
• Table 2 · p. 6
• Table 3 · p. 6
• Table 4 · p. 7
• Figure 4 · p. 8
• Figure 5 · p. 8
• Figure 6 · p. 9

Attached appendix

• Table 5 · p. 22
• Table 6 · p. 22
• Table 7 · p. 22
• Table 8 · p. 23
• Table 9 · p. 24
• Table 10 · p. 24
• Figure 7 · p. 25
• Figure 8 · p. 25
• Figure 9 · p. 26
• Table 11 · p. 26
• Table 12 · p. 26
• Table 13 · p. 27
• Table 14 · p. 27
• Figure 10 · p. 28
• Table 15 · p. 28
• Table 16 · p. 28
• Table 17 · p. 29
• Table 18 · p. 29
• Table 19 · p. 29
• Table 20 · p. 30
• Table 21 · p. 31
• Table 22 · p. 31

Argument and organization

1. Problem and claim

   Starts from a baseline that works in a restricted setting, then identifies the long, mixed-teacher regime where its ranking fails. The new rule answers a specific response-selection problem rather than replacing all likelihood-based selection.

2. Argument sequence

   Separates when global scoring fails, a hypothesis for why, the local scoring definition and two deployment uses: choosing a teacher and choosing individual responses. Diagnostic analysis follows the main comparisons.

3. Evidence and verification

   Within-teacher terciles provide a sanity check before the mixed-teacher comparison. Later window and segmenter controls ask whether the gain is specific to one truncation setting or segmentation model.

Problem, claim and contribution

Starts from a baseline that works in a restricted setting, then identifies the long, mixed-teacher regime where its ranking fails. The new rule answers a specific response-selection problem rather than replacing all likelihood-based selection.

Introduction and Sections 3–4, pp. 1–3 · Main paper

Argument and section order

Separates when global scoring fails, a hypothesis for why, the local scoring definition and two deployment uses: choosing a teacher and choosing individual responses. Diagnostic analysis follows the main comparisons.

Sections 4–7, pp. 3–9 · Main paper

Figures, captions and visual explanation

The context-window plot uses consistent teacher colors and marker shapes to show a ranking reversal; the token-category bar chart states its 440-response analysis set and prints percentages. These displays serve different explanatory jobs.

Figure 3, physical p. 5; Figure 4, physical p. 8; rasters inspected · Main paper

Tables, comparisons and evidence

Within-teacher terciles provide a sanity check before the mixed-teacher comparison. Later window and segmenter controls ask whether the gain is specific to one truncation setting or segmentation model.

Table 1, p. 3; Sections 7.1–7.2, pp. 6–7; Tables 11 and 21, pp. 26 and 31 · Main paper

Appendix and reproducibility

The toy theoretical model explicitly lists Markov and background-distribution simplifications. A separate details section records segmentation, sampling, SFT and evaluation, while additional tables expose teacher mixtures and expanded pools.

Appendix A limitations, p. 19; Appendices B–C, pp. 20–31 · Main paper

Clarity and writing conventions

Figure 2 is visibly a line-and-band plot although its caption calls each display a violin. Its strong generalization wording also exceeds what cosine similarity alone can establish; the main limitation correctly separates local learnability from objective correctness.

Figure 2, physical p. 5; Limitations, p. 10 · Main paper

What to reuse—and what not to infer

Transferable lesson: Preserve the baseline’s successful regime, diagnose the changed regime, and test whether the proposed selection rule survives reasonable implementation alternatives.

Boundary: Local likelihood is not a correctness verifier or a guarantee of long-range consistency; the paper does not evaluate it as an RL reward.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused scan of Sections 1–8, pp. 1–9, with limitations and impact on p. 10; baseline definition, failure regime, local score, teacher/response selection and diagnostic examples checked.

Appendix / supplement

Structural scan of Appendices A–C, pp. 15–31, with selected checks of toy-model limitations, segmentation implementation, training/evaluation settings, window and segmenter ablations. Not independent validation of the toy proofs or reported measurements.

Visually inspected PDF pages

5, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 31 pages