Writing study: DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and Its Loss’ Convexity is Dispensable)
← ICML 2026 oral writing corpus
Wenxuan Zhou, Shujian Zhang, Brice Magdalou, John Lambert, Ehsan Amid, Richard Nock, Andrew Hard · Official ICML 2026 oral record
Figure, table and section ledger
Editorial contribution type: Theory & formal analysis (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 24 physical pages
- Before references / appendix
- ≈ 9.25 page-equivalents
- Main figures / tables
- 1 / 1
- Attached appendix figures / tables
- 3 / 0
- Attached appendix span
- ≈ 10.92 page-equivalents
Source-bound correction: Restore the actual Appendix cover on physical p14 and roman-numbered appendix headings I-III. This is not an absent appendix.
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
| Top-level heading | Starts on PDF page | Estimated span | Share of entire PDF |
|---|---|---|---|
| 1. Introduction | 1 | ≈ 1.23 pp. | 5.13% |
| 2. Basic definitions | 2 | ≈ 0.55 pp. | 2.29% |
| 3. Bedrock | 3 | ≈ 1.77 pp. | 7.38% |
| 4. The Expanse | 4 | ≈ 1.87 pp. | 7.79% |
| 5. Fallout | 6 | ≈ 2.65 pp. | 11.04% |
| 6. Toy experiment | 9 | ≈ 0.26 pp. | 1.08% |
| 7. Conclusion: to be normative (or not to be) | 9 | ≈ 0.41 pp. | 1.71% |
| Acknowledgments | 10 | ≈ 0.05 pp. | 0.21% |
| Impact statement | 10 | ≈ 0.15 pp. | 0.63% |
| References | 10 | ≈ 3.84 pp. | 16% |
| Appendix | 14 | ≈ 1.42 pp. | 5.92% |
| I. Supplementary material on KLST∗ | 15 | ≈ 1.01 pp. | 4.21% |
| II. Supplementary material on proofs | 16 | ≈ 7.78 pp. | 32.42% |
| III. Supplementary material on toy experiment | 24 | ≈ 0.71 pp. | 2.96% |
All detected figure and table locations
Main text
Attached appendix
Argument and organization
- 1 Introduction; 2 Definitions; 3 Bedrock
Separates proper-loss theory from choice axioms and explains abstention with examples.
- 4 The Expanse; 5 Fallout
Generalizes the RLHF objective, reward link and final loss, then maps consequences and existing variants.
- 6 Toy experiment; 7 Conclusion
Offers a limited non-convex illustration and states which variants remain outside the framework.
Problem, claim and contribution
The claimed contribution is normative design freedom, not a new broadly benchmarked best training recipe or empirical validation of the human-choice axioms.
Abstract, p.1; Section 6, p.9 · Main paper
Argument and section order
The derivation is organized as three explicit DPO design steps; later prose distinguishes a universal compatibility claim from a weaker existence claim.
Sections 4.1-4.3, pp.4-5; Section 5.1, p.7 · Main paper
Figures, captions and visual explanation
A full-width mathematical table separates binary and multiclass properness with check/cross marks; a small loss-curve plot is paired with a three-row win-rate table rather than an architecture diagram.
Table 1, p.8; Figure 1, p.9 · Main paper
Tables, comparisons and evidence
The toy comparison names its base model, judge and two-run averaging and retains the parameter choice with a below-fifty-percent win rate.
Section 6 and Figure 1, p.9 · Main paper
Appendix and reproducibility
A page-numbered appendix contents list routes axiomatic illustrations, theorem proofs and toy setup; the experiment details explicitly disclaim firm conclusions.
Appendix contents, p.14; Section III, p.24 · Main paper
Clarity and writing conventions
Playful section titles need their numbered subsection signposts to remain navigable; the conclusion limits the framework by naming excluded DPO variants.
Sections 4-5, pp.4-9; Conclusion, p.9 · Main paper
What to reuse—and what not to infer
Transferable lesson: For a unification paper, expose each design degree of freedom and its assumptions, and use small experiments as illustrations without making them carry a broad performance claim.
Boundary: Normative compatibility is not behavioral validation, and dispensable loss convexity does not establish easy optimization or improved performance for arbitrary non-convex losses.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-10-07
- Reading status
- Writing-focused manuscript scan + selected visuals inspected
- Main text
- Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 4, 5, 6, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.
- Appendix / supplement
- Appendix organization and main-text pointers were checked; focused text inspection of physical pages 14, 23, 24. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.
- Visually inspected PDF pages
- 8, 9
This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.