Writing study: Why DPO is a Misspecified Estimator and How to Fix It

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Aditya Gopalan, Sayak Ray Chowdhury, Debangshu Banerjee · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal foundations (primary) · Learning & optimization algorithms.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

11 physical pages

Before references / appendix

≈ 9.18 page-equivalents

Main figures / tables

3 / 2

Attached appendix figures / tables

0 / 0

Attached appendix span

Separate file recorded below; not in this PDF total

Before references / appendix: ≈ 83.45%References / intervening material: ≈ 16.55%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2.03 pp.	18.45%
2 PRELIMINARIES	3	≈ 1 pp.	9.09%
3 REWARD MISSPECIFICATION IN DPO	4	≈ 2.05 pp.	18.64%
4 TOWARDS MITIGATING DPO’S PITFALLS	6	≈ 2.16 pp.	19.64%
5 EXPERIMENTS	8	≈ 1.32 pp.	12%
ACKNOWLEDGMENTS	10	≈ 0.07 pp.	0.64%
REFERENCES	10	≈ 1.82 pp.	16.55%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 5
• Figure 3 · p. 8
• Table 1 · p. 9
• Table 2 · p. 9

Attached appendix

No recognized caption labels.

Separate source files—excluded from canonical-PDF totals

A separate author version may repeat main text and references. Whole-file lengths and counts are not extra appendix material and are never added to the canonical PDF totals. Read the study's source scope for which sections received substantive reading versus layout inspection.

Additional source 1

Separate PDF (or ZIP containing it): whole-file layout of 10 physical pages; 2 detected figures and 5 tables. These may include overlapping main-text material.

Detected heading	PDF page	Estimated span

No top-level headings detected; no section lengths inferred.

• Figure 4 · p. 4
• Figure 5 · p. 6
• Table 3 · p. 7
• Table 4 · p. 7
• Table 5 · p. 9
• Table 6 · p. 9
• Table 7 · p. 9

Argument and organization

1. An ideal equivalence meets restricted policies

   The introduction turns the tabular-policy assumption into an explicit question about parametric DPO.

2. Counterexample, local geometry, repair

   Weighted projection and a three-response failure motivate null-space auxiliary variables before experiments.

Problem, claim and contribution

The contribution targets a specific gap between the unconstrained DPO derivation and a restricted policy class. Population-data counterexamples separate misspecification from finite-sample or optimizer failure.

Main pages 2–6, Sections 1 and 3. · Main paper · Additional source 2

Argument and section order

Named propositions create a traceable chain from KL projection to reward-equivalence classes and the augmented loss. The algorithm footnote explicitly limits its guarantee to sufficiently large beta.

Main Sections 3–4, especially page 8, Proposition 9 and footnote 2. · Main paper · Additional source 2

Figures, captions and visual explanation

Figure 1 uses a zoomed local manifold to introduce the geometry. Figures 2–3 reuse axes and reward points to show failure and repair, making the auxiliary direction visually interpretable.

Main pages 2, 5 and 8, Figures 1–3. · Main paper · Additional source 2

Tables, comparisons and evidence

Table 1 reports relative changes in pairwise preference accuracy; Table 2 reports generated-answer accuracy. These must not be merged into one metric. IPO exceeds AuxDPO for the Llama3.2-1B MMLU-Pro OOD entry despite the broader superiority wording.

Main page 9, Tables 1–2 and evaluation paragraph. · Main paper · Additional source 2

Appendix and reproducibility

The supplement separates proofs, capacity sweeps, a lambda sweep, implementation/memory accounting and synthetic checks. Full-replica state estimates are labeled theoretical, distinct from allocated/reserved device memory.

Supplement Appendix A and B.1–B.6; printed pages 18–19, Tables 3–4. · Main paper · Additional source 2

Clarity and writing conventions

Implementation prose requires reference-model gradients, but Table 4 describes reference passes as forward-only; penalty signs and transposes also vary. Concrete reproduction instructions need reconciliation before the practical loss is presented as unambiguous.

Supplement B.2, printed pages 17–18. · Main paper · Additional source 2

What to reuse—and what not to infer

Transferable lesson: Use a minimal counterexample and a recurring geometric diagram to connect a theoretical failure to an algorithm, while keeping local guarantees and practical implementation claims separate.

Boundary: The theory is local and assumes an accurately learned latent reward for the ideal RLHF comparison; it does not show all DPO runs fail. The finite-data batch penalty is approximate, reported metrics differ, and proofs were read rather than independently certified.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Main PDF pages 1–9, Sections 1–5; references on pages 10–11.

Appendix / supplement

Separate supplement, all 10 PDF pages (printed pages 12–21): Appendix A proofs and B.1–B.6 capacity ablations, implementation details, datasets, HH-RLHF, synthetic and balanced-frequency counterexamples.

Visually inspected PDF pages

2, 9

Visually inspected supplement pages: 7.

• ICLR 2026 final proceedings, main paper · 11 pages
• ICLR 2026 final proceedings, separate supplement · 10 pages