Writing study: What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Datasets, benchmarks & measurement resources.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

36 physical pages

Before references / appendix

≈ 10.27 page-equivalents

Main figures / tables

3 / 1

Attached appendix figures / tables

6 / 17

Attached appendix span

≈ 21.89 page-equivalents

Before references / appendix: ≈ 28.53%References / intervening material: ≈ 10.67%Attached appendix: ≈ 60.81%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 0.93 pp.	2.58%
2 EXPLAINING HUMAN FEEDBACK DATASETS	2	≈ 0.61 pp.	1.69%
3 METHODOLOGY: WHAT’S IN MY HUMAN FEEDBACK?	3	≈ 1.16 pp.	3.22%
4 LARGE-SCALE ANALYSIS OF PREFERENCE DATASETS WITH WIMHF	4	≈ 3.73 pp.	10.36%
5 STEERING MODEL BEHAVIOR WITH WIMHF	8	≈ 1.9 pp.	5.28%
6 RELATED WORK	10	≈ 0.54 pp.	1.5%
7 CONCLUSION	10	≈ 0.41 pp.	1.14%
ACKNOWLEDGMENTS	11	≈ 0.16 pp.	0.44%
REFERENCES	11	≈ 3.84 pp.	10.67%
A SPARSE AUTOENCODERS	15	≈ 2.73 pp.	7.58%
B DATA	17	≈ 0.88 pp.	2.44%
C ADDITIONAL EVALUATION	18	≈ 1.64 pp.	4.56%
D SUBJECTIVE PREFERENCES	20	≈ 0.49 pp.	1.36%
E LIMITATIONS AND FUTURE WORK	20	≈ 1.01 pp.	2.81%
F SUPPLEMENTARY FIGURES, TABLES, AND PROMPTS	21	≈ 15.14 pp.	42.06%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Table 1 · p. 5
• Figure 2 · p. 7
• Figure 3 · p. 9

Attached appendix

• Table 2 · p. 16
• Table 3 · p. 17
• Table 4 · p. 17
• Table 5 · p. 19
• Table 6 · p. 20
• Table 7 · p. 21
• Figure 4 · p. 22
• Figure 5 · p. 22
• Figure 6 · p. 23
• Figure 7 · p. 24
• Table 8 · p. 25
• Table 9 · p. 26
• Table 10 · p. 27
• Figure 8 · p. 27
• Table 11 · p. 28
• Figure 9 · p. 29
• Table 12 · p. 30
• Table 13 · p. 31
• Table 14 · p. 32
• Table 15 · p. 33
• Table 16 · p. 34
• Table 17 · p. 35
• Table 18 · p. 36

Argument and organization

1. Measurable versus expressed preferences

   Separates response variation from the labels assigned to it.

2. Discover, describe and regress

   Connects signed sparse features to interpreted associations.

3. Validation, conflicts and interventions

   Progresses from descriptive credibility to data curation and limited personalization.

Problem, claim and contribution

The central distinction between measurable and expressed preferences turns an ambiguous auditing goal into two separate questions. The probability factorization then links each question to a stage of dataset construction rather than relying on an interpretability slogan.

§2 and Eq.1, pp.2–3 · Main paper

Argument and section order

Three numbered method steps define feature learning, language descriptions and preference regression. The appendix supplies important mechanics omitted by the simplified overview: signed identity activations, a Matryoshka prefix loss, and selection among five candidate descriptions.

§3, pp.3–4; AppendicesA.1–A.3, pp.15–16 · Main paper

Figures, captions and visual explanation

Figure2 aligns dataset-colored estimates around a shared zero line, making reversals visible. Its caption specifies the prevalence exclusion and bootstrapped intervals; Figure3 separately identifies instance and annotator resampling units for two different experiments.

Figures2–3, pp.7,9 · Main paper

Tables, comparisons and evidence

The safety intervention increases a reward model's RewardBench2 safety accuracy from8.9% to46.2%, a37.3-point change rather than a37-percent relative gain. This is reward-model evaluation, not a demonstrated deployment-wide reduction in harmful generated responses.

§5.1 and Figure3a, pp.8–9 · Main paper

Appendix and reproducibility

Complete feature tables, interpretation prompts, negative matching examples and subjectivity-estimator checks make the audit inspectable. AppendixE narrows claims to English data, one personalization dataset and response-only representations that miss some prompt-dependent information.

AppendicesD–F, pp.20–36 · Main paper

Clarity and writing conventions

The causal caveat is explicit, but two local wording errors matter: exp(beta) multiplies odds, not log-odds; the preprocessing list says remove subjective pairs while the main text says retain subjective conversations. These need code-level reconciliation before reuse.

§3 and footnote3, p.4; AppendixB step5, p.18 · Main paper

What to reuse—and what not to infer

Transferable lesson: Define an audit's observable target first, triangulate learned concepts with independent evidence, and attach every intervention claim to its actual model, metric and unit of uncertainty.

Boundary: Feature descriptions depend on embedding and judge models; significant fidelity is not causal validity. A trained reward model is not an oracle upper bound, and overlapping confidence intervals do not establish equivalence or prove that other capabilities are unchanged.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

pp.1–10, §§1–7; acknowledgments and references pp.11–14

Appendix / supplement

pp.15–36, A–F: SAE design, preprocessing, validation, subjectivity, limitations, full feature tables and prompts

Visually inspected PDF pages

7, 9

• official ICLR 2026 proceedings PDF · 36 pages