Writing study: EigenBench: A Comparative Behavioral Measure of Value Alignment

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Jonathn Chang, Leonhard Piff, Suvadip Sana, Jasmine Li, Lionel Levine · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary) · Empirical analysis, diagnosis & auditing.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

29 physical pages

Before references / appendix

≈ 10.2 page-equivalents

Main figures / tables

3 / 4

Attached appendix figures / tables

7 / 14

Attached appendix span

≈ 16.9 page-equivalents

Before references / appendix: ≈ 35.17%References / intervening material: ≈ 6.55%Attached appendix: ≈ 58.28%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2.19 pp.	7.55%
2 RELATED WORK	3	≈ 0.83 pp.	2.86%
3 METHODOLOGY	4	≈ 2.64 pp.	9.1%
4 RESULTS	7	≈ 1.28 pp.	4.41%
5 BASELINES	8	≈ 1.14 pp.	3.93%
6 ROBUSTNESS	9	≈ 1.02 pp.	3.52%
7 CONCLUSION, LIMITATIONS, AND FUTURE DIRECTIONS	10	≈ 0.49 pp.	1.69%
8 ACKNOWLEDGMENTS	11	≈ 0.1 pp.	0.34%
REFERENCES	11	≈ 1.9 pp.	6.55%
Appendix	13	≈ 0.04 pp.	0.14%
A MODELS	13	≈ 0.5 pp.	1.72%
B CONSTITUTIONS, SCENARIOS, AND PERSONAS	13	≈ 0.52 pp.	1.79%
C DATA COLLECTION	14	≈ 0.4 pp.	1.38%
D PROMPTS FOR JUDGE SCAFFOLD	14	≈ 0.69 pp.	2.38%
E OPTIMIZATION	15	≈ 1.17 pp.	4.03%
F PROMPTED DISPOSITIONS VARIANCE CALCULATION	16	≈ 0.26 pp.	0.9%
G MODEL SURVEY DETAILS	16	≈ 0.12 pp.	0.41%
H HUMAN VALIDATION DETAILS	16	≈ 2.71 pp.	9.34%
I GPQA VALIDATION DETAILS	19	≈ 0.37 pp.	1.28%
J JUDGE QUALITY TESTS	19	≈ 0.87 pp.	3%
K LARGE POPULATION RUN	20	≈ 2.03 pp.	7%
L GREENBEARD EFFECT	22	≈ 7.23 pp.	24.93%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 3
• Table 1 · p. 4
• Figure 3 · p. 7
• Table 2 · p. 8
• Table 3 · p. 9
• Table 4 · p. 10

Attached appendix

• Table 5 · p. 13
• Figure 4 · p. 16
• Figure 5 · p. 17
• Table 6 · p. 18
• Figure 6 · p. 18
• Figure 7 · p. 19
• Table 7 · p. 19
• Table 8 · p. 21
• Figure 8 · p. 21
• Table 9 · p. 22
• Figure 9 · p. 23
• Figure 10 · p. 23
• Table 10 · p. 24
• Table 11 · p. 25
• Table 12 · p. 26
• Table 13 · p. 26
• Table 14 · p. 27
• Table 15 · p. 28
• Table 16 · p. 28
• Table 17 · p. 28
• Table 18 · p. 29

Argument and organization

1. Define the object being measured

   Specifies model population, constitution, and scenario distribution before explaining pairwise judgments and eigenvector aggregation.

2. Validation before robustness

   Contrasts revealed and stated preferences, checks human and objective-task agreement, then perturbs the benchmark inputs.

Problem, claim and contribution

The opening makes subjectivity a design constraint and clearly defines the score as comparative consensus over three inputs, not an absolute attribute of a model.

Sections 1 and 3.1–3.3 · Main paper

Argument and section order

A short comparison table distinguishes competing ranking systems by the question they answer, avoiding a superficial feature checklist.

Table 1, p. 4 · Main paper

Figures, captions and visual explanation

The pipeline pairs a concrete scenario and judge comparison with latent fitting and score aggregation. Its caption states the crucial information boundary: only judges receive the constitution.

Figure 1, p. 2 · Main paper

Tables, comparisons and evidence

Constitution-specific score plots show bootstrap intervals rather than just ranked bars. Human validation, GPQA rank recovery, and population perturbations test different aspects of the measurement claim.

Figure 3; Sections 5–6 · Main paper

Appendix and reproducibility

Exact model identifiers, full prompts, constitutions, judge-bias ablations, and sample-size stability are retained. The adversarial Greenbeard study exposes a limitation instead of assuming peer aggregation is manipulation-proof.

Appendices A–D, J–L · Main paper

Clarity and writing conventions

The premise that aligned behavior predicts judging quality is explicitly acknowledged as an assumption. The human validation is only seven people, including authors, and its human–human average includes zero self-distances.

Section 1, footnote 4; Appendix H · Main paper

What to reuse—and what not to infer

Transferable lesson: For a subjective benchmark, define the measurement population and criteria first, publish the elicitation protocol, and validate several distinct failure modes instead of treating one correlation as universal validity.

Boundary: Scores reflect the selected model community, scenarios, and constitution. GPQA agreement does not establish normative correctness. Figure 1 simplifies away tie terms used in Section 3.6, and the optimization appendix should not be treated as a general guarantee against bad local minima in low-rank fitting.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

PDF pp. 1–10, Sections 1–7, including all validation and robustness studies.

Appendix / supplement

PDF pp. 13–29, Appendices A–L, model IDs, constitutions, prompts, optimization, human validation, larger population, and Greenbeard experiment.

Visually inspected PDF pages

2, 7, 21

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 29 pages