Writing study: On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

25 physical pages

Before references / appendix

≈ 9.04 page-equivalents

Main figures / tables

5 / 7

Attached appendix figures / tables

1 / 19

Attached appendix span

≈ 11.91 page-equivalents

Before references / appendix: ≈ 36.16%References / intervening material: ≈ 16.2%Attached appendix: ≈ 47.64%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.26 pp.	5.04%
2. Background and Related Work	2	≈ 0.8 pp.	3.2%
3. Methods	3	≈ 2.08 pp.	8.32%
4. Results	5	≈ 2.97 pp.	11.88%
5. Discussion	8	≈ 0.88 pp.	3.52%
6. Conclusion	9	≈ 0.15 pp.	0.6%
Acknowledgment	9	≈ 0.09 pp.	0.36%
Impact Statement	9	≈ 0.18 pp.	0.72%
References	10	≈ 4.04 pp.	16.16%
A. Definition Texts Used	14	≈ 2.26 pp.	9.04%
B. Prompts	16	≈ 0.91 pp.	3.64%
C. Mixed Effects Regression Models	17	≈ 1.83 pp.	7.32%
D. Additional Experimental Results	19	≈ 2 pp.	8%
E. Familiarity Analysis Details	21	≈ 3 pp.	12%
F. Generalization Beyond Toxicity	24	≈ 1.07 pp.	4.28%
G. Model and Dataset Use Across Experiments	25	≈ 0.85 pp.	3.4%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Table 1 · p. 5
• Table 2 · p. 5
• Table 3 · p. 6
• Figure 3 · p. 6
• Table 4 · p. 6
• Table 5 · p. 7
• Table 6 · p. 7
• Figure 4 · p. 7
• Table 7 · p. 8
• Figure 5 · p. 8

Attached appendix

• Table 8 · p. 15
• Table 9 · p. 16
• Table 10 · p. 17
• Table 11 · p. 18
• Table 12 · p. 18
• Table 13 · p. 18
• Table 14 · p. 19
• Table 15 · p. 19
• Table 16 · p. 19
• Table 17 · p. 20
• Table 18 · p. 20
• Table 19 · p. 20
• Table 20 · p. 21
• Table 21 · p. 21
• Table 22 · p. 22
• Figure 6 · p. 23
• Table 23 · p. 23
• Table 24 · p. 24
• Table 25 · p. 24
• Table 26 · p. 25

Argument and organization

1. Problem and contribution

   Separates textual reproduction from alignment between an elicited concept and an operational labeling policy, treating annotation models as instruments that need policy-specific validation.

2. Argument sequence

   Three research questions govern metric definitions and result sections; discussion then separates definition sensitivity, rescue-corruption trade-offs and the limits of verbalized confidence.

3. Evidence and verification

   Encoder alternatives, memorization proxies, automatic prompting and non-toxicity checks test competing explanations. Figure 3 explicitly differs in unit of analysis between cell-level DSF and per-text memorization, while Table 4 compares a common 54-cell panel.

Problem, claim and contribution

Separates textual reproduction from alignment between an elicited concept and an operational labeling policy, treating annotation models as instruments that need policy-specific validation.

Introduction and Sections 3.1-3.2, pp. 1-4 · Main paper

Argument and section order

Three research questions govern metric definitions and result sections; discussion then separates definition sensitivity, rescue-corruption trade-offs and the limits of verbalized confidence.

Sections 3-5, pp. 3-9 · Main paper

Figures, captions and visual explanation

Raw and dataset-adjusted correlations are juxtaposed rather than letting dataset difficulty stand in for the target association; calibration and misalignment tables are colocated with their interpretation.

Figure 3 and Table 4, p. 6; Figure 5 and Table 7, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

Encoder alternatives, memorization proxies, automatic prompting and non-toxicity checks test competing explanations. Figure 3 explicitly differs in unit of analysis between cell-level DSF and per-text memorization, while Table 4 compares a common 54-cell panel.

Section 4.1, Figure 3 and Table 4, pp. 5-6; Appendices E-F, pp. 21-24 · Main paper

Appendix and reproducibility

The final coverage matrix maps six-model regressions, nine-model descriptive results and supplementary datasets to exact tables; the multi-turn rescue table discloses heavy refusal-related dropping for one model.

Appendix D.1 and Table 19, p. 20; G and Table 26, p. 25 · Main paper

Clarity and writing conventions

The limitations explicitly distinguish fidelity to dataset-specific policy from normative correctness and association from causation; these qualifications are narrower than some main-text language about internal priors and fundamental limits.

Discussion and limitations, pp. 8-9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Carry the analysis unit and population tier alongside correlations, publish the exact policy interventions, and pair error rescue with corruption of previously correct decisions.

Boundary: An elicited verbal definition is an operational proxy, not direct access to an internal concept; correlational DSF does not identify a training-stage cause, and binary annotation findings do not establish universal prompting limits.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-6, pp. 1-9: model-policy mismatch, familiarity and steerability definitions, model/dataset tiers, research-question results, confounding, discussion and limitations.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-G, pp. 14-25: exact definitions and prompts, mixed-effects specifications, extra results and multi-turn rescue, encoder/metric robustness, non-toxicity checks and analysis-coverage mapping.

Visually inspected PDF pages

6, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 25 pages