Writing study: Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, Samuel Marks · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

37 physical pages

Before references / appendix

≈ 9.12 page-equivalents

Main figures / tables

6 / 0

Attached appendix figures / tables

18 / 3

Attached appendix span

≈ 25.91 page-equivalents

Before references / appendix: ≈ 24.65%References / intervening material: ≈ 5.32%Attached appendix: ≈ 70.03%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Replace a prose cross-reference falsely detected as a caption with the actual caption from physical-page source text and coordinates.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.2 pp.	3.24%
2. Background	2	≈ 0.36 pp.	0.97%
3. Methods	3	≈ 1.66 pp.	4.49%
4. Evaluation on downstream Auditing Tasks	4	≈ 2.38 pp.	6.43%
5. Training Dataset Ablations	7	≈ 1 pp.	2.7%
6. Related Work	8	≈ 0.5 pp.	1.35%
7. Discussion and Limitations	8	≈ 0.48 pp.	1.3%
8. Conclusion	9	≈ 0.83 pp.	2.24%
Impact Statement	10	≈ 0.08 pp.	0.22%
References	10	≈ 1.97 pp.	5.32%
A. Extended Discussion and Limitations	12	≈ 0.57 pp.	1.54%
B. Training Details	12	≈ 3.96 pp.	10.7%
C. Additional Out-of-Distribution Generalization Experiments	16	≈ 1.77 pp.	4.78%
D. Separating Diversity from Data Quantity	18	≈ 0.39 pp.	1.05%
E. Confidence Calibration	18	≈ 0.9 pp.	2.43%
F. Training Dataset Details	19	≈ 1.03 pp.	2.78%
G. Evaluation Details	20	≈ 12.37 pp.	33.43%
H. Classification Datasets Construction	33	≈ 1.07 pp.	2.89%
I. PersonaQA Dataset Construction	34	≈ 2.93 pp.	7.92%
J. Anecdotal Hallucination Detection Experiment	37	≈ 0.37 pp.	1%
K. Sparse Autoencoder Dataset Experiments	37	≈ 0.54 pp.	1.46%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Figure 3 · p. 5
• Figure 4 · p. 6
• Figure 5 · p. 7
• Figure 6 · p. 8

Attached appendix

• Table 1 · p. 13
• Figure 7 · p. 14
• Figure 8 · p. 15
• Figure 9 · p. 16
• Figure 10 · p. 18
• Figure 11 · p. 19
• Figure 12 · p. 21
• Figure 13 · p. 23
• Figure 14 · p. 25
• Figure 15 · p. 26
• Figure 16 · p. 27
• Table 2 · p. 28
• Figure 17 · p. 29
• Figure 18 · p. 29
• Figure 19 · p. 30
• Figure 20 · p. 30
• Figure 21 · p. 31
• Figure 22 · p. 32
• Table 3 · p. 33
• Figure 23 · p. 35
• Figure 24 · p. 36

Argument and organization

1. 1 Introduction; 2 Background

   Defines the general-purpose activation-explanation goal and contrasts it with narrow prior training/evaluation.

2. 3 Methods; 4 Downstream Auditing

   Specifies the injection interface and dataset mixture before controlled auditing tasks.

3. 5 Dataset Ablations; 6 Related Work; 7 Discussion; 8 Conclusion

   Tests mixture composition and separates useful natural-language access from mechanistic explanation.

Problem, claim and contribution

The abstract emphasizes out-of-distribution evaluation and data diversity, and specifies matching white-box baselines on four tasks but the strongest overall baseline on only three.

Abstract, p.1 · Main paper

Argument and section order

Terminology separately defines target and oracle prompts before the injection equation, reducing ambiguity in a two-model interpretability setup.

Section 3.1, p.3 · Main paper

Figures, captions and visual explanation

Training and evaluation examples are arranged on opposite sides of a shared input/prompt/response grammar; the next display pairs task-level accuracy bars with the actual target and oracle questions.

Figure 2, p.4; Figure 3, p.5 · Main paper

Tables, comparisons and evidence

The secret-keeping display retains a stronger black-box baseline on some tasks, while data-scaling prose acknowledges near-ties and model-dependent improvements.

Figure 3, p.5; Section 5, p.7 · Main paper

Appendix and reproducibility

A fixed-sample-count ablation explicitly separates diversity from quantity, and the calibration discussion reports accuracy gains together with reduced prediction coverage.

Appendices D-E, p.18; implementation details, p.13 · Main paper

Clarity and writing conventions

The discussion says natural-language explanations do not expose mechanistic primitives and may assemble conclusions absent from the target representation, making the interface’s epistemic limits explicit.

Section 7, pp.8-9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Show precisely what crosses from training to evaluation, and separate convenient explanation interfaces from evidence of mechanistic faithfulness.

Boundary: The benchmarks use narrow fine-tunes, the oracle can guess incorrectly, and selective high-confidence accuracy comes with a coverage cost.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 5, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 13, 18. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

4, 5

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 37 pages