Writing study: Monitoring Monitorability

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Melody Y. Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y. Wei, Marcus Williams, Benjamin Arnav, Joost Huizinga, Ian D Kivlichan, Amelia Glaese, Jakub Pachocki, Bowen Baker · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

65 physical pages

Before references / appendix

≈ 9.28 page-equivalents

Main figures / tables

6 / 1

Attached appendix figures / tables

28 / 3

Attached appendix span

≈ 52.91 page-equivalents

Before references / appendix: ≈ 14.28%References / intervening material: ≈ 4.32%Attached appendix: ≈ 81.4%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Source-checked physical-page text restores wrapped top-level heading labels on pp. 6. Heading starts and measured positions are unchanged; this is label completeness, not a new section. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.98 pp.	1.51%
2. Evaluating Monitorability	2	≈ 1.44 pp.	2.22%
3. Evaluation Suite Details and Archetypes	3	≈ 2.6 pp.	4%
4. Monitorability of Frontier Reasoning Models	6	≈ 0.45 pp.	0.69%
5. Monitorability Scaling Trends	6	≈ 1.09 pp.	1.68%
6. Getting more monitorability out of CoT	7	≈ 0.96 pp.	1.48%
7. Related Work	8	≈ 0.55 pp.	0.85%
8. Discussion and Conclusion	9	≈ 0.62 pp.	0.95%
Acknowledgments	10	≈ 0.18 pp.	0.28%
Impact Statement	10	≈ 0.05 pp.	0.08%
References	10	≈ 2.81 pp.	4.32%
A. Sample CoTs	13	≈ 0.06 pp.	0.09%
B. Monitorability Metric Details	13	≈ 4.21 pp.	6.48%
C. Comparing monitorability across models and reasoning efforts	17	≈ 0.73 pp.	1.12%
D. Evaluation Details	18	≈ 22.28 pp.	34.28%
E. Correlation between different evaluation types	40	≈ 0.29 pp.	0.45%
F. Monitor Degeneracy Evals	40	≈ 5.7 pp.	8.77%
G. Frontier Model Comparison Details	46	≈ 0.31 pp.	0.48%
H. Unmonitorability Detection	46	≈ 10.43 pp.	16.05%
I. Additional monitor scaling analysis	57	≈ 1 pp.	1.54%
J. Getting More Monitorability Out of CoT	58	≈ 7.91 pp.	12.17%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 3
• Figure 2 · p. 7
• Figure 3 · p. 7
• Figure 4 · p. 8
• Figure 5 · p. 8
• Figure 6 · p. 9

Attached appendix

• Figure 7 · p. 14
• Figure 8 · p. 15
• Figure 9 · p. 16
• Figure 10 · p. 16
• Figure 11 · p. 18
• Table 2 · p. 20
• Table 3 · p. 25
• Figure 12 · p. 29
• Figure 13 · p. 29
• Figure 14 · p. 30
• Table 4 · p. 38
• Figure 15 · p. 41
• Figure 16 · p. 41
• Figure 17 · p. 42
• Figure 18 · p. 45
• Figure 19 · p. 46
• Figure 20 · p. 47
• Figure 21 · p. 48
• Figure 22 · p. 48
• Figure 23 · p. 52
• Figure 24 · p. 53
• Figure 25 · p. 53
• Figure 26 · p. 54
• Figure 27 · p. 55
• Figure 28 · p. 56
• Figure 29 · p. 57
• Figure 30 · p. 61
• Figure 31 · p. 62
• Figure 32 · p. 63
• Figure 33 · p. 64
• Figure 34 · p. 65

Argument and organization

1. Problem and contribution

   Defines monitorability as a relation between observable outputs and a particular monitor, expressly not a guarantee that externalized reasoning faithfully reproduces internal computation.

2. Argument sequence

   Three evaluation archetypes and three observation scopes precede the empirical comparisons, giving the reader independent axes for understanding later scaling plots.

3. Evidence and verification

   The metric is TPR times TNR with hierarchical-bootstrap errors, but process evaluations condition on correct responses and set TNR to one; per-figure common subsets can differ across figures.

Problem, claim and contribution

Defines monitorability as a relation between observable outputs and a particular monitor, expressly not a guarantee that externalized reasoning faithfully reproduces internal computation.

Section 2, p. 2 · Main paper

Argument and section order

Three evaluation archetypes and three observation scopes precede the empirical comparisons, giving the reader independent axes for understanding later scaling plots.

Sections 2-5, pp. 2-7 · Main paper

Figures, captions and visual explanation

The opening plot separates CoT length, access scope and model comparisons; compute/capability plots mark dominated points and iso-compute versus monitorability-tax moves.

Figure 1, p. 2; Figures 4-5, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

The metric is TPR times TNR with hierarchical-bootstrap errors, but process evaluations condition on correct responses and set TNR to one; per-figure common subsets can differ across figures.

Sections 2.1-2.2, p. 3; Section 3.2, p. 5 · Main paper

Appendix and reproducibility

Detailed prompts and degeneracy tests expose monitor shortcuts; a paired-outcome filtering step is explicitly described as a stopgap rather than an unbiased deployment protocol.

Appendix D.3, p. 30; F, pp. 40-45; J, pp. 58-65 · Main paper

Clarity and writing conventions

Limitations accompany both the evaluation taxonomy and the follow-up intervention, including novel solution paths, noisy labels and follow-ups that reduce monitorability.

Sections 2-3, pp. 2-5; Section 6, p. 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Separate the construct, observation access and labeling mechanism before plotting scaling trends, and explain when aggregation changes the evaluated subset.

Boundary: The aggregate score is neither a faithfulness certificate nor directly comparable across arbitrary figures; process and outcome archetypes use different labeling assumptions.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-7 and discussion, pp. 1-9: monitorability definition, metric, evaluation archetypes/scopes, model comparisons, scaling and follow-up interventions.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-J, pp. 13-65: metric/bootstrapping, evaluation and prompt details, degeneracy controls, model/obfuscation analysis and follow-up examples.

Visually inspected PDF pages

2, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 65 pages