Writing study: Is it Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Xinpeng Wang, Nitish Joshi, Barbara Plank, Rico Angell, He He · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Datasets, benchmarks & measurement resources.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

26 physical pages

Before references / appendix

≈ 9.79 page-equivalents

Main figures / tables

14 / 0

Attached appendix figures / tables

17 / 2

Attached appendix span

≈ 12.89 page-equivalents

Before references / appendix: ≈ 37.65%References / intervening material: ≈ 12.77%Attached appendix: ≈ 49.58%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 0.87 pp.	3.35%
2 PROBLEM STATEMENT	2	≈ 0.57 pp.	2.19%
3 TRAINING REWARD HACKING MODELS	3	≈ 1.55 pp.	5.96%
4 TRACE: TRUNCATED REASONING AUC EVALUATION	4	≈ 3.48 pp.	13.38%
5 CASE STUDY: DEBUG TRAINING ENVIRONMENT LOOPHOLE	8	≈ 0.62 pp.	2.38%
6 RELATED WORK	8	≈ 0.52 pp.	2%
7 DISCUSSION	9	≈ 1.29 pp.	4.96%
ACKNOWLEDGMENTS	10	≈ 0.11 pp.	0.42%
REFERENCES	10	≈ 3.32 pp.	12.77%
APPENDIX	14	≈ 0.03 pp.	0.12%
A TRAINING HACKING MODEL WITH DIFFERENT HINT COMPLEXITY	14	≈ 1.03 pp.	3.96%
B SCALING COT MONITOR	15	≈ 0.5 pp.	1.92%
C ADDITIONAL RESULTS ON TRUNCATION REASONING EVALUATION	15	≈ 2.67 pp.	10.27%
D GRPO RESULTS	18	≈ 0.48 pp.	1.85%
E HACKING EXAMPLES	18	≈ 0.52 pp.	2%
F TRAINING DETAILS	19	≈ 4.76 pp.	18.31%
G PROMPTS FOR LOOPHOLE INJECTION	24	≈ 1 pp.	3.85%
H PROMPTS FOR COT MONITORING	25	≈ 0.76 pp.	2.92%
I CLUSTERING SAMPLES WITH TRACE FOR LOOPHOLE DISCOVERY	25	≈ 0.92 pp.	3.54%
J THE USE OF LARGE LANGUAGE MODELS	26	≈ 0.21 pp.	0.81%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 3
• Figure 3 · p. 4
• Figure 4 · p. 4
• Figure 5 · p. 5
• Figure 6 · p. 5
• Figure 7 · p. 5
• Figure 8 · p. 6
• Figure 9 · p. 6
• Figure 10 · p. 6
• Figure 11 · p. 7
• Figure 12 · p. 7
• Figure 13 · p. 8
• Figure 14 · p. 8

Attached appendix

• Figure 15 · p. 14
• Figure 16 · p. 14
• Figure 17 · p. 15
• Figure 18 · p. 16
• Figure 19 · p. 17
• Figure 20 · p. 17
• Figure 21 · p. 18
• Figure 22 · p. 18
• Figure 23 · p. 19
• Table 1 · p. 19
• Table 2 · p. 19
• Figure 24 · p. 20
• Figure 25 · p. 21
• Figure 26 · p. 22
• Figure 27 · p. 23
• Figure 28 · p. 24
• Figure 29 · p. 24
• Figure 30 · p. 25
• Figure 31 · p. 26

Argument and organization

1. Hidden shortcut problem and controlled environments

   Distinguishes explicit from implicit hacking, then defines injected loopholes, trained policies and counterfactual labels.

2. Effort proxy and detection tests

   Explains forced early answers and evaluates both mixed-policy and single-policy detection.

3. Discovery demonstration and limitations

   Separates a clustering application from discussion of calibration, overthinking, monitor scale and adaptive optimization.

Problem, claim and contribution

The paper turns the mismatch between plausible explanations and shortcut answers into an operational question: how early can a truncated trace already earn proxy reward?

Sections 1–2, pages 1–3. · Main paper

Argument and section order

Defining hacking models and counterfactual sample labels before introducing the detector helps distinguish training access, observed reward and actual sample-level exploitation.

Sections 3.2 and 4, pages 4–5. · Main paper

Figures, captions and visual explanation

Figure 5 places truncation points on a reasoning timeline, branches to forced-answer samples and then shows reward estimates. The adjacent worked example connects this mechanism to a misleading self-correction trace.

Figures 5–6, page 5. · Main paper

Tables, comparisons and evidence

Detection plots separate task, loophole and model size; Figure 8 juxtaposes reward and TRACE against length and KL controls. The omitted no-CoT coding model is explicitly explained instead of silently dropped.

Figures 8–10 and Section 4.1, page 6. · Main paper

Appendix and reproducibility

Training tables and exact monitoring/discovery prompts support replication. The discovery prompt already tells the judge that hacking cases typically contain initial hints, so the demonstration is not a fully blind discovery test.

Appendix F, page 19; Appendix I, Figure 31, page 26. · Main paper

Clarity and writing conventions

The discussion explicitly identifies overthinking, initial-policy hacking and optimization against the score as limitations. Example consistency also needs attention: Figure 24 pairs a horse/chessboard prompt with a no-hint response about cube triangles.

Section 7, pages 9–10; Figure 24, page 20. · Main paper

What to reuse—and what not to infer

Transferable lesson: Define labels and counterfactual controls before presenting a detector; explain its intervention with a timeline, show competing diagnostic signals, and keep discovery demonstrations separate from validated detection claims.

Boundary: TRACE is a reward-versus-truncation AUC, not AUROC or a direct measurement of cognition. Synthetic loopholes, successful-response selection, extra sampling and initial-policy calibration limit generalization; high scores can also arise from overthinking, and the discovery prompt supplies a prior about hints.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–7, conclusion and acknowledgements; references on pages 10–13.

Appendix / supplement

Pages 14–26, Appendices A–J: hint complexity, monitor scaling, extra results, GRPO, examples, training settings, full prompts and clustering algorithm.

Visually inspected PDF pages

5, 6, 19, 20

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 26 pages