Writing study: CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Zhenhao Chen, Yongqiang Chen, Chenxi Liu, Junchi Yu, Xiangchen Song, Zijian Li, Jialin Li, Philip Torr, Bo Han, Kun Zhang · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

40 physical pages

Before references / appendix

≈ 9.23 page-equivalents

Main figures / tables

12 / 4

Attached appendix figures / tables

12 / 18

Attached appendix span

≈ 23.91 page-equivalents

Before references / appendix: ≈ 23.08%References / intervening material: ≈ 17.15%Attached appendix: ≈ 59.77%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Table 12 has a split label/caption at the top of physical p.28; source text and a rendered page confirm the omitted caption.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.55 pp.	3.88%
2. Related Work	2	≈ 0.9 pp.	2.25%
3. CausalGame Benchmark	3	≈ 2.06 pp.	5.15%
4. Experiments	5	≈ 3.99 pp.	9.98%
5. Conclusions	9	≈ 0.19 pp.	0.47%
Acknowledgements	10	≈ 0.11 pp.	0.27%
Impact Statement	10	≈ 0.08 pp.	0.2%
References	10	≈ 6.85 pp.	17.13%
A. Limitations and Future Works	17	≈ 0.31 pp.	0.78%
B. Related Work	17	≈ 1.29 pp.	3.23%
C. Details of CausalGame Benchmark	18	≈ 6.16 pp.	15.4%
D. Details of Experimental Results	24	≈ 7.34 pp.	18.35%
E. Rubrics used for evaluation	32	≈ 2.89 pp.	7.23%
F. Sample Agent Trajectories	35	≈ 1.77 pp.	4.42%
G. Additional Analysis	36	≈ 4.15 pp.	10.38%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Table 1 · p. 2
• Figure 2 · p. 3
• Figure 3 · p. 4
• Table 2 · p. 5
• Figure 4 · p. 6
• Figure 5 · p. 6
• Figure 6 · p. 6
• Figure 7 · p. 7
• Table 3 · p. 7
• Table 4 · p. 8
• Figure 8 · p. 8
• Figure 9 · p. 8
• Figure 10 · p. 9
• Figure 11 · p. 9
• Figure 12 · p. 9

Attached appendix

• Table 5 · p. 19
• Table 6 · p. 19
• Table 7 · p. 21
• Table 8 · p. 21
• Table 9 · p. 24
• Table 10 · p. 25
• Figure 13 · p. 25
• Figure 14 · p. 26
• Figure 15 · p. 26
• Figure 16 · p. 26
• Table 11 · p. 27
• Figure 17 · p. 27
• Table 12 · p. 28
• Table 13 · p. 28
• Table 14 · p. 29
• Table 15 · p. 29
• Table 16 · p. 30
• Table 17 · p. 30
• Figure 18 · p. 31
• Table 18 · p. 32
• Table 19 · p. 34
• Table 20 · p. 35
• Table 21 · p. 36
• Table 22 · p. 37
• Figure 19 · p. 37
• Figure 20 · p. 38
• Figure 21 · p. 39
• Figure 22 · p. 39
• Figure 23 · p. 40
• Figure 24 · p. 40

Argument and organization

1. 1 Introduction; 2 Related Work

   Motivates a causal-thinking evaluation gap in scientific-agent benchmarks.

2. 3 CausalGame Benchmark

   Defines interaction budgets, hidden mechanisms, biased observations and separate outcome/rubric scoring.

3. 4 Experiments; 5 Conclusions

   Organizes observations, execution-mode controls, cross-benchmark correlations and judge/trajectory failure diagnostics.

Problem, claim and contribution

The benchmark distinguishes obtaining a good final design from correctly explaining the mechanism, making the construct broader than reward maximization alone.

Sections 3.1 and 3.3, physical pp.4–5 · Main paper

Argument and section order

Simulator rules and exploration/evaluation budgets precede the leaderboard; rubric reliability is discussed before interpreting rubric-derived failure categories.

Sections 3–4, pp.3–8 · Main paper

Figures, captions and visual explanation

The pipeline explicitly joins observed history, active experiments and a final report; the analysis page places failure definitions alongside correlation and inter-rater plots.

Figure 2, p.3; Table 4 and Figures 8–9, p.8 · Main paper

Tables, comparisons and evidence

Main error bars summarize variation across scenarios, whereas experiments use three independent trials per model/scenario and appendices separately report uncertainty. The agent/prompt modes differ in several capabilities, not only interaction count.

Section 4.1 and Figures 4–6, p.6 · Main paper

Appendix and reproducibility

SCM tables make intervention, hidden variables, observed cues and win thresholds inspectable; the rubric table assigns explicit weights and the judge prompt asks for evidence-grounded decisions.

Tables 7–8, p.21; Table 18 and Appendix E, p.32 · Main paper

Clarity and writing conventions

A judge-independent configuration-path analysis complements LLM-scored explanations, separating an observable search failure from an inferred reasoning defect.

Configuration-based analysis and Figure 12, p.9 · Main paper

What to reuse—and what not to infer

Transferable lesson: For a reasoning benchmark, define the latent mechanism, permissible interventions, outcome score and process rubric separately, then validate the rubric with another observable analysis.

Boundary: Toy simulator failures are not proof that all scientific reasoning is absent or that medical deployment is safe/unsafe. Agreement among LLM judges is not ground-truth validity. The main 5–8-point threshold description does not cover Weather Noise’s 78% optimum versus 55% threshold in Table 8; preserve scenario-specific rules.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused scan across physical PDF pp.1–9, covering framing, section progression, definitions, method/evidence explanations, captions and scope. Physical pages extracted individually to avoid form-feed pagination errors. Selected full-page rasters inspected; not exhaustive line-by-line technical verification or experimental reproduction.

Appendix / supplement

Appendix heading/structure inventory checked across the source; selected detail reading of SCM specifications and optimal-survival definitions C.3–C.4 (p.21) and rubric/prompt E (p.32). Full prompts, all trajectory examples and numerical appendices were not exhaustively read or reproduced.

Visually inspected PDF pages

3, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 40 pages