Writing study: On the Role of Computation in Reinforcement Learning

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko, Benjamin Eysenbach · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Theory & formal analysis (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

15 physical pages

Before references / appendix

≈ 8.25 page-equivalents

Main figures / tables

7 / 1

Attached appendix figures / tables

0 / 2

Attached appendix span

≈ 2.91 page-equivalents

Before references / appendix: ≈ 55%References / intervening material: ≈ 25.6%Attached appendix: ≈ 19.4%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Wrapped main heading labels restored from the immediately following same-size source line in the physical PDF. Heading start positions are unchanged. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.92 pp.	6.13%
2. Related Work	2	≈ 0.63 pp.	4.2%
3. Preliminaries	2	≈ 0.48 pp.	3.2%
4. On the complexity of behaving optimally	3	≈ 0.92 pp.	6.13%
5. A Minimal recurrent architecture for studying the benefits of computation	4	≈ 0.59 pp.	3.93%
6. Experiments	4	≈ 3.51 pp.	23.4%
7. Conclusion	8	≈ 0.83 pp.	5.53%
Impact Statement	9	≈ 0.05 pp.	0.33%
References	9	≈ 3.84 pp.	25.6%
A. Proofs	13	≈ 2 pp.	13.33%
B. Implementation details	15	≈ 0.91 pp.	6.07%

All detected figure and table locations

Main text

• Figure 1 · p. 4
• Figure 2 · p. 5
• Figure 3 · p. 5
• Figure 4 · p. 5
• Table 1 · p. 6
• Figure 5 · p. 7
• Figure 6 · p. 7
• Figure 7 · p. 8

Attached appendix

• Table 2 · p. 15
• Table 3 · p. 15

Argument and organization

1. Problem and claim

   Opens with two explicit questions about compute and fixed parameter count, then explains why ordinary static-policy language conflates those axes. The framing gives the formalization a concrete purpose.

2. Argument sequence

   Definitions and existence theorems precede a deliberately minimal recurrent architecture. The experiment section lists questions before subsections on steps, generalization, value of compute, exploration and block choice.

3. Evidence and verification

   The comparison uses a parameter-oriented MLP baseline and a depth-oriented ResNet baseline, then reports five-seed results. Longer-horizon evaluation retains that IRU beats ResNet on four of six tasks, not all tasks.

Problem, claim and contribution

Opens with two explicit questions about compute and fixed parameter count, then explains why ordinary static-policy language conflates those axes. The framing gives the formalization a concrete purpose.

Abstract and Introduction, pp. 1–2 · Main paper

Argument and section order

Definitions and existence theorems precede a deliberately minimal recurrent architecture. The experiment section lists questions before subsections on steps, generalization, value of compute, exploration and block choice.

Sections 4–6, pp. 3–8 · Main paper

Figures, captions and visual explanation

Figure 1 locates the repeated state within the network and retains separate action/value outputs. Table 1 groups each environment into halfway and final-budget rows, making sample efficiency different from final performance.

Figure 1, physical p. 4; Table 1, physical p. 6; rasters inspected · Main paper

Tables, comparisons and evidence

The comparison uses a parameter-oriented MLP baseline and a depth-oriented ResNet baseline, then reports five-seed results. Longer-horizon evaluation retains that IRU beats ResNet on four of six tasks, not all tasks.

Sections 6.2–6.4, physical p. 6 · Main paper

Appendix and reproducibility

Short theorem-construction proofs are separated from algorithm-specific implementation tables. Main statements keep the existential quantifier and time-constructibility conditions visible rather than exporting all scope to the appendix.

Theorems 4.1–4.2, pp. 3–4; Appendices A–B, pp. 13–15 · Main paper

Clarity and writing conventions

The exploratory-data explanation is presented as a hypothesis and tested with yoked agents. The compute accounting still needs care: one paragraph calls the ResNet compute similar, while the next discusses approximately twice as much.

Sections 6.2–6.3, p. 6; Section 6.6, p. 7 · Main paper

What to reuse—and what not to infer

Transferable lesson: State which resource a baseline matches, and use a question-led experiment sequence to connect a formal distinction to observable behavior.

Boundary: Existence of MDPs benefiting from more computation does not prove monotonic gains on every task; parameter counts, effective depth and measured execution cost are distinct.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused scan of Sections 1–7, pp. 1–8: formal policy classes, theorem scope, architecture, question-led experiments and comparisons; impact statement p. 9.

Appendix / supplement

Structural and selected-detail scan of Appendices A–B, pp. 13–15: theorem constructions, assumptions and algorithm-specific hyperparameter tables. No independent proof verification.

Visually inspected PDF pages

4, 6

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 15 pages