Writing study: CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

ZHUN WANG, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, Dawn Song · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

46 physical pages

Before references / appendix

≈ 10.82 page-equivalents

Main figures / tables

8 / 2

Attached appendix figures / tables

17 / 7

Attached appendix span

≈ 30.89 page-equivalents

Before references / appendix: ≈ 23.52%References / intervening material: ≈ 9.33%Attached appendix: ≈ 67.15%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Caption audit 2026-09-19, 2YvbLQEdYt: rendered physical PDF page 5 confirms Figure 2, the OSS-Fuzz lifecycle diagram. Poppler splits Figure, 2:, OSS-Fuzz and lifecycle into separate same-baseline lines, not a contiguous caption line. Position uses yMin 579.484357 / 792 plus page index 4.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.59 pp.	3.46%
2 RELATED WORK	3	≈ 0.69 pp.	1.5%
3 CYBERGYM BENCHMARK	3	≈ 2.82 pp.	6.13%
4 EXPERIMENTAL EVALUATION	6	≈ 3 pp.	6.52%
5 FROM BENCHMARKING TO DIRECT SECURITY IMPACT	9	≈ 0.86 pp.	1.87%
6 CONCLUSION AND FUTURE WORK	10	≈ 0.74 pp.	1.61%
ETHICS STATEMENT	11	≈ 0.47 pp.	1.02%
REPRODUCIBILITY STATEMENT	11	≈ 0.14 pp.	0.3%
REFERENCES	11	≈ 4.29 pp.	9.33%
A THE USE OF LARGE LANGUAGE MODELS	16	≈ 0.14 pp.	0.3%
B DETAILS OF CYBERGYM BENCHMARK	16	≈ 2.86 pp.	6.22%
C DETAILS ON EXPERIMENTAL SETUP	19	≈ 4.22 pp.	9.17%
D ADDITIONAL EXPERIMENTAL RESULTS	23	≈ 4.02 pp.	8.74%
E ANALYSIS OF INCOMPLETE PATCHES AND ZERO-DAY VULNERABILITIES	27	≈ 0.28 pp.	0.61%
F ADDITIONAL CASE STUDIES	27	≈ 19.38 pp.	42.13%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 3
• Figure 2 · p. 5
• Figure 3 · p. 7
• Figure 4 · p. 7
• Figure 5 · p. 7
• Table 2 · p. 8
• Figure 6 · p. 8
• Figure 7 · p. 8
• Figure 8 · p. 9

Attached appendix

• Table 3 · p. 16
• Table 4 · p. 17
• Table 5 · p. 18
• Figure 9 · p. 19
• Figure 10 · p. 20
• Figure 11 · p. 20
• Figure 12 · p. 21
• Figure 13 · p. 21
• Table 6 · p. 22
• Table 7 · p. 22
• Figure 14 · p. 23
• Figure 15 · p. 24
• Figure 16 · p. 24
• Table 8 · p. 24
• Figure 17 · p. 25
• Table 9 · p. 26
• Figure 18 · p. 27
• Figure 19 · p. 29
• Figure 20 · p. 29
• Figure 21 · p. 32
• Figure 22 · p. 35
• Figure 23 · p. 37
• Figure 24 · p. 40
• Figure 25 · p. 46

Argument and organization

1. Benchmark limitations → reproducible task and oracle

   Defines inputs, success, difficulty levels, and collection filters before reporting rankings.

2. Capability measurements → practical impact → failure cases

   Separates benchmark scores from subsequently validated findings.

Problem, claim and contribution

Two explicit shortcomings—small evaluation sets and static outcomes—organize the introduction and map to the benchmark and discovery analyses.

Introduction, pages 1–2 · Main paper

Argument and section order

Scope is stated early: sanitizer-detectable issues in predominantly C/C++ projects. The conclusion revisits omitted vulnerability types and the difference between reproduction and exploitation.

Section 3.1, page 4; Section 6, page 10 · Main paper

Figures, captions and visual explanation

Figure 1 shows dataset breadth, agent-visible inputs, and pre/post/latest execution outcomes in separate columns; it communicates the evaluation contract before presenting performance.

Figure 1, page 2 · Main paper

Tables, comparisons and evidence

The results use distinct plots for backbone, thinking mode, and agent framework. The adjacent prose identifies the 300-instance subset used for thinking comparisons, avoiding direct confusion with full-benchmark scores.

Figures 3–5, page 7 · Main paper

Appendix and reproducibility

Model checkpoints, agent commits, prompts, hardware, and failure-mode counts are provided, alongside long success and failure traces rather than only favorable examples.

Appendices C–F, pages 19–46 · Main paper

Clarity and writing conventions

Final-manuscript consistency needs attention: introduction counts of incomplete patches and reproduction-derived zero-days differ from Section 5, and a JPEG trace is captioned as GIF. Such mismatches distract from an otherwise auditable benchmark.

Introduction, page 2; Section 5, pages 9–10; Figure 22, page 35 · Main paper

What to reuse—and what not to infer

Transferable lesson: A benchmark paper should make its scoring oracle inspectable, name the evaluation subset beside each comparison, and preserve negative agent traces with versioned configuration details.

Boundary: Non-significant cutoff comparisons do not prove absence of contamination. Sanitizer crashes measure the defined reproduction task, not the full spectrum of cybersecurity capability or exploitability.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–6, ethics and reproducibility; references on pages 11–15

Appendix / supplement

Pages 16–46, Appendices A–F, including dataset tables, complete printed prompts, settings, statistical analyses, and printed case traces

Visually inspected PDF pages

2, 7

• ICLR 2026 final proceedings · 46 pages