Writing study: CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Xianzhen Luo, Jingyuan Zhang, Shiqi Zhou, Jinyang Huang, Chuan Xiao, Qingfu Zhu, Zhiyuan Ma, Yue Xing, Yang Yue, Wencong Zeng, Wanxiang Che · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

32 physical pages

Before references / appendix

≈ 9.21 page-equivalents

Main figures / tables

5 / 4

Attached appendix figures / tables

1 / 7

Attached appendix span

≈ 19.91 page-equivalents

Before references / appendix: ≈ 28.78%References / intervening material: ≈ 9%Attached appendix: ≈ 62.22%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 0.96 pp.	3%
2. Related Work	2	≈ 0.61 pp.	1.91%
3. CVE-Factory	3	≈ 1.85 pp.	5.78%
4. Experiment	5	≈ 3.54 pp.	11.06%
5. Discussion	8	≈ 1.05 pp.	3.28%
6. Conclusion	9	≈ 0.19 pp.	0.59%
Impact Statement	10	≈ 0.17 pp.	0.53%
References	10	≈ 2.88 pp.	9%
A. Complete Example	13	≈ 4 pp.	12.5%
B. Sample Real-World Distribution	17	≈ 2.18 pp.	6.81%
C. LiveCVEbench	19	≈ 4.82 pp.	15.06%
D. Training Details	24	≈ 1 pp.	3.13%
E. Time and Cost Distribution of Agents	25	≈ 1 pp.	3.13%
F. Comparative Case Study: Manual vs. CVE-Factory (CVE-2021-21384)	26	≈ 6 pp.	18.75%
G. Limitations	32	≈ 0.91 pp.	2.84%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 3
• Table 1 · p. 5
• Figure 3 · p. 6
• Figure 4 · p. 7
• Table 2 · p. 7
• Table 3 · p. 8
• Figure 5 · p. 8
• Table 4 · p. 8

Attached appendix

• Table 5 · p. 17
• Table 6 · p. 17
• Table 7 · p. 19
• Table 8 · p. 19
• Table 9 · p. 20
• Table 10 · p. 23
• Table 11 · p. 24
• Figure 6 · p. 25

Argument and organization

1. Problem and contribution

   Defines a task package as description, runnable environment, tests and reference solution, making sparse vulnerability metadata's missing artifacts concrete.

2. Argument sequence

   Three generation stages and three verification stages lead to separate expert-agreement, real-world reproduction and training-utility evaluations rather than one aggregate success claim.

3. Evidence and verification

   Manual checking rejects 187 of 499 reported successful reproductions among 554 attempts; the quoted 66.2% excludes objectively irreproducible cases, and environment cross-validation also excludes stricter-test cases.

Problem, claim and contribution

Defines a task package as description, runnable environment, tests and reference solution, making sparse vulnerability metadata's missing artifacts concrete.

Introduction, p. 2; Section 3, p. 3 · Main paper

Argument and section order

Three generation stages and three verification stages lead to separate expert-agreement, real-world reproduction and training-utility evaluations rather than one aggregate success claim.

Sections 3-4, pp. 3-8 · Main paper

Figures, captions and visual explanation

The quality table identifies environment/test/solution substitutions; paired training rows retain absolute scores alongside the headline multiples, and stage-duration boxplots reveal heterogeneous workflow costs.

Table 1, p. 5; Tables 3-4 and Figure 5, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

Manual checking rejects 187 of 499 reported successful reproductions among 554 attempts; the quoted 66.2% excludes objectively irreproducible cases, and environment cross-validation also excludes stricter-test cases.

Sections 4.1-4.2, pp. 5-6 · Main paper

Appendix and reproducibility

Heuristic reproducibility scores and diversity caps expose deliberate sampling away from inaccessible hardware/proprietary settings; final limitations restrict the pipeline to public, patched and containerizable cases.

Appendix B, pp. 17-18; G, p. 32 · Main paper

Clarity and writing conventions

Strong expert-level and real-world labels need their actual verification contract. The introduction says 454 recent CVEs while Section 4.2 describes 554 attempts, so the source contains an unresolved count inconsistency.

Introduction, p. 2; Section 4.2, p. 6 · Main paper

What to reuse—and what not to infer

Transferable lesson: Report generated, automatically accepted, manually verified and excluded counts separately; cross-check independent task components rather than trusting an end-to-end self-check.

Boundary: Selected reproducible public CVEs are not the full vulnerability distribution, model-based test-quality ratings are not independent human equivalence judgments, and generated benchmarks do not certify arbitrary software security.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-6, pp. 1-9: executable-task motivation, staged generation/verification, expert cross-validation, recent-CVE filtering and manual verification, training utility and workflow ablations.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-G, pp. 13-32: example package, sampling rules, judge/provenance details, benchmark subgroup analysis, training configuration, comparative case and limitations; code and environments were not executed.

Visually inspected PDF pages

5, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 32 pages