Writing study: Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Zhicheng Fang, Jingjie Zheng, Chenxu Fu, Wei Xu · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

43 physical pages

Before references / appendix

≈ 8.75 page-equivalents

Main figures / tables

4 / 1

Attached appendix figures / tables

7 / 7

Attached appendix span

≈ 29.91 page-equivalents

Before references / appendix: ≈ 20.35%References / intervening material: ≈ 10.09%Attached appendix: ≈ 69.56%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Small-cap JBF-FORGE typography suppressed Appendix K and consequently L; restore printed headings on physical pp27 and42.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.03 pp.	2.4%
2. Background and Related Work	2	≈ 1.04 pp.	2.42%
3. Jailbreak Foundry (JBF)	3	≈ 1.66 pp.	3.86%
4. Results and Analysis of JBF	5	≈ 2.64 pp.	6.14%
5. Attack–Model Interaction Analysis	7	≈ 1.33 pp.	3.09%
6. Conclusion and Future Work	9	≈ 0.55 pp.	1.28%
Acknowledgements	9	≈ 0.05 pp.	0.12%
Impact Statement	9	≈ 0.11 pp.	0.26%
References	9	≈ 4.34 pp.	10.09%
A. Attack Taxonomy	14	≈ 1 pp.	2.33%
B. Auditor Acceptance Criteria and Spec/Contract Checks	15	≈ 2.51 pp.	5.84%
C. Audit–Iteration Cap T	17	≈ 0.49 pp.	1.14%
D. Complexity of JBF-FORGE	18	≈ 2 pp.	4.65%
E. Planner/Coder/Auditor Model Ablation	20	≈ 1 pp.	2.33%
F. Additional Fidelity Verification Beyond ASR	21	≈ 2 pp.	4.65%
G. Compression Ratio Variation	23	≈ 1 pp.	2.33%
H. Attack Budget Comparability	24	≈ 1 pp.	2.33%
I. JBF-LIB Core Interfaces and Reusable Components	25	≈ 1 pp.	2.33%
J. JBF-EVAL Standardized Benchmark Details	26	≈ 1 pp.	2.33%
K. JBF-FORGE	27	≈ 15 pp.	34.88%
L. Enhanced Refinement Pass	42	≈ 1.91 pp.	4.44%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 6
• Figure 2 · p. 6
• Figure 3 · p. 7
• Figure 4 · p. 8

Attached appendix

• Table 2 · p. 14
• Figure 5 · p. 17
• Table 3 · p. 18
• Table 4 · p. 20
• Table 5 · p. 20
• Table 6 · p. 21
• Table 7 · p. 21
• Table 8 · p. 23
• Figure 6 · p. 24
• Figure 7 · p. 30
• Figure 8 · p. 34
• Figure 9 · p. 41
• Figure 10 · p. 42
• Figure 11 · p. 43

Argument and organization

1. 1 Introduction; 2 Background

   Frames implementation drift and separates attack taxonomy from measurement infrastructure.

2. 3 Jailbreak Foundry

   Assigns contracts, synthesis and evaluation to three named components.

3. 4 Results; 5 Interaction Analysis; 6 Conclusion

   Separates reproduction fidelity, engineering reuse and standardized cross-model evaluation before limitations.

Problem, claim and contribution

The opening treats comparability and reproduction drift as the research problem, rather than presenting a higher attack score as the sole contribution.

Abstract and Introduction, p.1 · Main paper

Argument and section order

Paper-matched reproduction and the common-harness cross-model benchmark are distinct evaluation settings.

Sections 3.3 and 4.1, pp.4-5 · Main paper

Figures, captions and visual explanation

The overview separates synthesis, shared runtime and evaluation into three labeled blocks; the dense result table groups setup, implementation, taxonomy and scores under distinct headers.

Figure 1, p.2; Table 1, p.6 · Main paper

Tables, comparisons and evidence

The text reports both signed and absolute reproduction gaps, which prevents a near-zero mean signed gap from hiding large positive and negative discrepancies.

Section 4.2, p.5; Table 1, p.6 · Main paper

Appendix and reproducibility

Additional fidelity checks explicitly state that aggregate success rates are insufficient; a separate token-cost view explains why paper-matched budgets are not equal budgets.

Appendices F and H, pp.21,24 · Main paper

Clarity and writing conventions

The conclusion names unsupported attack interfaces and manual engineering needs, while the impact statement separates defensive evaluation value from misuse risk.

Conclusion and Impact Statement, p.9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Keep implementation fidelity, standardized effectiveness and engineering cost as separate claims with separate denominators.

Boundary: Matching aggregate success rates is not semantic equivalence, and paper-matched operating points are not equal-budget rankings; no attack procedures were executed in this writing study.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 4, 5, 6, 7, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 21, 24. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

2, 6

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 43 pages