Writing study: Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Thibaud Gloaguen, Mark Vero, Robin Staab, Martin Vechev · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Learning & optimization algorithms.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

40 physical pages

Before references / appendix

≈ 10.11 page-equivalents

Main figures / tables

4 / 9

Attached appendix figures / tables

31 / 11

Attached appendix span

≈ 24.4 page-equivalents

Before references / appendix: ≈ 25.27%References / intervening material: ≈ 13.73%Attached appendix: ≈ 61%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.07 pp.	2.68%
2 BACKGROUND AND RELATED WORK	2	≈ 1.03 pp.	2.58%
3 FAB: FINETUNING-ACTIVATED BEHAVIORS	3	≈ 1.77 pp.	4.42%
4 EVALUATION	5	≈ 4.74 pp.	11.85%
5 CONCLUSION AND LIMITATIONS	10	≈ 0.47 pp.	1.18%
ETHICS STATEMENT	10	≈ 0.2 pp.	0.5%
REPRODUCIBILITY STATEMENT	10	≈ 0.26 pp.	0.65%
REFERENCES	11	≈ 5.5 pp.	13.75%
A FURTHER EXPERIMENTS	16	≈ 15.79 pp.	39.48%
B ADDITIONAL SCENARIO	32	≈ 0.45 pp.	1.13%
C EXTENDED EXPERIMENTAL DETAILS	32	≈ 3.88 pp.	9.7%
D PROMPTS AND TEMPLATES	36	≈ 2.58 pp.	6.45%
E BROADER IMPACT AND RESOURCES	39	≈ 1.7 pp.	4.25%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 5
• Table 2 · p. 5
• Figure 2 · p. 5
• Table 3 · p. 6
• Table 4 · p. 6
• Table 5 · p. 7
• Table 6 · p. 7
• Table 7 · p. 8
• Table 8 · p. 9
• Table 9 · p. 9
• Figure 3 · p. 16
• Figure 4 · p. 16

Attached appendix

• Figure 5 · p. 17
• Figure 6 · p. 17
• Figure 7 · p. 18
• Figure 8 · p. 18
• Figure 9 · p. 19
• Figure 10 · p. 19
• Figure 11 · p. 19
• Figure 12 · p. 20
• Figure 13 · p. 20
• Figure 14 · p. 20
• Figure 15 · p. 21
• Figure 16 · p. 21
• Figure 17 · p. 22
• Figure 18 · p. 22
• Figure 19 · p. 22
• Figure 20 · p. 23
• Figure 21 · p. 23
• Figure 22 · p. 23
• Figure 23 · p. 24
• Figure 24 · p. 24
• Figure 25 · p. 25
• Figure 26 · p. 25
• Figure 27 · p. 25
• Figure 28 · p. 26
• Figure 29 · p. 26
• Figure 30 · p. 26
• Figure 31 · p. 27
• Figure 32 · p. 27
• Figure 33 · p. 27
• Figure 34 · p. 28
• Table 10 · p. 28
• Table 11 · p. 29
• Table 12 · p. 29
• Table 13 · p. 30
• Table 14 · p. 30
• Table 15 · p. 31
• Table 16 · p. 31
• Table 17 · p. 32
• Table 18 · p. 32
• Table 19 · p. 33
• Table 20 · p. 33
• Figure 35 · p. 36

Argument and organization

1. Threat model and method

   Separate attacker control from later user-controlled adaptation before describing the experimental method.

2. Evaluation

   Pair three behavioral scenarios with utility measurements, then stress-test adaptation choices.

3. Conclusion and limitations

   Separate demonstrated vulnerability from prospective detection and mitigation.

Problem, claim and contribution

The threat model distinguishes malicious model weights from poisoned user data and input-triggered backdoors. Its central novelty is the timing of activation, which the related-work section contrasts explicitly with superficially similar attacks.

§§1–3, pp.1–3 · Main paper

Argument and section order

Each scenario repeats setup then results, and attack-success tables are paired with utility tables. This reusable structure prevents behavioral activation and ordinary task quality from being conflated.

§§4.1–4.3, Tables1–6, pp.5–7 · Main paper

Figures, captions and visual explanation

The opening lifecycle diagram places the adversary, sharing platform and user in separate regions, making ownership boundaries and the before/after evaluation gap visible without requiring the optimization details.

Fig.1, p.2 · Main paper

Tables, comparisons and evidence

The robustness table names the varied setting, retains failed or weak conditions, and reports five-run variation. Its highlighted defaults help audit the extent to which the broad robustness claim depends on a favorable adaptation configuration.

Table7, §4.4, p.8 · Main paper

Appendix and reproducibility

The appendix retains full activation curves, evaluator definitions and a concrete judge failure. Excluding PubMedQA from over-refusal scoring is explained by formatting-induced judge errors, rather than silently dropping an inconvenient dataset.

Appendices A.1–A.3, C.4 and D.3, pp.16–28, 34–36, 38–39 · Main paper

Clarity and writing conventions

The limitations acknowledge model scale and substantial attacker tuning; the ethics discussion addresses misuse and restricted release. Proposed low-cost checks are presented after the empirical threat, but their observed success should not be rewritten as comprehensive defenses.

§5, p.10; Appendices A.8–A.9 and E, pp.30–31, 39–40 · Main paper

What to reuse—and what not to infer

Transferable lesson: State who controls each stage, measure the harmful behavior against an adapted benign baseline, and audit the evaluator before interpreting robustness.

Boundary: Experiments use models no larger than3B parameters, and some settings suppress activation. Utility cells allow substantial relative degradation; lack of obvious baseline change is not certified harmlessness. Utility evaluation selects the better of instruction/completion formats. The appendix's permissive-license wording includes noncommercial data, so it is not blanket reuse permission.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

PDF pp.1–10, §§1–5 and ethics/reproducibility; references pp.11–15

Appendix / supplement

Appendices A–E, PDF pp.16–40, including all ablation captions, evaluator prompts, example outputs and resource disclosures

Visually inspected PDF pages

2, 8

• official ICLR 2026 proceedings PDF · 40 pages