Writing study: EditBench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Wayne Chi, Valerie Chen, Ryan Shar, Aditya Mittal, Jenny Liang, Wei-Lin Chiang, Anastasios Angelopoulos, Ion Stoica, Graham Neubig, Ameet Talwalkar, Chris Donahue · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

27 physical pages

Before references / appendix

≈ 8.51 page-equivalents

Main figures / tables

5 / 3

Attached appendix figures / tables

10 / 5

Attached appendix span

≈ 13.89 page-equivalents

Before references / appendix: ≈ 31.52%References / intervening material: ≈ 17.04%Attached appendix: ≈ 51.44%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.72 pp.	6.37%
2 RELATED WORK	3	≈ 0.46 pp.	1.7%
3 BENCHMARK CONSTRUCTION	3	≈ 1.67 pp.	6.19%
4 BENCHMARK STATISTICS	5	≈ 1.08 pp.	4%
5 EVALUATION	6	≈ 2.57 pp.	9.52%
6 CONCLUSION, LIMITATIONS, AND FUTURE WORK	9	≈ 0.29 pp.	1.07%
ACKNOWLEDGMENTS	9	≈ 0.11 pp.	0.41%
REFERENCES	9	≈ 4.59 pp.	17%
A DATA COLLECTION DETAILS	14	≈ 2 pp.	7.41%
B EDITBENCHEXT INSTRUCTED EDITS DATA ANALYSIS	16	≈ 6 pp.	22.22%
C EDITBENCH DETAILS	22	≈ 1.49 pp.	5.52%
D EVALUATION SET-UP	23	≈ 2.51 pp.	9.3%
E ADDITIONAL EVALUATION RESULTS	26	≈ 1 pp.	3.7%
F FURTHER DISCUSSION	27	≈ 0.89 pp.	3.3%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 3
• Table 1 · p. 5
• Figure 3 · p. 5
• Table 2 · p. 6
• Figure 4 · p. 7
• Table 3 · p. 7
• Figure 5 · p. 8

Attached appendix

• Figure 6 · p. 16
• Figure 7 · p. 17
• Figure 8 · p. 17
• Figure 9 · p. 18
• Figure 10 · p. 18
• Table 4 · p. 19
• Table 5 · p. 20
• Figure 11 · p. 21
• Figure 12 · p. 22
• Table 6 · p. 23
• Figure 13 · p. 24
• Figure 14 · p. 24
• Figure 15 · p. 25
• Table 7 · p. 26
• Table 8 · p. 26

Argument and organization

1. Real-world editing gap

   Distinguishes single-file instructed edits from contest problems and repository-level agents.

2. Construction and evaluation

   Explains collection, manual test harnesses, context ablations and model comparisons.

3. Appendices A–F

   Documents privacy, representative requests, prompts and benchmark maintenance.

Problem, claim and contribution

The contribution is defined by an interaction—existing code plus an instruction and IDE context—rather than simply another leaderboard.

Section 1 and Figure 1, pp. 1–2 · Main paper

Figures, captions and visual explanation

Two side-by-side IDE screenshots make collection concrete; the marked region and resulting diff communicate input and output without a complex schematic.

Figure 2, p. 3 · Main paper

The sorted 40-model bar plot gives a clear overall ranking but tiny angled model labels make precise lookup difficult; a companion machine-readable table is important.

Figure 4, p. 7 · Main paper

Tables, comparisons and evidence

The context ablation separates code, highlighting and cursor position, showing that extra context can help or hurt instead of assuming all context is beneficial.

Table 3, p. 7 · Main paper

Appendix and reproducibility

Rejected ambiguous and trivial requests, full prompts and a failed fully automated test-harness attempt expose curation decisions and negative results.

Appendices C–F, pp. 22–27 · Main paper

Clarity and writing conventions

Several dataset descriptions disagree: natural-language lists differ, Appendix C gives 113 annotated problems versus the main 109, and the statement about longer hard-question highlights conflicts with Table 8.

Section 3.2; Appendix C, p. 22; Table 8, p. 26 · Main paper

What to reuse—and what not to infer

Transferable lesson: For an in-the-wild benchmark, show the actual collection interaction, rejected examples and a controlled context ablation; these establish what the benchmark measures beyond its headline score.

Boundary: Realistic inputs do not remove filtering bias. Reconcile corpus counts, language identities, model names and prose/table comparisons before reusing exact benchmark descriptions.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–9, Sections 1–6 including evaluation and limitations

Appendix / supplement

Pages 14–27, Appendices A–F: extension/privacy, corpus examples, filtering, prompts, additional evaluation, discussion

Visually inspected PDF pages

3, 7

• ICLR 2026 final proceedings · 27 pages