Writing study: Position: Stop Automating Peer Review Without Rigorous Evaluation

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Joachim Baumann, Jiaxin Pei, Sanmi Koyejo, Dirk Hovy · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Position & research agenda (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

32 physical pages

Before references / appendix

≈ 9.11 page-equivalents

Main figures / tables

6 / 1

Attached appendix figures / tables

6 / 6

Attached appendix span

≈ 17.91 page-equivalents

Before references / appendix: ≈ 28.47%References / intervening material: ≈ 15.56%Attached appendix: ≈ 55.97%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Join wrapped printed heading text using adjacent source lines; original heading start coordinates are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 Introduction	1	≈ 0.95 pp.	2.97%
2 Background: AI in peer review	2	≈ 1.44 pp.	4.5%
3 The AI reviewer hivemind effect	3	≈ 2.18 pp.	6.81%
4 Paper laundering: Gaming AI reviews is trivial	5	≈ 1.36 pp.	4.25%
5 Alternative views	7	≈ 0.96 pp.	3%
6 Call to action: Toward a science of peer review automation	8	≈ 1.42 pp.	4.44%
7 Conclusions	9	≈ 0.26 pp.	0.81%
Acknowledgements	9	≈ 0.23 pp.	0.72%
References	10	≈ 4.98 pp.	15.56%
A Limitations	15	≈ 0.39 pp.	1.22%
B Implementation details	15	≈ 6.3 pp.	19.69%
C AI reviewer score correlations	21	≈ 0.58 pp.	1.81%
D Common templates used in AI reviews	22	≈ 0.32 pp.	1%
E What laundering changes	22	≈ 1.16 pp.	3.62%
F Length statistics and embedding robustness	23	≈ 0.46 pp.	1.44%
G Ablations	24	≈ 1.48 pp.	4.63%
H AI-generated reviews	25	≈ 7.23 pp.	22.59%

All detected figure and table locations

Main text

• Table 1 · p. 3
• Figure 1 · p. 4
• Figure 2 · p. 5
• Figure 3 · p. 5
• Figure 4 · p. 6
• Figure 5 · p. 6
• Figure 6 · p. 7

Attached appendix

• Table 2 · p. 22
• Table 3 · p. 22
• Table 4 · p. 23
• Table 5 · p. 23
• Table 6 · p. 24
• Figure 7 · p. 24
• Figure 8 · p. 25
• Figure 9 · p. 26
• Figure 10 · p. 27
• Figure 11 · p. 28
• Table 7 · p. 28
• Figure 12 · p. 29

Argument and organization

1. 1 Introduction; 2 Context and Related Work

   States two necessary but explicitly insufficient conditions for peer-review automation.

2. 3 AI Reviewer Hivemind; 4 Paper Laundering

   Uses observational review data and controlled rewriting experiments to challenge those conditions.

3. 5 Alternative Views; 6 Call to Action; 7 Conclusions

   Answers objections and proposes evaluation, stakeholder, interaction and incentive research rather than a deployment recipe.

Problem, claim and contribution

A boxed pair of criteria makes the normative argument testable while the adjacent note explicitly denies that satisfying them alone would justify automation.

Introduction criteria box, p.1 · Main paper

Argument and section order

Empirical sections correspond to the two opening conditions, followed by counterarguments and four distinct research-agenda pillars.

Sections 3-6, pp.3-9 · Main paper

Figures, captions and visual explanation

The paired-score plot includes a zero-change baseline, confidence intervals and separate encodings for prompt and rewriter; a stacked outcome plot retains unchanged and worsened papers.

Figures 4-5, p.6 · Main paper

Tables, comparisons and evidence

The paper distinguishes detector-labeled reviews in the wild from a 60-paper controlled simulation, and reports effect sizes alongside significance rather than only large-sample p-values.

Section 3.1, p.3; Figures 2-3 and Section 3.5, p.5 · Main paper

Appendix and reproducibility

The limitations identify similarity as an imperfect proxy for viewpoint diversity and acknowledge detector labels, fixed prompts and one-venue sampling; manual rewrite inspection documents fabricated additions.

Appendix A, p.15; Appendix E.1, p.22 · Main paper

Clarity and writing conventions

Alternative Views narrows the thesis to unevaluated judgment automation and distinguishes validated assistance from replacing human assessment, avoiding a blanket prohibition on every AI use.

Sections 5.3-5.4 and 6.1, pp.7-8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Anchor a position in explicit criteria, test those criteria with complementary evidence, and distinguish necessary conditions from sufficient justification.

Boundary: Embedding similarity is not direct argumentative diversity, acceptance prediction is not ground-truth scientific quality, and simulated-review score changes should not become manuscript-writing tactics.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 5, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 15, 22, 25. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

1, 6

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 32 pages