Writing study: Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem
← ICML 2026 oral writing corpus
Qiyao Wei, Samuel Holt, Jing Yang, Markus Wulfmeier, Mihaela Van Der Schaar · Official ICML 2026 oral record
Figure, table and section ledger
Editorial contribution type: Position & research agenda (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 18 physical pages
- Before references / appendix
- ≈ 8.92 page-equivalents
- Main figures / tables
- 3 / 2
- Attached appendix figures / tables
- 0 / 1
- Attached appendix span
- ≈ 5.92 page-equivalents
Source-bound correction: Poppler merged left and right columns. Restore sections 4-9, the right-column p9 inline Impact Statement and References boundary, and p2 right-column Figure 1 using individual word coordinates. Full printed heading continuations retained. Physical p9 also rendered and visually checked.
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
| Top-level heading | Starts on PDF page | Estimated span | Share of entire PDF |
|---|---|---|---|
| 1. Introduction | 1 | ≈ 0.91 pp. | 5.06% |
| 2. The Cracks in the Current System | 2 | ≈ 0.98 pp. | 5.44% |
| 3. Related Work: The Evolving Role of AI in Peer Review | 3 | ≈ 0.63 pp. | 3.5% |
| 4. The Vision: An AI-Augmented Peer Review Ecosystem | 3 | ≈ 2.92 pp. | 16.22% |
| 5. The Critical Need for Richer Data: Fueling the AI-Augmented Ecosystem | 6 | ≈ 0.61 pp. | 3.39% |
| 6. Illustrative Experiments & Research Agenda | 7 | ≈ 0.83 pp. | 4.61% |
| 7. Alternative Views | 8 | ≈ 0.64 pp. | 3.56% |
| 8. Discussion and Conclusion | 8 | ≈ 0.44 pp. | 2.44% |
| 9. Limitations and Future Work | 9 | ≈ 0.53 pp. | 2.94% |
| Impact Statement | 9 | ≈ 0.07 pp. | 0.39% |
| References | 9 | ≈ 3.16 pp. | 17.56% |
| Appendix | 13 | ≈ 0.05 pp. | 0.28% |
| A. Extended Related Work | 13 | ≈ 1.99 pp. | 11.06% |
| B. Additional Notes on Illustrative Experimental Setup and Prompts | 15 | ≈ 3.88 pp. | 21.56% |
All detected figure and table locations
Attached appendix
Argument and organization
- 1 Introduction; 2 Current-System Problems; 3 Related Work
Frames review scale and deliberation quality as institutional pressures requiring a research response.
- 4 AI-Augmented Ecosystem; 5 Richer Data
Separates proposed author, reviewer and chair roles and specifies the missing process-data infrastructure.
- 6 Illustrative Experiments; 7 Alternative Views; 8-9 Conclusion and Limits
Uses exploratory tasks to motivate an agenda while retaining competing institutional reforms and validation needs.
Problem, claim and contribution
The abstract states a research/infrastructure priority and repeatedly distinguishes augmenting authors, reviewers and chairs from replacing final human judgment.
Abstract and Introduction, p.1 · Main paper
Argument and section order
The central proposal is organized by stakeholder roles and then by data requirements, rather than presenting a single monolithic reviewing agent.
Sections 4-5, pp.3-7 · Main paper
Figures, captions and visual explanation
A violin plot makes within-paper rating dispersion visible; the exploratory tables separately report component recall and rating errors with metric direction and interval definitions.
Figure 1, p.2; Tables 1-2, p.8 · Main paper
Tables, comparisons and evidence
The experiment discussion labels results illustrative and reports poorer weakness extraction and diminishing returns from extra in-context examples, using these as research questions rather than deployment readiness.
Section 6, p.7; Tables 1-2, p.8 · Main paper
Appendix and reproducibility
Extended related work is separated from an example reviewer report card and prompt/setup notes; the setup appendix explicitly calls for more rigorous implementation and evaluation.
Appendices A-B, pp.13-15 · Main paper
Clarity and writing conventions
Alternative Views treats non-evaluative assistance, publish-first reform, stronger automation and human-only approaches as distinct institutional choices before restating the authors’ preferred path.
Section 7, p.8 · Main paper
What to reuse—and what not to infer
Transferable lesson: For an infrastructure position, identify who acts, what information is missing, and which parts of the proposal are demonstrated versus prospective.
Boundary: Experimental status is not fully clear: the appendix calls setup conceptual while reporting a pilot, and the impact statement says no human-subject study. Do not treat these illustrations or proposed safeguards as validated deployment evidence.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-10-07
- Reading status
- Writing-focused manuscript scan + selected visuals inspected
- Main text
- Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 5, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.
- Appendix / supplement
- Appendix organization and main-text pointers were checked; focused text inspection of physical pages 13, 14, 15. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.
- Visually inspected PDF pages
- 2, 8
This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.