Writing study: Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Łukasz Borchmann, Jordy Van Landeghem, Michał Turski, Shreyansh Padarha, Ryan Othniel Kearns, Adam Mahdi, Niels Rogge, Clémentine Fourrier, Siwei Han, Huaxiu Yao, Artemis Llabrés, Yiming Xu, Dimosthenis Karatzas, Hao Zhang, Anupam Datta · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

64 physical pages

Before references / appendix

≈ 9.22 page-equivalents

Main figures / tables

8 / 3

Attached appendix figures / tables

17 / 22

Attached appendix span

≈ 50.88 page-equivalents

Before references / appendix: ≈ 14.41%References / intervening material: ≈ 6.09%Attached appendix: ≈ 79.5%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.99 pp.	3.11%
2. Dataset Construction and Validity	3	≈ 1.94 pp.	3.03%
3. Evaluation Protocol	5	≈ 0.98 pp.	1.53%
4. Baseline Approaches	6	≈ 0.57 pp.	0.89%
5. Results and Analysis	6	≈ 2.57 pp.	4.02%
6. Conclusion	9	≈ 0.28 pp.	0.44%
7. Limitations	9	≈ 0.35 pp.	0.55%
Impact Statement	10	≈ 0.18 pp.	0.28%
References	10	≈ 3.89 pp.	6.08%
Appendix	14	≈ 1.97 pp.	3.08%
A. Benchmark Specification	16	≈ 4 pp.	6.25%
B. Document Corpus	20	≈ 4 pp.	6.25%
C. Annotations and Human Baseline	24	≈ 5 pp.	7.81%
D. Principled Splits Creation	29	≈ 1 pp.	1.56%
E. Construct Validity Analyses	30	≈ 7 pp.	10.94%
F. Formal Definition of Metrics	37	≈ 7 pp.	10.94%
G. Baseline Implementation Details	44	≈ 7 pp.	10.94%
H. Extended Results	51	≈ 10 pp.	15.63%
I. Solution Space	61	≈ 1 pp.	1.56%
J. Landscape of Agentic Approaches to Documents	62	≈ 2.91 pp.	4.55%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Table 1 · p. 2
• Figure 2 · p. 3
• Table 2 · p. 4
• Figure 3 · p. 4
• Figure 4 · p. 5
• Figure 5 · p. 6
• Table 3 · p. 7
• Figure 6 · p. 8
• Figure 7 · p. 8
• Figure 8 · p. 9

Attached appendix

• Table 4 · p. 16
• Table 5 · p. 18
• Table 6 · p. 20
• Figure 9 · p. 22
• Figure 10 · p. 25
• Table 7 · p. 26
• Table 8 · p. 29
• Figure 11 · p. 29
• Table 9 · p. 30
• Table 10 · p. 30
• Table 11 · p. 33
• Table 12 · p. 34
• Table 13 · p. 39
• Table 14 · p. 39
• Table 15 · p. 41
• Table 16 · p. 42
• Table 17 · p. 42
• Table 18 · p. 42
• Table 19 · p. 43
• Table 20 · p. 46
• Table 21 · p. 51
• Table 22 · p. 51
• Figure 12 · p. 51
• Table 23 · p. 52
• Figure 13 · p. 53
• Table 24 · p. 53
• Figure 14 · p. 54
• Figure 15 · p. 55
• Figure 16 · p. 56
• Figure 17 · p. 56
• Figure 18 · p. 57
• Figure 19 · p. 58
• Figure 20 · p. 58
• Figure 21 · p. 59
• Figure 22 · p. 59
• Figure 23 · p. 60
• Figure 24 · p. 60
• Table 25 · p. 61
• Figure 25 · p. 62

Argument and organization

1. Problem and contribution

   Turns document QA into a grounded search-process evaluation: extractive answers, minimal evidence, multi-hop demands and visual requirements are explicit design properties.

2. Argument sequence

   Construction is followed by construct-validity checks before scoring or model results; later subsections distinguish search dynamics from human-agent comparison.

3. Evidence and verification

   Similar aggregate human/model accuracy is unpacked with item agreement and error categories, while step-budget curves reveal different early-search efficiency.

Problem, claim and contribution

Turns document QA into a grounded search-process evaluation: extractive answers, minimal evidence, multi-hop demands and visual requirements are explicit design properties.

Sections 1.2-2, pp. 2-4 · Main paper

Argument and section order

Construction is followed by construct-validity checks before scoring or model results; later subsections distinguish search dynamics from human-agent comparison.

Sections 2.3-5.2, pp. 4-8 · Main paper

Figures, captions and visual explanation

The dataset table separates documents, pages and questions. An intersection plot represents overlapping visual requirements; the cost plot uses color for model and shape for agent/RAG/RLM scaffold.

Table 2 and Figure 3, p. 4; Figures 6-7, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

Similar aggregate human/model accuracy is unpacked with item agreement and error categories, while step-budget curves reveal different early-search efficiency.

Section 5.2 and Figure 7, p. 8 · Main paper

Appendix and reproducibility

Annotation instructions define minimal evidence with concrete good/bad examples; the human baseline uses the same retrieval interface. Related-resource ratings receive individual explanations.

Appendix C.1-C.2, pp. 24-25; J.1, pp. 62-64 · Main paper

Clarity and writing conventions

Finding-led subheadings turn an extensive evaluation into specific comparisons, but measured associations between reformulation and success should remain distinct from causal explanation.

Section 5.1, p. 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Validate a benchmark's intended construct before ranking systems, then use process and item-level analyses to explain equal average scores.

Boundary: The appendix compares ordinary and oracle human access separately; preserve retrieval access, question subsets and denominators in any human-agent claim.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-7, pp. 1-9: design properties, dataset validity, metric/protocol definitions, baseline comparison, error taxonomy and human-agent analysis.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendix contents pp. 14-15 routes to specification, corpus, annotation, splits, validity, metrics, implementations, extended results and comparison rubric through p. 64; selected detailed annotation and comparison passages inspected.

Visually inspected PDF pages

4, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 64 pages