Writing study: Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations

NeurIPS overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← NeurIPS 2026 oral writing corpus

Junjue Wang, Weihao Xuan, Heli Qi, Pengyu Dai, Kunyi Liu, Hongruixuan Chen, Zhuo Zheng, Junshi Xia, Stefano Ermon, Naoto YOKOYA · Official NeurIPS 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

13 physical pages

Before references / appendix

≈ 9.8 page-equivalents

Main figures / tables

9 / 6

Attached appendix figures / tables

0 / 0

Attached appendix span

No boundary detected; not assumed zero

Before references / appendix: ≈ 75.38%References / intervening material: ≈ 24.62%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.

Source-bound correction: Source-checked 2026-10-02: recover all nine literal Figure. N. captions, including side-by-side Figures 3 and 4 sharing an XML line. Headings and reference boundary are correct; this 13-page author version has no appendix or checklist.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.69 pp.	13%
2. Related Work	3	≈ 0.89 pp.	6.85%
3. DORA Dataset	4	≈ 3.1 pp.	23.85%
4. Experiments	7	≈ 3.27 pp.	25.15%
5. Conclusion	10	≈ 0.13 pp.	1%
Acknowledgments	10	≈ 0.1 pp.	0.77%
References	10	≈ 3.2 pp.	24.62%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 3
• Figure 2 · p. 4
• Figure 3 · p. 5
• Figure 4 · p. 5
• Figure 5 · p. 5
• Table 2 · p. 6
• Figure 6 · p. 7
• Table 3 · p. 8
• Table 4 · p. 8
• Table 5 · p. 9
• Figure 7 · p. 9
• Figure 8 · p. 9
• Table 6 · p. 10
• Figure 9 · p. 10

Attached appendix

No recognized caption labels.

Argument and organization

1. Problem and contribution

   DORA shifts the claimed evaluation object from isolated remote-sensing perception to a multi-step disaster-observation agent workflow. It defines questions, data, reference tool trajectories and final answers, distinguishing the evaluator's reference trajectory from information available to the agent at inference.

2. Argument sequence

   The benchmark description progresses through disaster/data coverage, task formulation, tool inventory and annotation before defining scoring operators. Results then separate answer quality, trajectory matching, parameter correctness and execution efficiency rather than turning them into one undifferentiated agent score.

3. Evidence and verification

   The evaluation makes scalar, point and polygon matching thresholds explicit, including relative-error, pixel-distance and IoU criteria. Trajectory metrics distinguish any-order, in-order and exact matches. The gold-trajectory row does not imply perfect final answers: Table 4 reports 80.48 for that final-answer aggregate while trajectory matches are 100, so these outputs must remain distinct.

Problem, claim and contribution

DORA shifts the claimed evaluation object from isolated remote-sensing perception to a multi-step disaster-observation agent workflow. It defines questions, data, reference tool trajectories and final answers, distinguishing the evaluator's reference trajectory from information available to the agent at inference.

Introduction pp. 1–3; task formulation and data construction, §3, pp. 4–6 · Main paper

Argument and section order

The benchmark description progresses through disaster/data coverage, task formulation, tool inventory and annotation before defining scoring operators. Results then separate answer quality, trajectory matching, parameter correctness and execution efficiency rather than turning them into one undifferentiated agent score.

§3 and Tables 1–2, pp. 4–7; evaluation §4 and Tables 3–4, pp. 7–9 · Main paper

Figures, captions and visual explanation

Figure 2 combines geographic/event coverage with sensor characteristics and task-count charts. On p. 8, Table 3 puts operator formulas and tolerances beside their task types, while Table 4 groups final-answer metrics separately from trajectory, parameter and efficiency metrics. The visual hierarchy exposes different failure locations in an end-to-end chain.

Figure 2, p. 4; Tables 3–4, p. 8; rendered pages 4 and 8 · Main paper

Tables, comparisons and evidence

The evaluation makes scalar, point and polygon matching thresholds explicit, including relative-error, pixel-distance and IoU criteria. Trajectory metrics distinguish any-order, in-order and exact matches. The gold-trajectory row does not imply perfect final answers: Table 4 reports 80.48 for that final-answer aggregate while trajectory matches are 100, so these outputs must remain distinct.

Scoring protocol and Tables 3–4, p. 8; tool-selection, parameter and hint analyses pp. 8–9 · Main paper

Appendix and reproducibility

There is no appendix to supply missing assumptions in this version. Tool definitions, source data, annotation and metric details therefore need their main-text references retained; the reader should not be sent to an invented supplement for those evaluation conditions.

Task/tool inventory and annotation pp. 4–7; references pp. 10–13 · Main paper

Clarity and writing conventions

The most reusable pattern is to name the level of each claim: source coverage, tool use, argument correctness, final answer or efficiency. The paper's broad disaster-agent motivation should be narrowed to its offline benchmark evidence when writing conclusions.

Task formulation pp. 4–6; result analysis pp. 8–9; conclusion p. 10 · Main paper

What to reuse—and what not to infer

Transferable lesson: When a system executes a chain of tools, define the chain's intermediate contracts and score them separately; pair the result table with the exact operators and tolerances that produced it.

Boundary: Offline disaster-task performance is not evidence of real-emergency readiness. Tool-free, agent and reference-trajectory settings expose different information and should not be described as identical comparison conditions.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-02

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan across manuscript framing, section sequence, figure/table captions, selected argument/evaluation prose and conclusion; not exhaustive technical or proof verification. Sections 1–5, pp. 1–10; references pp. 10–13. Rendered the dataset/task inventory p. 4 and scoring operators plus multi-level results p. 8. Source is the exact public author version listed below, not a verified accepted/camera-ready revision.

Appendix / supplement

Writing-oriented appendix/documentation scan: No separate appendix is present in this 13-page public author manuscript; task construction, annotation and scoring descriptions occur in the main text. Source code, data, model outputs and external video assets were not executed or independently validated.

Visually inspected PDF pages

4, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Public author arXiv manuscript 2605.11633v1; not verified identical to the accepted or camera-ready OpenReview revision · 13 pages