Writing study: Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Chufan Shi, Cheng Yang, Yaokang Wu, Linghao Jin, Bo Shui, Taylor Berg-Kirkpatrick, Xuezhe Ma · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

27 physical pages

Before references / appendix

≈ 9.16 page-equivalents

Main figures / tables

4 / 9

Attached appendix figures / tables

8 / 9

Attached appendix span

≈ 14.91 page-equivalents

Before references / appendix: ≈ 33.93%References / intervening material: ≈ 10.85%Attached appendix: ≈ 55.22%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Source-checked physical-page text restores wrapped top-level heading labels on pp. 2, 3. Heading starts and measured positions are unchanged; this is label completeness, not a new section. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.49 pp.	5.52%
2. VISUALSWAP: A Diagnostic Framework for Visual Re-examination	2	≈ 0.83 pp.	3.07%
3. VS-BENCH: A Benchmark for Visual Re-examination	3	≈ 1.03 pp.	3.81%
4. Experiments	4	≈ 0.59 pp.	2.19%
5. Discussion	5	≈ 4.05 pp.	15%
6. Related Work	9	≈ 0.4 pp.	1.48%
7. Conclusion	9	≈ 0.14 pp.	0.52%
Acknowledgement	9	≈ 0.14 pp.	0.52%
Impact Statement	10	≈ 0.11 pp.	0.41%
References	10	≈ 2.93 pp.	10.85%
A. Case Study	13	≈ 1 pp.	3.7%
B. Models	14	≈ 0.31 pp.	1.15%
C. Experimental Setup	14	≈ 0.23 pp.	0.85%
D. Per-Task Baseline Results	14	≈ 1.11 pp.	4.11%
E. Reflection Prompt Variants	15	≈ 0.08 pp.	0.3%
F. Implementation Protocol for Image Swapping	15	≈ 1.44 pp.	5.33%
G. Human Baseline for Image Pair Distinguishability	17	≈ 0.11 pp.	0.41%
H. Decomposition of User Instruction Effects	17	≈ 0.5 pp.	1.85%
I. Stratified Analysis by Initial Correctness	17	≈ 0.72 pp.	2.67%
J. Control Experiment with Visually Distinct Images	18	≈ 0.82 pp.	3.04%
K. Evaluation on Closed-Source Models	19	≈ 8.6 pp.	31.85%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Table 1 · p. 4
• Table 2 · p. 6
• Figure 2 · p. 6
• Figure 3 · p. 7
• Table 3 · p. 7
• Table 4 · p. 7
• Table 5 · p. 8
• Table 6 · p. 8
• Figure 4 · p. 8
• Table 7 · p. 8
• Table 8 · p. 9
• Table 9 · p. 9

Attached appendix

• Table 10 · p. 14
• Table 11 · p. 15
• Table 12 · p. 15
• Table 13 · p. 16
• Table 14 · p. 16
• Table 15 · p. 17
• Table 16 · p. 18
• Table 17 · p. 19
• Table 18 · p. 19
• Figure 5 · p. 20
• Figure 6 · p. 21
• Figure 7 · p. 22
• Figure 8 · p. 23
• Figure 9 · p. 24
• Figure 10 · p. 25
• Figure 11 · p. 26
• Figure 12 · p. 27

Argument and organization

1. Problem and contribution

   Turns a claim of re-examining an image into a falsifiable behavior test by changing answer-critical visual content while preserving prior reasoning.

2. Argument sequence

   Protocol and metric definitions precede curation/quality checks; discussion then treats alternative-image difficulty, user turns, attention, context and natural triggers as separate questions.

3. Evidence and verification

   Independent inference on each image addresses difficulty; an implementation appendix explicitly states re-prefill and freshly encoded visual tokens rather than implying a hidden mid-cache replacement.

Problem, claim and contribution

Turns a claim of re-examining an image into a falsifiable behavior test by changing answer-critical visual content while preserving prior reasoning.

Introduction and VISUALSWAP overview, p. 2 · Main paper

Argument and section order

Protocol and metric definitions precede curation/quality checks; discussion then treats alternative-image difficulty, user turns, attention, context and natural triggers as separate questions.

Sections 2-5, pp. 2-9 · Main paper

Figures, captions and visual explanation

The curation page reports complementary similarity metrics with direction arrows; the diagnostic page aligns before/after attention, prompt variability, trigger frequency and context-length effects.

Table 1, p. 4; Tables 5-7 and Figure 4, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

Independent inference on each image addresses difficulty; an implementation appendix explicitly states re-prefill and freshly encoded visual tokens rather than implying a hidden mid-cache replacement.

Section 5.1, p. 5; Appendix F, pp. 15-16 · Main paper

Appendix and reproducibility

The supplement records exact reflection prompts and turn boundaries, separates human distinguishability and initial-correctness controls, and includes successful as well as failed visual re-grounding.

Appendices E-I, pp. 15-18; Figure 12, p. 27 · Main paper

Clarity and writing conventions

Question-led controls aid navigation, but the p. 8 prose says standard deviations are at most 1.1 while its Table 6 visibly reports 3.1–4.9; this should not be repeated as a verified bound.

Table 6 and Section 5.5 continuation, p. 8; PNG inspected · Main paper

What to reuse—and what not to infer

Transferable lesson: Document intervention mechanics and plausible confounds, then cross-check every prose summary against its neighboring numerical table.

Boundary: The demonstrated protocol-specific failure does not prove a universal structural impossibility; the prompt-variability numerical inconsistency remains unresolved in this source.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-7, pp. 1-9: image-swap protocol, dataset construction, model results, confound controls, attention/prompt/context analyses and intervention.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-K and extended examples, pp. 13-27: cases, models/setup, detailed tables, prompt variants, token/turn implementation, human and visually distinct controls, closed-model extension.

Visually inspected PDF pages

4, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 27 pages