Writing study: VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
← AAAI 2026 oral writing corpus
Shi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li, Xiaobin Zhu, Xu-Cheng Yin · Official AAAI 2026 oral record
Figure, table and section ledger
Editorial contribution type: Datasets, benchmarks & measurement (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 9 physical pages
- Before references / appendix
- ≈ 7.09 page-equivalents
- Main figures / tables
- 7 / 1
- Attached appendix figures / tables
- 0 / 0
- Attached appendix span
- No boundary detected; not assumed zero
Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.
Source-bound correction: Checked 2026-10-25. Unnumbered main headings are bound to the actual source columns. A physical p.2 raster confirms that Fig. 1. In summary begins a prose line, not a second figure caption; exclude that candidate. The genuine Figure 1 is on p.1. Remaining unique captions are Figures 1–7 and Table 1. Diagram category labels on p.3 are not sections.
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
All detected figure and table locations
Main text
Attached appendix
No recognized caption labels.
Argument and organization
- Introduction; Related Work
Explains why image-caption and broad video-understanding metrics miss fine-grained temporal, spatial and cinematographic attributes needed for detailed captions.
- The VCapsBench Benchmark
Introduces four major groups and 21 dimensions, then sources, model-assisted QA construction, human correction and operational definitions of accuracy, inconsistency and coverage.
- Experiment; Evaluation Analysis; Conclusion
Specifies captioning and TextQA roles, presents dimension-level results, then discusses annotation-model bias and the relationship between caption length, coverage and inconsistency.
Problem, claim and contribution
The benchmark is motivated by a measurement gap: captions can omit camera movement or temporal detail even when broad semantic agreement looks adequate. The abstract gives three distinct scale quantities, 5,677 videos, 109,796 QA pairs and 21 evaluation dimensions, instead of treating question count as the number of independent videos.
Abstract and Introduction, physical pp.1-2; Dataset Statistics, p.3 · Main paper
Argument and section order
The paper explains what is measured before ranking models: a taxonomy example precedes source distributions and two annotation pipelines, followed by ternary answer outcomes and explicit metric denominators. This sequence lets readers distinguish missing content from contradictory content before reaching the results table.
Figures 2-4, pp.3-4; Captioning Evaluation and Equations 1-3, pp.4-5 · Main paper
Figures, captions and visual explanation
Figure 4 uses separate colored routes for caption-mediated QA generation and direct video-to-QA generation, then joins them before filtering and human correction. Table 1 groups attributes and repeats metric-direction arrows. Figure 7 is a cautionary contrast: word counts and percentage rates share one axis labeled Values, making a scale comparison visually ambiguous.
Figure 4, physical p.4; Table 1, p.6; Figure 7, p.7; rendered pages inspected · Main paper
Tables, comparisons and evidence
The evaluation protocol distinguishes the caption-producing VLM from the TextQA judge, and Table 1 names GPT-4.1 as its judge. Accuracy and coverage use all questions as denominator, whereas inconsistency uses questions answered positively or negatively. The analysis acknowledges that Gemini-generated captions contributed to QA creation and may inflate Gemini coverage.
Equations 1-3 and Experimental Setup, p.5; Table 1 caption, p.6; Evaluation Analysis, p.7 · Main paper
Appendix and reproducibility
The main manuscript contains the annotation pipeline and metric formulas rather than delegating their meaning to a supplement, and provides a first-page repository link. No attached appendix was inspected. For a reusable benchmark paper, exact prompts, model revisions, reviewer instructions and agreement measurements should be separately locatable; their availability is not established by the link alone.
Code link, p.1; Annotation Details, p.4; evaluation protocol, p.5; references, pp.8-9 · Main paper
Clarity and writing conventions
Positive, negative and unanswerable outcomes give the metrics a useful verbal interpretation before the equations. Keep that precision when describing rankings: the table's highest accuracy and coverage do not imply the lowest inconsistency, and caption-evaluation scores should not be described as direct evidence of generated-video quality.
Captioning Evaluation, pp.4-5; Table 1, p.6; Evaluation Analysis and Conclusion, p.7 · Main paper
What to reuse—and what not to infer
Transferable lesson: For a benchmark contribution, explain the unit of analysis, annotation decisions and metric denominators before the leaderboard. Show both the data-production pipeline and a concrete annotated example, and disclose overlap between annotation models and evaluated models.
Boundary: The selected model comparisons are judge- and protocol-dependent, and the main paper acknowledges annotation-model bias. The three rates are not independent, higher coverage is not automatically better factuality, and a mixed word-count/percentage plot should not be copied as a formatting standard. No independent dataset-quality validation was performed here.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-10-25
- Reading status
- Writing-focused manuscript scan + selected visuals inspected
- Main text
- Writing-focused text scan of physical PDF pp.1-7 covering the benchmark motivation, taxonomy, acquisition and annotation pipeline, metric definitions, evaluation protocol, results and acknowledged bias. Physical pp.4, 6 and 7 were rendered and visually inspected; the acknowledgments/reference boundary on p.8 was checked. Dataset records and evaluations were not independently reproduced.
- Appendix / supplement
- The inspected nine-page proceedings PDF contains no attached appendix; its final two pages are acknowledgments and references. No main-text appendix pointer was identified. The first-page code link was recorded but repository contents, annotation instructions and any separate supporting files were not inspected.
- Visually inspected PDF pages
- 4, 6, 7
This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.