Skip to content
Computational Pathology Reading Roadmap

Vision–language evaluation, data, and safety

Vision–language evaluation must determine which modality and evidence produced an answer. A high aggregate score can combine perception, OCR, retrieval, language prior, format compliance, and evaluator preference without revealing any of them. This chapter treats datasets and evaluation protocols as scientific instruments.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

ECCV 2026 conference additions and metadata checked through 2026-09-12; the broader literature audit boundary above is unchanged.

Prerequisite path

Section titled “Prerequisite path”
  1. Complete splitting, uncertainty, calibration, subgroup analysis, and reproducibility in ML foundations.
  2. Read the task contracts for caption/VQA, grounding, MLLMs, and generation/video/agents.
  3. Be able to distinguish validation-based prompt/model selection from a sealed final test and to document pretraining-data overlap.

Output, supervision, and evidence taxonomy

Section titled “Output, supervision, and evidence taxonomy”
Evaluation target Output scored Reference/supervision Evidence question
Retrieval/alignment ranked media/text items paired positives, relevance judgments does similarity recover semantically relevant pairs, including one-to-many matches?
Caption/generation free-form text or media references, attributes, human preferences are required facts present and unsupported claims absent?
VQA/reasoning short/free answer or rationale answers, programs, human judgments was the relevant visual evidence necessary?
Grounding point, box, mask, track spatial annotations is the referent correctly localized, and can the system reject absence?
MLLM capability heterogeneous responses benchmark answer keys/judges which sub-capability and modality caused success/failure?
Safety response, refusal, action harm taxonomy, policy, human adjudication is the system safe without excessive refusal of benign inputs?
Agent behavior state/action trajectory environment success and safety constraints were perception, plan, actions, recovery, and cost acceptable?

Evaluation evidence has levels:

  1. Internal: random held-out samples from the same source.
  2. Compositional/interventional: controlled image/text changes and null inputs.
  3. External: new source, culture, language, device, environment, or time.
  4. Human: blinded, rubric-defined review with agreement and expertise reported.
  5. Operational: versioned environment or prospective workflow with failures logged.

Method detail: dataset and benchmark record

Section titled “Method detail: dataset and benchmark record”

Before using a benchmark, record:

Field Questions
Unit and provenance image, video, page, question, dialogue, trajectory; who created each component?
Annotation human, templated, model-generated, weak web pairing; annotator population and agreement?
Split random, source, entity, time, creator, environment; are near duplicates removed?
Accessibility license, download, version/hash, removed items, personal/sensitive data?
Task contract answerable inputs, acceptable outputs, ambiguity, rejection behavior?
Metrics exact implementation, normalization, judge model/prompt/version, uncertainty?
Contamination was the benchmark public before model cutoff; can images/captions be searched or regenerated?
Selection were prompts, checkpoints, or decoding tuned on the final test?
Coverage languages, geographies, cultures, demographics, object scales, media quality?

Data scale cannot repair an invalid split, unclear license, or circular synthetic teacher. “Public on the web” is not equivalent to consent for model training.

Staged evaluation route

Section titled “Staged evaluation route”

Caption-metric validity

Section titled “Caption-metric validity”
Order Primary paper and year Venue / status Read it for
1 CIDEr (2015) CVPR, peer reviewed human triplet consensus, high-reference test sets, TF–IDF n-gram weighting, and CIDEr-D anti-gaming controls
2 SPICE (2016) ECCV, peer reviewed scene-graph proposition F1, object/relation/attribute decomposition, and system- versus caption-level human correlation

Validate lexical consensus and parsed semantic overlap against their stated human constructs, but separately test pixel grounding, unsupported objects and relations, reference omissions, parser perturbations, and rare high-severity errors.

Task construction and shortcut control

Section titled “Task construction and shortcut control”
Order Primary paper and year Venue / status Read it for
1 CLEVR (2017) CVPR, peer reviewed controlled visual programs that expose compositional failure by operation
2 VQA v2 (2017) CVPR, peer reviewed complementary images and reduced question-conditioned answer priors
3 VizWiz (2018) CVPR, peer reviewed authentic distribution shift, image-quality failures, and unanswerability
4 CHAIR (2018) EMNLP, peer reviewed object hallucination in image captioning
5 GQA (2019) CVPR, peer reviewed structured compositional questions, consistency, and scene-graph biases
6 Winoground (2022) CVPR, peer reviewed minimal image/caption pairs that test word order and composition
7 ARO (2023) ICLR, peer reviewed attribution, relation, and order sensitivity versus bag-of-words behavior
8 SugarCrepe (2023) NeurIPS Datasets and Benchmarks, peer reviewed hard negatives refined to remove fluency and plausibility shortcuts
9 POPE (2023) EMNLP, peer reviewed polling-based object-hallucination evaluation for large VLMs

Training-data scale and curation audits

Section titled “Training-data scale and curation audits”
Order Primary paper and year Venue / status Read it for
1 LAION-5B (2022) NeurIPS Datasets and Benchmarks, peer reviewed access, filtering, language, provenance, opt-out, and duplication at open web scale
2 MetaCLIP (2024) ICLR, peer reviewed controlled metadata balancing and data-distribution experiments

Documents, charts, and answerability

Section titled “Documents, charts, and answerability”
Order Primary paper and year Venue / status Read it for
1 DocVQA (2021) WACV, peer reviewed OCR/layout decomposition on document images
2 ChartQA (2022) Findings of ACL, peer reviewed visual and arithmetic reasoning over charts
3 A-OKVQA (2022) ECCV, peer reviewed world-knowledge questions with rationales and answer-format alternatives
4 ChartQAPro (2025) Findings of ACL, peer reviewed real-world chart diversity, dashboards, conversations, and unanswerable cases
5 POLYCHARTQA (2026) ACL, peer reviewed multilingual chart QA and performance disparities across ten languages

Broad capability suites

Section titled “Broad capability suites”
Order Primary paper and year Venue / status Read it for
1 MMMU (2024) CVPR, peer reviewed expert-level questions across disciplines and image types
2 MM-Vet (2024) ICML, peer reviewed integrated capability combinations with open-ended answers
3 SEED-Bench (2024) CVPR, peer reviewed image/video comprehension across multiple dimensions
4 MathVista (2024) ICLR, peer reviewed visual mathematical reasoning across diagrams and problems
5 MMStar (2024) NeurIPS, peer reviewed multimodal samples filtered for visual dependence and leakage
6 MMBench (2024) ECCV, peer reviewed capability taxonomy, circular option remapping, and reproducible MLLM comparison
7 BLINK (2024) ECCV, peer reviewed perception tasks that remain difficult even when language mediation is easy
8 Video-MME (2025) CVPR, peer reviewed short-to-long video, subtitles, duration, and broad task coverage
9 UniM (2026) CVPR, peer reviewed interleaved inputs/outputs across seven modalities

An aggregate leaderboard should be accompanied by per-capability results, confidence intervals, failure examples, cost, model/data disclosure, and an explicit statement of what the benchmark does not test.

Hallucination, grounding, and evaluator quality

Section titled “Hallucination, grounding, and evaluator quality”
Order Primary paper and year Venue / status Read it for
1 HallusionBench (2024) CVPR, peer reviewed separating visual illusion, knowledge, and language hallucination
2 VL-RewardBench (2025) CVPR, peer reviewed whether multimodal reward models judge hallucination and reasoning correctly
3 VidHalluc (2025) CVPR, peer reviewed action, temporal-order, and scene-transition hallucination
4 PhD (2025) CVPR, peer reviewed object, attribute, sentiment, position, and counting hallucinations
5 GroundingME (2026) CVPR, peer reviewed difficult discrimination, spatial relations, limited evidence, and rejection
6 RealUnify (2026) CVPR, peer reviewed whether unified understanding and generation produce synergy

Generative-model alignment and safety

Section titled “Generative-model alignment and safety”
Order Primary paper and year Venue / status Read it for
1 GenEval (2023) NeurIPS Datasets and Benchmarks, peer reviewed object presence, count, color, position, and attribute binding in generated images
2 T2ISafety (2025) CVPR, peer reviewed fairness, toxicity, and privacy as distinct text-to-image safety axes

Multimodal and agent safety

Section titled “Multimodal and agent safety”
Order Primary paper and year Venue / status Read it for
1 MM-SafetyBench (2024) ECCV, peer reviewed image-based manipulations that bypass text-aligned safeguards
2 RiOSWorld (2025) NeurIPS, peer reviewed risky computer tasks split by user and environment origin and by intent/completion
3 LLaVAShield (2026) CVPR, peer reviewed accumulated, concealed, and cross-modal risk in multi-turn dialogue
4 AGENTSAFE (2026) CVPR, peer reviewed hazardous instructions across perception, planning, and execution
5 SecureWebArena (2026) Findings of ACL, peer reviewed six adversarial web environments and failure decomposition across agent stages

Safety evaluation needs both attack success and over-refusal on benign controls. Policy wording, judge identity, refusal parsing, and multi-turn state must be versioned.

Claim-to-metric map

Section titled “Claim-to-metric map”
Claim Primary measurements What they miss alone
aligned retrieval Recall@K, mAP, nDCG fine-grained grounding, calibration, one-to-many relevance
zero-shot recognition per-class accuracy/AUROC, calibration prompt search, pretraining overlap, spatial evidence
factual caption human/entity/relation correctness, CHAIR-like hallucination completeness and style unless separately scored
correct VQA soft accuracy/exact match image necessity, confidence, answerability
grounded answer answer score plus point/box/mask metric causal use unless evidence is intervened on
robust MLLM capability-specific external/interventional suite deployment workflow and safety
safe dialogue harmfulness, policy compliance, benign helpfulness long-horizon agent actions
safe agent task success, hazard rate, intervention, recovery untested environments and rare irreversible failures

Use paired bootstrap confidence intervals or an appropriate hierarchical analysis when multiple questions share images/videos. Report the unit of resampling.

Controlled experiment and evaluation protocol

Section titled “Controlled experiment and evaluation protocol”

Build an audit panel from one task with eight conditions:

  1. original image and question;
  2. relevant counterfactual image;
  3. random image;
  4. blank image;
  5. image with the relevant region masked;
  6. paraphrased question;
  7. false-premise/unanswerable question;
  8. cross-source or culturally shifted example.

Freeze model/checkpoint, prompt, decoding, evaluator prompt/version, and normalization. Collect answer, confidence or token likelihood when accessible, spatial evidence, latency, tokens, and refusal. Obtain blinded human review for a stratified subset and measure judge agreement with humans before scaling automated judging.

For an agent, replay fixed initial states and log every observation, tool input/output, action, environment transition, safety intervention, and recovery.

Failure modes

Section titled “Failure modes”
  • Random example splits leak images, entities, creators, or near duplicates.
  • Public test data influences pretraining, instruction data, prompts, or synthetic derivatives.
  • A benchmark’s answer distribution supports a strong language-only baseline.
  • A model selects an answer through elimination without correct visual reasoning.
  • Exact match penalizes valid paraphrases; a language-model judge rewards eloquence.
  • The same model family generates training data and judges outputs.
  • Prompt/checkpoint selection occurs on the reported test set.
  • Missing confidence intervals exaggerate small leaderboard differences.
  • Image perturbations change difficulty or visual quality beyond the target concept.
  • Safety improves by refusing harmless requests.
  • Agent success ignores invalid intermediate actions, retries, or human rescue.
  • Dataset licenses and consent are assumed rather than verified.

Self-check and exit task

Section titled “Self-check and exit task”
  1. What is the independent sampling unit for a dataset with many questions per image?
  2. Which controls show that a task genuinely requires the image?
  3. How do factuality and faithfulness differ for retrieved or generated evidence?
  4. When is model-as-judge acceptable, and what validation must precede it?
  5. How would you detect prompt and checkpoint selection on a test benchmark?
  6. Why must safety report both harmfulness and benign helpfulness?
  7. Which agent failures disappear inside final task success?

Exit task: write a preregistered evaluation card for one model and claim. Include data provenance, split unit, overlap audit, interventions, metrics, uncertainty, human protocol, cost, failure taxonomy, and promotion criterion.

Bridge to computational pathology

Section titled “Bridge to computational pathology”

Clinical validity needs patient-level splitting, institution/scanner/stain shift, clinically meaningful endpoints, calibration, subgroup analysis, expert review, and clear intended use. Public slides and reports can overlap across foundation-model pretraining and downstream benchmarks even when file names differ. Natural-image caption/VQA scores do not validate report fidelity or diagnostic safety. Continue to computational-pathology evaluation and translation and apply the WSI-specific audit before making clinical claims.

Counterexamples for spatial, semantic and action grounding

Use matched distance counterexamples, cross-category part correspondence, action-absent video clips and background-consistent concealment to test different kinds of visual evidence. Preserve each study's denominator: answer probability, correspondence success, absence/presence accuracy and newly introduced ungrounded nouns are not interchangeable.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Why Far Looks Up: Probing Spatial Representation in Vision-Language ModelsCheolhong Min, Jaeyun Jung, Daeun Lee, Hyeonseong Jeon, Yu Su, Jonathan Tremblay, Chan Hee Song, and Jaesik Park2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #10791 checked on 2026-09-12. Publisher version of record was not independently verified. The project page describes an oral; the catalog follows the official Spotlight record.Why Far Looks Up diagnoses vertical-distance entanglement: a VLM may treat higher image position as evidence of greater depth. Controlled behavioral tests are paired with hidden-state contrastive probes, separating a correct answer from the representation that supports it. This is a diagnostic study and benchmark, not a training algorithm that guarantees three-dimensional reasoning.ECCV Spotlight program · Conference manuscript · Official project or implementation
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation ModelsOlaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #349 checked on 2026-09-12. Publisher version of record was not independently verified.SOCO benchmarks whether a representation matches the correct object part, not merely the correct kind of part. Its taxonomy separates concept correspondence, identity-specific semantic object correspondence and cross-category transfer. Language descriptions also let VLMs be tested on text-guided localization versus visual-reference matching.ECCV Spotlight program · Conference manuscript · Official project or implementation
Learning to Deny: Action Denial in Multimodal Large Language ModelsRaiyaan Abdullah, Shehreen Azad, and Yogesh Singh Rawat2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #3692 checked on 2026-09-12. Publisher version of record was not independently verified.Learning to Deny introduces UCF101-AD to test whether video-language models reject an action whose defining motion never occurs despite plausible context. Its CausalAct framework organizes persons, objects, relations, interactions and motion into an explicit verification order. The paper combines a denial benchmark with a structured-prompt and fine-tuning baseline.ECCV Spotlight program · Conference manuscript · Official project or implementation
Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language ModelsAmira Guesmi and Muhammad Shafique2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #12081 checked on 2026-09-12. Publisher version of record was not independently verified.Background-Consistent Re-encoding (BCR) studies how a vision-language model can stop naming a selected object without filling its missing representation with an invented replacement. It perturbs the input image so object-region features resemble surrounding background features while retaining the encoder's ordinary token layout. The scientific claim concerns continuity of visual evidence, not literal elimination of every hallucination.ECCV Spotlight program · Conference manuscript

Reading question: Does an apparent improvement survive counterexamples that preserve superficial cues while changing the correct spatial, semantic or temporal conclusion?

Evidence boundary: A small bias gap can coexist with near-chance performance. Action denial is not safety refusal, detector-based concealment is not a privacy guarantee, and benchmark correlations are not universal causal explanations.

Reasoning budgets, executable outputs and scientific edits

Test-time scaling studies how extra reasoning changes perception and mathematics; EmbedCopilot checks whether plausible embedded-system answers actually execute; GRADE requires domain-correct and readable image edits. Together they motivate measuring compute, implementation success and constraint satisfaction rather than judging prose alone.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
On Test-Time Scaling for Vision-Language ModelsFawaz Sammani, Tzoulio Chamiti, and Nikos Deligiannis2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #4968 checked on 2026-09-12. Publisher version of record was not independently verified.This empirical study tests when extra inference computation helps vision-language models. Nine prompting and sampling strategies are compared across thirteen instruction-tuned models. The main distinction is task-dependent: more reasoning can help mathematical or logical questions while harming perception-focused judgments. It is not a new trained reasoning model.ECCV Spotlight program · Conference manuscript
EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System DevelopmentDongsheng Yuan, Yimo Deng, and Huangxun Chen2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #9127 checked on 2026-09-12. Publisher version of record was not independently verified. The PDF uses EmbedCopilot-Bench; the program shortens this to EmbedCopilot.EmbedCopilot-Bench evaluates whether a vision-language assistant can turn hardware and software screenshots into useful embedded-development steps and functioning code. It combines rubric-based response quality with execution success, explicitly separating exact pin-grounded operation from behavior that works only after pin correction. Fluent instructions alone are insufficient.ECCV Spotlight program · Conference manuscript · Official project or implementation
GRADE: Benchmarking Discipline-Informed Reasoning in Image EditingMingxin Liu, Ziqian Fan, Zhaokai Wang, Leyao Gu, Zirun Zhu, Yiguo He, Yuchen Yang, Changyao Tian, Xiangyu Zhao, Ning Liao, Shaofeng Zhang, Qibing Ren, Zhihang Zhong, Xuanhe Zhou, Junchi Yan, and Xue Yang2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #2706 checked on 2026-09-12. Publisher version of record was not independently verified.GRADE evaluates whether an image editor can use disciplinary knowledge to make a scientifically correct visual change, preserve relevant context, and keep the result logically readable. Its 520 image–instruction–reference triplets span ten academic disciplines. The central distinction is between drawing a plausible image and rendering the right answer to a domain-specific problem.ECCV Spotlight program · Conference manuscript · Official project or implementation

Reading question: What external test can falsify a persuasive explanation: a perception answer, a running circuit, or a scientifically valid image?

Evidence boundary: More tokens can reduce perceptual accuracy. Judge scores and physical execution use different protocols; GRADE strict accuracy is a conjunction of three maxima, not its mean reasoning score.

Recovering, steering and testing visual concepts

Intermediate features can expose information hidden by the final representation. Compare frozen-backbone negation adaptation, language-directed representations, circuit interventions and geometry-subspace probes; distinguish information accessibility from causal control.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate FeaturesChen-Yi Lu, Yueh-Shao Chen, and Somali Chaterji2026 · ECCV · acceptedECCV 2026 Oral verified in the official program (event 6084); scientific claims use the conference-hosted manuscript #10575. Publisher proceedings/version of record was not verified on 2026-09-12.PeakPatch recovers negation cues from frozen intermediate CLIP text features instead of updating the encoders. A learned embedding residual restores distinctions between affirmative and negated captions; a separate, bounded score correction improves image-caption decisions. The contribution is a diagnosis-plus-adapter design for compositional failure, not a new image encoder or an inference-time language parser.ECCV program · Conference manuscript · Author project

Reading question: Which layer, token, subspace or circuit changes, and does the intervention preserve unrelated capabilities?

Evidence boundary: A probe may measure accessibility rather than semantic understanding; intervention and natural distribution shift are different tests.

Long-video memory, scripts and entity continuity

Retrieval-based episodic memory, audio-visual script generation and entity-centric evaluation answer different long-video questions. Track what is stored, how evidence is retrieved, and whether identities and events remain consistent across time.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative UnderstandingHyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, and Gargi Chakraborty2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official program (event 6177); scientific claims use the conference-hosted manuscript #10972. Publisher proceedings/version of record was not verified on 2026-09-12. Earlier arXiv versions used the subtitle Evaluating Video Language Models Beyond the Frame; this is the same canonical work.NarrativeTrack evaluates whether a video-language model keeps track of the same person through time, changing attributes, and visually confusable characters. Its Compositional Reasoning Progression moves from entity existence to state changes and then identity ambiguity. This makes narrative understanding a set of testable temporal-perceptual operations rather than one broad caption-quality score.ECCV program · Conference manuscript · Author project · Dataset

Reading question: Can the system recover the right episode and preserve who did what, when, and in which scene?

Evidence boundary: Compressed memory can omit decisive events; fluent scripts and pooled benchmark scores can conceal identity or temporal failures.

Distilling visual context and sensitivity to evidence

SCOPD teaches a sparse-context student from a full-context EMA teacher on student rollouts. CW-OPD instead tests how predictions change when answer-critical visual evidence is edited between two worlds.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language ModelsAhmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, and Babak Taati2026 · arXiv · preprintPreprint checked 2026-10-02; project page still labels code and models as forthcoming. Visual-token retention is not an optical-magnification intervention.SCOPD treats visual-token pruning partly as a representation-utilization problem. A full-context teacher supervises responses generated by the sparse student; SCOPD+ concentrates the supervision on visually sensitive positions.Versioned full text · Author project page
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language ModelsYuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, and Xinbing Wang2026 · arXiv · preprintPreprint checked 2026-10-02. The paper calls the method CW-OPD; its linked code repository uses the name CWAD. No conference acceptance inferred.CW-OPD supervises how a VLM prediction changes when answer-critical visual evidence changes. It combines ordinary per-world distillation with a cross-world belief-transition target, rather than assuming a correct answer proves the student used the right evidence.Versioned full text · Author implementation (repository name CWAD)

Reading question: Does the student learn visual evidence, or merely the teacher’s answer distribution?

Evidence boundary: Token pruning and counterfactual image editing are not WSI acquisition changes. Track EMA updates, teacher-only hints, editor quality and training cost before comparing distillation methods.

Density-weighted residual alignment in hybrid VLMs

HEED aligns teacher and hybrid-student decoder residual streams, weighting visual positions by frozen-ViT local self-dissimilarity. This density proxy is not semantic saliency. Distinguish the hybrid architecture’s inference benefit from density weighting’s incremental benefit in a matched training recipe.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model DistillationYihao Liang and Niraj K. Jha2026 · NeurIPS · acceptedOfficial NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested.HEED protects locally distinctive visual positions when converting an attention-based VLM to a hybrid Mamba-2/attention model. Its contribution is density-weighted residual alignment: the student matches the teacher's residual stream more strongly at patches whose frozen visual features differ from their neighbors. The deployed architecture is unchanged by this loss; HEED does not delete visual tokens.Versioned full text · Official acceptance identity · Official NeurIPS program

Reading question: Does density weighting outperform uniform residual alignment when data, trainable parameters and architecture are held fixed?

Evidence boundary: The C3→C4 OCRBench v2 gain is 4.7 points; the larger headline change includes other training changes. The manuscript disagrees about Stage 3 trainable parameters. Neither the speedup nor this feature residual establishes WSI cross-magnification transfer.

Test semantic coherence, concept supervision, and hierarchical steering

Hierarchical CLIP SAEs and VLM SAE studies connect feature interpretation with retrieval, bias analysis, or interventions on generated descriptions. Concept Bottleneck SAE adds supplied concept structure; SAE++ builds a hierarchy over dictionary atoms. Human alignment, control strength, and preserving unrelated content are separate questions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Sparse Autoencoders Learn Monosemantic Features in Vision-Language ModelsMateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata2025 · NeurIPS · peer-reviewedNeurIPS 2025 proceedings volume 38; the repository BibTeX uses publication year 2026. Catalog year follows the conference edition.This work evaluates whether sparse vision features are semantically coherent and whether manipulating them changes a VLM output. Its MonoSemanticity score measures activation-weighted image similarity and is checked against human pairwise judgments; separate CLIP-feature interventions steer LLaVA. Interpretability and controllability are related experiments, not interchangeable definitions.Proceedings PDF · Code and human-comparison benchmark · arXiv
Interpretable and Steerable Concept Bottleneck Sparse AutoencodersAkshay Kulkarni, Tsui-Wei Weng, Vivek Narayanaswamy, Shusen Liu, Wesam A. Sakla, and Kowshik Thopalli2026 · CVPR · peer-reviewedCB-SAE combines an unsupervised sparse dictionary with a user-specified concept bottleneck. It scores existing neurons for interpretability and downstream steerability, prunes low-utility neurons, then trains a parallel concept autoencoder while freezing retained SAE weights. This recovers useful reconstruction and supplies concepts missing from the retained dictionary.Proceedings PDF · Supplement · Author repository (code not yet released) · arXiv
SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMsYusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, and Hao Wang2026 · NeurIPS · acceptedNeurIPS 2026 Accept (poster) verified from official program event 151133, OpenReview cl4vXMku1D, matching all five authors. Program title omits SAE++ and joins LearnMulti-Level; review uses arXiv v3 revised 7 October 2026. Proceedings version not yet verified.SAE++ learns a hierarchy by training a second SAE on the decoder atoms of a first SAE. Level 1 decomposes model activations into sparse features; Level 2 groups the learned feature directions into higher-level concepts. It is not a residual-error cascade and does not feed Level-1 sparse sample codes into a conventional stacked bottleneck.Reviewed manuscript v3 · Official NeurIPS poster · OpenReview · Code

Reading question: Can the target concept change while unrelated visual content and answer quality remain intact?

Evidence boundary: A semantic-coherence score is not a causal or clinical certificate. Report coverage, duplicate features, judge dependence, and both intended and collateral intervention effects.

Document understanding needs component and layout-aware evaluation

OmniDocBench separates end-to-end parsing from text, formula, table and reading-order evaluation across document types. Matching rules and ignored elements matter as much as the leaderboard. The original paper and later benchmark releases must be cited as distinct versions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive AnnotationsLinke Ouyang et al.2025 · CVPR · peer-reviewedReview pins the CVPR 2025 paper and supplement (981 pages). The live repository has later data, matching and scoring revisions; its current leaderboard is not the paper benchmark.OmniDocBench evaluates document parsing as structured extraction rather than just readable OCR. It separates text, tables, formulas and reading order, then breaks results down by page type and visual attributes. The paper benchmark contains 981 pages; newer repository versions are separate evaluation configurations.Proceedings PDF · Supplement · Author code and version history

Reading question: Does a better overall parser still fail on the document type or element your application needs?

Evidence boundary: The reviewed CVPR paper uses 981 pages; newer releases change coverage and scoring. Parsing fidelity does not establish factual, clinical or scientific correctness of a generated answer.

Separate concept recognition from execution of logical queries

LCSE keeps a dual encoder frozen, calibrates concept scores, evaluates an explicit soft logical expression and applies a correction to the holistic image–text score. It improves tested conjunction, disjunction and negation decisions without retraining the encoder.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language ModelsSultan Alshehri, Zhantao Yang, Han Zhang, and Marios Savvides2026 · ICML · peer-reviewedLCSE separates concept evidence from constraint execution in frozen dual-encoder VLMs. It parses a query into concepts and logical operators, scores concept presence, applies polarity-aware aggregation, and edits the original image–text similarity. This is an inference interface, not a newly trained visual encoder or a guarantee of exact logical consistency.Proceedings PDF · Author project

Reading question: Is a retrieval failure caused by missing visual concepts or by composing recognized concepts incorrectly?

Evidence boundary: Calibration uses labeled data and parsing is a separate dependency. The holistic residual retains many logical-equivalence violations. Proposed pathology queries need new tile/slide-level labels and evaluation; no pathology transfer was demonstrated.

Audit teacher reliability before stronger on-policy imitation

The VLM teacher-inheritance study compares filtered SFT with full-vocabulary OPD on ChartQA and Geometry3K. It separates answer accuracy from completion, decomposes paired failures and shows OPD can erase completion already learned through SFT. This is an empirical warning, not a new KD objective.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language ModelsFrank F. Yang, Kefei Yan, Yiren Wang, and Yiyang Wang2026 · arXiv · preprintVersion 1 empirical study. The current author README describes an anonymized code-only supplement and explicitly excludes evaluation records, logs, adapters, tests and figure scripts despite the manuscript release statement.This is an empirical boundary study, not a new distillation objective. On-policy distillation can transfer a teacher’s failure to finish an answer, even undoing completion learned through filtered supervised fine-tuning. The central distinction is between final-answer accuracy, presence of an answer line and actual termination. Agreement with a teacher is not evidence of correct or complete reasoning.Version 1 paper and appendices · Author code and current availability scope

Reading question: Does the teacher provide a usable answer on the states the student actually visits?

Evidence boundary: One model family and two main tasks; updates are fixed within channels, not compute-matched between them. Teacher probes are not a unique causal attribution. Current author code omits the evaluation records advertised in the manuscript.

Complete maintained paper route

Section titled “Complete maintained paper route”

82 canonical readings · 2002–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 82 readings in maintained order

  53. #542026NeurIPSAccepted · proceedings pendingL3

    VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

    Luca L. Weishaupt, Simone de Brot, Javier Asin, Llorenç Grau-Roma, Nic G. Reitsam, Andrew H. Song, Dongmin Bang, Stefan T. Kaluziak, Long Phi Le, Jakob Nikolas Kather, Faisal Mahmood, and Guillaume Jaume

    Acceptance independently verified 2026-10-08 through official NeurIPS poster 139742: exact title and matching 12-author list, linked OpenReview 7986DZ9rPe. Scientific review and artifact checks remain tied to arXiv:2608.26382v1 and the 2026-08-30 audit, not a refreshed full-text/code/data inspection. VIPER evaluates rat toxicologic regions rather than whole-slide images; it is an ROI benchmark, not a WSI benchmark.

    Official artifacts: Official evaluation code and documentation · Official benchmark dataset · Official NeurIPS 2026 acceptance record

    Metadata checked

  54. #552026Findings of EMNLPAccepted · proceedings pendingL3

    MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models

    Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan, Muhammad Akhtar Munir Sujair Ibrahim, Mohamed Rafeek Mareer Ahamed, Yutong Xie, Imran Razzak, and Muhammad Haris Khan

    The arXiv record reports acceptance to Findings of EMNLP 2026. No final ACL Anthology record was available when checked on 30 August 2026, so claims were verified against arXiv v1 and the official repository.

    Official artifacts: Official implementation

    Metadata checked

  56. #572026ACLPeer reviewedL2–L3

    Afri-MCQA: Multimodal Cultural Question Answering for African Languages

    Atnafu Lambebo Tonja, Srija Anand, Emilio Villa-Cueva, Israel Abebe Azime, Jesujoba Oluwadara Alabi, Muhidin A. Mohamed, Debela Desalegn Yadeta, Negasi Haile Abadi, Abigail Oppong, Nnaemeka Casmir Obiefuna, Idris Abdulmumin, Naome A Etori, Eric Peter Wairagala, Kanda Patrick Tshinu, Imanigirimbabazi Emmanuel, Gabofetswe Malema, Alham Fikri Aji, David Ifeoluwa Adelani, and Thamar Solorio

    Peer-reviewed ACL 2026 long paper and Best Social Impact Paper award recipient.

    Official artifacts: Author manuscript · Official dataset

    Metadata checked

  62. #632026arXivPreprintL2–L3

    LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training

    Andreas Hochlehnert, Marianna Nezhurina, Mehdi Cherti, Andrej Radonjic, Thaddäus Wiedemer, Christoph Schuhmann, Romain Beaumont, Wieland Brendel, Bernhard Schölkopf, A. Sophia Koepke, Jenia Jitsev, and Matthias Bethge

    arXiv preprint first submitted 25 August 2026; not peer reviewed as of the audit date.

    Official artifacts: Official project · Official data tooling · Official URL dataset

    Metadata checked

  63. #642026arXivPreprintL2–L3

    VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

    Junxiang Xu, Ruisi Wang, Fanyi Pu, Maijunxian Wang, Ran Ji, Tongxi Zhou, Chenyang Gu, Jing Zuo, Hongcan Xiao, Yimeng Geng, Wanqi Yin, Wei Chen, Oscar Qian, Zhengan Yan, Ziqi Huang, Haiwen Diao, Liang Pan, Bo Li, Xiangyu Fan, Dezhi Luo, Fengyuan Yu, Zehong Zhao, Qingying Gao, Tinghui Zhu, Yilan Zhang, Jingqi Tong, Pinyuan Feng, Zhengze Jiang, Letian Wang, Ziyu Guo, Renrui Zhang, Jieneng Chen, Sonia Joseph, Constantin Venhoff, Saman Motamed, Mengyue Yang, Chandra Sripada, Alan Yuille, Philip Torr, Lvmin Zhang, Vikash Kumar, Daniel Khashabi, Nikolaus Kriegeskorte, Raphaël Millière, Vincent C. Müller, Anyi Rao, Quan Wang, Ziwei Liu, Dahua Lin, Lei Yang, Hokin Deng, and Zhongang Cai

    arXiv preprint first submitted 26 August 2026; not peer reviewed as of the audit date.

    Official artifacts: Official project · Official code and benchmark

    Metadata checked

  65. #662026EMNLPAccepted · proceedings pendingL3

    IntroConformal: Conformal Factuality Guarantees for Large Vision-Language Models via Introspective Signals

    Md. Atabuzzaman, Christian Alexander, and Chris Thomas

    The arXiv v1 record submitted 1 September 2026 reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology record was available when checked on 2 September 2026, so publication claims remain author-reported and scientific claims use the arXiv manuscript.

    Official artifacts: Official implementation

    Metadata checked

  67. #682026ECCVAccepted · proceedings pendingL3

    NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

    Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, and Gargi Chakraborty

    ECCV 2026 Spotlight verified in the official program (event 6177); scientific claims use the conference-hosted manuscript #10972. Publisher proceedings/version of record was not verified on 2026-09-12. Earlier arXiv versions used the subtitle Evaluating Video Language Models Beyond the Frame; this is the same canonical work.

    Official artifacts: ECCV program · Conference manuscript · Author project · Dataset

    Metadata checked

  68. #692026ECCVAccepted · proceedings pendingL3

    Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

    Cheolhong Min, Jaeyun Jung, Daeun Lee, Hyeonseong Jeon, Yu Su, Jonathan Tremblay, Chan Hee Song, and Jaesik Park

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #10791 checked on 2026-09-12. Publisher version of record was not independently verified. The project page describes an oral; the catalog follows the official Spotlight record.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  70. #712026ECCVAccepted · proceedings pendingL3

    SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

    Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #349 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  71. #722026ECCVAccepted · proceedings pendingL3

    EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System Development

    Dongsheng Yuan, Yimo Deng, and Huangxun Chen

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #9127 checked on 2026-09-12. Publisher version of record was not independently verified. The PDF uses EmbedCopilot-Bench; the program shortens this to EmbedCopilot.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  72. #732026ECCVAccepted · proceedings pendingL3

    Learning to Deny: Action Denial in Multimodal Large Language Models

    Raiyaan Abdullah, Shehreen Azad, and Yogesh Singh Rawat

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #3692 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  74. #752026ECCVAccepted · proceedings pendingL3

    GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing

    Mingxin Liu, Ziqian Fan, Zhaokai Wang, Leyao Gu, Zirun Zhu, Yiguo He, Yuchen Yang, Changyao Tian, Xiangyu Zhao, Ning Liao, Shaofeng Zhang, Qibing Ren, Zhihang Zhong, Xuanhe Zhou, Junchi Yan, and Xue Yang

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #2706 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  75. #762026arXivPreprintL3

    SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

    Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, and Babak Taati

    Preprint checked 2026-10-02; project page still labels code and models as forthcoming. Visual-token retention is not an optical-magnification intervention.

    Official artifacts: Versioned full text · Author project page

    Metadata checked

  77. #782026NeurIPSAccepted · proceedings pendingL3

    HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation

    Yihao Liang and Niraj K. Jha

    Official NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested.

    Official artifacts: Versioned full text · Official acceptance identity · Official NeurIPS program

    Metadata checked

  79. #802026NeurIPSAccepted · proceedings pendingL3

    SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

    Yusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, and Hao Wang

    NeurIPS 2026 Accept (poster) verified from official program event 151133, OpenReview cl4vXMku1D, matching all five authors. Program title omits SAE++ and joins LearnMulti-Level; review uses arXiv v3 revised 7 October 2026. Proceedings version not yet verified.

    Official artifacts: Reviewed manuscript v3 · Official NeurIPS poster · OpenReview · Code

    Metadata checked

  81. #822026arXivPreprintL3

    Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models

    Frank F. Yang, Kefei Yan, Yiren Wang, and Yiyang Wang

    Version 1 empirical study. The current author README describes an anonymized code-only supplement and explicitly excludes evaluation records, logs, adapters, tests and figure scripts despite the manuscript release statement.

    Official artifacts: Version 1 paper and appendices · Author code and current availability scope

    Metadata checked