Skip to content
Computational Pathology Reading Roadmap

Grounding and open-vocabulary vision

Visual grounding links language to a spatial entity. Depending on the task, the output may be a point, box, mask, track, or phrase-linked region. Open-vocabulary recognition uses language to define categories beyond a fixed supervised label set. These ideas overlap, but a zero-shot class score is not a localization result and a box does not prove that a generated explanation used the boxed evidence.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

ECCV 2026 conference additions and metadata checked through 2026-09-12; the broader literature audit boundary above is unchanged.

Prerequisite path

Section titled “Prerequisite path”
  1. Complete detection, semantic/instance segmentation, IoU, AP, and matching from the computer-vision route.
  2. Read global and token-level alignment in image–text representation.
  3. Understand referring expressions, noun phrases, tokenization, and attention from NLP.

At research level, you should be able to define the referent, spatial output, annotation granularity, absent-referent behavior, and open-set boundary before choosing a model.

Output, supervision, and evidence taxonomy

Section titled “Output, supervision, and evidence taxonomy”
Problem Output Supervision Evidence and claim
Referring-expression comprehension one box/region for a phrase expression–box pairs localizes a described instance among distractors
Phrase grounding boxes linked to several phrase spans captions plus phrase–box links aligns language spans with regions
Open-vocabulary detection labeled boxes from arbitrary text classes detection labels plus image–text/caption data recognizes and localizes categories beyond a closed training vocabulary
Open-vocabulary segmentation text-defined semantic/panoptic masks masks plus language or pretrained VLM features assigns pixels to language-defined concepts
Promptable segmentation mask from point/box/text/reference prompt large mask corpus and prompt simulation responds to a prompt; semantic naming may require another model
Reasoning segmentation mask plus language answer/rationale instructions, masks, reasoning data resolves a compositional request and emits pixel evidence
Grounded dialogue response with points/boxes/masks localized instructions/conversations binds generated claims to inspectable regions

An absent or impossible query is part of the task. Systems that must always return a box can appear accurate on closed benchmarks while being unsafe in open use.

Method detail

Section titled “Method detail”

Language-conditioned detection

Section titled “Language-conditioned detection”

DETR-style models use object queries and bipartite matching. Language-grounded variants condition queries, class scores, or cross-modal features on text. Important design choices include:

  • whether the text is a class list, caption, or free-form expression;
  • whether image and text interact before or after object proposals;
  • how words/phrases map to boxes;
  • how unmatched queries and “no object” are trained;
  • whether category synonyms share supervision;
  • which detection, grounding, and caption corpora are mixed.

Dense and promptable segmentation

Section titled “Dense and promptable segmentation”

A promptable segmenter can delineate an object from a point or box without assigning a semantic label. An open-vocabulary segmenter uses language-aligned features to name pixels or masks. A grounded MLLM may reason in text and delegate masks to a segmentation decoder. Keep these modules and their evidence separate.

Grounded language generation

Section titled “Grounded language generation”

Special coordinate/location tokens can serialize points or boxes into text. Other models read internal word-to-pixel attention or connect an MLLM to a frozen mask decoder. Coordinate tokens permit end-to-end generation but quantization and decoding affect precision; attention extraction can preserve dialogue capability but attention is not automatically calibrated evidence.

Staged paper route

Section titled “Staged paper route”

Phrase and region alignment

Section titled “Phrase and region alignment”
Order Primary paper and year Venue / status Read it for
1 Referring expressions (2016) CVPR, peer reviewed jointly generating and resolving unambiguous object descriptions
2 MDETR (2021) ICCV, peer reviewed end-to-end detection conditioned on free-form text and phrase alignment
3 RegionCLIP (2022) CVPR, peer reviewed region-level language pretraining transferred to object detection
4 GLIP (2022) CVPR, peer reviewed unifying detection and grounding through phrase–region alignment
5 OWL-ViT (2022) ECCV, peer reviewed transferring image–text representations to open-vocabulary detection
6 Grounding DINO (2024) ECCV, peer reviewed language-guided query selection and cross-modal detector fusion

Open-vocabulary and promptable masks

Section titled “Open-vocabulary and promptable masks”
Order Primary paper and year Venue / status Read it for
1 GroupViT (2022) CVPR, peer reviewed semantic groups and zero-shot masks emerging from text supervision
2 CLIPSeg (2022) CVPR, peer reviewed dense prediction conditioned by arbitrary text or image prompts
3 Detic (2022) ECCV, peer reviewed expanding detector vocabulary with image-level supervision
4 ODISE (2023) CVPR, peer reviewed using diffusion representations for open-vocabulary panoptic segmentation
5 Segment Anything (2023) ICCV, peer reviewed promptable masks, data engine, and ambiguity-aware outputs
6 SEEM (2023) NeurIPS, peer reviewed a general prompt interface across semantic, spatial, and referring inputs
7 Florence-2 (2024) CVPR, peer reviewed one sequence interface for caption, detection, grounding, and segmentation
8 SAM 2 (2025) ICLR, peer reviewed promptable segmentation extended from images to persistent video objects

Grounded multimodal language models

Section titled “Grounded multimodal language models”
Order Primary paper and year Venue / status Read it for
1 Kosmos-2 (2024) ICLR, peer reviewed location tokens and grounded image–text data for referring and generation
2 LISA (2024) CVPR, peer reviewed a segmentation token connecting LLM reasoning to a mask decoder
3 GLaMM (2024) CVPR, peer reviewed region-level understanding, generation, and pixel grounding
4 Qwen2-VL (2024) arXiv preprint dynamic-resolution inputs and coordinate outputs in a general MLLM
5 Qwen2.5-VL (2025) arXiv preprint object grounding extended to documents, charts, and long video
6 F-LMM (2025) CVPR, peer reviewed extracting word–pixel correspondences from frozen MLLM attention
7 Argus (2025) CVPR, peer reviewed object-centric grounded steps for visual reasoning
8 Grounding Everything in Tokens (2026) CVPR, peer reviewed tokenized grounding outputs in an MLLM interface
9 VGent (2026) CVPR, peer reviewed separating language reasoning from efficient spatial prediction

Evaluation that changes the claim

Section titled “Evaluation that changes the claim”
Order Primary paper and year Venue / status Read it for
1 Referring-expression evaluation in MDETR (2021) ICCV, peer reviewed task-specific box accuracy and phrase grounding
2 GroundingME (2026) CVPR, peer reviewed hard distractors, spatial relations, limited evidence, and rejection
3 Perceval (2026) CVPR, peer reviewed claim-level visual error localization as process supervision

Evaluation contract

Section titled “Evaluation contract”
Output Metric Required audit
Point pointing accuracy / distance object size, ambiguity, absent target
Box Acc@IoU, AP by rare/common/frequent class synonymous prompts, distractors, calibration, no-object
Mask mIoU, Dice, boundary/object metrics small objects, empty masks, prompt type, decoder dependency
Phrase–region set phrase grounding recall/accuracy span tokenization, one-to-many matches, duplicate boxes
Grounded response answer correctness plus localization claim–region entailment and intervention
Open vocabulary base/novel class AP vocabulary source, class hierarchy, prompt tuning, pretraining overlap

Qualitative heatmaps or attention overlays are useful diagnostics but not sufficient metrics. Evaluate the emitted spatial object and whether manipulating it changes the language claim.

Controlled experiment

Section titled “Controlled experiment”

Choose a detection/grounding dataset with object boxes and referring expressions. Compare:

  1. CLIP scoring over detector proposals;
  2. an open-vocabulary detector;
  3. a grounded MLLM that emits coordinates;
  4. the same MLLM coupled to a mask decoder.

Hold visual backbone, resolution, class/expression list, split, and prompt templates fixed where architectures allow. Create matched cases with:

  • one clear referent;
  • two visually similar distractors;
  • a relational expression;
  • a tiny or partially occluded referent;
  • an absent referent.

Report box/mask accuracy, rejection AUROC, calibration, latency, visual-token count, and language-answer correctness. Perform a causal test by masking the predicted region and measuring the change in the claimed answer.

Failure modes

Section titled “Failure modes”
  • Closed benchmarks omit absent referents and reward forced localization.
  • Class-name prompts contain benchmark-specific synonyms selected on test data.
  • Detector pretraining already includes the “novel” category or near-duplicate images.
  • Global CLIP scores are upsampled into persuasive but poorly localized heatmaps.
  • Small objects disappear through resizing or visual-token compression.
  • A segmentation decoder supplies the spatial ability attributed to the LLM.
  • The model chooses the correct object from linguistic priors despite wrong pixels.
  • IoU penalizes multiple valid masks without accounting for annotation ambiguity.
  • Generated coordinates are parsed differently across evaluation toolchains.
  • A grounded rationale remains unchanged after its stated evidence is removed.

Self-check and exit task

Section titled “Self-check and exit task”
  1. How do referring comprehension, phrase grounding, and open-vocabulary detection differ?
  2. Why is Segment Anything promptable but not automatically language semantic?
  3. What must an absent-referent example test?
  4. Which module produces masks in LISA or a SAM-connected MLLM?
  5. What does an attention heatmap fail to establish?
  6. How would you measure whether grounding improves answer correctness?

Exit task: construct a rejection-aware grounding benchmark slice and compare at least one proposal-scoring, detector, and generative-coordinate approach under the same image and text inputs. Include region-removal interventions.

Bridge to computational pathology

Section titled “Bridge to computational pathology”

WSI grounding must specify scale and coordinate system: cell, tile, region, slide, or case. A tile heatmap cannot establish that a report sentence is supported at the cellular level, and a high-attention tile is not a validated diagnostic explanation. Record microns per pixel, pyramid level, tissue mask, tile sampling, coordinate transforms, annotator granularity, and whether the target is absent. Then use the pathology VLM and cell/tissue modelling chapters for domain-specific grounding evidence.

Counterexamples for spatial, semantic and action grounding

Use matched distance counterexamples, cross-category part correspondence, action-absent video clips and background-consistent concealment to test different kinds of visual evidence. Preserve each study's denominator: answer probability, correspondence success, absence/presence accuracy and newly introduced ungrounded nouns are not interchangeable.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation ModelsOlaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #349 checked on 2026-09-12. Publisher version of record was not independently verified.SOCO benchmarks whether a representation matches the correct object part, not merely the correct kind of part. Its taxonomy separates concept correspondence, identity-specific semantic object correspondence and cross-category transfer. Language descriptions also let VLMs be tested on text-guided localization versus visual-reference matching.ECCV Spotlight program · Conference manuscript · Official project or implementation
Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language ModelsAmira Guesmi and Muhammad Shafique2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #12081 checked on 2026-09-12. Publisher version of record was not independently verified.Background-Consistent Re-encoding (BCR) studies how a vision-language model can stop naming a selected object without filling its missing representation with an invented replacement. It perturbs the input image so object-region features resemble surrounding background features while retaining the encoder's ordinary token layout. The scientific claim concerns continuity of visual evidence, not literal elimination of every hallucination.ECCV Spotlight program · Conference manuscript

Reading question: Does an apparent improvement survive counterexamples that preserve superficial cues while changing the correct spatial, semantic or temporal conclusion?

Evidence boundary: A small bias gap can coexist with near-chance performance. Action denial is not safety refusal, detector-based concealment is not a privacy guarantee, and benchmark correlations are not universal causal explanations.

Recovering, steering and testing visual concepts

Intermediate features can expose information hidden by the final representation. Compare frozen-backbone negation adaptation, language-directed representations, circuit interventions and geometry-subspace probes; distinguish information accessibility from causal control.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Steerable Visual RepresentationsJona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano2026 · ECCV · acceptedECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.SteerViT lets text change the visual representation while the image is being encoded. A frozen ViT receives lightweight gated cross-attention to a frozen language encoder, producing prompt-aware global features and dense patch features. The objective is a reusable vision-centric representation that can be directed toward small objects or attributes, rather than a language model that answers a question after vision encoding has finished.ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints

Reading question: Which layer, token, subspace or circuit changes, and does the intervention preserve unrelated capabilities?

Evidence boundary: A probe may measure accessibility rather than semantic understanding; intervention and natural distribution shift are different tests.

Perceptual experts, geometry and explicit grounding tokens

Continuous expert-supervised visual tokens, geometry-aware fusion and coarse-to-fine grounding alter how a language model accesses visual evidence. Compare the representation interface, auxiliary supervision and deployed computation before comparing answer scores.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
MolmoPoint: Better Pointing for VLMs with Grounding TokensChristopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, and Ranjay Krishna2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official program (event 6028); scientific claims use the conference-hosted manuscript #8311. Publisher proceedings/version of record was not verified on 2026-09-12. The program spells the name Molmo-Point; the conference PDF uses MolmoPoint.MolmoPoint makes a VLM point by selecting its own visual tokens, then refining the selected location. Three grounding tokens replace a coordinate string: choose a coarse patch, select a finer ViT subpatch, and choose a position inside that subpatch. The mechanism ties spatial output to visual evidence and reduces the need to learn a global textual coordinate system.ECCV program · Conference manuscript · Author project

Reading question: Does a claimed reasoning gain arise from more visual evidence, better alignment, a changed output space or a larger inference budget?

Evidence boundary: Expert distillation is not proof of faithful reasoning; stronger pointing or spatial QA need not improve every tracking metric.

Complete maintained paper route

Section titled “Complete maintained paper route”

35 canonical readings · 2016–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 35 readings in maintained order

  12. #122023ICCVPeer reviewedL2–L3

    Segment Anything

    Alexander Kirillov et al.

    Metadata checked

  31. #312026ECCVAccepted · proceedings pendingL3

    Steerable Visual Representations

    Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano

    ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.

    Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints

    Metadata checked

  32. #322026ECCVAccepted · proceedings pendingL3

    MolmoPoint: Better Pointing for VLMs with Grounding Tokens

    Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, and Ranjay Krishna

    ECCV 2026 Spotlight verified in the official program (event 6028); scientific claims use the conference-hosted manuscript #8311. Publisher proceedings/version of record was not verified on 2026-09-12. The program spells the name Molmo-Point; the conference PDF uses MolmoPoint.

    Official artifacts: ECCV program · Conference manuscript · Author project

    Metadata checked

  33. #332026ECCVAccepted · proceedings pendingL3

    SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

    Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #349 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked