Natural-image vision and language

Vision–language research asks how visual evidence and language can share representations, exchange information, and support outputs such as retrieval scores, answers, regions, generated images, or actions. This section follows the natural-image lineage. Its purpose is to explain the methods that later enter scientific and clinical domains, without treating a natural-image benchmark result as evidence of pathology validity.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

This is a curated, prerequisite-aware route rather than a claim that every paper using the term “multimodal” is equally important. Each chapter records peer-review status and links to a primary paper. New work belongs in its scientific lineage; the review date changes without creating a dated catch-all section. Every detailed chapter ends with a catalog-generated Complete maintained paper route; its shorter staged table is the teaching spine, not the full bibliography.

Prerequisite path

Level	Read first	You are ready when you can…
L0 · orientation	ML and deep-learning foundations	distinguish a dataset, objective, architecture, checkpoint, and evaluation task
L1 · visual and textual representations	Computer vision and NLP	trace a CNN/ViT feature and a token embedding through a model
L2 · multimodal systems	attention, contrastive learning, causal language modelling, transfer learning	write an image–text loss and identify every trained and frozen component
L3 · research synthesis	calibration, distribution shift, data provenance, human evaluation	design a comparison that holds data, compute, prompts, and adaptation budget fixed

The shortest useful sequence is:

1. CLIP (2021) for global dual-encoder alignment;
2. ALBEF (2021) and BLIP (2022) for fusion and multiple pretraining losses;
3. CoCa (2022) for contrastive representation plus caption generation;
4. GLIP (2022) for language-supervised regions and open-vocabulary detection;
5. Flamingo (2022) and BLIP-2 (2023) for bridges to frozen language models;
6. LLaVA (2023) for visual instruction tuning;
7. Grounding DINO (2024) and Florence-2 (2024) for promptable, spatial output;
8. Qwen2-VL (2024) and Molmo (CVPR 2025; preprint 2024) for resolution, data, pointing, and openness;
9. SigLIP 2 (2025) and AIMv2 (2025) for modern vision-encoder recipes;
10. choose a branch: unified generation, video-language, agents, or trustworthy evaluation.

Output, supervision, and evidence taxonomy

Do not classify a model only by its name. First record what enters, what leaves, and which supervision connects them.

Family	Typical output	Supervision	Visual evidence actually exposed
Dual encoder	image/text embeddings or similarity	paired image–text data, contrastive or sigmoid loss	usually a global image embedding; dense features require separate evidence
Fusion encoder	matched-pair score, answer, region-conditioned representation	matching, masked modelling, captions, task labels	region proposals or patch tokens jointly attended with text
Conditional text generator	caption, answer, dialogue, rationale	captions, QA, instructions, preferences	visual tokens selected by a connector, resampler, or cross-attention module
Grounded predictor	point, box, phrase, mask, or dense label	phrase–region pairs, boxes, masks, referring expressions	explicit spatial features at region or pixel level
Visual generator	image/video or edit	text–image/video pairs, denoising or next-token targets, preferences	text and optional reference/control images condition a visual decoder
Unified multimodal model	interleaved text and visual outputs	mixed understanding and generation objectives	depends on its visual tokenizer and whether understanding preserves spatial detail
Vision–language–action model	tool call, click, navigation step, robot action	demonstrations, trajectories, rewards	current observation plus a possibly lossy history of earlier observations

“Grounded” should mean that an output is linked to inspectable visual evidence, not merely that the model received an image. Likewise, a generated explanation is not a faithful reasoning trace unless interventions show that it changes with the relevant evidence.

Staged reading map

Stage	Primary paper and year	Venue / status	What it establishes
Visual-semantic precursor	DeViSE (2013)	NeurIPS, peer reviewed	mapping images into a language-derived semantic space for zero-shot transfer
Global alignment	CLIP (2021)	ICML, peer reviewed	symmetric image–text contrastive learning and prompt-based zero-shot recognition
Scaled weak pairs	ALIGN (2021)	ICML, peer reviewed	scale and noisy web alt-text as a viable training regime
Fusion objectives	ALBEF (2021)	NeurIPS, peer reviewed	align before cross-modal fusion, momentum distillation, and hard negatives
Unified understanding/generation	BLIP (2022)	ICML, peer reviewed	contrastive, matching, and language-modelling objectives with caption filtering
Language-grounded regions	GLIP (2022)	CVPR, peer reviewed	reformulating detection as phrase grounding
Few-shot multimodality	Flamingo (2022)	NeurIPS, peer reviewed	gated cross-attention and a resampler around a frozen language model
Frozen-model bridge	BLIP-2 (2023)	ICML, peer reviewed	a Q-Former connecting frozen image and language models
Visual instruction	LLaVA (2023)	NeurIPS, peer reviewed	synthetic multimodal instruction data and connector-based tuning
Open spatial vocabulary	Grounding DINO (2024)	ECCV, peer reviewed	language-guided object detection with open-set transfer
Unified task interface	Florence-2 (2024)	CVPR, peer reviewed	prompt-to-sequence formulation for captions, boxes, and masks
Open model and data	Molmo and PixMo (2025; preprint 2024)	CVPR, peer reviewed	human-collected dense captions, QA, pointing, weights, and data
Instruction-based editing	InstructPix2Pix (2023)	CVPR, peer reviewed	synthetic edit instructions and conditional diffusion for source-image transformation
Encoder synthesis	SigLIP 2 (2025)	arXiv preprint	sigmoid alignment plus captioning, SSL, curation, multilingual and dense features
Multimodal autoregression	AIMv2 (2025)	CVPR, peer reviewed	joint image-patch and text-token prediction for a scalable vision encoder
Open vision–language–action policy	OpenVLA (2025; CoRL 2024)	PMLR version of record, peer reviewed	an open VLA policy trained across robot embodiments with efficient adaptation
Compositional audit	SugarCrepe (2023)	NeurIPS Datasets and Benchmarks, peer reviewed	hard-negative construction that explicitly audits language-only benchmark shortcuts
Grounding audit	GroundingME (2026)	CVPR, peer reviewed	discriminative, spatial, limited-visibility, and rejection tests
Unified-model audit	RealUnify (2026)	CVPR, peer reviewed	tests whether understanding and generation actually help one another

Follow the detailed branches:

• Image–text representation and fusion
• Captioning, VQA, OCR, and reasoning
• Grounding and open-vocabulary vision
• Multimodal LLMs
• Generation, video, and agents
• Evaluation, data, and safety

Method detail: how the families connect

paired image–text data
    ├── separate encoders ── similarity ── retrieval / zero-shot recognition
    ├── patch + word fusion ── matching / masking ── VQA / reasoning
    ├── region supervision ── boxes / masks ── grounding / open vocabulary
    ├── visual connector + LLM ── causal tokens ── caption / dialogue / tool call
    ├── visual tokenizer or diffusion path ── pixels ── generation / editing
    └── temporal observations + policy ── actions ── video agent / embodied agent

The branches share datasets and components, but they do not establish the same claim. A CLIP retrieval score does not prove fine-grained grounding. A language model that answers a VQA item does not necessarily use the image. A unified generator may produce excellent images while its “understanding” stream discards spatial detail.

Controlled experiment: field integration

Use one public image–text dataset and one held-out distribution. Compare:

1. a frozen CLIP dual encoder;
2. the same visual encoder with a small cross-modal fusion head;
3. a frozen LLM connected through a trainable projection;
4. an image-ablated and shuffled-image control for every language output.

Hold the split, image resolution, text normalization, prompt set, parameter-update budget, and evaluation examples fixed. Report:

• image-to-text and text-to-image Recall@K;
• zero-shot class accuracy and expected calibration error;
• VQA accuracy with correct, shuffled, blank, and counterfactual images;
• phrase localization or pointing accuracy where annotations permit;
• latency, peak memory, number of visual tokens, and trainable parameters.

The purpose is not to declare one family universally best. It is to identify which capability appears only when the architecture receives the corresponding evidence and supervision.

Failure modes to carry through every chapter

• Text shortcut: the question, caption template, or class frequency predicts the answer without vision.
• Global-to-local overclaim: a global embedding is presented as pixel-level evidence.
• False-negative alignment: semantically compatible image–text pairs are repelled because they are not the annotated pair.
• Data leakage: benchmark images, near duplicates, captions, or generated derivatives occur in pretraining.
• Prompt instability: zero-shot results depend strongly on class wording, language, or prompt order.
• Resolution illusion: more input pixels do not imply that the connector preserves small objects or text.
• Language-prior hallucination: fluent objects, attributes, or relations are not supported by the image.
• Metric substitution: similarity, lexical overlap, or model-as-judge preference replaces task-specific correctness.
• Synthetic-data circularity: a proprietary teacher supplies both training data and evaluation judgments.
• Unsafe action boundary: a visually grounded plan can still be incorrect, irreversible, or harmful.

Self-check and exit task

You are ready to specialize when you can answer all of the following:

1. Which outputs require a dual encoder, fusion encoder, generator, spatial decoder, or policy?
2. Where do positives and negatives enter CLIP and SigLIP?
3. Which visual tokens reach the language model, and what spatial information can the connector discard?
4. What experiment distinguishes visual grounding from language plausibility?
5. Why must retrieval, generation, grounding, and action use different metrics?
6. How would you audit pretraining overlap and synthetic-data provenance?

Exit task: choose one system and draw its complete data flow from raw media and text through preprocessing, objectives, frozen/trainable modules, decoding, and evaluation. Add one intervention for each claimed capability.

Bridge to computational pathology

Natural-image methods supply useful mechanisms, not automatic clinical evidence. Before transfer to a whole-slide image (WSI), replace “image” with the actual hierarchy:

patient → case → slide → region → tile/cell → visual tokens → language output

Then ask which level is paired with the report, whether the text describes the whole case or one image region, how magnification and stain enter the evidence, and whether patients or reports overlap across pretraining and evaluation. Continue to pathology vision–language models for the domain-specific reading route.

Dedicated data-rich routes

Document intelligence treats OCR, layout, charts, schemas, and structured decoding as separate evidence layers. Multimodal embedding and retrieval follows dual encoders, data curation, modality bridges, task-conditioned embeddings, reranking, and measured retrieval systems.