Image–text representation learning
Dual encoders, normalization, temperature, and data define embedding geometry.
Multimodal embedding and retrieval
nDCG (Järvelin and Kekäläinen, 2002) is the graded-ranking prerequisite for multimodal retrieval. It discounts relevant items by rank and normalizes by the ideal ordering. Report gain mapping, logarithmic discount convention, cutoff, query aggregation, judgment coverage, and treatment of unjudged image–text pairs; nDCG does not replace Recall@K, latency, or a false-negative audit.
Retrieval quality depends on the query, corpus, relevance relation, index, instruction, reranking stages, and measured latency—not only the embedding name.
21 maintained papers · literature reviewed through 2 October 2026
A shared embedding can support zero-shot classification, image–text retrieval, composed queries, multi-vector late interaction, or multimodal documents, but these are different contracts. Positives, negatives, instructions, corpus provenance, index construction, reranking, and latency determine the reported retrieval system.
Prerequisite contract
Retrieval evaluation
Candidate pools, relevance, duplicates, and hard negatives define Recall and nDCG.
How the field developed
2002
Graded ranked-retrieval evaluation
nDCG accumulates graded relevance with rank discount and normalizes against the ideal ordering, making cutoff, gain, judgments, and unjudged items part of every multimodal-retrieval claim.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Cumulated Gain-Based Evaluation of IR Techniques Järvelin and Kekäläinen evaluate a ranked list as the relevance gain a user accumulates while moving down it. CG adds graded relevance; DCG discounts gain retrieved late; normalized variants divide the observed curve by an ideal ordering built from the topic's full recall base. The result credits both highly relevant documents and early placement, while making the gain mapping, discount, cutoff, and judgment pool explicit parts of the evaluation model. — Work: Compute DCG, ideal DCG, and nDCG for one query with graded relevance, then change cutoff and missing judgments.
Exit evidence: State gain, discount, cutoff, query aggregation, and judgment coverage beside every nDCG score.
2018–2023
Hard-negative and web-scale dual encoders
VSE++ selects the most violating mini-batch negatives for ranking; CLIP learns a web-scale shared space; LiT locks the image tower; TinyCLIP distils affinity and inherits weights; SigLIP replaces global softmax with pairwise sigmoid loss.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 VSE++: Improving Visual-Semantic Embeddings with Hard Negatives VSE++ learns a joint image–caption embedding with a small but consequential change to the bidirectional ranking loss: for each matched pair in a mini-batch, it optimizes against the highest-scoring mismatched caption and image rather than summing all violating negatives. Concentrating gradient on the hardest sampled confusions makes the embedding resolve fine semantic distinctions that an average-negative objective can underweight. Official implementation 02.02 Learning Transferable Visual Models From Natural Language Supervision CLIP jointly trains an image encoder and a text encoder to recognize which image and text belong together among all pairings in a large batch. The shared embedding space is then reused without retraining the encoders: descriptions of downstream classes are encoded into text-generated classifier weights, and a new image is assigned to the most similar class embedding. This turns natural language into a flexible classification interface while retaining a separate frozen image representation for supervised linear-probe evaluation. — 02.03 LiT: Zero-Shot Transfer with Locked-image Text Tuning LiT separates learning a good visual representation from learning an open-vocabulary text interface. Start with a strong pretrained image encoder, lock every image-tower weight, attach a trainable text tower, and use paired image-text contrastive learning to make language read out the fixed visual space. In the paper's notation this strict configuration is Lu: locked pretrained image, unlocked randomly initialized text. The key mechanism is preserving visual geometry while learning cross-modal alignment, rather than letting noisy web pairs rewrite both towers. — 02.04 TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance TinyCLIP compresses both towers of CLIP by transferring the teacher's full image-text affinity structure and inheriting a compact subset of teacher weights. Affinity mimicking matches bidirectional probability matrices over all image-text pairs in a batch, including non-diagonal relations. Weight inheritance initializes a thinner or shallower student from selected teacher parameters, reducing the optimization burden of learning vision-language alignment from scratch. Official code and checkpoints 02.05 Sigmoid Loss for Language Image Pre-Training SigLIP retains the CLIP-style two-tower representation space but changes the training question. CLIP uses two globally normalized softmax classifications—each image selects its paired text and each text selects its paired image from the whole batch. SigLIP instead treats every image–text combination as an independent binary decision: the observed diagonal pair is positive and every off-diagonal combination is negative. A learned sigmoid loss is applied once to the signed pair logit. This removes the loss's need for a global softmax denominator, enables chunk-by-chunk distributed evaluation, and performs especially well at modest batch sizes; it does not remove the encoders, the negative pairs, or the quadratic number of pair interactions. — Work: Reproduce bidirectional retrieval while fixing data, towers, batch, negative selection, preprocessing, and student capacity.
Exit evidence: Report bidirectional relevance, false-negative audit, model size, latency, and quality retention.
2023–2024
Data quality and modality expansion
ImageBind aligns six modalities through image-paired data; DataComp and DFN isolate data-pool and learned-filter effects.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 ImageBind: One Embedding Space To Bind Them All ImageBind learns one embedding space for image/video, text, audio, depth, thermal imagery, and inertial measurements without collecting every possible modality pair. The image representation is the hub: each new modality is contrastively aligned only with its naturally co-occurring image or video. Because text is already aligned with the same frozen image space, audio and text—or IMU and text—can become comparable even though those pairs were never training positives. The paper calls this emergent alignment; it is a geometric consequence tested by zero-shot classification and retrieval, not a guarantee that every cross-modal relation is semantically complete. — 03.02 DataComp: In Search of the Next Generation of Multimodal Datasets DataComp turns web-scale dataset design into a controlled experiment: participants choose training examples, while the CLIP architecture, optimization budget, and 38-task evaluation suite stay fixed. The benchmark offers a filtering track over a common 12.8B-pair pool and a Bring Your Own Data track. Its scientific object is therefore a data-selection policy measured through the model it produces, not merely a downloadable corpus or a leaderboard of arbitrary systems. — 03.03 Data Filtering Networks Data Filtering Networks reframes web-scale dataset construction as a learned-model problem. Train a small image-text model on relatively high-quality pairs, use its image-caption similarity to retain examples from a much larger uncurated pool, then train a new target model only on the retained set. Crucially, the filter is not judged by its own ImageNet or retrieval score: filtering performance means the downstream performance of the model induced by its selected data. The paper's central finding is that those two rankings can differ sharply. — Work: Audit modality bridges, filtering, duplicates, licenses, and source-specific retrieval.
Exit evidence: Separate objective, data quantity, data quality, and modality coverage.
2025
Multi-vector visual document retrieval
ColPali transfers ColBERT-style late interaction to visual-language page tokens so document retrieval can preserve layout, figures, and text without an OCR-only index.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 ColPali: Efficient Document Retrieval with Vision Language Models ColPali retrieves a document page directly from its rendered image instead of first extracting OCR text, layout blocks, tables, and captions. PaliGemma contextualizes 1,024 visual patch tokens and query text tokens in a shared language-conditioned space; a learned projection keeps one 128-dimensional vector per patch or query token. At search time, ColBERT-style late interaction lets every query token choose its best-matching page patch, then sums those maxima. The page index is computed offline, so fine-grained query–patch matching remains fast enough for retrieval while preserving text, layout, figures, and tables. Official code · Official ViDoRe models and benchmark Work: Compare OCR text, single-vector page embeddings, and multi-vector late interaction on the same corpus and latency budget.
Exit evidence: Report index size, retrieval latency, and page-level relevance together.
2025–2026
Instruction-conditioned general embeddings
VLM2Vec trains VLMs across broad embedding tasks; SigLIP 2 broadens dense and multilingual transfer; Qwen3-VL-Embedding adds retrieval and reranking interfaces.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks VLM2Vec turns an instruction-following vision-language model into a universal multimodal bi-encoder. A query and a candidate may each contain text, an image, or both; the same VLM encodes either side, its final token becomes a normalized vector, and cosine similarity ranks candidates. The accompanying MMEB benchmark tests this contract across 36 datasets and four meta-task families rather than only image–caption retrieval. Official code · Project page 05.02 SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features SigLIP 2 is a multilingual two-tower image–text encoder family built by extending SigLIP's pairwise sigmoid alignment with a unified staged recipe: a training-only captioning and localization decoder, late image self-distillation and masked patch prediction, bias-aware multilingual data mixing, optional resolution adaptation, and active-curation distillation for the smallest models. The released encoders target both global retrieval/classification and local dense or grounding transfer. Models 05.03 Qwen3-VL-Embedding: Advancing Multimodal Retrieval and Reranking Qwen3-VL-Embedding is a 2B/8B family with two complementary retrieval interfaces: a bi-encoder maps text, images, video, visual documents, or mixtures into reusable vectors, while a cross-encoder reranker jointly reads a query–candidate pair and scores whether it matches. Both inherit Qwen3-VL instruction following; staged contrastive learning, reranker distillation, matryoshka dimensions, and quantization make the representation configurable for accuracy and serving cost. Official code · Official project article Work: Evaluate unseen task instructions, corpus scale, modality mixtures, embedding versus reranker, and cost.
Exit evidence: Publish task-by-modality retrieval and latency matrices.
2026
VideoLLM embeddings and open video corpora
ViLL-E adds an adaptive embedding head and staged generative/contrastive training; LAION-BVD supplies scale, captions, code, and models while requiring provenance and access auditing.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 ViLL-E: Video LLM Embeddings for RetrievalPeer-reviewed ACL 2026 long paper and Outstanding Paper award recipient. ViLL-E adds a discriminative embedding path to a generative VideoLLM. Video frames and a prompt pass through the multimodal decoder, which may generate a variable number of internal or textual tokens before an EOS trigger. A learned attentive pooling head then converts those tokens into a compact video embedding. Joint generative and contrastive training lets one model answer questions, caption, retrieve videos, and localize moments instead of maintaining separate decoder and dual-encoder systems. Author manuscript · Official publication page 06.02 LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-trainingarXiv preprint first submitted 25 August 2026; not peer reviewed as of the audit date. LAION-BVD is a web-scale video resource for multimodal pretraining. The pipeline starts from roughly 1.3 billion URLs, reports about 80 million downloaded videos totaling 10 million hours, and derives around 55 million scene clips with audio, keyframes, metadata, and synthetic captions. The release separates open URL and tooling artifacts from gated raw-media access, so “open dataset” does not mean every video byte is freely redistributable. Official project · Official data tooling · Official URL dataset Work: Compare a generative VideoLLM with and without an adaptive embedding head across retrieval and temporal localization, then measure how corpus scale, captions, provenance, rights, and link availability change results.
Exit evidence: Report query/corpus unit, negative sampling, temporal granularity, recall and localization metrics, index and reranking cost, caption noise, deduplication, link decay, and license/access strata.
2025–2026
Recovering, steering and testing visual concepts
Intermediate features can expose information hidden by the final representation. Compare frozen-backbone negation adaptation, language-directed representations, circuit interventions and geometry-subspace probes; distinguish information accessibility from causal control.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate FeaturesECCV 2026 Oral verified in the official program (event 6084); scientific claims use the conference-hosted manuscript #10575. Publisher proceedings/version of record was not verified on 2026-09-12. PeakPatch recovers negation cues from frozen intermediate CLIP text features instead of updating the encoders. A learned embedding residual restores distinctions between affirmative and negated captions; a separate, bounded score correction improves image-caption decisions. The contribution is a diagnosis-plus-adapter design for compositional failure, not a new image encoder or an inference-time language parser. ECCV program · Conference manuscript · Author project 07.02 Steerable Visual RepresentationsECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'. SteerViT lets text change the visual representation while the image is being encoded. A frozen ViT receives lightweight gated cross-attention to a frozen language encoder, producing prompt-aware global features and dense patch features. The objective is a reusable vision-centric representation that can be directed toward small objects or attributes, rather than a language model that answers a question after vision encoding has finished. ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints Work: Which layer, token, subspace or circuit changes, and does the intervention preserve unrelated capabilities?
Exit evidence: A probe may measure accessibility rather than semantic understanding; intervention and natural distribution shift are different tests.
2025–2026
Long-video memory, scripts and entity continuity
Retrieval-based episodic memory, audio-visual script generation and entity-centric evaluation answer different long-video questions. Track what is stored, how evidence is retrieved, and whether identities and events remain consistent across time.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video UnderstandingECCV 2026 Oral verified in the official program (event 6047); scientific claims use the conference-hosted manuscript #10831. Publisher proceedings/version of record was not verified on 2026-09-12. MERIT uses multi-key episodic memory for ultra-long video QA: preserve short, detailed episodes when building memory, then compose query-specific temporal context only after the question is known. Each episode has event, dialogue, object, and summary keys; retrieval uses its strongest matching key, followed by local temporal expansion and question-aware filtering. ECCV program · Conference manuscript · Author project · Code Work: Can the system recover the right episode and preserve who did what, when, and in which scene?
Exit evidence: Compressed memory can omit decisive events; fluent scripts and pooled benchmark scores can conceal identity or temporal failures.
2026
Prompt-conditioned audiovisual embeddings
WAVE combines speech and sound pathways, layer-wise non-text fusion and joint retrieval/QA embedding training. Its output is similarity ranking, not unrestricted text generation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMOfficial ICLR 2026 oral and full proceedings manuscript checked. Multimodal embeddings and candidate-answer ranking, not unrestricted report generation. Official paper and released training configuration checked; unequal pooling contracts, negative findings and unresolved overall-score discrepancy are documented. No WSI evaluation. WAVE turns a multimodal generative backbone into a prompt-conditioned embedding model for text, video, audio and audiovisual inputs. Its central design combines complementary speech and sound encoders with hierarchical feature fusion across LLM layers. Retrieval and multiple-choice QA are both formulated as embedding comparisons, not unrestricted answer generation. Official proceedings record · OpenReview · Official implementation Work: How do the task prompt and pooling rule change what information the embedding retains?
Exit evidence: Modality coverage is finite; question-conditioned candidate accuracy is not evidence of report-generation reliability.
2026
Separate concept recognition from execution of logical queries
LCSE keeps a dual encoder frozen, calibrates concept scores, evaluates an explicit soft logical expression and applies a correction to the holistic image–text score. It improves tested conjunction, disjunction and negation decisions without retraining the encoder.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 10.01 Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models LCSE separates concept evidence from constraint execution in frozen dual-encoder VLMs. It parses a query into concepts and logical operators, scores concept presence, applies polarity-aware aggregation, and edits the original image–text similarity. This is an inference interface, not a newly trained visual encoder or a guarantee of exact logical consistency. Proceedings PDF · Author project Work: Is a retrieval failure caused by missing visual concepts or by composing recognized concepts incorrectly?
Exit evidence: Calibration uses labeled data and parsing is a separate dependency. The holistic residual retains many logical-equivalence violations. Proposed pathology queries need new tile/slide-level labels and evaluation; no pathology transfer was demonstrated.
2026
Retrieve evidence inside the answer-generating model
URaG uses early shared MLLM states for ColBERT-style page ranking, retains selected page tokens and all query tokens, then continues deeper generation. Its staged retrieval and LoRA training connect evidence selection to answer modeling without a separate retrieval backbone.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 11.01 URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document UnderstandingAAAI 2026 proceedings paper; FLOP reductions are not measured latency. Code repository states non-commercial research use. URaG makes page retrieval an internal operation of a multimodal language model. Early layers contextualize the full document and question; a lightweight head ranks pages, and deeper layers see only selected page tokens plus the query. This preserves fine-grained tokens on retained pages while removing irrelevant pages, rather than compressing every page uniformly or running an independent external retriever. Proceedings PDF · Author code and usage terms Work: Where should evidence be selected so detailed tokens survive without paying full deep-layer cost?
Exit evidence: Joint training always keeps ground-truth evidence pages; inference must select them. FLOP tests extend documents by duplication and are not latency measurements. Multi-page reasoning and WSI transfer need separate evidence.
Research branches
Global dual-encoder retrieval
Can one compact vector per item preserve the relevance relation, and which negatives should constrain it?
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives · Learning Transferable Visual Models From Natural Language Supervision · LiT: Zero-Shot Transfer with Locked-image Text Tuning · TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance · Sigmoid Loss for Language Image Pre-Training
Multimodal shared spaces
Which modality pairs are observed directly and which are bridged indirectly?
Multi-vector late interaction
Which query and item tokens survive indexing, and how is their interaction aggregated?
ColPali: Efficient Document Retrieval with Vision Language Models
Task-conditioned embedding and reranking
Does the instruction change representation, similarity, or only the reranker?
VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks · Qwen3-VL-Embedding: Advancing Multimodal Retrieval and Reranking
VideoLLM embeddings and open video corpora
Does the system learn a reusable retrieval representation, and which properties of the indexed video corpus make that claim reproducible?
ViLL-E: Video LLM Embeddings for Retrieval · LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training
Recovering, steering and testing visual concepts
Which layer, token, subspace or circuit changes, and does the intervention preserve unrelated capabilities?
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features · Steerable Visual Representations
Long-video memory, scripts and entity continuity
Can the system recover the right episode and preserve who did what, when, and in which scene?
Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
Prompt-conditioned audiovisual embeddings
How do the task prompt and pooling rule change what information the embedding retains?
WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM
Separate concept recognition from execution of logical queries
Is a retrieval failure caused by missing visual concepts or by composing recognized concepts incorrectly?
Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models
Retrieve evidence inside the answer-generating model
Where should evidence be selected so detailed tokens survive without paying full deep-layer cost?
URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Single-vector embedding | Precomputes compact corpus vectors for efficient search and can be trained by ranking, contrastive, or sigmoid objectives. | Hard-negative noise and compression can discard fine-grained relevance or teacher alignment. | 2018 · 2021 · 2023 · 2023 |
| Data/filter recipe | Changes what semantic relations enter the representation. | Duplicates and opaque filters contaminate evaluation. | 2023 · 2024 |
| Multi-vector index | Keeps token-level item vectors and applies late interaction with the query. | Higher index size and interaction cost can be hidden behind retrieval scores. | 2025 |
| Instruction / reranker | Uses task context or cross-item interaction after candidate retrieval. | Reranking cost and first-stage recall can be hidden. | 2025 · 2026 |
| VideoLLM embeddings and open video corpora | ViLL-E adds adaptive embeddings and staged generative/contrastive training to a VideoLLM; LAION-BVD supplies a large indexed video source pool, captions, tooling, and released models. | Retrieval gains can conflate head design with data scale, while web-video rights, unavailable URLs, caption noise, duplication, and benchmark overlap can undermine a reproducible open-corpus claim. | 2026 · 2026 |
| Recovering, steering and testing visual concepts | Intermediate features can expose information hidden by the final representation. Compare frozen-backbone negation adaptation, language-directed representations, circuit interventions and geometry-subspace probes; distinguish information accessibility from causal control. | A probe may measure accessibility rather than semantic understanding; intervention and natural distribution shift are different tests. | 2026 · 2026 |
| Long-video memory, scripts and entity continuity | Retrieval-based episodic memory, audio-visual script generation and entity-centric evaluation answer different long-video questions. Track what is stored, how evidence is retrieved, and whether identities and events remain consistent across time. | Compressed memory can omit decisive events; fluent scripts and pooled benchmark scores can conceal identity or temporal failures. | 2026 |
| Prompt-conditioned audiovisual embeddings | WAVE combines speech and sound pathways, layer-wise non-text fusion and joint retrieval/QA embedding training. Its output is similarity ranking, not unrestricted text generation. | Modality coverage is finite; question-conditioned candidate accuracy is not evidence of report-generation reliability. | 2026 |
| Separate concept recognition from execution of logical queries | LCSE keeps a dual encoder frozen, calibrates concept scores, evaluates an explicit soft logical expression and applies a correction to the holistic image–text score. It improves tested conjunction, disjunction and negation decisions without retraining the encoder. | Calibration uses labeled data and parsing is a separate dependency. The holistic residual retains many logical-equivalence violations. Proposed pathology queries need new tile/slide-level labels and evaluation; no pathology transfer was demonstrated. | 2026 |
| Retrieve evidence inside the answer-generating model | URaG uses early shared MLLM states for ColBERT-style page ranking, retains selected page tokens and all query tokens, then continues deeper generation. Its staged retrieval and LoRA training connect evidence selection to answer modeling without a separate retrieval backbone. | Joint training always keeps ground-truth evidence pages; inference must select them. FLOP tests extend documents by duplication and are not latency measurements. Multi-page reasoning and WSI transfer need separate evidence. | 2026 |
Concept lineage
VSE++: Improving Visual-Semantic Embeddings with Hard NegativesLearning Transferable Visual Models From Natural Language Supervision
Mini-batch hard-negative max-margin retrieval gives way to symmetric large-batch contrastive classification at web scale.
Learning Transferable Visual Models From Natural Language SupervisionSigmoid Loss for Language Image Pre-Training
Shared-space contrastive retrieval is reformulated with independent pairwise sigmoid terms.
Learning Transferable Visual Models From Natural Language SupervisionTinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance
A web-scale dual encoder becomes a compact student through affinity mimicking and weight inheritance.
Learning Transferable Visual Models From Natural Language SupervisionColPali: Efficient Document Retrieval with Vision Language Models
One-vector image–text alignment is specialized into multi-vector page embeddings with token-level late interaction for document retrieval.
DataComp: In Search of the Next Generation of Multimodal DatasetsVLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Controlled data studies lead into broad task-conditioned embedding evaluation beyond one image–caption contract.
Failure modes to audit
- Using false negatives as known irrelevant items.
- Leaking duplicate items across query and corpus splits.
- Reporting only Recall@K without corpus size and latency.
- Tuning prompts or instructions on the test set.
- Comparing embeddings with unequal reranking stages.
Questions for every paper
- What exactly is a relevant pair?
- How was the corpus constructed?
- Which stage uses the instruction?
- What quality–latency point is reported?
Complete maintained paper route
Section titled “Complete maintained paper route”21 canonical readings · 2002–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 21 readings in maintained order
Cumulated Gain-Based Evaluation of IR Techniques
Kalervo Järvelin and Jaana Kekäläinen
Metadata checked
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler
Official artifacts: Official implementation
Metadata checked
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford et al.
Metadata checked
LiT: Zero-Shot Transfer with Locked-image Text Tuning
Xiaohua Zhai et al.
Metadata checked
TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance
Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Chen, Xinggang Wang, Hongyang Chao, and Han Hu
Official artifacts: Official code and checkpoints
Metadata checked
Sigmoid Loss for Language Image Pre-Training
Xiaohua Zhai et al.
Metadata checked
ImageBind: One Embedding Space To Bind Them All
Rohit Girdhar et al.
Metadata checked
DataComp: In Search of the Next Generation of Multimodal Datasets
Samir Yitzhak Gadre et al.
Metadata checked
Data Filtering Networks
Alex Fang et al.
Metadata checked
VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Ting Jiang et al.
Official artifacts: Official code · Project page
Metadata checked
ColPali: Efficient Document Retrieval with Vision Language Models
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo
Official artifacts: Official code · Official ViDoRe models and benchmark
Metadata checked
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
Michael Tschannen et al.
Official artifacts: Models
Metadata checked
Qwen3-VL-Embedding: Advancing Multimodal Retrieval and Reranking
Qwen Team
Official artifacts: Official code · Official project article
Metadata checked
ViLL-E: Video LLM Embeddings for Retrieval
Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei, Sheng Liu, Son Tran, and Mubarak Shah
Peer-reviewed ACL 2026 long paper and Outstanding Paper award recipient.
Official artifacts: Author manuscript · Official publication page
Metadata checked
LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training
Andreas Hochlehnert, Marianna Nezhurina, Mehdi Cherti, Andrej Radonjic, Thaddäus Wiedemer, Christoph Schuhmann, Romain Beaumont, Wieland Brendel, Bernhard Schölkopf, A. Sophia Koepke, Jenia Jitsev, and Matthias Bethge
arXiv preprint first submitted 25 August 2026; not peer reviewed as of the audit date.
Official artifacts: Official project · Official data tooling · Official URL dataset
Metadata checked
Steerable Visual Representations
Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano
ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.
Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints
Metadata checked
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Chen-Yi Lu, Yueh-Shao Chen, and Somali Chaterji
ECCV 2026 Oral verified in the official program (event 6084); scientific claims use the conference-hosted manuscript #10575. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV program · Conference manuscript · Author project
Metadata checked
Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
Yeeun Choi, Youngbeom Yoo, Joon-Young Lee, Hyolim Kang, and Seon Joo Kim
ECCV 2026 Oral verified in the official program (event 6047); scientific claims use the conference-hosted manuscript #10831. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV program · Conference manuscript · Author project · Code
Metadata checked
WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM
Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, Chao Zhang
Official ICLR 2026 oral and full proceedings manuscript checked. Multimodal embeddings and candidate-answer ranking, not unrestricted report generation. Official paper and released training configuration checked; unequal pooling contracts, negative findings and unresolved overall-score discrepancy are documented. No WSI evaluation.
Official artifacts: Official proceedings record · OpenReview · Official implementation
Metadata checked
Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models
Sultan Alshehri, Zhantao Yang, Han Zhang, and Marios Savvides
Official artifacts: Proceedings PDF · Author project
Metadata checked
URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding
Yongxin Shi, Jiapeng Wang, Zeyu Shan, Dezhi Peng, Zening Lin, and Lianwen Jin
AAAI 2026 proceedings paper; FLOP reductions are not measured latency. Code repository states non-commercial research use.
Official artifacts: Proceedings PDF · Author code and usage terms
Metadata checked