Skip to content
Computational Pathology Reading Roadmap

Image–text representation and fusion

Image–text pretraining can learn a common retrieval space, a joint fused representation, or both. These outputs are easily conflated. A dual encoder is efficient because images and texts can be indexed independently; a fusion encoder is expressive because every text token can interact with visual tokens, but it normally cannot precompute a single reusable similarity vector.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Individual ICLR oral-paper studies retain their 19 September 2026 source-check dates; the later discovery audit does not re-date that evidence.

ECCV 2026 conference additions and metadata checked through 2026-09-12; the broader literature audit boundary above is unchanged.

Prerequisite path

Section titled “Prerequisite path”
  1. Complete contrastive learning and vision-transformer material in computer vision.
  2. Be able to derive softmax cross-entropy, cosine similarity, and multi-head attention from the ML and NLP routes.
  3. Read the field overview and distinguish an embedding, patch token, fused state, and generated token.

At research level, you should be able to explain why batch construction, caption quality, duplicate removal, and prompt design are part of the learned model.

Output, supervision, and evidence taxonomy

Section titled “Output, supervision, and evidence taxonomy”
Architecture Output Main supervision Evidence retained
Global dual encoder one normalized vector per image/text paired contrastive or sigmoid binary loss global similarity; patch evidence is not guaranteed
Token-aware dual encoder word and patch vectors plus global score fine-grained/token contrast, distillation local similarities without full cross-modal fusion
Two-stream fusion modality-specific encoders plus cross-attention matching, masked tokens, QA/caption labels patch/region and word interactions
Single-stream fusion concatenated visual/text sequence masked modelling, matching, task losses joint states whose modality-specific reuse is limited
Contrastive captioner retrieval vectors and autoregressive text contrastive plus captioning loss global alignment and generative semantics
Multimodal vision encoder reusable visual features captions plus SSL/reconstruction/distillation global and, when explicitly trained, dense visual features

Method detail

Section titled “Method detail”

Symmetric contrastive alignment

Section titled “Symmetric contrastive alignment”

For normalized image vector v_i, text vector t_j, and temperature τ, CLIP optimizes image-to-text and text-to-image cross-entropy over a similarity matrix:

s(i,j) = v_iᵀ t_j / τ
L = 1/2 [CE(rows(s), paired_text) + CE(cols(s), paired_image)]

Every unpaired item in the batch is treated as a negative. Large batches provide many comparisons but create false negatives when different captions describe compatible content. Temperature, normalization, duplicate policy, and distributed gathering all change the objective.

SigLIP replaces the batch-wide softmax with pairwise sigmoid classification. It reduces dependence on global normalization and scales differently with batch size, but still inherits the scientific problem of deciding which pairs are positive or negative.

Fusion objectives

Section titled “Fusion objectives”

Common objectives answer different questions:

  • Image–text contrastive (ITC): can separate encoders retrieve the paired item?
  • Image–text matching (ITM): can a fused encoder distinguish a matched pair from a hard negative?
  • Masked language/image modelling: can context reconstruct hidden tokens?
  • Captioning/language modelling: can visual states condition the next text token?
  • Self-distillation/reconstruction: can the vision branch retain transferable local structure in addition to language alignment?

A paper combining losses must report their weights, sampling, negative miner, and which modules receive each gradient. “Multitask pretraining” alone is not a reproducible method.

Staged paper route

Section titled “Staged paper route”

Joint fusion before web-scale dual encoders

Section titled “Joint fusion before web-scale dual encoders”
Order Primary paper and year Venue / status Read it for
1 ViLBERT (2019) NeurIPS, peer reviewed two visual/language streams connected by co-attention
2 LXMERT (2019) EMNLP, peer reviewed object, relation, language, and cross-modal pretraining losses
3 VisualBERT (2020) ACL, peer reviewed a single Transformer over region and word embeddings
4 UNITER (2020) ECCV, peer reviewed conditional masking, matching, optimal-transport alignment, and hard negatives
5 OSCAR (2020) ECCV, peer reviewed object tags as semantic anchors between region features and language
6 ViLT (2021) ICML, peer reviewed removing detector regions and convolutions to expose the cost of modality-specific visual processing

These systems typically depend on detector-produced regions. Record the detector, label vocabulary, frozen status, and whether region extraction itself used target benchmark categories.

Scalable separate encoders

Section titled “Scalable separate encoders”
Order Primary paper and year Venue / status Read it for
1 DeViSE (2013) NeurIPS, peer reviewed the visual-semantic embedding and ranking-loss precursor to modern dual encoders
2 VSE++ (2018) BMVC, peer reviewed selecting the hardest mismatched item in a mini-batch for max-margin image–text retrieval
3 CLIP (2021) ICML, peer reviewed symmetric contrastive alignment and text-defined zero-shot classifiers
4 ALIGN (2021) ICML, peer reviewed scaling noisy alt-text pairs with a simple dual encoder
5 LiT (2022) CVPR, peer reviewed locking a strong image tower while learning text alignment
6 FILIP (2022) ICLR, peer reviewed late interaction between image patches and text tokens
7 SLIP (2022) ECCV, peer reviewed combining language supervision with image-only self-supervision
8 SigLIP (2023) ICCV, peer reviewed pairwise sigmoid loss and scaling behavior

Alignment plus fusion or generation

Section titled “Alignment plus fusion or generation”
Order Primary paper and year Venue / status Read it for
1 ALBEF (2021) NeurIPS, peer reviewed alignment before fusion, momentum teacher, and hard-negative mining
2 X-VLM (2022) ICML, peer reviewed aligning text with visual concepts at image, region, and object-label granularity before downstream fusion
3 BLIP (2022) ICML, peer reviewed ITC, ITM, language modelling, and caption bootstrapping/filtering
4 FLAVA (2022) CVPR, peer reviewed unimodal and multimodal objectives in one foundation model
5 CoCa (2022) TMLR, peer reviewed contrastive representation and caption generation from one model
6 GroupViT (2022) CVPR, peer reviewed whether text-supervised grouping can make dense semantic regions emerge
7 ImageBind (2023) CVPR, peer reviewed binding image, text, audio, depth, thermal, and IMU through image-paired data
8 BEiT-3 (2023) CVPR, peer reviewed a Multiway Transformer and masked-data objective shared across vision, language, and paired inputs

Data and modern encoder recipes

Section titled “Data and modern encoder recipes”
Order Primary paper and year Venue / status Read it for
1 LAION-5B (2022) NeurIPS Datasets and Benchmarks, peer reviewed the access, provenance, deduplication, and governance questions created by open web-scale pairs
2 DataComp (2023) NeurIPS Datasets and Benchmarks, peer reviewed controlled data-pool, filtering, and compute-scale experiments
3 MetaCLIP (2024) ICLR, peer reviewed metadata-balanced data curation as a controlled alternative to opaque filtering
4 Data Filtering Networks (2024) ICLR, peer reviewed learning a data-quality filter rather than using similarity alone
5 TinyCLIP (2023) ICCV, peer reviewed affinity mimicking and weight inheritance for compact CLIP retrieval; measure retained bidirectional retrieval and actual latency
6 MobileCLIP (2024) CVPR, peer reviewed distillation and reinforced datasets for efficient encoders
7 SigLIP 2 (2025) arXiv preprint captioning, self-distillation, masked prediction, online curation, multilingual and dense transfer
8 AIMv2 (2025) CVPR, peer reviewed autoregressive prediction of image patches and text tokens
9 COSMOS (2025) CVPR, peer reviewed local/global image–text views and cross-modal self-distillation
10 TIPSv2 (2026) CVPR, peer reviewed patch-level distillation for dense patch–text alignment
11 FALCON (2026) CVPR, peer reviewed learned scheduling of negative hardness versus false-negative risk

Reading and reproduction record

Section titled “Reading and reproduction record”

For every representation paper, capture:

Axis Required record
Data source, pair count, languages, licenses, filtering, deduplication, synthetic captions
Image tower architecture, resolution/aspect policy, initialization, frozen/trainable
Text tower tokenizer, context length, architecture, initialization
Loss exact positives/negatives, temperature, loss weights, teacher updates
Batch local/global batch, distributed feature gathering, hard-negative policy
Output global vector, word/patch vectors, fused tokens, generative decoder
Transfer zero-shot prompts, frozen probe, fine-tuning, dense prediction
Evidence nearest neighbors, token/patch localization, counterfactual sensitivity
Access exact checkpoint, preprocessing transform, code, weights, license

Controlled experiment

Section titled “Controlled experiment”

Train small CLIP-style and SigLIP-style models on exactly the same curated subset. Use identical towers, augmentations, optimizer, number of observed pairs, and compute. Vary only:

  1. symmetric softmax versus pairwise sigmoid loss;
  2. random batches versus semantic-neighbor batches;
  3. one caption versus multiple compatible captions per image.

Evaluate image↔text Recall@1/5/10, zero-shot classification with a frozen prompt list, calibration, nearest-neighbor metadata enrichment, and dense patch–text localization. Report false-negative rate estimated from human review of the hardest “negative” pairs. Repeat with one held-out data source.

Failure modes

Section titled “Failure modes”
  • Web captions mention context, photographer intent, or metadata absent from pixels.
  • Duplicate images in different splits make retrieval appear easier.
  • A large batch adds more negatives but also more semantically valid false negatives.
  • Prompt selection is tuned on the test classes.
  • Data/filter quality and model objective change together, defeating causal claims.
  • Global alignment improves recognition while degrading precise spatial features.
  • A fusion model is compared with a dual encoder at unequal resolution or compute.
  • Closed training data makes contamination and subgroup analysis impossible.
  • “Open weights” is reported as “open model” despite unavailable data or training code.

Self-check and exit task

Section titled “Self-check and exit task”
  1. Derive both directions of CLIP loss and identify how a false negative contributes.
  2. Explain how SigLIP changes normalization without eliminating pair-label errors.
  3. When is late token interaction cheaper than full cross-attention?
  4. Which objectives in ALBEF, BLIP, and CoCa produce reusable embeddings?
  5. What evidence would demonstrate that a patch embedding, not only a class token, is language aligned?

Exit task: reproduce one dual-encoder baseline, document every pairing/filtering decision, and write a model card separating global retrieval, zero-shot recognition, dense localization, and out-of-domain evidence.

Bridge to computational pathology

Section titled “Bridge to computational pathology”

Pathology reports usually describe a specimen, slide, or case rather than one random tile. Therefore a tile–sentence “positive” can be weak or false even when patient IDs match. Before importing a CLIP recipe, define the pairing level, stain and magnification, sampling distribution, report section, and negative hierarchy (tile/slide/case/patient/site). Compare a natural-image encoder and a pathology encoder under the same preprocessing and downstream protocol, then continue to pathology VLMs and foundation encoders.

Readable prompt ensembles, local concepts and selective adaptation

TPB reweights mistakes across discrete prompt classifiers, CFM exposes spatially grounded sparse concepts, and Robotic Steering selects action-predictive attention heads. Compare what is selected or trained, how frozen features are reused, and whether the final intervention preserves unrelated capabilities.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
AdaBoosting Text Prompts for Vision-Language ModelsSeokhee Jin, Changhwan Sung, Sunung Mun, Hoyoung Kim, and Jungseul Ok2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #5256 checked on 2026-09-12. Publisher version of record was not independently verified.Text Prompt Boosting (TPB) adapts a frozen CLIP-style classifier by sequentially constructing readable prompt banks that focus on previously misclassified images. The weak learners are entire class-wise text-prompt classifiers; their confidence outputs are combined through SAMME.R boosting. The result is an error-correcting ensemble in language space, not a smaller student trained to copy teacher features.ECCV Spotlight program · Conference manuscript · Official project or implementation
CFM: Language-aligned Concept Foundation Model for VisionKai Wittenmayer, Sukrut Rao, Amin Parchami-Araghi, Bernt Schiele, and Jonas Fischer2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #4902 checked on 2026-09-12. Publisher version of record was not independently verified.CFM converts dense language-aligned visual features into spatially grounded sparse concepts. A Matryoshka sparse autoencoder reconstructs patch features at nested dictionary capacities, separating coarse concepts from finer residual detail. Data-derived parent-child relationships then improve concept naming and support interpretable classification, captioning and open-vocabulary segmentation.ECCV Spotlight program · Conference manuscript · Official project or implementation

Reading question: Is the adaptation unit a prompt bank, a concept dictionary, an attention head, or another trainable branch?

Evidence boundary: Frozen backbones do not imply label-free or zero-cost adaptation. Probe predictability is not causal proof, and Robotic Steering also trains FFN LoRA and its action head.

Frequency-domain visual token budgets

Fourier Compressor uses fixed DCT truncation on feature grids, followed by VLM adaptation. Distinguish parameter-free compression from training-free deployment, and average retention from fine-detail fidelity.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language ModelsHuanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, Zhouhan Lin2026 · ECCV · acceptedECCV 2026 spotlight verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Parameter-free compression is NOT training-free: Section 5.1 and Table 2 require adaptation. The 31.2% number concerns reduced time to first token, not a measured 31.2% increase in autoregressive decoding throughput.Fourier Compressor removes visual tokens by shrinking their spatial frequency representation. It applies a two-dimensional type-II DCT independently across feature channels, keeps a low-frequency rectangle, and applies an inverse DCT on the smaller grid. The compressor adds no learned parameters, but the surrounding VLM is trained to accommodate the altered features.Official ECCV spotlight record · Official code

Reading question: Which frequencies and tasks lose evidence at a matched training and token budget?

Evidence boundary: TextVQA and video subtasks can regress despite strong average retention; TTFT is not decoding throughput.

Teacher-class mixtures and privileged residual routing

MoE-KD reformulates classification through latent teacher-class experts and EM. MARS uses full-minus-partial features to teach a deployable router. Specify whether knowledge resides in label mixtures, prototypes, routing decisions or a residual target.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Residual-Guided Expert Specialization for Incomplete Multimodal LearningSeunghun Baek, Jihwan Park, Jaeyoon Sim, Minjae Jeong, Hoseok Lee, Won Hwa Kim2026 · ECCV · acceptedECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. MARS distills a privileged residual router into an available-feature router; it is not a cascade of residual-error student networks. Full modalities are assumed available during training. Food-101 full-modality accuracy is below SimMLM despite higher missing-modality and average scores.MARS uses the difference between complete- and partial-modality representations to teach expert routing. A privileged residual router learns which experts address each missingness pattern; a second router learns to imitate those decisions from available features alone. Thus the residual is a training signal, while deployment uses a practical feature router without the missing modalities.Official ECCV poster record · Official code

Reading question: Which privileged information is available only during training, and exactly what remains in the student?

Evidence boundary: Residual-guided routing is not a residual-error student cascade; conditional optimization arguments do not certify best accuracy.

Recovering, steering and testing visual concepts

Intermediate features can expose information hidden by the final representation. Compare frozen-backbone negation adaptation, language-directed representations, circuit interventions and geometry-subspace probes; distinguish information accessibility from causal control.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate FeaturesChen-Yi Lu, Yueh-Shao Chen, and Somali Chaterji2026 · ECCV · acceptedECCV 2026 Oral verified in the official program (event 6084); scientific claims use the conference-hosted manuscript #10575. Publisher proceedings/version of record was not verified on 2026-09-12.PeakPatch recovers negation cues from frozen intermediate CLIP text features instead of updating the encoders. A learned embedding residual restores distinctions between affirmative and negated captions; a separate, bounded score correction improves image-caption decisions. The contribution is a diagnosis-plus-adapter design for compositional failure, not a new image encoder or an inference-time language parser.ECCV program · Conference manuscript · Author project
Steerable Visual RepresentationsJona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano2026 · ECCV · acceptedECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.SteerViT lets text change the visual representation while the image is being encoded. A frozen ViT receives lightweight gated cross-attention to a frozen language encoder, producing prompt-aware global features and dense patch features. The objective is a reusable vision-centric representation that can be directed toward small objects or attributes, rather than a language model that answers a question after vision encoding has finished.ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints

Reading question: Which layer, token, subspace or circuit changes, and does the intervention preserve unrelated capabilities?

Evidence boundary: A probe may measure accessibility rather than semantic understanding; intervention and natural distribution shift are different tests.

Historical adapter anchors and analytic continual classifiers

Historical LoRA anchors constrain encoder drift while text-guided accumulated statistics update a classifier without raw-example replay. This is a teacher-state and classifier-memory contract, not the residual-error cascade used by other KD methods.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual LearningQier Meng, De Cheng, Jiahao Li, and Cheng Deng2026 · ECCV · acceptedECCV 2026 Spotlight verified via the official related presentation event; the poster identity itself says Accept Poster. No Springer version of record or public runnable implementation was verified. Section 3.5's printed closed-form equation omits a term implied by its objective; the review flags this reproducibility issue without silently repairing the authors' formula.Multi-anchor distillation restrains accumulated CLIP feature drift while a text-guided analytic classifier avoids gradient-based overwriting of classifier weights. Historical LoRA states supply visual anchors; accumulated sufficient statistics and frozen text prototypes supply the classifier. This exemplar-free approach stores model state and statistics rather than old raw images.Official ECCV 2026 Spotlight

Reading question: Which stored states prevent forgetting, and does the analytic argument still apply when the feature encoder changes?

Evidence boundary: Stable-feature equivalence is conditional; check the printed objective/closed-form mismatch and separate average from final accuracy.

Description-mediated transfer for image quality

RALI transfers teacher quality descriptions into a CLIP encoder, then fits compressed score bases. Read its image-only deployment path separately from the paper’s multi-domain RACT procedure.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality AssessmentShijie Zhao, Xuanyu Zhang, Weiqi Li, Junlin Li, Li Zhang, Tianfan Xue, Jian Zhang2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Teacher-generated quality descriptions align a compact image encoder; deployed scoring uses PCA and fitted bases without an LLM. Keep RALI separate from multi-domain RACT. Dataset-specific trade-offs, causal-interpretation limits and the conflicting memory-reduction claim are qualified. No WSI or cross-magnification evaluation.RALI transfers quality-specific descriptions from a reasoning-based image quality assessment teacher into a CLIP image encoder. The deployed scorer does not generate a reasoning trace or load the teacher LLM: it compares a compressed image embedding with learned quality bases and combines their scalar scores.Official proceedings record · OpenReview · Official implementation

Reading question: Which teacher information is transferred, which modules disappear at inference, and which domains lose performance?

Evidence boundary: Comparable aggregate IQA performance is not universal replacement or WSI validation; account for offline description generation and the measured memory ratio.

Prompt-conditioned audiovisual embeddings

WAVE combines speech and sound pathways, layer-wise non-text fusion and joint retrieval/QA embedding training. Its output is similarity ranking, not unrestricted text generation.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMChangli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, Chao Zhang2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Multimodal embeddings and candidate-answer ranking, not unrestricted report generation. Official paper and released training configuration checked; unequal pooling contracts, negative findings and unresolved overall-score discrepancy are documented. No WSI evaluation.WAVE turns a multimodal generative backbone into a prompt-conditioned embedding model for text, video, audio and audiovisual inputs. Its central design combines complementary speech and sound encoders with hierarchical feature fusion across LLM layers. Retrieval and multiple-choice QA are both formulated as embedding comparisons, not unrestricted answer generation.Official proceedings record · OpenReview · Official implementation

Reading question: How do the task prompt and pooling rule change what information the embedding retains?

Evidence boundary: Modality coverage is finite; question-conditioned candidate accuracy is not evidence of report-generation reliability.

Locate visual concepts and align representations across models

PatchSAE examines spatial visual features and how adaptation changes their use. Universal SAE learns cross-model sparse correspondences from representations of shared images. The first audits adaptation; the second aligns feature spaces, rather than training a distilled student.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Sparse autoencoders reveal selective remapping of visual concepts during adaptationHyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider2025 · ICLR · peer-reviewedPatchSAE turns intermediate CLIP image-token activations into a sparse concept dictionary with spatial locations. It then uses the same dictionary to compare CLIP with prompt-adapted MaPLe: much of the observed adaptation benefit comes from remapping already recognized concepts to task labels, rather than discovering an entirely new visual vocabulary.Proceedings PDF · Code and demo
Universal Sparse Autoencoders: Interpretable Cross-Model Concept AlignmentHarrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos G. Derpanis2025 · ICML · peer-reviewedUniversal Sparse Autoencoders learn a shared sparse concept space across multiple frozen vision models. Each model keeps its own encoder and decoder, but a code obtained from any one model must reconstruct the activations of all the others on the same input. This aligns concept coordinates during learning instead of matching independently discovered dictionaries afterward.Proceedings PDF · Code · Project

Reading question: Which concepts persist across image locations, adaptation settings, and encoders?

Evidence boundary: Readable examples and cross-decoding do not prove a complete or unique concept basis. Preserve the source-specific activation unit and the actual downstream intervention.

Test semantic coherence, concept supervision, and hierarchical steering

Hierarchical CLIP SAEs and VLM SAE studies connect feature interpretation with retrieval, bias analysis, or interventions on generated descriptions. Concept Bottleneck SAE adds supplied concept structure; SAE++ builds a hierarchy over dictionary atoms. Human alignment, control strength, and preserving unrelated content are separate questions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Interpreting CLIP with Hierarchical Sparse AutoencodersVladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek2025 · ICML · peer-reviewedZaigrajew et al. multi-TopK CLIP method, distinct from Bussmann et al. fixed-prefix Matryoshka SAE. Author repository discloses a post-acceptance text-standardization correction; review distinguishes proceedings measurements from revised code.Zaigrajew and colleagues train a CLIP sparse autoencoder at several activation-ranked TopK budgets simultaneously. Every budget uses the same encoder and decoder, but retains a different number of the current input’s strongest activations. This multi-budget objective seeks useful reconstructions from coarse through richer sparse descriptions; it is not the fixed-prefix Matryoshka SAE of Bussmann and colleagues.Proceedings PDF · Author code and correction notice · Original ICML implementation · arXiv
Sparse Autoencoders Learn Monosemantic Features in Vision-Language ModelsMateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata2025 · NeurIPS · peer-reviewedNeurIPS 2025 proceedings volume 38; the repository BibTeX uses publication year 2026. Catalog year follows the conference edition.This work evaluates whether sparse vision features are semantically coherent and whether manipulating them changes a VLM output. Its MonoSemanticity score measures activation-weighted image similarity and is checked against human pairwise judgments; separate CLIP-feature interventions steer LLaVA. Interpretability and controllability are related experiments, not interchangeable definitions.Proceedings PDF · Code and human-comparison benchmark · arXiv
Interpretable and Steerable Concept Bottleneck Sparse AutoencodersAkshay Kulkarni, Tsui-Wei Weng, Vivek Narayanaswamy, Shusen Liu, Wesam A. Sakla, and Kowshik Thopalli2026 · CVPR · peer-reviewedCB-SAE combines an unsupervised sparse dictionary with a user-specified concept bottleneck. It scores existing neurons for interpretability and downstream steerability, prunes low-utility neurons, then trains a parallel concept autoencoder while freezing retained SAE weights. This recovers useful reconstruction and supplies concepts missing from the retained dictionary.Proceedings PDF · Supplement · Author repository (code not yet released) · arXiv
SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMsYusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, and Hao Wang2026 · NeurIPS · acceptedNeurIPS 2026 Accept (poster) verified from official program event 151133, OpenReview cl4vXMku1D, matching all five authors. Program title omits SAE++ and joins LearnMulti-Level; review uses arXiv v3 revised 7 October 2026. Proceedings version not yet verified.SAE++ learns a hierarchy by training a second SAE on the decoder atoms of a first SAE. Level 1 decomposes model activations into sparse features; Level 2 groups the learned feature directions into higher-level concepts. It is not a residual-error cascade and does not feed Level-1 sparse sample codes into a conventional stacked bottleneck.Reviewed manuscript v3 · Official NeurIPS poster · OpenReview · Code

Reading question: Can the target concept change while unrelated visual content and answer quality remain intact?

Evidence boundary: A semantic-coherence score is not a causal or clinical certificate. Report coverage, duplicate features, judge dependence, and both intended and collateral intervention effects.

Separate concept recognition from execution of logical queries

LCSE keeps a dual encoder frozen, calibrates concept scores, evaluates an explicit soft logical expression and applies a correction to the holistic image–text score. It improves tested conjunction, disjunction and negation decisions without retraining the encoder.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language ModelsSultan Alshehri, Zhantao Yang, Han Zhang, and Marios Savvides2026 · ICML · peer-reviewedLCSE separates concept evidence from constraint execution in frozen dual-encoder VLMs. It parses a query into concepts and logical operators, scores concept presence, applies polarity-aware aggregation, and edits the original image–text similarity. This is an inference interface, not a newly trained visual encoder or a guarantee of exact logical consistency.Proceedings PDF · Author project

Reading question: Is a retrieval failure caused by missing visual concepts or by composing recognized concepts incorrectly?

Evidence boundary: Calibration uses labeled data and parsing is a separate dependency. The holistic residual retains many logical-equivalence violations. Proposed pathology queries need new tile/slide-level labels and evaluation; no pathology transfer was demonstrated.

Complete maintained paper route

Section titled “Complete maintained paper route”

55 canonical readings · 2013–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 55 readings in maintained order

  35. #352025ICMLPeer reviewedL3

    Interpreting CLIP with Hierarchical Sparse Autoencoders

    Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek

    Zaigrajew et al. multi-TopK CLIP method, distinct from Bussmann et al. fixed-prefix Matryoshka SAE. Author repository discloses a post-acceptance text-standardization correction; review distinguishes proceedings measurements from revised code.

    Official artifacts: Proceedings PDF · Author code and correction notice · Original ICML implementation · arXiv

    Metadata checked

  43. #432026arXivPreprintL2–L3

    LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training

    Andreas Hochlehnert, Marianna Nezhurina, Mehdi Cherti, Andrej Radonjic, Thaddäus Wiedemer, Christoph Schuhmann, Romain Beaumont, Wieland Brendel, Bernhard Schölkopf, A. Sophia Koepke, Jenia Jitsev, and Matthias Bethge

    arXiv preprint first submitted 25 August 2026; not peer reviewed as of the audit date.

    Official artifacts: Official project · Official data tooling · Official URL dataset

    Metadata checked

  44. #442026ECCVAccepted · proceedings pendingL3

    Steerable Visual Representations

    Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano

    ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.

    Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints

    Metadata checked

  46. #462026ECCVAccepted · proceedings pendingL3

    Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual Learning

    Qier Meng, De Cheng, Jiahao Li, and Cheng Deng

    ECCV 2026 Spotlight verified via the official related presentation event; the poster identity itself says Accept Poster. No Springer version of record or public runnable implementation was verified. Section 3.5's printed closed-form equation omits a term implied by its objective; the review flags this reproducibility issue without silently repairing the authors' formula.

    Official artifacts: Official ECCV 2026 Spotlight

    Metadata checked

  48. #482026ECCVAccepted · proceedings pendingL3

    CFM: Language-aligned Concept Foundation Model for Vision

    Kai Wittenmayer, Sukrut Rao, Amin Parchami-Araghi, Bernt Schiele, and Jonas Fischer

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #4902 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  49. #492026ECCVAccepted · proceedings pendingL3

    Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models

    Huanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, Zhouhan Lin

    ECCV 2026 spotlight verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Parameter-free compression is NOT training-free: Section 5.1 and Table 2 require adaptation. The 31.2% number concerns reduced time to first token, not a measured 31.2% increase in autoregressive decoding throughput.

    Official artifacts: Official ECCV spotlight record · Official code

    Metadata checked

  50. #502026ECCVAccepted · proceedings pendingL3

    Residual-Guided Expert Specialization for Incomplete Multimodal Learning

    Seunghun Baek, Jihwan Park, Jaeyoon Sim, Minjae Jeong, Hoseok Lee, Won Hwa Kim

    ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. MARS distills a privileged residual router into an available-feature router; it is not a cascade of residual-error student networks. Full modalities are assumed available during training. Food-101 full-modality accuracy is below SimMLM despite higher missing-modality and average scores.

    Official artifacts: Official ECCV poster record · Official code

    Metadata checked

  51. #512026ICLRPeer reviewedL3

    WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM

    Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, Chao Zhang

    Official ICLR 2026 oral and full proceedings manuscript checked. Multimodal embeddings and candidate-answer ranking, not unrestricted report generation. Official paper and released training configuration checked; unequal pooling contracts, negative findings and unresolved overall-score discrepancy are documented. No WSI evaluation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked

  52. #522026ICLRPeer reviewedL3

    Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessment

    Shijie Zhao, Xuanyu Zhang, Weiqi Li, Junlin Li, Li Zhang, Tianfan Xue, Jian Zhang

    Official ICLR 2026 oral and full proceedings manuscript checked. Teacher-generated quality descriptions align a compact image encoder; deployed scoring uses PCA and fitted bases without an LLM. Keep RALI separate from multi-domain RACT. Dataset-specific trade-offs, causal-interpretation limits and the conflicting memory-reduction claim are qualified. No WSI or cross-magnification evaluation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked

  54. #542026NeurIPSAccepted · proceedings pendingL3

    SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

    Yusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, and Hao Wang

    NeurIPS 2026 Accept (poster) verified from official program event 151133, OpenReview cl4vXMku1D, matching all five authors. Program title omits SAE++ and joins LearnMulti-Level; review uses arXiv v3 revised 7 October 2026. Proceedings version not yet verified.

    Official artifacts: Reviewed manuscript v3 · Official NeurIPS poster · OpenReview · Code

    Metadata checked