Skip to content
Computational Pathology Reading Roadmap

Visual representation and foundation models

A visual representation maps an image or region into features that another model can reuse. The important questions are not only backbone size and benchmark score, but which invariances, targets, data, teachers, and resolution policy shaped the feature space.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Individual ICLR oral-paper studies retain their 19 September 2026 source-check dates; the later discovery audit does not re-date that evidence.

ECCV 2026 conference additions and metadata checked through 2026-09-12; the broader literature audit boundary above is unchanged.

Scaled encoders remain inside their objective families; there is no separate “recent foundation model” bucket.

Prerequisites and objectives

Section titled “Prerequisites and objectives”

Complete image recognition and understand the transformer block in the ML foundations. You should be able to:

  • derive patch and attention shapes and their resolution-dependent cost;
  • distinguish invariance, equivariance, alignment, reconstruction, and prediction;
  • explain collapse prevention in contrastive and negative-free objectives;
  • classify an encoder as label-supervised, image-only, image–text, autoregressive, teacher-distilled, or hybrid;
  • evaluate global and dense features under controlled adaptation.

Tokenized vision and backbone structure

Section titled “Tokenized vision and backbone structure”

For patch size P, an H × W image produces N = HW/P² patch tokens before optional class or register tokens. Standard self-attention is

Attention(Q, K, V) = softmax(QKᵀ / √d)V

with O(N²) pairwise scores. Doubling each spatial dimension produces roughly four times as many tokens and sixteen times as many attention scores. Windowing, hierarchy, sparse attention, token reduction, convolutions, or state-space scans change this trade-off.

Stage Primary reading · year · venue/status Architectural lesson
T1 An Image is Worth 16×16 Words — 2021 · ICLR, peer reviewed a minimally modified transformer can learn image representations when data and regularization are sufficient
T2 Training Data-Efficient Image Transformers — 2021 · ICML, peer reviewed strong augmentation, regularization, and distillation change the data requirement
T3 Swin Transformer — 2021 · ICCV, peer reviewed shifted local windows create a hierarchical dense-prediction backbone
T4 Pyramid Vision Transformer — 2021 · ICCV, peer reviewed spatial reduction and feature pyramids connect transformers to dense tasks
T5 A ConvNet for the 2020s — 2022 · CVPR, peer reviewed a modernized convolutional control is required before attributing gains to attention
T6 VMamba — 2024 · NeurIPS, peer reviewed 2D selective scans adapt a linear-time sequence mechanism to images
T7 MambaVision — 2025 · CVPR, peer reviewed hybrid state-space and attention blocks expose where each mixer helps

Asymptotic complexity is not measured latency. Kernel maturity, memory movement, token layout, batch size, numerical precision, and hardware determine realized speed.

Distillation as a controlled representation intervention

Section titled “Distillation as a controlled representation intervention”

Distillation is not one mechanism. These papers vary the transferred tensor, capacity bridge, augmentation consistency, and optimization contract; compare them with one fixed teacher–student pair before attributing gains to the method name.

Primary reading · year · venue/status Representation question to isolate
Improved Knowledge Distillation via Teacher Assistant — 2020 · AAAI, peer reviewed when does an intermediate-capacity teacher repair a teacher–student capacity gap?
Distilling Knowledge via Knowledge Review — 2021 · CVPR, peer reviewed how do cross-stage connections and hierarchical context move knowledge across feature levels?
A Good Teacher Is Patient and Consistent — 2022 · CVPR, peer reviewed how much of the gain comes from long training and identical teacher/student augmentations?
Decoupled Knowledge Distillation — 2022 · CVPR, peer reviewed what changes when target-class and non-target-class knowledge are optimized separately?
Masked Generative Distillation — 2022 · ECCV, peer reviewed can masked feature reconstruction transfer dense representations without rigid pointwise matching?
Learning Task-Agnostic Representations through Multi-Teacher Distillation — 2025 · NeurIPS, peer reviewed can a majority-vote objective combine heterogeneous teachers without a task-specific label contract?
Streamlined Knowledge Distillation — 2026 · CVPR, peer reviewed do instance-wise KL and direction-wise logit geometry suffice without redundant transfer objectives?

Report teacher and student training tokens, augmentations, architecture, parameter count, latency, calibration, and external transfer together. A smaller parameter count is not evidence of faster or safer deployment.

What a self-supervised target teaches

Section titled “What a self-supervised target teaches”

Let t₁, t₂ be two views of an image and z_i = g(f(t_i(x))). A contrastive objective makes paired views similar relative to other samples:

ℓ_i = −log[ exp(sim(z_i, z_i⁺) / τ) / Σ_j exp(sim(z_i, z_j) / τ) ]

The augmentation distribution is therefore part of the target. Cropping, color jitter, blur, masking, and multi-crop views specify what the model should ignore or preserve.

Spatial pretexts and instance discrimination

Section titled “Spatial pretexts and instance discrimination”
Stage Primary reading · year · venue/status Read it for
P1 Context Prediction — 2015 · ICCV, peer reviewed spatial context as a pseudo-label and the risk of low-level shortcuts
P2 Jigsaw Puzzles — 2016 · ECCV, peer reviewed part arrangement, permutation prediction, and shortcut prevention
P3 Rotation Prediction — 2018 · ICLR, peer reviewed transformation prediction and when orientation is semantic
P4 Non-Parametric Instance Discrimination — 2018 · CVPR, peer reviewed treating each image as a class and using a memory bank
P5 Contrastive Predictive Coding — 2018 · arXiv preprint InfoNCE and prediction of future latent structure

Contrast, prototypes, and negative-free learning

Section titled “Contrast, prototypes, and negative-free learning”
Family Primary reading · year · venue/status Mechanism to compare
queued contrast MoCo — 2020 · CVPR, peer reviewed momentum teacher plus a queue of negatives
large-batch contrast SimCLR — 2020 · ICML, peer reviewed projection head, augmentation composition, temperature, and batch negatives
labelled multi-positive contrast Supervised Contrastive Learning — 2020 · NeurIPS, peer reviewed same-class positives, batch composition, and a controlled comparison with cross-entropy
online prototypes SwAV — 2020 · NeurIPS, peer reviewed balanced cluster assignment and multi-crop learning
bootstrap target BYOL — 2020 · NeurIPS, peer reviewed online/target asymmetry without explicit negatives
stop-gradient Siamese SimSiam — 2021 · CVPR, peer reviewed collapse prevention from stop-gradient and predictor asymmetry
redundancy reduction Barlow Twins — 2021 · ICML, peer reviewed; VICReg — 2022 · ICLR, peer reviewed cross-correlation versus explicit variance and covariance regularization
self-distillation DINO — 2021 · ICCV, peer reviewed centering, sharpening, momentum teacher, multi-crop views, and emergent dense structure

Negative-free does not mean collapse-free by default. Stop-gradient, teacher updates, centering, predictors, variance terms, normalization, and data diversity are structural safeguards.

Masked and predictive representation learning

Section titled “Masked and predictive representation learning”

Masked models differ in their target: pixels, discrete visual tokens, teacher features, or latent context. A low reconstruction loss can prioritize texture rather than downstream semantics.

Stage Primary reading · year · venue/status Target and asymmetry
M1 BEiT — 2022 · ICLR, peer reviewed predict discrete tokenizer codes for masked patches
M2 Masked Autoencoders — 2022 · CVPR, peer reviewed encode visible patches and reconstruct pixels with a lightweight decoder
M3 iBOT — 2022 · ICLR, peer reviewed online tokenizer and self-distillation at patch and image levels
M4 I-JEPA — 2023 · CVPR, peer reviewed predict abstract target-block representations from visible context

For each method, write down the online encoder, target encoder or tokenizer, mask policy, prediction head, stop-gradient boundary, and components discarded after pretraining.

Scaling to reusable visual encoders

Section titled “Scaling to reusable visual encoders”

“Foundation” encoders use different supervision and should not be collapsed into one leaderboard.

Scientific branch Primary reading · year · venue/status What to record
curated image-only SSL DINOv2 — 2024 · TMLR, peer reviewed data curation, distillation, registers/layers, resolution, and frozen dense transfer
scaled image-only SSL DINOv3 — 2026 · TMLR, peer reviewed Gram anchoring, scale, post-hoc resolution/model adaptation, and global versus dense features
multi-teacher agglomeration RADIOv2.5 — 2025 · CVPR, peer reviewed teacher coverage, balancing, resolution modes, and token compression
multimodal autoregression AIMv2 — 2025 · CVPR, peer reviewed autoregressive image-patch and text targets, scale, and localization transfer
language-aligned image/video Perception Encoder — 2025 · NeurIPS, peer reviewed which intermediate layer is used and how language/spatial alignment is added
language-free scaling Scaling Language-Free Visual Representation Learning — 2025 · ICCV, peer reviewed data and model scale without captions; compare against equally scaled language-aligned controls

CLIP and SigLIP are essential language-aligned controls, but zero-shot class-name accuracy does not establish strong dense features. Conversely, image-only SSL does not automatically provide an open vocabulary.

Evaluation ladder

Section titled “Evaluation ladder”

Use several adaptation regimes because each asks a different question:

  1. nearest-neighbour retrieval and qualitative neighbourhoods;
  2. frozen k-NN and linear probes;
  3. low-shot and parameter-efficient adaptation;
  4. full fine-tuning under a fixed budget;
  5. dense transfer to detection, segmentation, depth, or correspondence;
  6. cross-dataset and corruption transfer;
  7. resolution and layer-selection sweeps;
  8. overlap, license, memory, latency, and energy audit.

Keep preprocessing and evidence fixed. An encoder with a stronger decoder, higher resolution, extra text, or more target labels has not won an encoder-only comparison.

Controlled lab

Section titled “Controlled lab”

Select one CNN, one ViT, and three pretrained objectives, for example supervised ResNet, SimCLR, DINO/DINOv2, and MAE.

  1. Freeze exact checkpoints and official normalization.
  2. Extract global and patch features from the same images and resolution.
  3. Run k-NN and linear probes with identical train/validation/test examples.
  4. Fine-tune under fixed label and update budgets.
  5. Attach the same lightweight segmentation or depth decoder to each frozen backbone.
  6. Measure accuracy, macro metrics, calibration, label efficiency, dense quality, peak memory, extraction time, and representation storage.
  7. Visualize nearest neighbours, patch correspondences, principal components, and failures; do not use an attractive attention map as quantitative grounding.

Failure modes

Section titled “Failure modes”
  • augmentation destroys information needed by the downstream task;
  • false negatives push semantically related images apart;
  • a negative-free model collapses or loses feature dimensions;
  • masked reconstruction learns low-level detail but weak semantics;
  • the chosen class token hides useful intermediate or patch features;
  • one method receives a larger input or a stronger task decoder;
  • web data overlap makes a frozen benchmark appear more transferable;
  • model scale, data scale, teacher supervision, and objective change together;
  • checkpoint preprocessing or license is misreported;
  • one global score hides dense-feature degradation.

Self-check and exit task

Section titled “Self-check and exit task”
  1. Why does doubling image height and width roughly multiply dense attention scores by sixteen?
  2. Which mechanism prevents collapse in BYOL, SimSiam, VICReg, and DINO?
  3. How do pixel, token, teacher-feature, and latent-prediction targets differ?
  4. Why can intermediate features outperform the final embedding for dense tasks?
  5. Which comparison isolates the pretraining objective from scale and decoder?

Exit task: create a representation scorecard with rows for supervision, data, objective, target, global/dense token, layer, resolution, frozen probe, low-shot, full tuning, dense transfer, shift, compute, and license. End with one claim the evidence supports and one it does not.

Pathology bridge

Section titled “Pathology bridge”

Pathology encoders inherit these objectives but change view validity, scale, stain, and evaluation. A color augmentation may erase evidence; random crops may exclude rare morphology; web-scale natural images may supply shape priors but not tissue semantics. Continue to pathology SSL and foundation encoders, using the same frozen/shallow/full adaptation ladder and patient-level overlap audit.

Counterexamples for spatial, semantic and action grounding

Use matched distance counterexamples, cross-category part correspondence, action-absent video clips and background-consistent concealment to test different kinds of visual evidence. Preserve each study's denominator: answer probability, correspondence success, absence/presence accuracy and newly introduced ungrounded nouns are not interchangeable.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation ModelsOlaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #349 checked on 2026-09-12. Publisher version of record was not independently verified.SOCO benchmarks whether a representation matches the correct object part, not merely the correct kind of part. Its taxonomy separates concept correspondence, identity-specific semantic object correspondence and cross-category transfer. Language descriptions also let VLMs be tested on text-guided localization versus visual-reference matching.ECCV Spotlight program · Conference manuscript · Official project or implementation

Reading question: Does an apparent improvement survive counterexamples that preserve superficial cues while changing the correct spatial, semantic or temporal conclusion?

Evidence boundary: A small bias gap can coexist with near-chance performance. Action denial is not safety refusal, detector-based concealment is not a privacy guarantee, and benchmark correlations are not universal causal explanations.

Spatial-frequency and response-token distillation targets

SFKD selectively aligns wavelet-localized Fourier amplitudes across backbone families. TRAG aligns response-to-vision attention using token-specific entropy-weighted KL. Both teach structure beyond final logits, but use different correspondence assumptions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral InteractionCuipeng Wang, Haipeng Wang2026 · ECCV · acceptedECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The linked official repository still says source code will be released soon. This is heterogeneous-architecture feature KD, not demonstrated cross-magnification transfer. The selected CIFAR-100 table includes a ViT-S→MobileNetV2 case where FBT is better.SFKD transfers knowledge between different backbone families by separating local spatial structure from global spectral information. It decomposes intermediate maps into Haar-wavelet subbands, refines the student bands through convolutional and Transformer streams, and matches Gaussian-filtered Fourier amplitudes to teacher bands. Reconstruction and logit supervision keep the transfer connected to the classification task.Official ECCV poster record · Official code

Reading question: Are the grids, semantic units and probability supports genuinely aligned before applying the loss?

Evidence boundary: SFKD has architecture-specific regressions; TRAG shares a visual encoder and resolution. Neither establishes cross-magnification performance.

Recovering, steering and testing visual concepts

Intermediate features can expose information hidden by the final representation. Compare frozen-backbone negation adaptation, language-directed representations, circuit interventions and geometry-subspace probes; distinguish information accessibility from causal control.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Steerable Visual RepresentationsJona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano2026 · ECCV · acceptedECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.SteerViT lets text change the visual representation while the image is being encoded. A frozen ViT receives lightweight gated cross-attention to a frozen language encoder, producing prompt-aware global features and dense patch features. The objective is a reusable vision-centric representation that can be directed toward small objects or attributes, rather than a language model that answers a question after vision encoding has finished.ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints
“World Knowledge” in the Weights: Reading Concept Circuits of Vision TransformersYanlin Chen, Tang Li, and Xi Peng2026 · ECCV · acceptedECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #8168. Publisher proceedings/version of record was not verified on 2026-09-12.VisionCLT adapts cross-layer transcoders to visualize how sparse concepts interact inside a ViT. It offers a global circuit assembled from learned encoder/decoder connections and an instance circuit based on active concepts for an image. These are complementary approximations to the model's computation, useful for testing shortcuts and interventions rather than merely highlighting salient pixels.ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript
Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace InterventionWeichen Zhou, Yawen Zou, Chunzhi Gu, Ran Dong, Haoran Xie, and Chao Zhang2026 · ECCV · acceptedECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #7101. Publisher proceedings/version of record was not verified on 2026-09-12.This study asks how geometry is encoded, not simply which backbone has the best depth score. It decomposes a trained linear probe by SVD, projects frozen ViT features into selected subspaces, and reuses the same frozen head. Matched aligned, random and residual projections test which directions carry the geometric information that this particular linear decoder can read.ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript

Reading question: Which layer, token, subspace or circuit changes, and does the intervention preserve unrelated capabilities?

Evidence boundary: A probe may measure accessibility rather than semantic understanding; intervention and natural distribution shift are different tests.

Masked generation and diffusion-friendly latent spectra

MIMFlow combines masked image modeling with normalizing flows; SSVAE changes the spectral structure of video latents to improve downstream diffusion. Separate tokenizer reconstruction quality, likelihood or generation objectives, and final sample quality.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image GenerationYang Chen, Xiaowei Xu, Shuai Wang, Xinwen Zhang, Qiushi Guo, Tiezheng Ge, and Limin Wang2026 · ECCV · acceptedECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #2272. Publisher proceedings/version of record was not verified on 2026-09-12.MIMFlow makes a masked visual tokenizer and its normalizing-flow prior learn together. A fixed 128-token bottleneck extracts global information from incomplete images; the invertible flow models those semantic latents, while a separate decoder reconstructs detailed pixels. The important change is where modeling capacity is spent, not simply making a larger flow.ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript

Reading question: Which representation is easier to model, and what fidelity or sampling cost is exchanged for that advantage?

Evidence boundary: Good reconstruction alone does not establish good diffusability, and normalizing flows are not interchangeable with flow matching.

Volume tokens, persistent geometry and registration

Compare learned volume representations, streaming geometric context, flow-based rigid registration and dense image matching. Each chooses a different correspondence unit and computation path; learned priors complement rather than abolish geometric constraints.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
RoMa v2: Harder Better Faster Denser Feature MatchingJohan Edstedt, David Nordström, Yushan Zhang, Georg Bökman, Jonathan Astermark, Anders Heyden, Viktor Larsson, Mårten Wadenbäck, Michael Felsberg, and Fredrik Kahl2026 · ECCV · acceptedAccepted to ECCV 2026; spotlight presentation verified in the official program. This record and review use the conference-hosted manuscript checked 2026-09-12; a separate Springer proceedings record was not verified. The first preprint is from 2025; 2026 reflects the verified conference. Author order and table values follow the ECCV-hosted manuscript, not the earlier repository BibTeX.RoMa v2 combines frozen DINOv3 features with a cross-view coarse matcher and separately trained subpixel refiners. It aims to retain foundation-feature generalization while improving difficult correspondences, throughput, and uncertainty estimation. The output is a dense warp with overlap confidence and predictive covariance, not a camera pose or a generated 3D scene by itself.Official ECCV spotlight presentation · Official ECCV paper record · arXiv record · Official code

Reading question: Which coordinates, features or correspondences are predicted, and how are rigidity, memory growth and distribution shift tested?

Evidence boundary: Compact per-frame context still accumulates; multi-step integration is not one-pass inference, and improved average matching can hide benchmark regressions.

Diffusion-boundary self-distillation and negative transfer

TRACE turns trajectory-derived attention boundaries into one-pass predictions through LoRA and an edge decoder. Read its natural-image gains alongside the reported MoNuSeg/TNBC degradation.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
TRACE: Your Diffusion Model Is Secretly an Instance Edge DetectorSanghyun Jo, Ziseok Lee, Wooyeol Lee, Jonghyun Choi, Jaesik Park, Kyungsu Kim2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Diffusion-boundary self-distillation improves selected natural-image segmentation but degrades MoNuSeg/TNBC cell segmentation. One-pass speed is edge extraction only, with the large backbone retained. No WSI or cross-magnification validation; official code/checkpoints remain promised rather than released at the checked revision.TRACE extracts instance-boundary cues from self-attention inside a pretrained diffusion model, then self-distills a slower trajectory-based edge procedure into one backbone pass. Natural-image mask refinement improves, but the paper also reports worse cell segmentation on MoNuSeg and TNBC.OpenReview · Official project · Official repository — release pending

Reading question: Does the teacher prior delineate the target objects before it is distilled, and what remains in the deployed model?

Evidence boundary: A tiny decoder does not remove the large diffusion backbone. Edge-extraction speed is not end-to-end speed, and negative cell-level transfer is not WSI validation.

Distributional geometry of contrastive representations

InfoNCE Induces Gaussian Distribution separates a plateau/projection argument from an explicitly regularized surrogate. Its diagnostic table is a lesson in testing, rather than assuming, representation distributions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
InfoNCE Induces Gaussian DistributionRoy Betser, Eyal Gofer, Meir Yossef Levi, Guy Gilboa2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Conditional theory and representation diagnostics, not proof that every InfoNCE encoder is jointly Gaussian. Separate fixed-dimensional projection limits from a regularized population surrogate and coordinate-wise non-rejection from joint normality.This paper investigates when contrastive representations have approximately Gaussian low-dimensional marginals. It gives two different analytical routes: an alignment/uniformity argument under plateau and concentration assumptions, and an explicitly regularized population surrogate. The title is broader than a theorem that arbitrary InfoNCE training always produces jointly Gaussian embeddings.Official proceedings record · OpenReview

Reading question: Which assumptions make the distributional claim possible, and what does a univariate normality test actually establish?

Evidence boundary: Coordinate-wise non-rejection is not joint Gaussianity; the paper does not prove that arbitrary SGD reaches the described optimum.

Predictive representations for reward-selected control

TD-JEPA connects non-contrastive latent prediction to temporal-difference learning. Read the state/action input contract and downstream task-vector inference before interpreting zero-shot control.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement LearningMarco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, Andrea Tirinzoni2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Reward-free state–action transitions, not action-free video. Zero-shot policy selection still requires reward observations for task-vector inference. Theoretical assumptions, benchmark-specific actor inputs and mixed OGBench comparisons are qualified in the source-checked review; no WSI evaluation.TD-JEPA learns state and task representations from reward-free transitions so that a new reward can select a policy without task-specific policy fine-tuning. Its distinguishing move is temporal-difference bootstrapping in a joint-embedding predictive architecture: predicting accumulated future task features connects representation learning to zero-shot control.Official proceedings record · OpenReview · Official implementation

Reading question: What future quantity is predicted, what prevents collapse, and what reward information is still needed at deployment?

Evidence boundary: Restricted theory and simulator-specific actor inputs do not prove arbitrary video or pathology-control validity.

Which feature directions should distillation preserve?

SpecMatch balances reconstruction along teacher principal components rather than letting high-variance directions dominate an L2 objective. Low variance need not mean low downstream value.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation DistillationKuniaki Saito and Yoshitaka Ushiku2026 · arXiv · preprintarXiv v1 checked 2 October 2026. No official code/model release or conference acceptance verified. Camelyon17 is patch-level evaluation, not whole-slide cross-magnification validation. PE teacher/student labels and parameter counts differ between main text/tables and Appendix C; this review uses unambiguous DINOv2-to-MobileNet Table 1 rows.SpecMatch changes how feature distillation allocates its learning signal. A fixed teacher PCA basis separates representation directions, and a sublinear function of each direction’s reconstruction error reduces the dominance of high-variance coordinates. The aim is to preserve useful low-variance information, not merely make the total feature MSE small.

Reading question: Does a smaller global feature error hide worse transfer on discriminative but low-variance directions?

Evidence boundary: Patch-level Camelyon17 is not a WSI endpoint. Spectral balancing is not evidence that unobserved high-magnification information can be recovered.

Locate visual concepts and align representations across models

PatchSAE examines spatial visual features and how adaptation changes their use. Universal SAE learns cross-model sparse correspondences from representations of shared images. The first audits adaptation; the second aligns feature spaces, rather than training a distilled student.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Sparse autoencoders reveal selective remapping of visual concepts during adaptationHyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider2025 · ICLR · peer-reviewedPatchSAE turns intermediate CLIP image-token activations into a sparse concept dictionary with spatial locations. It then uses the same dictionary to compare CLIP with prompt-adapted MaPLe: much of the observed adaptation benefit comes from remapping already recognized concepts to task labels, rather than discovering an entirely new visual vocabulary.Proceedings PDF · Code and demo
Universal Sparse Autoencoders: Interpretable Cross-Model Concept AlignmentHarrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos G. Derpanis2025 · ICML · peer-reviewedUniversal Sparse Autoencoders learn a shared sparse concept space across multiple frozen vision models. Each model keeps its own encoder and decoder, but a code obtained from any one model must reconstruct the activations of all the others on the same input. This aligns concept coordinates during learning instead of matching independently discovered dictionaries afterward.Proceedings PDF · Code · Project

Reading question: Which concepts persist across image locations, adaptation settings, and encoders?

Evidence boundary: Readable examples and cross-decoding do not prove a complete or unique concept basis. Preserve the source-specific activation unit and the actual downstream intervention.

Test semantic coherence, concept supervision, and hierarchical steering

Hierarchical CLIP SAEs and VLM SAE studies connect feature interpretation with retrieval, bias analysis, or interventions on generated descriptions. Concept Bottleneck SAE adds supplied concept structure; SAE++ builds a hierarchy over dictionary atoms. Human alignment, control strength, and preserving unrelated content are separate questions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Interpreting CLIP with Hierarchical Sparse AutoencodersVladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek2025 · ICML · peer-reviewedZaigrajew et al. multi-TopK CLIP method, distinct from Bussmann et al. fixed-prefix Matryoshka SAE. Author repository discloses a post-acceptance text-standardization correction; review distinguishes proceedings measurements from revised code.Zaigrajew and colleagues train a CLIP sparse autoencoder at several activation-ranked TopK budgets simultaneously. Every budget uses the same encoder and decoder, but retains a different number of the current input’s strongest activations. This multi-budget objective seeks useful reconstructions from coarse through richer sparse descriptions; it is not the fixed-prefix Matryoshka SAE of Bussmann and colleagues.Proceedings PDF · Author code and correction notice · Original ICML implementation · arXiv

Reading question: Can the target concept change while unrelated visual content and answer quality remain intact?

Evidence boundary: A semantic-coherence score is not a causal or clinical certificate. Report coverage, duplicate features, judge dependence, and both intended and collateral intervention effects.

Distillation projectors: dimension matching versus optimization interference

A projector can separate the student’s task representation from the space used to imitate a teacher even when both feature dimensions match. PEFD averages parallel nonlinear projector outputs before direction matching; the extended study adds logit, representation-similarity and calibration analyses.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Understanding the Effects of Projectors in Knowledge DistillationYudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, Brano Kusy, and Zi Huang2026 · IEEE TPAMI · peer-reviewedIEEE publisher metadata identifies TPAMI 48(8), 9078–9092, August 2026. The accessible scientific text checked here is arXiv:2310.17183v1 (26 October 2023); final journal numerical identity was not established. This extends the authors’ NeurIPS 2022 Projector Ensemble Feature Distillation work, rather than originating PEFD in 2026.A distillation projector can do more than resize a feature vector: it can buffer the conflict between learning the label task and matching a teacher. This study tests projectors even when dimensions already agree, extends the idea to logits, and analyzes a parallel projector ensemble. The scientific summary follows the accessible 2023 manuscript; the catalog also identifies the 2026 journal publication.Inspected author manuscript (v1) · Author code and earlier PEFD implementation

Reading question: Does the alignment head protect useful student features, or merely fit the teacher target more closely?

Evidence boundary: PEFD originated in 2022. The accessible 2023 manuscript supports the reviewed scientific details; final 2026 journal numerical identity was not verified. More projectors or greater depth do not guarantee improvement.

Global semantic anchors and local patch targets

Two Global Crops Suffice separates same-image global-view alignment from local crops and iBOT refinement in controlled DINO-family experiments. CODI instead changes the patch target: weakly mix the same teacher view’s class distribution into its local patch distribution. The diagnostic study and target-side method answer related questions, but neither independently validates the other.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised LearningBasavaraj Sunagad, Artur Jesslen, and Adam Kortylewski2026 · arXiv · preprintarXiv:2609.28187v1, submitted 23 September 2026, inspected 26 October 2026. Empirical DINO-family ablations on ImageNet-1k, not a universal necessity theorem or an evaluation of DINOv3/pathology. The review retains a discrepancy between the main text’s fixed-scale depth claim and Appendix Table A3.This is a controlled diagnosis of DINO-style learning, not a new foundation model. In the tested ImageNet-1k regime, matching two geometrically different global crops from the same image already produces semantic patch correspondence. iBOT’s masked-patch objective improves that structure when combined with the global objective, while extra local crops chiefly improve classification in the DINO-only comparison. The practical lesson is to separate semantic anchoring, refinement and evaluation rather than treat a successful recipe as an indivisible block.Inspected manuscript and appendices (v1)
CODI: Improving Self-Supervised Vision Transformers with Cross DistillationSiran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, and Qingming Huang2026 · arXiv · preprintarXiv:2610.30639v1, submitted 22 October 2026, inspected 26 October 2026. The manuscript has a NeurIPS 2026 footer, but no official acceptance record was independently located in this audit; retained as a preprint. The author repository contains only a Coming soon README, not released implementation code.CODI changes a masked patch’s teaching target, not the Vision Transformer architecture. It mixes 90% of the momentum teacher’s local patch distribution with 10% of that same view’s global class-token distribution. This gives low-attention patches a direct semantic training signal while retaining mostly local supervision. It is self-distillation during unlabeled image pretraining, not compression from a separately pretrained large teacher and not cross-magnification transfer.Inspected manuscript (v1) · Author repository — implementation pending at inspection

Reading question: Which signal creates useful spatial semantics, which refines it, and do your global and dense evaluations agree?

Evidence boundary: Both inspected sources are preprints. Crop ablations match iterations, not all computation; the depth-scale narrative has an appendix exception. CODI’s gradient statement is local/logit-level, its alpha is task-dependent, and its code was not released at inspection. Neither reports WSI or cross-magnification validation.

Complete maintained paper route

Section titled “Complete maintained paper route”

86 canonical readings · 2006–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 86 readings in maintained order

  9. #92018CVPRPeer reviewedL2–L3

    Deep Mutual Learning

    Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu

    Official artifacts: Official code

    Metadata checked

  10. #102018ICMLPeer reviewedL2–L3

    Born Again Neural Networks

    Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar

    Metadata checked

  33. #332021IEEE Transactions on Image ProcessingPeer reviewedL3

    Resolution-Aware Knowledge Distillation for Efficient Inference

    Zhanxiang Feng, Jianhuang Lai, and Xiaohua Xie

    Peer-reviewed journal article, volume 30, pages 6985–6996. The paper calls its framework RKD, meaning resolution-aware—not relational—knowledge distillation. Teacher and student use the same architecture but paired high- and low-resolution inputs; only the low-resolution student is needed at inference.

    Official artifacts: PubMed record

    Metadata checked

  34. #342021NeurocomputingPeer reviewedL3

    Residual error based knowledge distillation

    Mengya Gao, Yujun Wang, and Liang Wan

    Published in Neurocomputing 433 (14 April 2021), pages 154–161, DOI 10.1016/j.neucom.2020.10.113. The differently authored arXiv:2002.09168 is recorded only as a related precursor, not as this journal version.

    Metadata checked

  63. #632025ICMLPeer reviewedL3

    Interpreting CLIP with Hierarchical Sparse Autoencoders

    Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek

    Zaigrajew et al. multi-TopK CLIP method, distinct from Bussmann et al. fixed-prefix Matryoshka SAE. Author repository discloses a post-acceptance text-standardization correction; review distinguishes proceedings measurements from revised code.

    Official artifacts: Proceedings PDF · Author code and correction notice · Original ICML implementation · arXiv

    Metadata checked

  72. #722026ECCVAccepted · proceedings pendingL3

    MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image Generation

    Yang Chen, Xiaowei Xu, Shuai Wang, Xinwen Zhang, Qiushi Guo, Tiezheng Ge, and Limin Wang

    ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #2272. Publisher proceedings/version of record was not verified on 2026-09-12.

    Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript

    Metadata checked

  73. #732026ECCVAccepted · proceedings pendingL3

    Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention

    Weichen Zhou, Yawen Zou, Chunzhi Gu, Ran Dong, Haoran Xie, and Chao Zhang

    ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #7101. Publisher proceedings/version of record was not verified on 2026-09-12.

    Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript

    Metadata checked

  75. #752026ECCVAccepted · proceedings pendingL3

    Steerable Visual Representations

    Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, and Yuki M. Asano

    ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #14503. Publisher proceedings/version of record was not verified on 2026-09-12. The official program title is 'Steerable Vision Transformers'; the conference-hosted manuscript, arXiv and author project use 'Steerable Visual Representations'.

    Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoints

    Metadata checked

  76. #762026ECCVAccepted · proceedings pendingL3

    RoMa v2: Harder Better Faster Denser Feature Matching

    Johan Edstedt, David Nordström, Yushan Zhang, Georg Bökman, Jonathan Astermark, Anders Heyden, Viktor Larsson, Mårten Wadenbäck, Michael Felsberg, and Fredrik Kahl

    Accepted to ECCV 2026; spotlight presentation verified in the official program. This record and review use the conference-hosted manuscript checked 2026-09-12; a separate Springer proceedings record was not verified. The first preprint is from 2025; 2026 reflects the verified conference. Author order and table values follow the ECCV-hosted manuscript, not the earlier repository BibTeX.

    Official artifacts: Official ECCV spotlight presentation · Official ECCV paper record · arXiv record · Official code

    Metadata checked

  77. #772026ECCVAccepted · proceedings pendingL3

    SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

    Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, and Adam Kortylewski

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #349 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  78. #782026ECCVAccepted · proceedings pendingL3

    SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction

    Cuipeng Wang, Haipeng Wang

    ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The linked official repository still says source code will be released soon. This is heterogeneous-architecture feature KD, not demonstrated cross-magnification transfer. The selected CIFAR-100 table includes a ViT-S→MobileNetV2 case where FBT is better.

    Official artifacts: Official ECCV poster record · Official code

    Metadata checked

  79. #792026ICLRPeer reviewedL3

    TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning

    Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, Andrea Tirinzoni

    Official ICLR 2026 oral and full proceedings manuscript checked. Reward-free state–action transitions, not action-free video. Zero-shot policy selection still requires reward observations for task-vector inference. Theoretical assumptions, benchmark-specific actor inputs and mixed OGBench comparisons are qualified in the source-checked review; no WSI evaluation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked

  80. #802026ICLRPeer reviewedL3

    InfoNCE Induces Gaussian Distribution

    Roy Betser, Eyal Gofer, Meir Yossef Levi, Guy Gilboa

    Official ICLR 2026 oral and full proceedings manuscript checked. Conditional theory and representation diagnostics, not proof that every InfoNCE encoder is jointly Gaussian. Separate fixed-dimensional projection limits from a regularized population surrogate and coordinate-wise non-rejection from joint normality.

    Official artifacts: Official proceedings record · OpenReview

    Metadata checked

  81. #812026ICLRPeer reviewedL3

    TRACE: Your Diffusion Model Is Secretly an Instance Edge Detector

    Sanghyun Jo, Ziseok Lee, Wooyeol Lee, Jonghyun Choi, Jaesik Park, Kyungsu Kim

    Official ICLR 2026 oral and full proceedings manuscript checked. Diffusion-boundary self-distillation improves selected natural-image segmentation but degrades MoNuSeg/TNBC cell segmentation. One-pass speed is edge extraction only, with the large backbone retained. No WSI or cross-magnification validation; official code/checkpoints remain promised rather than released at the checked revision.

    Official artifacts: OpenReview · Official project · Official repository — release pending

    Metadata checked

  82. #822026arXivPreprintL3

    The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation

    Kuniaki Saito and Yoshitaka Ushiku

    arXiv v1 checked 2 October 2026. No official code/model release or conference acceptance verified. Camelyon17 is patch-level evaluation, not whole-slide cross-magnification validation. PE teacher/student labels and parameter counts differ between main text/tables and Appendix C; this review uses unambiguous DINOv2-to-MobileNet Table 1 rows.

    Metadata checked

  83. #832026IEEE TPAMIPeer reviewedL3

    Understanding the Effects of Projectors in Knowledge Distillation

    Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, Brano Kusy, and Zi Huang

    IEEE publisher metadata identifies TPAMI 48(8), 9078–9092, August 2026. The accessible scientific text checked here is arXiv:2310.17183v1 (26 October 2023); final journal numerical identity was not established. This extends the authors’ NeurIPS 2022 Projector Ensemble Feature Distillation work, rather than originating PEFD in 2026.

    Official artifacts: Inspected author manuscript (v1) · Author code and earlier PEFD implementation

    Metadata checked

  84. #842026arXivPreprintL3

    CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

    Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, and Qingming Huang

    arXiv:2610.30639v1, submitted 22 October 2026, inspected 26 October 2026. The manuscript has a NeurIPS 2026 footer, but no official acceptance record was independently located in this audit; retained as a preprint. The author repository contains only a Coming soon README, not released implementation code.

    Official artifacts: Inspected manuscript (v1) · Author repository — implementation pending at inspection

    Metadata checked

  85. #852026arXivPreprintL3

    Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

    Basavaraj Sunagad, Artur Jesslen, and Adam Kortylewski

    arXiv:2609.28187v1, submitted 23 September 2026, inspected 26 October 2026. Empirical DINO-family ablations on ImageNet-1k, not a universal necessity theorem or an evaluation of DINOv3/pathology. The review retains a discrepancy between the main text’s fixed-scale depth claim and Appendix Table A3.

    Official artifacts: Inspected manuscript and appendices (v1)

    Metadata checked

  86. #862026TMLRPeer reviewedResearch

    DINOv3

    Meta AI et al.

    Metadata checked