Skip to content
Computational Pathology Reading Roadmap

Multimodal LLMs and visual instruction

A multimodal large language model (MLLM) usually combines a visual encoder, a vision-to-language interface, and a language model. The interface can be a linear projection, perceiver/resampler, Q-Former, cross-attention layers, or native multimodal pretraining. The phrase “the LLM sees the image” hides the key research question: which visual information survives this interface?

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Individual ICLR oral-paper studies retain their 19 September 2026 source-check dates; the later discovery audit does not re-date that evidence.

ECCV 2026 conference additions and metadata checked through 2026-09-12; the broader literature audit boundary above is unchanged.

Prerequisite path

Section titled “Prerequisite path”
  1. Complete causal masking, transformer decoding, instruction tuning, and preference optimization in NLP and large-language-model foundations.
  2. Read image–text representation for the visual encoder and caption/VQA for task contracts.
  3. Read grounding before interpreting a free-form answer as spatially supported.

At research level, you should be able to reconstruct every training stage and label each component frozen, initialized, distilled, adapted, or trained from scratch.

Output, supervision, and evidence taxonomy

Section titled “Output, supervision, and evidence taxonomy”
Design Output Supervision Evidence bottleneck
Frozen vision + frozen LLM bridge text conditioned on learned visual queries image–text alignment and generation fixed query count may compress regions and small text
Projector-based MLLM visual tokens inserted into causal sequence image–caption pretraining then visual instructions projector and resolution determine which patch information reaches the LLM
Cross-attention MLLM text states attend to resampled visual tokens interleaved image/text sequences cross-attention placement and resampler capacity
Native multimodal pretraining one model trained jointly on text and multimodal data next-token/mixed objectives data mixture may trade language quality against visual grounding
Grounded MLLM text plus coordinates, points, or masks localized instructions and spatial labels spatial decoder or location-token precision
Reasoning/post-trained MLLM answer plus rationale/tool use SFT, preferences, process/outcome rewards verifier quality and whether the trace cites actual visual evidence
Diffusion-language MLLM iteratively denoised text conditioned on vision masked diffusion plus visual instruction data non-causal generation changes latency, bidirectional context, and scoring

Method detail and architecture record

Section titled “Method detail and architecture record”

For any MLLM, write this tuple before comparing results:

(vision encoder, input resolution/aspect policy, visual-token count,
 connector/resampler, language backbone/tokenizer, fusion positions,
 pretraining data/objectives, instruction data, preference/RL stage,
 decoding policy, exact checkpoint and license)

Visual token policy

Section titled “Visual token policy”

Fixed resizing is cheap but can erase small objects and text. Dynamic resolution, tiling, multi-crop inputs, and video sampling preserve more evidence at the cost of longer sequences. Token pooling/pruning changes the evidence distribution. Always report both original pixels and post-connector tokens.

Alignment and instruction stages

Section titled “Alignment and instruction stages”

A common pipeline is:

image–caption alignment → multimodal pretraining → instruction SFT
→ preference or reinforcement post-training → task-time prompting/tools

Stage boundaries matter. Alignment teaches a representation interface; instruction tuning teaches response formats and conversational behavior; preference optimization changes which responses are selected. Later stages can improve apparent helpfulness while weakening visual sensitivity.

Staged paper route

Section titled “Staged paper route”

Bridging pretrained vision and language

Section titled “Bridging pretrained vision and language”
Order Primary paper and year Venue / status Read it for
1 Frozen (2021) NeurIPS, peer reviewed prefixing a frozen language model with visual representations for few-shot transfer
2 Flamingo (2022) NeurIPS, peer reviewed Perceiver resampling, gated cross-attention, interleaved data, few-shot prompting
3 BLIP-2 (2023) ICML, peer reviewed two-stage Q-Former learning between frozen image and language models
4 PaLI (2023) ICLR, peer reviewed joint language/vision scaling across multilingual task mixtures
5 Kosmos-1 (2023) NeurIPS, peer reviewed multimodal language modelling and in-context perception
6 InstructBLIP (2023) NeurIPS, peer reviewed instruction-aware visual feature extraction through a Q-Former
7 PaLM-E (2023) ICML, peer reviewed continuous visual and sensor observations injected into an embodied language model

Visual instruction tuning

Section titled “Visual instruction tuning”
Order Primary paper and year Venue / status Read it for
1 LLaVA (2023) NeurIPS, peer reviewed language-model-generated instruction data and a simple visual projector
2 Qwen-VL (2023) arXiv preprint multilingual dialogue, OCR, grounding, and a staged recipe
3 CogVLM (2024; preprint 2023) NeurIPS, peer reviewed trainable visual expert modules inside the language backbone
4 MM1 (2024) ECCV, peer reviewed controlled ablations of data mixture, image encoder, resolution, token count, and connector
5 VILA (2024) CVPR, peer reviewed pretraining-stage choices, interleaved data, and preserving language ability
6 Improved Baselines with Visual Instruction Tuning (2024) CVPR, peer reviewed LLaVA-1.5 data, resolution, MLP connector, and stronger evaluation
7 InternVL (2024) CVPR, peer reviewed scaling a large vision encoder before language alignment
8 PaliGemma (2024) arXiv preprint a compact open vision-language model intended for transfer
9 Cambrian-1 (2024) NeurIPS, peer reviewed a vision-centric, open study of encoders, connector design, and spatial aggregation

Resolution, openness, and native multimodality

Section titled “Resolution, openness, and native multimodality”
Order Primary paper and year Venue / status Read it for
1 Qwen2-VL (2024) arXiv preprint dynamic resolution, multimodal rotary position encoding, images and video
2 Molmo and PixMo (2025; preprint 2024) CVPR, peer reviewed open weights, data, detailed captions, pointing, and human evaluation
3 Pixtral (2024) arXiv preprint variable image sizes and an open-weight multimodal decoder
4 Qwen2.5-VL (2025) arXiv preprint document/chart understanding, spatial grounding, and long video
5 Gemma 3 (2025) arXiv technical report lightweight multimodal models, long context, local/global attention, distillation
6 Open-Qwen2VL (2025) arXiv preprint an inspectable academic-scale pretraining reproduction
7 InternVL3 (2025) arXiv preprint native multimodal pretraining, position encoding, preference tuning, test-time scaling
8 Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models (2026) CVPR, peer reviewed uncertainty-weighted transfer and an energy-based prior; audit whether uncertainty improves calibration as well as compact-model accuracy

Technical reports and arXiv model releases are valuable system records, but their benchmark claims have not necessarily received peer review. Keep status visible.

Reasoning, grounding, and alternative decoding

Section titled “Reasoning, grounding, and alternative decoding”
Order Primary paper and year Venue / status Read it for
1 Multimodal Chain-of-Thought (2024) TMLR, peer reviewed separating rationale generation and answer inference
2 LISA (2024) CVPR, peer reviewed language reasoning connected to pixel masks
3 F-LMM (2025) CVPR, peer reviewed spatial evidence extracted without changing the base conversational model
4 Argus (2025) CVPR, peer reviewed grounded, object-centric reasoning steps tied to visual evidence
5 Cross-modal Information Flow (2025) CVPR, peer reviewed layer-wise analysis of where image and text information combine
6 LLaDA-V (2026) CVPR, peer reviewed masked-diffusion language decoding with visual instructions
7 ROSE (2026) CVPR, peer reviewed parameter-space visual injection instead of long visual token sequences
8 VGent (2026) CVPR, peer reviewed a modular split between language reasoning and spatial prediction
9 Perceval (2026) CVPR, peer reviewed claim-level visual error feedback during reinforcement learning

Evaluation before chat demonstrations

Section titled “Evaluation before chat demonstrations”

Measure component capabilities independently:

Claim Minimum evidence
“understands images” perception tests with image swaps, small objects, attributes, relations, OCR, and calibration
“uses the image” answer changes correctly under controlled visual interventions
“grounds language” points/boxes/masks with absent-target rejection
“reasons visually” visually necessary tasks plus verified intermediate claims, not rationale fluency
“supports long context” accuracy, latency, memory, and evidence retrieval versus length
“is open” weights, code, preprocessing, training data/provenance, recipe, and usable license reported separately
“is safe” harmfulness and over-refusal under text, image, and mixed attacks

Controlled experiment

Section titled “Controlled experiment”

Choose one open language backbone and one vision encoder. Train or evaluate four interfaces:

  1. linear projector with fixed visual tokens;
  2. learned query/resampler with a fixed output count;
  3. cross-attention layers;
  4. dynamic-resolution tokens with a matched maximum token budget.

Use identical alignment and instruction examples, optimizer budget, prompt, and decoding. Freeze the language model for one comparison and adapt it identically for another. Evaluate:

  • retrieval/alignment before instruction tuning;
  • VQA, OCR, spatial relations, and grounding;
  • correct, shuffled, blank, low-resolution, and cropped images;
  • language-only regression from the original checkpoint;
  • latency, KV-cache memory, trainable parameters, and tokens per image.

Perform an evidence bottleneck sweep: vary visual tokens while keeping original image resolution constant, then vary resolution while keeping tokens constant.

Failure modes

Section titled “Failure modes”
  • Instruction data teaches answer style more strongly than visual use.
  • A proprietary MLLM generates training captions, instructions, and evaluation scores.
  • Model comparisons change the vision encoder, language backbone, data, and resolution simultaneously.
  • Tiling increases pixels but loses cross-tile spatial relationships.
  • Visual token pruning removes small or rare evidence.
  • Language ability drops after multimodal training but is not re-evaluated.
  • Reasoning traces confidently elaborate an incorrect initial perception.
  • Coordinate strings are accepted even when the described object is absent.
  • A benchmark is contaminated through public images, captions, or synthetic variants.
  • “Open source” describes weights while training data and recipe remain unavailable.

Self-check and exit task

Section titled “Self-check and exit task”
  1. Compare the Perceiver resampler, Q-Former, linear projection, and cross-attention.
  2. Which modules are frozen in Flamingo and BLIP-2?
  3. Why can instruction tuning improve chat quality without improving perception?
  4. How do dynamic resolution and visual-token count differ?
  5. What experiment detects damage to the language backbone?
  6. How would you verify that a chain-of-thought claim is image supported?

Exit task: produce a component-and-stage audit for one MLLM, then run an image-intervention suite and a visual-token-budget sweep. Report evidence-sensitive accuracy separately from ordinary benchmark accuracy.

Bridge to computational pathology

Section titled “Bridge to computational pathology”

A WSI cannot be represented faithfully by resizing it into one natural-image input. Pathology MLLMs must specify tile encoder, magnification, spatial sampler, slide aggregator, visual token budget, report pairing, and whether cellular/region evidence survives the connector. Natural-image visual instruction data does not establish pathology terminology or diagnostic calibration. Continue to pathology VLMs after auditing the general connector and post-training mechanisms here.

Anatomical token selection and staged CT report generation

A coarse abnormality language model guides structure-specific evidence selection for a second report model. Training-only anatomical masks and a downstream report reward align the stages. Inspect which component GRPO updates and which signals come from segmentation supervision.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Mimicking Radiologists: A Coarse-to-Fine Framework with Structural Sparse Tokens for Dual-LLM Computed Tomography Report GenerationHong Liu, Dong Wei, Yefeng Zheng, Xian Wu, and Liansheng Wang2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #9339. Publisher proceedings/version of record was not verified on 2026-09-12. The paper-linked CTRG repository README still describes the earlier Structure Observation Driven Image-Text Contrastive Learning work and LLaMA-2-7B. A matching dual-LLM/GRPO release was not verified; do not treat this link as reproduction evidence.This dual-LLM CT reporting system first proposes abnormalities from coarse structural tokens, then selects local evidence to generate the full report. AP-LTF combines text–token similarity, learned scoring and attention weights, while training-only anatomical masks discourage attention outside the intended structure. GRPO refines the proposal policy using the downstream report reward.ECCV Spotlight program · Conference manuscript · Paper-linked repository (earlier method; matching release unverified)

Reading question: Can an early abnormality proposal remove evidence needed by the final report, and how is that failure measured?

Evidence boundary: Labeler-based clinical efficacy scores are proxy metrics, not clinical validation. The linked repository documents an earlier method; a matching release is not yet verified.

Reasoning budgets, executable outputs and scientific edits

Test-time scaling studies how extra reasoning changes perception and mathematics; EmbedCopilot checks whether plausible embedded-system answers actually execute; GRADE requires domain-correct and readable image edits. Together they motivate measuring compute, implementation success and constraint satisfaction rather than judging prose alone.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
On Test-Time Scaling for Vision-Language ModelsFawaz Sammani, Tzoulio Chamiti, and Nikos Deligiannis2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #4968 checked on 2026-09-12. Publisher version of record was not independently verified.This empirical study tests when extra inference computation helps vision-language models. Nine prompting and sampling strategies are compared across thirteen instruction-tuned models. The main distinction is task-dependent: more reasoning can help mathematical or logical questions while harming perception-focused judgments. It is not a new trained reasoning model.ECCV Spotlight program · Conference manuscript

Reading question: What external test can falsify a persuasive explanation: a perception answer, a running circuit, or a scientifically valid image?

Evidence boundary: More tokens can reduce perceptual accuracy. Judge scores and physical execution use different protocols; GRADE strict accuracy is a conjunction of three maxima, not its mean reasoning score.

Frequency-domain visual token budgets

Fourier Compressor uses fixed DCT truncation on feature grids, followed by VLM adaptation. Distinguish parameter-free compression from training-free deployment, and average retention from fine-detail fidelity.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language ModelsHuanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, Zhouhan Lin2026 · ECCV · acceptedECCV 2026 spotlight verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Parameter-free compression is NOT training-free: Section 5.1 and Table 2 require adaptation. The 31.2% number concerns reduced time to first token, not a measured 31.2% increase in autoregressive decoding throughput.Fourier Compressor removes visual tokens by shrinking their spatial frequency representation. It applies a two-dimensional type-II DCT independently across feature channels, keeps a low-frequency rectangle, and applies an inverse DCT on the smaller grid. The compressor adds no learned parameters, but the surrounding VLM is trained to accommodate the altered features.Official ECCV spotlight record · Official code

Reading question: Which frequencies and tasks lose evidence at a matched training and token budget?

Evidence boundary: TextVQA and video subtasks can regress despite strong average retention; TTFT is not decoding throughput.

Spatial-frequency and response-token distillation targets

SFKD selectively aligns wavelet-localized Fourier amplitudes across backbone families. TRAG aligns response-to-vision attention using token-specific entropy-weighted KL. Both teach structure beyond final logits, but use different correspondence assumptions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge DistillationJaehyun Jang, Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Mark A. Hasegawa-Johnson, and Chang D. Yoo2026 · ECCV · acceptedECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Teacher and student share a frozen SigLIP vision encoder and resolution; this is not cross-resolution KD. The official code repository lists training/evaluation scripts as planned. Attention similarity correlations are not causal guarantees of faithful reasoning.TRAG distills how a multimodal teacher attends to visual tokens while producing a response. It aggregates middle-layer response-to-vision attention, renormalizes over visual keys and assigns each response token its own forward/reverse KL mixture using teacher attention entropy. This supplements language and unimodal distillation losses rather than replacing the full training recipe.Official ECCV poster record · Official code

Reading question: Are the grids, semantic units and probability supports genuinely aligned before applying the loss?

Evidence boundary: SFKD has architecture-specific regressions; TRAG shares a visual encoder and resolution. Neither establishes cross-magnification performance.

Perceptual experts, geometry and explicit grounding tokens

Continuous expert-supervised visual tokens, geometry-aware fusion and coarse-to-fine grounding alter how a language model accesses visual evidence. Compare the representation interface, auxiliary supervision and deployed computation before comparing answer scores.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual TokensYiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, and XuDong Wang2026 · ECCV · acceptedECCV 2026 Oral verified in the official program (event 6085); scientific claims use the conference-hosted manuscript #5952. Publisher proceedings/version of record was not verified on 2026-09-12. First preprint appeared in November 2025; stable ID preserves that origin.CoVT teaches a VLM to generate continuous visual thoughts alongside ordinary language. These latent tokens acquire perceptual meaning through supervision from segmentation, depth, edge, and self-supervised feature experts. The answer conditions on this compact visual reasoning sequence; dense predictions are optional explanatory outputs rather than mandatory inference-time tool calls.ECCV program · Conference manuscript · Author project · Code
Make Geometry Matter for Spatial ReasoningShihua Zhang, Qiuhong Shen, Shizun Wang, Tianbo Pan, and Xinchao Wang2026 · ECCV · acceptedECCV 2026 Oral verified in the official program (event 6086); scientific claims use the conference-hosted manuscript #482. Publisher proceedings/version of record was not verified on 2026-09-12.GeoSR asks whether a spatial-reasoning VLM actually uses its geometry tokens. It combines training-time masking of appearance tokens with learned token-and-channel-wise fusion, so geometric evidence becomes useful rather than an ignored auxiliary stream. The contribution is controlled utilization of pretrained geometry, not simply adding another encoder.ECCV program · Conference manuscript · Author project · Code
MolmoPoint: Better Pointing for VLMs with Grounding TokensChristopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, and Ranjay Krishna2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official program (event 6028); scientific claims use the conference-hosted manuscript #8311. Publisher proceedings/version of record was not verified on 2026-09-12. The program spells the name Molmo-Point; the conference PDF uses MolmoPoint.MolmoPoint makes a VLM point by selecting its own visual tokens, then refining the selected location. Three grounding tokens replace a coordinate string: choose a coarse patch, select a finer ViT subpatch, and choose a position inside that subpatch. The mechanism ties spatial output to visual evidence and reduces the need to learn a global textual coordinate system.ECCV program · Conference manuscript · Author project

Reading question: Does a claimed reasoning gain arise from more visual evidence, better alignment, a changed output space or a larger inference budget?

Evidence boundary: Expert distillation is not proof of faithful reasoning; stronger pointing or spatial QA need not improve every tracking metric.

Long-video memory, scripts and entity continuity

Retrieval-based episodic memory, audio-visual script generation and entity-centric evaluation answer different long-video questions. Track what is stored, how evidence is retrieved, and whether identities and events remain consistent across time.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic VideoJunfu Pu, Yuxin Chen, Teng Wang, and Ying Shan2026 · ECCV · acceptedECCV 2026 Oral verified in the official program (event 6045); scientific claims use the conference-hosted manuscript #9919. Publisher proceedings/version of record was not verified on 2026-09-12.OmniScript converts cinematic audio and video into a hierarchical scene-event script. The model first reasons about plot and character relationships, then emits time-grounded fields for actions, dialogue, expressions, and audio cues. Its research contribution couples an audio-visual architecture, structured training data, and an evaluation protocol that separately checks what happened and when.ECCV program · Conference manuscript · Author project · Code

Reading question: Can the system recover the right episode and preserve who did what, when, and in which scene?

Evidence boundary: Compressed memory can omit decisive events; fluent scripts and pooled benchmark scores can conceal identity or temporal failures.

Prompt-conditioned audiovisual embeddings

WAVE combines speech and sound pathways, layer-wise non-text fusion and joint retrieval/QA embedding training. Its output is similarity ranking, not unrestricted text generation.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMChangli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, Chao Zhang2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Multimodal embeddings and candidate-answer ranking, not unrestricted report generation. Official paper and released training configuration checked; unequal pooling contracts, negative findings and unresolved overall-score discrepancy are documented. No WSI evaluation.WAVE turns a multimodal generative backbone into a prompt-conditioned embedding model for text, video, audio and audiovisual inputs. Its central design combines complementary speech and sound encoders with hierarchical feature fusion across LLM layers. Retrieval and multiple-choice QA are both formulated as embedding comparisons, not unrestricted answer generation.Official proceedings record · OpenReview · Official implementation

Reading question: How do the task prompt and pooling rule change what information the embedding retains?

Evidence boundary: Modality coverage is finite; question-conditioned candidate accuracy is not evidence of report-generation reliability.

Diverse representatives and temporal token merging

FlashVID preserves representative video tokens and merges the unselected remainder into similarity trees. Compare token budgets, compression overhead and complete time to first token separately.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token MergingZiyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, Zhuotao Tian2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Inference-time token selection/merging, not residual-error KD. Full configuration includes segmentation, budget adjustment and hybrid LLM pruning. Report encoding and compression overhead as well as LLM savings; natural-video evidence is not cross-magnification pathology validation.FlashVID compresses video tokens without training the target Video LLM. Attention-guided Diverse Token Selection keeps representative tokens, while Tree-based Spatiotemporal Merging pools redundant tokens across adjacent frames. This is token selection and merging, not teacher–student knowledge distillation.Official proceedings record · OpenReview · Official implementation

Reading question: Which information survives selection or merging, and where is compression cost included?

Evidence boundary: Residual tokens are not residual-error KD; large LLM-only savings are not the same as end-to-end acceleration.

Distilling visual context and sensitivity to evidence

SCOPD teaches a sparse-context student from a full-context EMA teacher on student rollouts. CW-OPD instead tests how predictions change when answer-critical visual evidence is edited between two worlds.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language ModelsAhmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, and Babak Taati2026 · arXiv · preprintPreprint checked 2026-10-02; project page still labels code and models as forthcoming. Visual-token retention is not an optical-magnification intervention.SCOPD treats visual-token pruning partly as a representation-utilization problem. A full-context teacher supervises responses generated by the sparse student; SCOPD+ concentrates the supervision on visually sensitive positions.Versioned full text · Author project page
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language ModelsYuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, and Xinbing Wang2026 · arXiv · preprintPreprint checked 2026-10-02. The paper calls the method CW-OPD; its linked code repository uses the name CWAD. No conference acceptance inferred.CW-OPD supervises how a VLM prediction changes when answer-critical visual evidence changes. It combines ordinary per-world distillation with a cross-world belief-transition target, rather than assuming a correct answer proves the student used the right evidence.Versioned full text · Author implementation (repository name CWAD)

Reading question: Does the student learn visual evidence, or merely the teacher’s answer distribution?

Evidence boundary: Token pruning and counterfactual image editing are not WSI acquisition changes. Track EMA updates, teacher-only hints, editor quality and training cost before comparing distillation methods.

Distill visual grounding by withholding textual shortcuts

Hide to See masks parts of the student’s reasoning prefix while the teacher sees the complete prefix. The asymmetric distillation setup asks the student to rely more on the image rather than copy answers from preceding text.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM DistillationSeonghoon Yu, Dongjun Nam, Byung-Kwan Lee, and Jeany Son2026 · NeurIPS · acceptedNeurIPS 2026 acceptance is author-reported by NVIDIA Research and the official implementation, checked 8 October 2026. No exact title or author match was found in the dated official oral/poster snapshot; presentation tier and final proceedings are unverified. Review uses arXiv v4. Training-time textual-prefix masking is not WSI magnification transfer.Masking-KD distills a reasoning VLM while hiding selected textual shortcuts from the student. The teacher sees the complete image, question, and earlier response; the student loses access to its most attended earlier reasoning tokens and must predict the same next-token distribution with the image still available. The intervention is reasoning-prefix masking, not image masking or an inference-time pruning algorithm.Versioned full text · Official implementation · Author project · Author acceptance record

Reading question: Does masking improve image-dependent reasoning, or only alter the difficulty of next-token prediction?

Evidence boundary: Acceptance is author-reported; no exact match was located in the inspected official program snapshot. Prefix masking changes training context, not image resolution, and image-dependence proxies are not clinical grounding tests.

Density-weighted residual alignment in hybrid VLMs

HEED aligns teacher and hybrid-student decoder residual streams, weighting visual positions by frozen-ViT local self-dissimilarity. This density proxy is not semantic saliency. Distinguish the hybrid architecture’s inference benefit from density weighting’s incremental benefit in a matched training recipe.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model DistillationYihao Liang and Niraj K. Jha2026 · NeurIPS · acceptedOfficial NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested.HEED protects locally distinctive visual positions when converting an attention-based VLM to a hybrid Mamba-2/attention model. Its contribution is density-weighted residual alignment: the student matches the teacher's residual stream more strongly at patches whose frozen visual features differ from their neighbors. The deployed architecture is unchanged by this loss; HEED does not delete visual tokens.Versioned full text · Official acceptance identity · Official NeurIPS program

Reading question: Does density weighting outperform uniform residual alignment when data, trainable parameters and architecture are held fixed?

Evidence boundary: The C3→C4 OCRBench v2 gain is 4.7 points; the larger headline change includes other training changes. The manuscript disagrees about Stage 3 trainable parameters. Neither the speedup nor this feature residual establishes WSI cross-magnification transfer.

Test semantic coherence, concept supervision, and hierarchical steering

Hierarchical CLIP SAEs and VLM SAE studies connect feature interpretation with retrieval, bias analysis, or interventions on generated descriptions. Concept Bottleneck SAE adds supplied concept structure; SAE++ builds a hierarchy over dictionary atoms. Human alignment, control strength, and preserving unrelated content are separate questions.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Sparse Autoencoders Learn Monosemantic Features in Vision-Language ModelsMateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata2025 · NeurIPS · peer-reviewedNeurIPS 2025 proceedings volume 38; the repository BibTeX uses publication year 2026. Catalog year follows the conference edition.This work evaluates whether sparse vision features are semantically coherent and whether manipulating them changes a VLM output. Its MonoSemanticity score measures activation-weighted image similarity and is checked against human pairwise judgments; separate CLIP-feature interventions steer LLaVA. Interpretability and controllability are related experiments, not interchangeable definitions.Proceedings PDF · Code and human-comparison benchmark · arXiv
Interpretable and Steerable Concept Bottleneck Sparse AutoencodersAkshay Kulkarni, Tsui-Wei Weng, Vivek Narayanaswamy, Shusen Liu, Wesam A. Sakla, and Kowshik Thopalli2026 · CVPR · peer-reviewedCB-SAE combines an unsupervised sparse dictionary with a user-specified concept bottleneck. It scores existing neurons for interpretability and downstream steerability, prunes low-utility neurons, then trains a parallel concept autoencoder while freezing retained SAE weights. This recovers useful reconstruction and supplies concepts missing from the retained dictionary.Proceedings PDF · Supplement · Author repository (code not yet released) · arXiv
SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMsYusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, and Hao Wang2026 · NeurIPS · acceptedNeurIPS 2026 Accept (poster) verified from official program event 151133, OpenReview cl4vXMku1D, matching all five authors. Program title omits SAE++ and joins LearnMulti-Level; review uses arXiv v3 revised 7 October 2026. Proceedings version not yet verified.SAE++ learns a hierarchy by training a second SAE on the decoder atoms of a first SAE. Level 1 decomposes model activations into sparse features; Level 2 groups the learned feature directions into higher-level concepts. It is not a residual-error cascade and does not feed Level-1 sparse sample codes into a conventional stacked bottleneck.Reviewed manuscript v3 · Official NeurIPS poster · OpenReview · Code

Reading question: Can the target concept change while unrelated visual content and answer quality remain intact?

Evidence boundary: A semantic-coherence score is not a causal or clinical certificate. Report coverage, duplicate features, judge dependence, and both intended and collateral intervention effects.

Audit teacher reliability before stronger on-policy imitation

The VLM teacher-inheritance study compares filtered SFT with full-vocabulary OPD on ChartQA and Geometry3K. It separates answer accuracy from completion, decomposes paired failures and shows OPD can erase completion already learned through SFT. This is an empirical warning, not a new KD objective.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language ModelsFrank F. Yang, Kefei Yan, Yiren Wang, and Yiyang Wang2026 · arXiv · preprintVersion 1 empirical study. The current author README describes an anonymized code-only supplement and explicitly excludes evaluation records, logs, adapters, tests and figure scripts despite the manuscript release statement.This is an empirical boundary study, not a new distillation objective. On-policy distillation can transfer a teacher’s failure to finish an answer, even undoing completion learned through filtered supervised fine-tuning. The central distinction is between final-answer accuracy, presence of an answer line and actual termination. Agreement with a teacher is not evidence of correct or complete reasoning.Version 1 paper and appendices · Author code and current availability scope

Reading question: Does the teacher provide a usable answer on the states the student actually visits?

Evidence boundary: One model family and two main tasks; updates are fixed within channels, not compute-matched between them. Teacher probes are not a unique causal attribution. Current author code omits the evaluation records advertised in the manuscript.

Retrieve evidence inside the answer-generating model

URaG uses early shared MLLM states for ColBERT-style page ranking, retains selected page tokens and all query tokens, then continues deeper generation. Its staged retrieval and LoRA training connect evidence selection to answer modeling without a separate retrieval backbone.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document UnderstandingYongxin Shi, Jiapeng Wang, Zeyu Shan, Dezhi Peng, Zening Lin, and Lianwen Jin2026 · AAAI · peer-reviewedAAAI 2026 proceedings paper; FLOP reductions are not measured latency. Code repository states non-commercial research use.URaG makes page retrieval an internal operation of a multimodal language model. Early layers contextualize the full document and question; a lightweight head ranks pages, and deeper layers see only selected page tokens plus the query. This preserves fine-grained tokens on retained pages while removing irrelevant pages, rather than compressing every page uniformly or running an independent external retriever.Proceedings PDF · Author code and usage terms

Reading question: Where should evidence be selected so detailed tokens survive without paying full deep-layer cost?

Evidence boundary: Joint training always keeps ground-truth evidence pages; inference must select them. FLOP tests extend documents by duplication and are not latency measurements. Multi-page reasoning and WSI transfer need separate evidence.

Complete maintained paper route

Section titled “Complete maintained paper route”

66 canonical readings · 2021–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 66 readings in maintained order

  18. #182024arXivTechnical reportL2

    Pixtral 12B

    Pravesh Agrawal et al.

    Metadata checked

  31. #312025ICLRPeer reviewedL2–L3

    LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation

    Fangxun Shu, Yue Liao, Lei Zhang, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Zhelun Yu, Si Liu, Hongsheng Li, and Hao Jiang

    Peer-reviewed ICLR 2025 conference paper.

    Official artifacts: Official code

    Metadata checked

  46. #462026arXivPreprintL2–L3

    VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

    Junxiang Xu, Ruisi Wang, Fanyi Pu, Maijunxian Wang, Ran Ji, Tongxi Zhou, Chenyang Gu, Jing Zuo, Hongcan Xiao, Yimeng Geng, Wanqi Yin, Wei Chen, Oscar Qian, Zhengan Yan, Ziqi Huang, Haiwen Diao, Liang Pan, Bo Li, Xiangyu Fan, Dezhi Luo, Fengyuan Yu, Zehong Zhao, Qingying Gao, Tinghui Zhu, Yilan Zhang, Jingqi Tong, Pinyuan Feng, Zhengze Jiang, Letian Wang, Ziyu Guo, Renrui Zhang, Jieneng Chen, Sonia Joseph, Constantin Venhoff, Saman Motamed, Mengyue Yang, Chandra Sripada, Alan Yuille, Philip Torr, Lvmin Zhang, Vikash Kumar, Daniel Khashabi, Nikolaus Kriegeskorte, Raphaël Millière, Vincent C. Müller, Anyi Rao, Quan Wang, Ziwei Liu, Dahua Lin, Lei Yang, Hokin Deng, and Zhongang Cai

    arXiv preprint first submitted 26 August 2026; not peer reviewed as of the audit date.

    Official artifacts: Official project · Official code and benchmark

    Metadata checked

  48. #482026ECCVAccepted · proceedings pendingL3

    Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

    Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, and XuDong Wang

    ECCV 2026 Oral verified in the official program (event 6085); scientific claims use the conference-hosted manuscript #5952. Publisher proceedings/version of record was not verified on 2026-09-12. First preprint appeared in November 2025; stable ID preserves that origin.

    Official artifacts: ECCV program · Conference manuscript · Author project · Code

    Metadata checked

  51. #512026ECCVAccepted · proceedings pendingL3

    MolmoPoint: Better Pointing for VLMs with Grounding Tokens

    Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, and Ranjay Krishna

    ECCV 2026 Spotlight verified in the official program (event 6028); scientific claims use the conference-hosted manuscript #8311. Publisher proceedings/version of record was not verified on 2026-09-12. The program spells the name Molmo-Point; the conference PDF uses MolmoPoint.

    Official artifacts: ECCV program · Conference manuscript · Author project

    Metadata checked

  52. #522026ECCVAccepted · proceedings pendingL3

    Mimicking Radiologists: A Coarse-to-Fine Framework with Structural Sparse Tokens for Dual-LLM Computed Tomography Report Generation

    Hong Liu, Dong Wei, Yefeng Zheng, Xian Wu, and Liansheng Wang

    ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #9339. Publisher proceedings/version of record was not verified on 2026-09-12. The paper-linked CTRG repository README still describes the earlier Structure Observation Driven Image-Text Contrastive Learning work and LLaMA-2-7B. A matching dual-LLM/GRPO release was not verified; do not treat this link as reproduction evidence.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Paper-linked repository (earlier method; matching release unverified)

    Metadata checked

  54. #542026ECCVAccepted · proceedings pendingL3

    Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models

    Huanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, Zhouhan Lin

    ECCV 2026 spotlight verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Parameter-free compression is NOT training-free: Section 5.1 and Table 2 require adaptation. The 31.2% number concerns reduced time to first token, not a measured 31.2% increase in autoregressive decoding throughput.

    Official artifacts: Official ECCV spotlight record · Official code

    Metadata checked

  55. #552026ECCVAccepted · proceedings pendingL3

    Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

    Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Mark A. Hasegawa-Johnson, and Chang D. Yoo

    ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Teacher and student share a frozen SigLIP vision encoder and resolution; this is not cross-resolution KD. The official code repository lists training/evaluation scripts as planned. Attention similarity correlations are not causal guarantees of faithful reasoning.

    Official artifacts: Official ECCV poster record · Official code

    Metadata checked

  56. #562026ICLRPeer reviewedL3

    WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM

    Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, Chao Zhang

    Official ICLR 2026 oral and full proceedings manuscript checked. Multimodal embeddings and candidate-answer ranking, not unrestricted report generation. Official paper and released training configuration checked; unequal pooling contracts, negative findings and unresolved overall-score discrepancy are documented. No WSI evaluation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked

  57. #572026ICLRPeer reviewedL3

    FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging

    Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, Zhuotao Tian

    Official ICLR 2026 oral and full proceedings manuscript checked. Inference-time token selection/merging, not residual-error KD. Full configuration includes segmentation, budget adjustment and hybrid LLM pruning. Report encoding and compression overhead as well as LLM savings; natural-video evidence is not cross-magnification pathology validation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked

  58. #582026arXivPreprintL3

    SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

    Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, and Babak Taati

    Preprint checked 2026-10-02; project page still labels code and models as forthcoming. Visual-token retention is not an optical-magnification intervention.

    Official artifacts: Versioned full text · Author project page

    Metadata checked

  60. #602026NeurIPSAccepted · proceedings pendingL3

    Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation

    Seonghoon Yu, Dongjun Nam, Byung-Kwan Lee, and Jeany Son

    NeurIPS 2026 acceptance is author-reported by NVIDIA Research and the official implementation, checked 8 October 2026. No exact title or author match was found in the dated official oral/poster snapshot; presentation tier and final proceedings are unverified. Review uses arXiv v4. Training-time textual-prefix masking is not WSI magnification transfer.

    Official artifacts: Versioned full text · Official implementation · Author project · Author acceptance record

    Metadata checked

  61. #612026NeurIPSAccepted · proceedings pendingL3

    HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation

    Yihao Liang and Niraj K. Jha

    Official NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested.

    Official artifacts: Versioned full text · Official acceptance identity · Official NeurIPS program

    Metadata checked

  63. #632026NeurIPSAccepted · proceedings pendingL3

    SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

    Yusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, and Hao Wang

    NeurIPS 2026 Accept (poster) verified from official program event 151133, OpenReview cl4vXMku1D, matching all five authors. Program title omits SAE++ and joins LearnMulti-Level; review uses arXiv v3 revised 7 October 2026. Proceedings version not yet verified.

    Official artifacts: Reviewed manuscript v3 · Official NeurIPS poster · OpenReview · Code

    Metadata checked

  64. #642026arXivPreprintL3

    Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models

    Frank F. Yang, Kefei Yan, Yiren Wang, and Yiyang Wang

    Version 1 empirical study. The current author README describes an anonymized code-only supplement and explicitly excludes evaluation records, logs, adapters, tests and figure scripts despite the manuscript release statement.

    Official artifacts: Version 1 paper and appendices · Author code and current availability scope

    Metadata checked