Skip to content
Computational Pathology Reading Roadmap

Multimodal pathology and vision–language models

Pathology is multimodal at several different levels. A model may align one cropped micrograph with a caption, retrieve a whole slide from a report, fuse a WSI with RNA sequencing, answer a question about a selected field, generate a report, or actively navigate a gigapixel slide. Those are different scientific problems even when every paper calls its system a vision–language or multimodal foundation model.

Discovery update: 26 October 2026. Targeted new readings were added; existing paper reviews retain their individual source-check dates. This is not a fresh full-text audit of every model.

Chapter coverage: dated primary readings, separated by publication status, with the complete maintained route below; an architecture and supervision taxonomy; prerequisite-aware routes for image–text retrieval, assistants, WSI–language learning, report generation, omics fusion, grounding, and agentic search; verified author artifacts where available; access cautions; and a comparison protocol.

This is a research guide, not clinical guidance. None of the models should be treated as a diagnostic device merely because it can produce fluent pathology language.

1. Scope, evidence labels, and boundaries

Section titled “1. Scope, evidence labels, and boundaries”

This chapter includes a work when it introduced an influential pathology-specific multimodal resource or method, materially changed the unit of visual–language alignment, connected WSIs to another clinically relevant modality, or exposed an important evaluation failure. It is a curated comprehensive route, not a claim that every pathology VLM preprint is included.

Status labels are deliberately strict:

  • Peer reviewed means a journal or archival conference/workshop version of record was located.
  • Preprint means an arXiv paper was located but no peer-reviewed version of record was verified by the review date.
  • Model artifact means code or weights exist; it does not upgrade a preprint into a peer-reviewed paper.
  • Author artifact links only to a repository, model card, project page, or dataset identified by the authors or the version of record.

The paper year is the peer-reviewed version-of-record year. A preprint revised in a later year displays both dates when that distinction matters. Counts always retain their unit: a patch, image–caption pair, slide, specimen, patient, question, and question–answer turn are not interchangeable.

Read self-supervised learning for visual objectives, foundation encoders for checkpoint-level comparison, WSI engineering for tiling and coordinates, and spatial/molecular modeling for deeper biomarker and spatial assay coverage.

2. First identify the actual multimodal task

Section titled “2. First identify the actual multimodal task”
patient → case/specimen → slide → regions → tiles → visual encoder
    │                         │        │          │
    │                         └── coordinates ───┘
    ├── report / caption / question → text encoder or autoregressive LLM
    ├── RNA / DNA / pathways         → molecular encoder
    └── clinical context / outcomes  → structured-data encoder


visual tokens → pool / Perceiver / long-context slide encoder
             → alignment, retrieval, classification, grounding, or generation

This is an original schematic. It emphasizes that the alignment unit and the inference-time inputs must be stated before architecture names.

Family Typical output Defensible first claim Frequent category error
Dual image–text encoder normalized image and text embeddings retrieval or prompt-based recognition under a fixed protocol calling cosine similarity “reasoning”
Contrastive plus captioning encoder–decoder embeddings and generated text joint representation learning and conditional generation assuming the released artifact contains the decoder
Visual adapter plus autoregressive LLM open-ended text instruction following over the provided visual unit evaluating mostly text-answerable questions and calling it visual understanding
WSI–language encoder slide embedding, report embedding, or report slide-level cross-modal alignment treating a bag of preselected features as autonomous WSI examination
Multimodal fusion model risk, class, or molecular endpoint complementary predictive signal with all stated modalities available comparing it with a histology-only model while hiding extra inference inputs
Grounding model box, mask, region score, or navigation trace phrase-to-region localization treating attention as a validated localization result
Retrieval/agentic system evidence set plus response evidence acquisition under a fixed action and tool budget crediting the generator for evidence supplied by an oracle crop

The supervision ladder

Section titled “The supervision ladder”

From weakest spatial correspondence to strongest:

  1. Unpaired domain image and text corpora teach broad domain statistics.
  2. Figure–caption or patch–caption pairs align a small visual field with text, but captions may describe multi-panel figures or facts outside the crop.
  3. WSI–report pairs provide case-level language but only weak localization.
  4. Instruction and VQA pairs teach response format; synthetic answers can amplify a teacher model’s errors and style.
  5. Phrase–box or phrase–mask pairs test explicit entity grounding.
  6. Navigation traces and magnification choices supervise where to look and in what order.
  7. Image–report–omics tuples add biological knowledge but may reduce the number of complete cases and change the inference contract.

More supervision is not automatically better. It can be noisier, more private, less reproducible, or contain stronger target leakage.

3. Prerequisites and shortest routes

Section titled “3. Prerequisites and shortest routes”
Level Prerequisite You are ready when you can…
L0 · visual language CNNs/ViTs, tokenization, cosine similarity, cross-entropy distinguish an image encoder, text encoder, projector, decoder, and downstream head
L1 · objectives contrastive learning, transformers, autoregressive decoding write the positive-pair unit and trace which parameters receive each loss
L2 · pathology systems WSI pyramids, MPP, MIL, patient-level splitting distinguish patch, region, slide, specimen, and patient supervision
L3 · research audit retrieval metrics, calibration, survival analysis, clinical NLP audit visual necessity, data overlap, unobservable report content, and access constraints

Twelve-paper minimum route

Section titled “Twelve-paper minimum route”
  1. CLIP (2021) for symmetric contrastive image–text alignment.
  2. CoCa (2022) for contrastive and captioning objectives in one system.
  3. PLIP (2023) and Quilt-1M (2023) for pathology data-source tradeoffs.
  4. MI-Zero (2023) for non-parametric WSI aggregation of language-aligned tiles.
  5. CONCH (2024) for pathology contrastive-caption pretraining.
  6. PathChat (2024) for the projector-plus-LLM copilot design and expert open-response evaluation.
  7. PathMMU (2024) for benchmark construction and the text-shortcut problem.
  8. TITAN (2025) for region-to-slide language alignment with coordinates.
  9. HistoGPT (2025) for WSI report generation and human assessment.
  10. mSTAR (2025) for slide–report–RNA pretraining.
  11. CPathAgent (2025) and Pathology-CoT (2026) for active multiscale search and behavior supervision.
  12. PathBind (2026 preprint) for separating answer accuracy from visual binding.

The rest of the page expands each branch. Do not read chronologically without first writing the input unit, pairing rule, objective, evaluation unit, and inference-time modalities for every paper.

4. General visual–language foundations

Section titled “4. General visual–language foundations”
ID Primary paper Year / status Author artifact Read for
M01 Learning Transferable Visual Models From Natural Language Supervision (CLIP) 2021 · ICML, peer reviewed Code and models symmetric contrastive alignment, prompt ensembling, temperature, and zero-shot classification
M02 CoCa: Contrastive Captioners are Image-Text Foundation Models 2022 · TMLR, peer reviewed No author implementation located in this review one image encoder with contrastive and autoregressive captioning losses
M03 Visual Instruction Tuning (LLaVA) 2023 · NeurIPS, peer reviewed Code and models a visual encoder, learned projector, LLM, and synthetic instruction tuning; the template inherited by many pathology assistants
M04 BiomedCLIP 2023 · arXiv preprint; released model artifact Official model card broad biomedical figure–caption pretraining on 15 million PMC pairs and why biomedical is not equivalent to pathology-specialized
M05 Towards Visual Question Answering on Pathology Images (PathVQA) 2021 · ACL-IJCNLP Short Papers, peer reviewed; dataset preprint first released in 2020 Author data and code an early pathology VQA dataset; inspect question templates, answer priors, image provenance, and visual necessity before using accuracy as evidence

Checkpoint: derive the symmetric contrastive loss for a batch of image–text pairs. Then explain why multiple captions for one morphology, multiple regions in one caption, and semantically similar “negative” cases violate a one-to-one batch assumption.

5. Image–text pretraining, retrieval, and prompt transfer

Section titled “5. Image–text pretraining, retrieval, and prompt transfer”

These systems mostly answer: can language organize pathology visual representations? They do not by themselves establish WSI reasoning or report-level diagnostic safety.

ID Primary paper Year / status Author artifact Scientific contribution and audit question
M06 A visual–language foundation model for pathology image analysis using medical Twitter (PLIP) 2023 · Nature Medicine, peer reviewed Code · weights OpenPath contains 208,414 pathology image–text pairs. Study retrieval and zero-shot transfer, then audit social-media selection, duplicates, captions, and consent/provenance.
M07 Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images (MI-Zero) 2023 · CVPR, peer reviewed Code non-parametric aggregation of language-aligned tile scores for zero-shot WSI classification; separate the pretrained encoder from the aggregation rule
M08 Quilt-1M: One Million Image-Text Pairs for Histopathology 2023 · NeurIPS Datasets and Benchmarks, peer reviewed Dataset/code · QuiltNet a one-million-pair resource assembled largely from educational video frames and narration; inspect temporal near-duplicates, narrator context, and restricted data distribution
M09 A visual-language foundation model for computational pathology (CONCH) 2024 · Nature Medicine, peer reviewed Code · gated weights contrastive-caption pretraining on 1.17 million image–caption pairs across classification, retrieval, captioning, and segmentation; note that the public checkpoint omits the multimodal decoder
M10 CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision–Language Alignment 2024 · CVPR, peer reviewed Project page many-to-many visual/text bags and a pathology dictionary; audit how retrieved or generated bag members can inject the label
M11 PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration 2025 · ICLR oral, peer reviewed Data, PathGen-CLIP, and code a multi-agent caption pipeline for 1.6 million pathology pairs plus instruction data; distinguish scalable synthetic supervision from independent expert ground truth
M12 Knowledge-enhanced pretraining for vision-language pathology foundation model on cancer diagnosis (KEEP) 2026 · Cancer Cell, peer reviewed Code · model card disease-knowledge-graph organization of image–text pairs into hierarchical semantic groups; audit knowledge-source coverage, rare-class prompt sensitivity, and private-cohort comparability
M13 A vision–language foundation model for precision oncology (MUSK) 2025 · Nature, peer reviewed Code and gated weights masked image/text learning followed by pathology alignment and broad patch/slide evaluation; separate the contribution of unpaired corpora, paired data, model scale, and task adaptation
M14 Melan-Dx: a knowledge-enhanced vision-language framework improves differential diagnosis of melanocytic neoplasm pathology 2026 · npj Digital Medicine, peer reviewed Code retrieval-augmented knowledge for 40 melanocytic differential classes on patch and WSI tasks; test whether gains come from relevant evidence or class-name/atlas overlap
M15 MR-PLIP: Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation 2025 · CVPR, peer reviewed Follow the official paper’s project/code links if present multi-resolution text-guided representations; audit synthetic descriptions, TCGA overlap, and leakage across resolutions
M16 MUSE: Harnessing Precise and Diverse Semantics for Few-Shot Whole-Slide Classification 2026 · CVPR, peer reviewed Follow the official paper’s project/code links if present LLM-generated and retrieved class descriptions for few-shot WSI learning; isolate gains from semantic class priors from gains grounded in slide evidence

How to study this branch

Section titled “How to study this branch”

For each model, make one row with:

visual_unit: patch | figure | region | slide
text_unit: label | caption | narration | report | synthetic description
pair_source: exact | weak | retrieved | generated
objective: contrastive | captioning | masked | combined
public_benchmark_overlap: none | possible | known | unknown
released_components: image_encoder | text_encoder | decoder | projector
zero_shot_protocol: prompts + templates + ensembling + calibration

Retrieval requires both directions. Report image-to-text and text-to-image Recall@K, median rank, query count, candidate-pool construction, duplicate handling, and patient/specimen exclusions. A five-candidate retrieval problem is not comparable with cohort-scale search.

6. Assistants, instruction tuning, reasoning, and agents

Section titled “6. Assistants, instruction tuning, reasoning, and agents”

Fluent answers can come from language priors without using the image. Every assistant evaluation therefore needs image-only, text-only, shuffled-image, blank-image, and relevant-versus-irrelevant-context controls.

ID Primary paper Year / status Author artifact Read for
M17 PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology 2024 · AAAI, peer reviewed Code PathCLIP plus a 13B language model, pathology instruction construction, and specialist tools; distinguish routed tool output from native visual competence
M18 Quilt-LLaVA 2024 · CVPR, peer reviewed Project · gated model localized narratives from educational videos, pathology pretraining, and synthetic visual instruction tuning; audit frame/narration alignment and benchmark overlap
M19 A multimodal generative AI copilot for human pathology (PathChat) 2024 · Nature, peer reviewed Training code and public PathQABench resources UNI-derived vision alignment, a projector, Llama 2 13B, 456,916 instructions, and blinded pathologist evaluation of open responses
M20 PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology 2024 · ECCV, peer reviewed Project and data 33,428 multiple-choice questions with 24,067 images, expert validation of held-out portions, answer priors, source overlap, and the gap between exam QA and slide diagnosis
M21 CPath-Omni 2025 · CVPR, peer reviewed Repository a 15B system spanning patch and WSI classification, VQA, captioning, and visual referring, plus CPath-CLIP; verify component availability rather than assuming the full model was released
M22 SlideChat 2025 · CVPR, peer reviewed Code, model, SlideInstruction, and SlideBench CONCH patch features, a slide encoder, projector, LLM, 4.2K slide captions, and 176K WSI VQA pairs; study the cost of serializing pre-extracted slide features
M23 CPathAgent 2025 · NeurIPS, peer reviewed No author release verified in this review learned zoom/move actions across patch, region, and WSI scales plus PathMMU-HR²; evaluate navigation success separately from final-answer quality
M24 Patho-R1 2026 · AAAI, peer reviewed Code and model links 3.5 million image–text continued-pretraining pairs, 500K chain-of-thought samples, and reinforcement learning; reasoning-token plausibility is not proof of faithful visual causation
M25 PathFound 2026 · Medical Image Analysis, peer reviewed Author repository named by the publisher an evidence-seeking system with a slide highlighter, visual interpreter, and diagnostic reasoner; verify actual repository contents before claiming reproducibility
M26 Evidence-based diagnostic reasoning with multi-agent copilot for human pathology (PathChat+ and SlideSeek) 2025; revised 2026 · arXiv preprint No public weights verified in this review a PathChat successor with over one million instructions and nearly 5.5 million QA turns, plus hierarchical multi-agent WSI search; retain preprint status
M27 Pathology-CoT: learning visual chain-of-thought agents from expert whole-slide image diagnosis behaviour 2026 · Nature Biomedical Engineering, peer reviewed Code, behavioral data, and models pathologist viewport/navigation traces, “where to look” plus “why it matters” supervision, and an external WSI cohort; compare behavior imitation with outcome optimization
M28 PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology 2025 · ICCV, peer reviewed Follow the official paper’s project/code links if present multi-agent diagnostic decision-making; separate region acquisition, evidence interpretation, and final synthesis in evaluation
M29 Act Like a Pathologist: Tissue-Aware Whole-Slide Image Reasoning (HistoSelect) 2026 · CVPR, peer reviewed Follow the official paper’s project/code links if present tissue-aware slide navigation and reasoning; compare its selected evidence with oracle, random, and fixed-budget regions
M30 Patho-AgenticRAG 2026 · AAAI, peer reviewed Follow the official paper’s project/code links if present retrieval-augmented agentic pathology; quantify evidence retrieval separately from visual interpretation and answer generation

Bayesian evidence acquisition rather than relevance alone

Section titled “Bayesian evidence acquisition rather than relevance alone”

BEACON (2026, arXiv preprint) maintains a diagnostic belief distribution and selects the next patch/resolution by expected information gain. This is a different contract from retrieving the patch most semantically similar to the question: a relevant patch can still be non-discriminative between plausible diagnoses. Read the author code with the paper, and report oracle candidate generation, acquisition rounds, belief calibration, stopping rules, and final-answer accuracy separately.

Evidence-adaptive movement and magnification

Section titled “Evidence-adaptive movement and magnification”

PathPilot (2026, arXiv preprint) separates a question-agnostic localization prior (CrossMagLoc) from question-conditioned evidence search (EviRegionSearch). A pathology VLM describes the selected region; a language model updates known and missing evidence before choosing Move, Zoom In, Zoom Out, or Stop. The system retrieves real 256×256 ROIs from thumbnail/1.25×/2.5×/5×/10× views, rather than distilling them into a low-only student. Its separate MIL ablation tests the usefulness of selected patches, not a new bag-pooling rule.

Read its full method and tables with three controls in mind: GACC is a graded language-model judge score, deterministic repeatability is not patient-level uncertainty, and morphology evaluation on TCGA-SKCM/LIHC uses just 35/31 questions from 9/7 slides. Code is promised on publication; the separately referenced supplement was not available in the inspected PDF. Keep evidence acquisition, answer generation, judging and actual compute costs separate when comparing this branch with fixed sampling or low-magnification KD.

PathChat is an important access case

Section titled “PathChat is an important access case”

The PathChat paper releases training code and public benchmark material for non-commercial academic research, but states that the trained PathChat weights cannot be released because proprietary patient text was used and privacy/intellectual property restrictions apply. A public paper and public code therefore do not imply a runnable checkpoint. Record this explicitly in comparisons.

Agent evaluation has three separate denominators

Section titled “Agent evaluation has three separate denominators”
  1. Acquisition: did the system reach a diagnostic region at the needed scale?
  2. Interpretation: given the acquired region, did it describe the visible evidence correctly?
  3. Synthesis: given all evidence and context, was the conclusion supported, calibrated, and appropriately uncertain?

Report all three. An oracle-region experiment measures interpretation, not acquisition. A correct final answer with the wrong trace is not faithful reasoning.

7. Whole-slide vision–language representation

Section titled “7. Whole-slide vision–language representation”

A patch VLM plus mean pooling is not automatically a WSI VLM. Whole-slide systems must define spatial coordinates, tile selection, maximum tile count, empty tissue, multiple slides per specimen, and how a case-level report is paired to slide-level visual evidence.

ID Primary paper Year / status Author artifact Read for
M31 PathAlign: A Vision–Language Model for Whole Slide Images in Histopathology 2024 · MICCAI COMPAY workshop, peer reviewed workshop paper No author release located early WSI–report alignment and the consequences of weak slide/report pairing
M32 PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology 2024 · arXiv preprint; released model artifact Gated model Virchow tile embeddings, a Perceiver slide encoder, BioGPT, and contrastive/captioning objectives over large private WSI–report data
M33 A multimodal whole-slide foundation model for pathology (TITAN) 2025 · Nature Medicine, peer reviewed Code · gated weights region-level visual SSL, ROI-caption alignment, WSI-report alignment, coordinate-aware slide encoding, retrieval, zero-shot transfer, and report generation
M34 PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue 2025 · arXiv preprint; released model artifact Gated model a Perceiver/Phi-3 encoder–decoder trained with 2.3M WSIs and report-derived dialogue; separate released artifact evidence from peer review
M35 PathFLIP 2026 · AAAI, peer reviewed Follow the official paper’s project/code links if present fine-grained region–language alignment within WSIs rather than treating one compressed slide token as the only cross-modal unit
M36 MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole-Slide Image Understanding 2026 · CVPR, peer reviewed Follow the official paper’s project/code links if present cell→patch→region→slide alignment; audit the context and supervision introduced at every level

A source-checked WSI–report data resource

Section titled “A source-checked WSI–report data resource”
ID Primary paper Year / status Official data Read for
D01 TUM-Uteria: Paired Uterine Whole-Slide Images and Pathology Reports 2026 · arXiv preprint (v2); no peer-reviewed version verified Gated Hugging Face dataset card 216 cases and 455 H&E SVS–report pairs; German-to-English translation; LLM-assisted but manually reviewed slide pairing; case IDs; six report-derived diagnostic groups; single-center spectrum; gated DUA. The live card displayed 674 GB on 20 August 2026—treat that size as mutable card metadata, not a paper claim.

The WSI–report pairing problem

Section titled “The WSI–report pairing problem”

A report can contain:

  • findings visible on the paired H&E slide;
  • findings visible only on a different block, level, stain, IHC, or molecular assay;
  • specimen/site and clinical history;
  • measurements, margins, grade, or stage assembled across multiple slides;
  • a final diagnosis whose name is not visually recoverable from one selected WSI.

For every corpus, specify whether the relation is one slide ↔ one report, many slides ↔ one specimen report, or many specimens ↔ one case report. Preprocess text into visible morphology, non-visual context, and integrated diagnosis fields rather than silently treating the full report as an image caption. TUM-Uteria makes this mapping problem unusually explicit: 71 cases required manual judgment, and the slide-level paired release retained cases with no more than three WSIs. Use its case identifiers to split before any translation, text processing, retrieval, or feature extraction; the curation workflow improves traceability but does not make every report statement visible on every paired slide.

8. Language-conditioned histology generation

Section titled “8. Language-conditioned histology generation”

This direction reverses the usual report-generation arrow: language and retrieved visual evidence condition a synthetic pathology image. The claim is therefore not “the model understood the report” unless requested findings are visible and verified in the generated morphology.

ID Primary paper Year / status Author artifact Read for
G01 PathLDM: Text Conditioned Latent Diffusion Model for Histopathology 2024 · WACV, peer reviewed Official code GPT report summaries, cyclic 154-token positions, PLIP conditioning, VQ-f4 detail preservation, and the weak slide-report-to-patch pairing contract
G02 Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Control (UniPath) 2026 · CVPR, peer reviewed Official code and data diagnostic semantic tokens from Patho-R1, raw-text semantics, CONCH/keyword prototype retrieval, flow-matched DiT generation, and prototype/patient-leakage controls

For both, split patients before report processing or prototype construction; report which statements are visually observable in the target patch; compare aligned, shuffled, text-only, image-only, and prototype-only conditions; and evaluate morphology, condition adherence, distribution coverage, memorization, and real-case utility separately. FID or a language-model preference is not clinical validity.

9. Captioning and report generation

Section titled “9. Captioning and report generation”
ID Primary paper Year / status Author artifact Read for
M37 HistGen: Histopathology Report Generation via Local–Global Feature Encoding and Cross-modal Context Interaction 2024 · MICCAI, peer reviewed Code and benchmark resources hierarchical WSI encoding and report generation on roughly 7,800 WSI–report pairs
M38 Generating dermatopathology reports from gigapixel whole slide images with HistoGPT 2025 · Nature Communications, peer reviewed Code · weights multi-slide patient input, a pathology encoder connected to an LLM, saliency, rare-disease failure, and multicenter pathologist evaluation
M39 HARE: a Hierarchical Approach for Histopathology Report Evaluation 2025 · Findings of EMNLP, peer reviewed Code entity- and relation-aware report evaluation on hospital and TCGA reports; why n-gram similarity misses clinical errors
M40 On the Importance of Text Preprocessing for Multimodal Representation Learning and Pathology Report Generation 2026 · MICCAI COMPAY workshop proceedings, peer reviewed workshop paper No author release located full reports can reward hallucination of patient history and other image-unobservable facts; morphology-only text improves factual support while changing retrieval behavior
M41 PathGenIC: Multimodal In-Context Learning for WSI Report Generation 2026 · MIDL, peer reviewed Follow the paper’s related-material links retrieval of similar WSI–report examples, multimodal in-context generation, and adaptive feedback; audit train/test retrieval leakage
M42 Democratising Pathology Co-Pilots 2026 · MIDL, peer reviewed ANTONI-alpha, Polysome, and HISTAI-Instruct an open pipeline producing more than 1.1M instructions from 24,259 public HISTAI slides; a reproducible counterpoint to private WSI–report corpora
M43 PolyPath 2025 · Modern Pathology, peer reviewed Follow the publisher’s related-material links case-level report generation from multiple slides; preserve the specimen/case hierarchy and audit facts that are unavailable in the selected pixels
M44 BiGen 2025 · MICCAI, peer reviewed Follow the official paper’s project/code links if present historical-report retrieval for generation; test whether retrieval helps factual morphology or leaks diagnosis and patient-specific context
M45 PathReportEval 2026 · arXiv preprint, submitted July 2026 Follow author links on arXiv pathology-specific report scoring; compare it with entity/relation metrics and blinded pathologist judgments before adopting it as a standard
M46 Pathologist attention-aligned report generation 2026 · MICCAI, peer reviewed; official proceedings verified Official code repository behavior supervision for report generation; attention alignment is useful evidence but is not proof of faithful causal reasoning
M46a An end-to-end-trained VLM for native-language prostate pathology report generation 2026 · arXiv preprint No public training code, weights, or paired corpus verified a German tokenizer/model trained from scratch on 17,344 core–report pairs from 2,402 cases; audit patient/case splits, report pairing, attribute factuality, unsupported findings, and language coverage

Risk-controlled core-biopsy reporting

Section titled “Risk-controlled core-biopsy reporting”

CorePath (2026, arXiv preprint) specializes a PRISM-derived system with 7,901 paired core-needle-biopsy WSIs and reports, then separates free generation from a conformal release rule designed to control unsupported output. Its reported hallucination reduction and zero non-breast hallucinations among released outputs are properties of the specified cohorts, score, threshold, and abstention policy—not a universal guarantee. The authors state that code will be released upon publication; no implementation is counted as available at this review.

A report scorecard

Section titled “A report scorecard”

Do not collapse report quality into one BLEU, ROUGE, BERTScore, or LLM-judge number. Measure:

Axis Example measurement
Morphology support expert-labeled visible finding precision/recall with linked regions
Diagnostic correctness entity and relation extraction plus expert adjudication
Hallucination unsupported visible claims and unobservable clinical/molecular claims, separately
Omission missing critical positive finding, negative finding, uncertainty, or limitation
Consistency contradiction within the report or across sections
Localization phrase-to-region box/mask overlap or evidence-retrieval hit rate
Calibration confidence/abstention against correctness and case difficulty
Utility blinded pathologist correctness, editing time, edit distance, and failure severity

Always disclose whether the reference report itself contains copied templates, incomplete findings, or facts not inferable from the model input.

Native-language generation is a separate transfer question from translating an English report. The prostate preprint trains the text system for German clinical prose rather than assuming an English tokenizer and post-hoc translation. That design must still be validated with blinded clinical adjudication and external linguistic and institutional shift before it can support a reporting workflow.

10. Histology with genomics, transcriptomics, and pathways

Section titled “10. Histology with genomics, transcriptomics, and pathways”

This branch is multimodal learning, but not always a language model. It matters because many slide foundation systems now treat reports, expression, and genomic profiles as complementary pretraining signals.

ID Primary paper Year / status Author artifact Read for
M47 HE2RNA: A deep learning model to predict RNA-seq expression of tumours from whole slide images 2020 · Nature Communications, peer reviewed Code the morphology-to-bulk-expression task, tile aggregation, gene-wise validation, and why prediction is not assay replacement
M48 Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images (MCAT) 2021 · ICCV, peer reviewed Code genomic tokens attending to histology for censored survival; identify what modalities are required at inference
M49 Pan-cancer integrative histology-genomic analysis via multimodal deep learning (PORPOISE) 2022 · Cancer Cell, peer reviewed Code weakly supervised histology–genomics fusion across 14 cancers and modality attribution; audit pan-cancer confounding and missing data
M50 Multimodal Optimal Transport-based Co-Attention Transformer (MOTCat) 2023 · ICCV, peer reviewed Code computationally tractable histology–genomics token matching and global structure consistency
M51 Transcriptomics-guided Slide Representation Learning in Computational Pathology (TANGLE) 2024 · CVPR, peer reviewed Code contrastive slide–expression pretraining; distinguish RNA-guided representation learning from RNA-required downstream fusion
M52 Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (SurvPath) 2024 · CVPR, peer reviewed Code pathway tokens, dense cross-modal interactions, and memory-aware survival modeling
M53 A multimodal knowledge-enhanced whole-slide pathology foundation model (mSTAR) 2025 · Nature Communications, peer reviewed Code · gated weights slide-level contrastive learning across WSI, report, and RNA-seq, followed by self-taught transfer to a patch encoder
M54 Molecular-driven Foundation Model for Oncologic Pathology (THREADS) 2025 · arXiv preprint Follow links on the preprint page; no release status assumed here contrastive alignment of H&E sections with genomic/transcriptomic profiles across 54 downstream tasks; retain preprint status and audit complete-case selection
M55 Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics (UMPIRE) 2026 · Pattern Recognition, peer reviewed Code gene-encoder pretraining followed by alignment of pathology image–gene pairs; inspect gene tokenization, matching unit, and cross-cohort transfer
M56 PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images 2025 · ICCV, peer reviewed Follow the official paper’s project/code link if present reports, histology, and biological pathways for survival; compare against equally informed unimodal and bimodal baselines
M56a PANDA: Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning 2026 · arXiv preprint Code promised upon acceptance, not public at the cutoff align partially paired WSI and RNA representations to class prototypes, then deploy with WSI alone; preserve complete-case selection, missingness, shuffled-pair, histology-only, and uncertainty controls

Fair multimodal ablations

Section titled “Fair multimodal ablations”

For a fusion paper, require:

  1. each unimodal model with a comparable capacity and tuning budget;
  2. a simple late-fusion baseline;
  3. missing-modality evaluation, including informative missingness;
  4. shuffled-pair and permuted-modality controls;
  5. complete-case versus all-case cohort accounting;
  6. performance with each modality removed at test time;
  7. calibration and survival metrics appropriate to censoring;
  8. an explicit statement of which assays and reports are required at deployment.

An omics-supervised histology encoder can be used with H&E alone after pretraining; a histology–omics fusion predictor generally cannot. Do not merge those inference contracts.

11. Grounding, hallucination, and visual-necessity evaluation

Section titled “11. Grounding, hallucination, and visual-necessity evaluation”
ID Primary paper Year / status Author artifact Read for
M57 PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathology 2025 · Findings of EMNLP, peer reviewed Follow official paper links PathoHR-Bench for hierarchy and compositional semantics rather than isolated label recognition
M58 PathVG: A New Benchmark and Dataset for Pathology Visual Grounding 2025 · MICCAI, peer reviewed Code and data RefPath with 27,610 images and 33,500 language-grounded boxes; the paper reports that generated-box quality control required rejection or revision for most candidates, so audit one-box assumptions and 20×/40× scope
M59 AcZeroTS: Active Learning for Zero-shot Tissue Segmentation in Pathology Images 2025 · ICCV, peer reviewed Follow the official paper’s code link if present language-guided tissue segmentation and active selection; a bridge from similarity to spatially testable output
M60 PathGLS 2026 · arXiv preprint Code reference-free Grounding, Logic, and Stability checks; useful research signals that still require prospective expert validation
M61 PathAgentBench 2026 · arXiv preprint, submitted 21 July Author links on arXiv 1,822 TCGA WSIs with multiscale diagnostic paths plus a private cohort; evidence interpretation, verification, localization, and multiscale integration
M62 Do Pathology Vision-Language Models Truly See Pathology? (PathBind) 2026 · arXiv preprint, submitted 23 July Author links on arXiv visual-necessity, multimodal-gain, teaching-atlas, and grounding tests; a current warning that answer accuracy can exceed visual binding
M63 VIPER: An Expert-Curated Benchmark for VLMs in Veterinary Pathology 2026 · arXiv preprint Code · data 1,251 expert questions over 419 rat H&E regions test veterinary/toxicologic knowledge and visual grounding; this is a 1,024-pixel ROI benchmark, not WSI evaluation
M64 MVC-Bench: Benchmarking Calibration of Medical VLMs 2026 · Findings of EMNLP acceptance reported on arXiv; final ACL record pending Code calibration across histopathology, fundus, and chest-radiography classification with ECE/MCE/ACE and margin regularization; not evidence for report, VQA, segmentation, or workflow calibration

VIPER and MVC-Bench answer incompatible questions. VIPER tests domain-specific expert knowledge and grounding outside human pathology; MVC-Bench tests probability calibration in controlled medical classification. Keep answer accuracy, visual binding, calibration, domain transfer, and clinical utility as separate axes.

PathView-Bench adds fine-grained multiscale pathology questions with vision-anchored evaluation, so answer correctness and localization/scale use can be inspected separately. ZIO adds a complementary deployment question: can a pathology VLM define visual and text prototypes that reject slide- and patch-level out-of-distribution inputs without fitting an OOD detector? Use the ZIO code to reproduce the declared training-free protocol, while keeping prototype construction, one-shot support, shrinkage, and threshold selection inside the evaluation boundary.

M60–M62 are recent preprints, not settled benchmark standards. They are included because they formalize failures that should already be tested independently.

Hallucination taxonomy

Section titled “Hallucination taxonomy”

Tag each generated claim as one of:

  • unsupported morphology: a visible feature is asserted but absent;
  • unobservable fact: history, molecular result, margin, stage, treatment, or another-slide finding is asserted without that input;
  • wrong localization: a correct term points to the wrong tissue or scale;
  • false quantity: count, size, grade, proportion, or measurement is invented;
  • contradiction: the response conflicts with itself or supplied evidence;
  • spurious rationale: the answer is correct but the stated visual evidence is wrong;
  • unsafe certainty: uncertainty, differential diagnosis, or need for ancillary testing is omitted.

Do not call every incorrect answer a hallucination. Also do not let a correct final label erase an unsupported rationale.

Minimum visual-binding controls

Section titled “Minimum visual-binding controls”

For every VQA or assistant benchmark, run:

A. question + correct image
B. question only
C. question + blank image
D. question + random in-domain image
E. question + same-class different case
F. question + correct image with diagnostic region masked
G. question + diagnostic region only
H. paraphrased question and shuffled answer order

Report the gain from A over B–E, not only A’s absolute accuracy. For grounding, compare with center, tissue-mask, saliency, and random-box baselines. Validate boxes or masks at a declared MPP; an ROI that is plausible at thumbnail scale may miss the diagnostic cells.

12. Leakage and provenance audit

Section titled “12. Leakage and provenance audit”

Multimodal corpora have more leakage channels than image-only cohorts:

ID Primary paper Year / status Read for
M63 Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks 2026 · arXiv preprint, submitted July 2026 provenance, memorization, and train/test identity checks before interpreting a leaderboard; treat its proposed audit as emerging evidence rather than a settled standard
Leakage channel Required check
Patient/specimen/slide overlap hash identifiers and keep every related slide, level, stain, crop, and report in one split
Near-duplicate figures/video frames perceptual hashes and temporal/source grouping before splitting
Public benchmark reuse compare pretraining sources with TCGA, CPTAC, PANDA, CAMELYON, PAIP, PathVQA, PathMMU, Quilt, and downstream source lists
Text contains target remove or separately score diagnosis names, answer choices, synoptic codes, and explicit biomarkers
Synthetic-teacher contamination record teacher, prompt, retrieval corpus, date, and whether the teacher could have seen the evaluation source
Report-to-slide mismatch map report statements to the exact specimen, block, stain, and slide where possible
Instruction/evaluation reuse deduplicate question stems, images, captions, reports, and source documents across stages
Site/scanner shortcut quantify institution, scanner, stain, and tissue-preparation predictability
Private-data opacity label overlap as unknown, not “none,” when identifiers or source lists cannot be audited

For internet-derived data, also record copyright/license, removal requests, usernames/identifiers, and whether redistribution differs from model-training permission.

13. A fair comparison protocol

Section titled “13. A fair comparison protocol”

There is no defensible global VLM leaderboard. Compare one capability at a time.

A. Freeze the task contract

Section titled “A. Freeze the task contract”

Prespecify:

  • visual unit and MPP/magnification;
  • one or multiple images/slides per case;
  • text/context available at inference;
  • candidate labels, prompt templates, and prompt-development split;
  • tile encoder, slide aggregator, coordinates, tile budget, and selection rule;
  • frozen, linear-probe, adapter, or full-finetuning regime;
  • parameter, GPU-hour, and annotation budget;
  • patient/site splits and pretraining-overlap status.

B. Match the baselines

Section titled “B. Match the baselines”
Claim Minimum baselines
Better representation same downstream head on frozen visual SSL, general biomedical VLM, and pathology VLM encoders
Better zero-shot classifier identical class ontology, prompts, temperature/calibration, tile sampling, and WSI aggregation
Better assistant same visual encoder with text-only, blank-image, shuffled-image, and oracle-region controls
Better WSI reasoning same tile budget and resolution ladder; oracle ROI versus autonomous search
Better report generation retrieval/template baseline, image-only model, text-context-only model, and expert-written reference quality audit
Better multimodal fusion unimodal, simple late fusion, missing-modality, and shuffled-pair baselines
Better grounding center/tissue/random boxes, same phrase set, and expert-adjudicated region labels

C. Match metrics to the task

Section titled “C. Match metrics to the task”
  • Classification: macro AUROC/AUPRC, balanced accuracy, per-class sensitivity, specificity, calibration, confidence intervals, and external sites.
  • Retrieval: both directions, Recall@K, median rank, candidate-pool size, and patient-level duplicate exclusion.
  • VQA: answer accuracy by question family plus visual-necessity gain and abstention.
  • Open response: blinded pathologist correctness, severity, preference, inter-rater agreement, and a structured error taxonomy.
  • Grounding: box/mask IoU, point accuracy, recall at an operating threshold, and scale-aware expert review.
  • Report generation: HARE-style entities/relations, factual support, omissions, hallucination, calibration, and editing utility in addition to text similarity.
  • Survival: concordance with confidence intervals, time-dependent AUC, integrated Brier score, calibration, censoring protocol, and external validation.

D. Report uncertainty and multiplicity

Section titled “D. Report uncertainty and multiplicity”

Use patient-level bootstrap intervals or repeated patient-level folds. Do not treat tiles as independent samples. Correct or clearly account for repeated testing across many tasks, prompts, models, and endpoints. Report seeds and hyperparameter-search budgets.

14. Access, license, and reproducibility snapshot

Section titled “14. Access, license, and reproducibility snapshot”

Checked 20 August 2026; re-check every card and dependency before use. Paper access, code license, weight terms, source-data terms, and base-model license are separate.

Artifact Access observed Important constraint
PLIP public repository and weights official card frames it as research use; deployment and source-data rights require separate review
QuiltNet / Quilt-1M public code and model; full dataset access is restricted repository/model licensing does not grant redistribution of the underlying media
CONCH gated weights with institutional registration CC BY-NC-ND 4.0, non-commercial academic research, no redistribution; released checkpoint excludes the multimodal decoder
MUSK public code, gated weights repository states CC BY-NC-ND 4.0 and non-commercial academic research
Quilt-LLaVA gated model model card states non-commercial terms and inherits base-model obligations; inspect all component licenses
PathChat public training code and some benchmark material trained weights are not released because of proprietary patient-text privacy/IP restrictions
CPath-Omni CPath-CLIP delta/reconstruction resources available official repository still said the full CPath-Omni model would be released soon when checked
SlideChat code, model, instruction data, and benchmark linked by authors Apache-2.0 repository code does not override licenses of CONCH, LLMs, WSIs, or derived data
PRISM / PRISM2 gated model cards CC BY-NC-ND 4.0, non-commercial academic research, no redistribution, and explicit non-clinical restrictions
TITAN gated model card and public code CC BY-NC-ND 4.0, non-commercial academic research, institutional registration, no redistribution
mSTAR public code with gated weights author repository states CC BY-NC-ND 4.0 and non-commercial academic research
HistoGPT public code and weights code is Apache-2.0; inspect the weight card, training data, and base-model terms separately
ANTONI-alpha public pipeline/data/model links verify each HISTAI-derived artifact and base model rather than assuming one license covers the stack

“Open access paper,” “public repository,” “downloadable weights,” and “open-source model” are four different statements. Preserve the exact one supported by the source.

15. Practical exercises

Section titled “15. Practical exercises”

Exercise 1 — reproduce a small retrieval study

Section titled “Exercise 1 — reproduce a small retrieval study”

Use a public pathology image set with patient IDs and manually written morphology phrases. Compare PLIP, QuiltNet, and one general biomedical VLM using their official preprocessing. Report both retrieval directions, patient-deduplicated candidate pools, prompt variants, uncertainty, throughput, memory, access terms, and failure clusters.

Exercise 2 — prompt sensitivity without label leakage

Section titled “Exercise 2 — prompt sensitivity without label leakage”

Build prompts in three stages: bare class names, morphology descriptions written without looking at test images, and an independently curated ontology. Freeze the prompt set before evaluation. Quantify rank changes, calibration, and per-class variance. Do not ask an LLM to generate prompts from benchmark examples.

Exercise 3 — prove that a VQA model uses the image

Section titled “Exercise 3 — prove that a VQA model uses the image”

Implement controls A–H from the visual-binding section. Stratify by microscopy, diagnosis, clinical knowledge, ancillary testing, counting, and localization. Manually adjudicate at least 100 disagreements between correct-image and text-only conditions.

Exercise 4 — audit one WSI–report corpus

Section titled “Exercise 4 — audit one WSI–report corpus”

For 50 specimens, map every report sentence to visible H&E morphology, another slide/stain, clinical context, molecular data, or unknown provenance. Estimate the upper bound on image-supported report content and redesign the training targets.

Exercise 5 — compare oracle and autonomous WSI search

Section titled “Exercise 5 — compare oracle and autonomous WSI search”

Give the same assistant: an expert ROI, a random tissue ROI, a center crop, and an autonomously selected ROI under equal tile and action budgets. Score acquisition, interpretation, and synthesis separately; save all coordinates and magnifications.

Exercise 6 — multimodal missingness

Section titled “Exercise 6 — multimodal missingness”

Train histology-only, omics-only, simple late-fusion, and learned-fusion survival models. Evaluate complete cases, all cases with modality missingness, shuffled pairs, and an external site. Report whether missingness itself predicts outcome.

Exercise 7 — build a factual report evaluator

Section titled “Exercise 7 — build a factual report evaluator”

Create an ontology of specimen, tissue, architecture, cytology, grade, invasion, margin, stain, biomarker, and uncertainty entities. Link each generated claim to a region or mark it non-visual. Compare n-gram, embedding, HARE-style, LLM-judge, and pathologist ratings; publish disagreements, not only correlations.

16. Current research gaps

Section titled “16. Current research gaps”
  1. Visual binding still trails answer fluency. Recent grounding and visual-necessity studies show that high VQA accuracy can coexist with weak dependence on diagnostic pixels.
  2. Evidence acquisition is under-measured. Most benchmarks supply a salient crop or pre-extracted features; autonomous localization on gigapixel WSIs remains much harder.
  3. Pair semantics are poorly standardized. Figure captions, narrations, patch descriptions, slide reports, and case diagnoses are often mixed as “text.”
  4. Unobservable report content is a training hazard. Models are rewarded for producing clinical, molecular, and multi-slide facts unavailable in the pixels.
  5. Rare disease evidence is fragile. Small private cohorts, ontology choices, and retrieval corpora can dominate apparent gains.
  6. Synthetic instruction provenance is incomplete. Teacher identity, model version, prompts, retrieval sources, filtering, and benchmark exposure are not consistently recorded.
  7. Grounding labels are scarce and scale-dependent. Boxes may not capture diffuse, exclusionary, or relational findings, while attention is not a substitute.
  8. Agent traces are not automatically faithful. A readable chain of thought can rationalize a language prior; interventions on images and actions are needed.
  9. Cross-institution and temporal validation is limited. Scanner, stain, reporting style, prevalence, and workflow changes can break both modalities.
  10. Complete-case multimodal cohorts are selected populations. Missing reports, assays, or slides can encode care pathways and bias fusion results.
  11. Licensing blocks reproducibility and translation. Many leading checkpoints are gated, non-commercial, non-derivative, or unavailable.
  12. Clinical utility is rarely tested prospectively. Accuracy does not measure pathologist editing burden, automation bias, escalation, safety, or patient impact.

17. Maintenance protocol

Section titled “17. Maintenance protocol”

New papers should first enter the Paper Radar and move here only after:

  1. title, authors, date, and peer-review status are verified at a primary source;
  2. the new scientific role is stated and the paper is inserted directly into the corresponding task subsection—not into a month- or date-based update bucket;
  3. training/evaluation units and modality availability are recorded;
  4. code, weights, data, and licenses are checked independently;
  5. overlap and synthetic-data provenance are assessed;
  6. a prerequisite location and at least one audit question are assigned.

Create a new scientific subsection only when the work introduces a genuinely distinct task, input/output contract, or evaluation problem that the existing taxonomy cannot describe.

For preprints, store the arXiv version and review date. Re-check for a version of record, changed cohort counts, withdrawn claims, and revised artifacts. Keep every preprint clearly labeled as a research lead until peer review and independent use provide stronger evidence; keep accepted papers labeled as proceedings forthcoming until a version of record exists.

Reading-note template

Section titled “Reading-note template”
paper_id: M00
status_checked: YYYY-MM-DD
visual_unit: patch | region | slide | multi_slide | active_wsi
text_or_other_unit: caption | report | question | dialogue | omics | clinical
pairing: exact | weak | retrieved | generated
objectives: []
inference_modalities: []
patients_slides_pairs_questions: {}
public_overlap: none | possible | known | unknown
visual_necessity_controls: []
external_sites: []
artifacts:
  code: ""
  weights: ""
  data: ""
  licenses_checked: []
main_contribution: ""
strongest_evidence: ""
largest_validity_threat: ""
next_experiment: ""

The central habit is simple: never ask only whether a pathology VLM answered correctly. Ask what evidence it received, what evidence it acquired, whether its language is grounded in that evidence, and whether the comparison kept every other advantage fixed.

Taxonomy-conditioned diagnostic label generation

TaxoMIL generates coarse and fine diagnostic strings using a shared decoder, branch-specific prefixes and hierarchy-aware losses. Its target is a known label pair, so compare it with hierarchical MIL and exact-match diagnosis rather than free-text report generation.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image AnalysisChaeyeon Lee, Khang Nguyen Quoc, Jinsol Song, Yosep Chong, Kwangil Yim, and Jin Tae Kwak2026 · ECCV · acceptedOfficial ECCV 2026 poster and conference-hosted manuscript verify acceptance; no Springer version of record was verified. Author spelling follows the paper rather than the abbreviated program entry. Output is coarse/fine diagnostic label text, not full clinical reports. GastWSI is private.TaxoMIL learns to generate a coherent pair of coarse and fine diagnostic labels from a whole slide. It combines MIL aggregation, two granularity-specific visual prefixes, a shared GPT-2 decoder with separate heads, and taxonomy-aware training losses. The key teaching idea is to structure both the label space and its alignment with visual evidence before judging hierarchical predictions.Official ECCV 2026 Poster · Official implementation · arXiv manuscript

Reading question: Does each prediction satisfy both granularity-specific correctness and the supplied parent–child taxonomy?

Evidence boundary: Taxonomy-aware training is not a formal guarantee for arbitrary generated strings; private cohorts and predefined labels limit transfer claims.

Tissue distillation: masks, features, hierarchy and relations

Compare staged pixel/feature KD from Virchow2, PAGET’s hierarchical multi-teacher pseudo-labels, and ConStruct’s transfer of SegFormer structural affinities into adapted CONCH features for prototype-guided segmentation. Their annotation contracts and deployed components differ.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology SegmentationKhang Le, Ha Thach, Anh M. Vu, Trang T. K. Vo, Han H. Huynh, David Yang, Minh H. N. Le, Thanh-Huy Nguyen, Akash Awasthi, Chandra Mohan, Zhu Han, and Hien Van Nguyen2026 · MIDL · peer-reviewedPeer-reviewed MIDL 2026 proceedings, PMLR 315:2893–2905; official record verified 2026-10-02.ConStruct adds spatial organization to a pathology vision–language model for weakly supervised tissue segmentation. A frozen SegFormer teacher supplies token-to-token relations; trainable adapters refine frozen CONCH image features, and text-initialized prototypes turn the refined features into class activation maps. Training needs image-level tissue labels rather than dense pixel masks. The reported experiment is on BCSS-WSSS patches, not a whole-slide classification or cross-magnification benchmark.Official proceedings PDF · Official implementation

Reading question: Which supervision remains observable at student resolution, and which teacher or post-processing components still run at inference?

Evidence boundary: PAGET-H retains HoverNet. ConStruct’s residual adapters are not residual-error targets. Pixel segmentation, patch throughput and patient-level WSI outcomes are distinct evidence.

Case-level composites and diagnostic text transfer

WILSON combines actual multi-magnification tissue views into composite images and studies patient-level embedding and text-generation variants. Separate image-embedding, report-distillation and generative branches.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
WILSON - a pathology foundation model framework for patient-level analysis and diagnostic text generationSaghir Alfasly, Wataru Uegami, Sobhan Hemati, Wenchao Han, Xiaojia Tang, Kevin Thompson, Daniel Stone, Ghazal Alabtah, Saba Yasir, Michael R. Lucas, Eric W. Klee, Cheryl L. Willman, Judy C. Boughey, Matthew P. Goetz, Krishna R. Kalari, and H. R. Tizhoosh2026 · arXiv · preprintarXiv v1 submitted 2026-09-20. No peer-reviewed record was verified. The manuscript says model release is planned after publication and institutional clearance; Mayo training WSIs/reports are not publicly downloadable.WILSON makes a slide or a multi-slide patient case into a compact multi-magnification composite image, then learns one visual representation aligned with pathology-report supervision. It combines an organ-aware region-selection pipeline, a convolutional encoder, and alternative dense-text, keyword or CLIP alignment branches; a CoCa-style decoder adds caption generation. The central idea is to change the computational unit from thousands of separately encoded tiles to a structured image summary.Full manuscript and supplementary material

Reading question: What images and reports are needed at training and deployment, and which compute path was measured?

Evidence boundary: The system still reads high-power tissue; analytic embedding TFLOPs are not end-to-end speed. It is a preprint with restricted training data and a planned model release.

From fixed tissue graphs to evolving concept-guided fields

TMEvolve groups frozen CONCHv1.5 patch features into soft semantic regions, exchanges messages on a spatial graph, and lets regional and local states repeatedly update each other before slide prediction. Concept guidance and task supervision do not make its latent regions measured cell types or biological dynamics.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Modeling Whole-Slide Images as Dynamic Tumor Microenvironment FieldsLei Wu, Jiashuai Liu, Di Zhang, Zhangpeng Gong, Yingkang Zhan, Yi Niu, Jiusong Ge, Chunze Yang, Kai Yi, Mireia Crispin-Ortuzar, Chen Li, and Zeyu Gao2026 · NeurIPS · acceptedAccepted as a NeurIPS 2026 poster: official conference program event 149764 matches the exact title and 12 authors, checked 2026-10-08. Final proceedings were not verified; review uses arXiv v1 (2026-09-28). Official method code is available under a noncommercial research license, but does not include the WSI feature-extraction pipeline. Tissue-tile pixel size, extraction magnification/MPP, and image preprocessing remain unreported in the inspected paper and release.TMEvolve replaces a static bag of patch features with adaptive tissue regions whose states are repeatedly refined. Similar-region neighbors exchange visual information; heterogeneous boundaries carry directional visual and pathological-concept messages. The final regions are pooled for slide-level prediction. This is reaction-diffusion-inspired representation learning, not a simulation of tumor progression or a generative diffusion model.Reviewed manuscript and supplement (v1) · Author publication and acceptance statement · Official NeurIPS 2026 poster · Official method implementation (noncommercial research)

Reading question: Does adaptive region–patch interaction add value beyond the same encoder with set pooling or a fixed spatial graph?

Evidence boundary: Tile pixel size, extraction magnification and MPP are not reported in the inspected paper or method release. The 512 setting is a hidden feature width, not pixels; offline feature extraction is excluded from the reported method cost.

Surprise-guided navigation and reusable slide memory

PathNavigate uses a low-power map to guide higher-power evidence collection, reusing scan statistics during answer synthesis. Predictive surprise directs exploration while the answering model reads selected tissue evidence; local high-power memory is separate from the global scan memory.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQAChunze Yang, Qidong Liu, Wenjie Zhao, Yue Tang, Jiusong Ge, Di Zhang, Jiashuai Liu, Lei Wu, Junbo Lu, Ni Zhang, Xian Wu, Zeyu Gao, and Chen Li2026 · NeurIPS · acceptedOfficial NeurIPS 2026 poster 154224 verifies acceptance; program title shortens 'Whole-Slide Image VQA' to 'Whole-Slide VQA'. Review uses arXiv:2605.23559v1, not an uninspected camera-ready. Backbone-frozen navigation includes test-time memory optimization and high-magnification evidence; not 5×-only deployment.PathNavigate first establishes what is unusual within the current slide, then asks which of those regions is relevant to the question. A frozen pathology encoder feeds a small per-slide reconstruction memory; its pre-update gradient norm proposes regions, PLIP reranks that pool, and frozen perception and language models answer from selected high-magnification evidence. The same scan statistics provide compact answer-time context. Training-free here means no task-specific backbone training, not absence of test-time optimization.Official NeurIPS 2026 program · Versioned full manuscript · Official inference implementation

Reading question: How much of the answer requires newly sampled high-power tissue, and what does the shared scan summary contribute beyond local evidence?

Evidence boundary: Training-free refers to frozen backbones, not zero optimization: the memory MLP learns online. Released nominal magnifications depend on a hard-coded native 40× assumption; verify scanner metadata. This is not 5×-only inference.

Question-conditioned evidence acquisition across magnifications

PathPilot separates question-agnostic diagnostic localization from question-dependent evidence accumulation, movement, zoom and stopping. The agent accesses real multi-scale ROIs; it is neither low-only distillation nor a new MIL pooling operator. Its MIL experiment evaluates whether selected regions remain useful.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
PathPilot: An Adaptive Question-Conditioned Multi-Magnification Navigation Agent for Whole-Slide Image AnalysisZhengyang Xu, Han Li, Jingsong Liu, Ayako Ura, Shihui Zu, Shaohua Kevin Zhou, Nassir Navab, and Peter Schüffler2026 · arXiv · preprintarXiv:2610.16734v1, submitted 12 October 2026; manuscript says under review, with no independently verified acceptance. Full 24-page public manuscript inspected; referenced supplementary material was not attached or independently obtained. Code is promised on publication, not a verified release. Adaptive thumbnail/1.25×/2.5×/5×/10× exploration is not knowledge distillation or low-resolution-only inference. QA scores use a language-model judge; deterministic repeated runs do not establish population uncertainty.PathPilot separates where tissue looks diagnostically promising from what evidence a particular question still needs. CrossMagLoc learns question-agnostic region priorities across magnifications; EviRegionSearch uses a VLM, language-model memory and explicit evidence gaps to move, zoom or stop. This is adaptive acquisition of real multi-magnification evidence, not distillation into a low-resolution-only student.Inspected arXiv v1 manuscript

Reading question: What evidence is missing, which region and scale can supply it, and what justifies stopping?

Evidence boundary: Small morphology subsets and a language-model judge limit inference. A same-institution cohort called external is not independent-site validation; deterministic repeatability is not patient uncertainty.

Complete maintained paper route

Section titled “Complete maintained paper route”

81 canonical readings · 2021–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 81 readings in maintained order

  52. #522026MICCAIPeer reviewedL3

    Pathologist Attention–Aligned Report Generation for Prostate Histopathology

    Ruoyu Xue, Suryakant Singh, Souradeep Chakraborty, Pierre Marza, Oksana Yaskiv, Constantin Friedman, Natallia Sheuka, Paul Friedman, Bharat Ramlal, Beatrice Knudsen, Rajarsi Gupta, Joel Saltz, Prateek Prasanna, Gregory Zelinsky, and Dimitris Samaras

    Official MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02; Springer DOI and pagination remain pending.

    Official artifacts: Code and attention data · Official MICCAI 2026 proceedings · Official accepted-version PDF

    Metadata checked

  58. #582026NeurIPSAccepted · proceedings pendingL3

    PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?

    Zongyi Chen, Yu Liang, Jie Lin, and Liansheng Wang

    Acceptance independently verified 2026-10-08: official NeurIPS poster 139172 exactly matches the title and four authors and links OpenReview gZJHNK7GHw. The scientific review remains tied to arXiv:2607.28318v1 and its 2026-08-08 full-text audit; this metadata refresh did not re-audit code or dataset availability.

    Official artifacts: Official NeurIPS 2026 acceptance record

    Metadata checked

  64. #642026arXivPreprintL2–L3

    Paired Uterine Whole-Slide Images and Pathology Reports for Multimodal Computational Pathology

    Han Li, Jingsong Liu, Ayako Ura, Junlin Hou, Zhengyang Xu, Azar Kazemi, Oskar Thaeter, Christian Grashei, Fabian Gülhan, Reza Nasirigerdeh, Xun Ma, Rui Yan, Hao Chen, S. Kevin Zhou, Nassir Navab, Carolin Mogler, and Peter Schüffler

    arXiv preprint submitted 4 July 2026 and revised as v2 on 17 July 2026; no peer-reviewed version was verified on 20 August 2026.

    Official artifacts: Official gated dataset card

    Metadata checked

  68. #682026NeurIPSAccepted · proceedings pendingL3

    VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

    Luca L. Weishaupt, Simone de Brot, Javier Asin, Llorenç Grau-Roma, Nic G. Reitsam, Andrew H. Song, Dongmin Bang, Stefan T. Kaluziak, Long Phi Le, Jakob Nikolas Kather, Faisal Mahmood, and Guillaume Jaume

    Acceptance independently verified 2026-10-08 through official NeurIPS poster 139742: exact title and matching 12-author list, linked OpenReview 7986DZ9rPe. Scientific review and artifact checks remain tied to arXiv:2608.26382v1 and the 2026-08-30 audit, not a refreshed full-text/code/data inspection. VIPER evaluates rat toxicologic regions rather than whole-slide images; it is an ROI benchmark, not a WSI benchmark.

    Official artifacts: Official evaluation code and documentation · Official benchmark dataset · Official NeurIPS 2026 acceptance record

    Metadata checked

  69. #692026Findings of EMNLPAccepted · proceedings pendingL3

    MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models

    Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan, Muhammad Akhtar Munir Sujair Ibrahim, Mohamed Rafeek Mareer Ahamed, Yutong Xie, Imran Razzak, and Muhammad Haris Khan

    The arXiv record reports acceptance to Findings of EMNLP 2026. No final ACL Anthology record was available when checked on 30 August 2026, so claims were verified against arXiv v1 and the official repository.

    Official artifacts: Official implementation

    Metadata checked

  70. #702026arXivPreprintL3

    Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation

    Yumi Lee, Harim Oh, Hyoryung Kim, Minji Kim, Eunsu Kim, Hyeseong Lee, Junya Fukuoka, Andrey Bychkov, Jijgee Munkhdelger, Rajiv Kumar Kaushal, Ayushi Sahay, Rajni Yadav, Bharathi Prabakaran, Sulen Sarioglu, Serdar Balcı, Ilknur Turkmen, Yuri Tolkach, Christian Harder, Julian Westerdorf, Reinhard Buettner, Audun Ljone Henriksen, Sepp De Raedt, Byung Hyun Lee, Sungjin Lim, Joohoon Lee, Gwanghyun Kim, Se Young Chun, Suryakant Singh, Saarthak Kapse, Prateek Prasanna, Kyung A Kim, Yousun Kang, Sehwan Yoo, Sungman Hong, Shubham Innani, Michael Feldman, Spyridon Bakas, Ujjwal Baid, Prasad Dutande, Suhas Gajare, Bhakti Baheti, Serkan Sökmen, Ece Tuğba Cebeci, Ahmet Halıcı, Musa Balcı, Kardelen Peçenek, Srividhya Sainath, Kyongseok Jang, Messi H. J. Lee, Noorul Wahab, Bodong Du, Jiaming Zhang, Qixiang Zhang, Jang-Hwan Choi, and Sangjeong Ahn

    arXiv v1 submitted 1 September 2026 reports the results of the MICCAI REG 2025 challenge. No final peer-reviewed proceedings or journal record was verified on 2 September 2026. The approximately 10,500 institutional WSI-report pairs are not a generally downloadable public dataset; the released repositories expose benchmark information and team artifacts.

    Official artifacts: Official challenge repository · Official winning-system repository · Official challenge site

    Metadata checked

  72. #722026ECCVAccepted · proceedings pendingL3

    TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image Analysis

    Chaeyeon Lee, Khang Nguyen Quoc, Jinsol Song, Yosep Chong, Kwangil Yim, and Jin Tae Kwak

    Official ECCV 2026 poster and conference-hosted manuscript verify acceptance; no Springer version of record was verified. Author spelling follows the paper rather than the abbreviated program entry. Output is coarse/fine diagnostic label text, not full clinical reports. GastWSI is private.

    Official artifacts: Official ECCV 2026 Poster · Official implementation · arXiv manuscript

    Metadata checked

  73. #732026arXivPreprintL3

    WILSON - a pathology foundation model framework for patient-level analysis and diagnostic text generation

    Saghir Alfasly, Wataru Uegami, Sobhan Hemati, Wenchao Han, Xiaojia Tang, Kevin Thompson, Daniel Stone, Ghazal Alabtah, Saba Yasir, Michael R. Lucas, Eric W. Klee, Cheryl L. Willman, Judy C. Boughey, Matthew P. Goetz, Krishna R. Kalari, and H. R. Tizhoosh

    arXiv v1 submitted 2026-09-20. No peer-reviewed record was verified. The manuscript says model release is planned after publication and institutional clearance; Mayo training WSIs/reports are not publicly downloadable.

    Official artifacts: Full manuscript and supplementary material

    Metadata checked

  74. #742026MIDLPeer reviewedL3

    ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology Segmentation

    Khang Le, Ha Thach, Anh M. Vu, Trang T. K. Vo, Han H. Huynh, David Yang, Minh H. N. Le, Thanh-Huy Nguyen, Akash Awasthi, Chandra Mohan, Zhu Han, and Hien Van Nguyen

    Peer-reviewed MIDL 2026 proceedings, PMLR 315:2893–2905; official record verified 2026-10-02.

    Official artifacts: Official proceedings PDF · Official implementation

    Metadata checked

  75. #752026NeurIPSAccepted · proceedings pendingL3

    Modeling Whole-Slide Images as Dynamic Tumor Microenvironment Fields

    Lei Wu, Jiashuai Liu, Di Zhang, Zhangpeng Gong, Yingkang Zhan, Yi Niu, Jiusong Ge, Chunze Yang, Kai Yi, Mireia Crispin-Ortuzar, Chen Li, and Zeyu Gao

    Accepted as a NeurIPS 2026 poster: official conference program event 149764 matches the exact title and 12 authors, checked 2026-10-08. Final proceedings were not verified; review uses arXiv v1 (2026-09-28). Official method code is available under a noncommercial research license, but does not include the WSI feature-extraction pipeline. Tissue-tile pixel size, extraction magnification/MPP, and image preprocessing remain unreported in the inspected paper and release.

    Official artifacts: Reviewed manuscript and supplement (v1) · Author publication and acceptance statement · Official NeurIPS 2026 poster · Official method implementation (noncommercial research)

    Metadata checked

  76. #762026NeurIPSAccepted · proceedings pendingL3

    PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA

    Chunze Yang, Qidong Liu, Wenjie Zhao, Yue Tang, Jiusong Ge, Di Zhang, Jiashuai Liu, Lei Wu, Junbo Lu, Ni Zhang, Xian Wu, Zeyu Gao, and Chen Li

    Official NeurIPS 2026 poster 154224 verifies acceptance; program title shortens 'Whole-Slide Image VQA' to 'Whole-Slide VQA'. Review uses arXiv:2605.23559v1, not an uninspected camera-ready. Backbone-frozen navigation includes test-time memory optimization and high-magnification evidence; not 5×-only deployment.

    Official artifacts: Official NeurIPS 2026 program · Versioned full manuscript · Official inference implementation

    Metadata checked

  77. #772026arXivPreprintL3

    PathPilot: An Adaptive Question-Conditioned Multi-Magnification Navigation Agent for Whole-Slide Image Analysis

    Zhengyang Xu, Han Li, Jingsong Liu, Ayako Ura, Shihui Zu, Shaohua Kevin Zhou, Nassir Navab, and Peter Schüffler

    arXiv:2610.16734v1, submitted 12 October 2026; manuscript says under review, with no independently verified acceptance. Full 24-page public manuscript inspected; referenced supplementary material was not attached or independently obtained. Code is promised on publication, not a verified release. Adaptive thumbnail/1.25×/2.5×/5×/10× exploration is not knowledge distillation or low-resolution-only inference. QA scores use a language-model judge; deterministic repeated runs do not establish population uncertainty.

    Official artifacts: Inspected arXiv v1 manuscript

    Metadata checked