Optimization, regularization, and capacity
A better-trained student can look like a better distillation method.
Knowledge distillation & model transfer
Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.
Knowledge distillation is not one loss. It is a family of teacher–student interfaces that transfer outputs, intermediate representations, relations, generated sequences, synthesized transfer inputs, or student-sampled training trajectories. This chapter keeps those interfaces and their data-access requirements separate and asks when a smaller or specialized student actually retains the teacher’s useful behaviour.
140 maintained papers · literature reviewed through 2 October 2026
Distillation transfers a teacher-dependent training signal, not an abstract guarantee of “knowledge.” Soft outputs, hidden features, pairwise relations, synthesized surrogate inputs, generated sequences, student rollouts, self-teachers, and multiple domain experts expose different information and failure modes. A valid comparison fixes data access, sampling policy, augmentation, optimization, student capacity, inference budget, and external evaluation before attributing a gain to distillation.
Prerequisite contract
Loss and objective design
Temperature, divergence, feature normalization, and loss weighting define the transferred signal.
How the field developed
2006–2015
From ensemble function matching to softened targets
Model Compression first trains a fast model on a large pseudo-labelled transfer set; Ba and Caruana show that a shallower network can reproduce a deeper ensemble function; Hinton et al. formalize temperature-scaled soft targets. Read them together so “distillation” is not reduced to one KL formula.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Model CompressionPeer-reviewed KDD 2006 proceedings paper, pages 535–541. Model Compression asks whether a deployable neural network can reproduce a cumbersome but accurate ensemble. A high-capacity ensemble labels a much larger synthetic transfer set with continuous predictions, and one compact multilayer perceptron learns that function. The paper therefore establishes an early knowledge-distillation pattern: separate the model used to discover a strong decision function from the model used to serve it. — 01.02 Do Deep Nets Really Need to Be Deep?Peer-reviewed NeurIPS 2014 proceedings paper. Ba and Caruana show that a shallow network can approximate part of the function learned by a deep convolutional teacher when it receives teacher logits on many inputs. The pre-softmax vector preserves relative class evidence that hard labels and saturated probabilities hide. The result is primarily about supervision and optimization: a function difficult to discover from labels may be easier to fit after a teacher exposes its response geometry. Author manuscript 01.03 Distilling the Knowledge in a Neural Network Knowledge distillation trains a deployable student to match a teacher or ensemble's softened class distribution, usually while also learning hard labels. Raising softmax temperature reveals relative probabilities among incorrect classes—the teacher's dark knowledge—and the student returns to temperature one at inference. The method transfers predictive behavior, not teacher parameters, hidden features, or a guarantee of equal calibration. — Work: Rebuild hard-label, ensemble-regression, raw-logit, and temperature-softened baselines with the same student and transfer set.
Exit evidence: Explain which information comes from extra examples, ensemble averaging, logits, temperature, and student capacity.
2015–2024
Responses, divergences, hints, attention, and compact states
Soft distributions compress teacher behaviour; hints, attention maps, relations, and Transformer states move the interface inside the network. DistilBERT, MiniLM, DistiLLM, and Wasserstein KD show that preserved state, sampling policy, and discrepancy all matter.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Distilling the Knowledge in a Neural Network Knowledge distillation trains a deployable student to match a teacher or ensemble's softened class distribution, usually while also learning hard labels. Raising softmax temperature reveals relative probabilities among incorrect classes—the teacher's dark knowledge—and the student returns to temperature one at inference. The method transfers predictive behavior, not teacher parameters, hidden features, or a guarantee of equal calibration. — 02.02 FitNets: Hints for Thin Deep Nets FitNets makes a thin student trainable even when it is deeper than its wide teacher by giving the student an intermediate target before asking it to imitate final predictions. A selected teacher hidden layer supplies a hint; a selected student layer plus a learned regressor predicts that hint. Only after this initialization does the whole student undergo ordinary label-and-teacher output distillation. Official code 02.03 Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer Attention Transfer compresses a convolutional feature tensor into a spatial map that says where a network responds, then trains a student to reproduce the normalized maps of a stronger teacher at corresponding resolutions. The student still learns the task label; the added loss transfers spatial emphasis without demanding channel-by-channel equality, so teachers and students may differ in width and depth. Official code 02.04 DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and LighterWorkshop paper and arXiv preprint; not a main-track NeurIPS proceedings paper. DistilBERT compresses BERT during language-model pretraining rather than only after task fine-tuning. A six-layer student initialized from alternating BERT-base layers learns from three simultaneous signals: the teacher token distribution, masked-token ground truth, and cosine alignment of hidden states. Removing half the Transformer blocks, token-type embeddings, and the pooler yields a 66M-parameter encoder intended to preserve much of BERT's general English representation at lower inference cost. Official distillation code 02.05 TinyBERT: Distilling BERT for Natural Language Understanding TinyBERT compresses BERT through two distinct teaching periods. General distillation transfers a pretrained teacher on unlabeled Wikipedia text; task-specific distillation then transfers a fine-tuned teacher on labeled and augmented task examples. At both stages the student imitates embeddings, hidden states, attention matrices, and—during task transfer—prediction logits, so compression covers the transformer's internal computation rather than only its final output. Official implementation 02.06 MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers MiniLM compresses a pretrained Transformer before task-specific fine-tuning by matching relations inside the teacher's final self-attention layer. It distills both attention distributions and a newly defined value-relation matrix. Because supervision is taken from only the last layer, teacher and student may differ in depth and hidden size without hand-designed layer mapping. Official implementation and checkpoints 02.07 DistiLLM: Towards Streamlined Distillation for Large Language ModelsPeer-reviewed ICML 2024 proceedings paper. DistiLLM targets two coupled failures in autoregressive language-model distillation: unstable token-distribution matching and train/test exposure mismatch. It replaces ordinary forward or reverse KL with skew KL divergences whose mixture reference controls estimator behavior, then adaptively trains on student-generated sequences and reuses recent generations through an off-policy buffer. The student therefore learns both from fixed instruction data and from states its own policy is likely to visit. Official code 02.08 Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation WKD asks whether knowledge distillation must compare teacher and student with Kullback-Leibler divergence or pointwise feature regression. Its answer is a two-level optimal-transport design: WKD-L transports probability mass between semantically related classes instead of charging every class mismatch equally, while WKD-F compares the distributions of spatial feature vectors with the closed-form 2-Wasserstein distance between Gaussians. The two losses can be used separately or together. The essential reading rule is therefore: logits are treated as structured categorical distributions; features are treated as geometric distributions, not coordinate-by-coordinate targets. Official project page · Official code Work: Reproduce logit, feature, attention, relation, KL, skew-KL, and Wasserstein transfer with one fixed teacher–student pair.
Exit evidence: State exact tensors, discrepancy, normalization, temperature, layer mapping, sampling distribution, and gradient recipients.
2016
Distillation as old-task preservation
Learning without Forgetting records the old model outputs on new-task images and preserves them while fitting new labels, avoiding direct access to old training files. It transfers an old decision function into a continually adapted model rather than compressing a larger teacher into a smaller student.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Learning Without ForgettingPeer-reviewed ECCV 2016 proceedings paper; the canonical locator is the open author manuscript. Learning without Forgetting adds a new task to a pretrained classifier without storing old task images. Before updating, the old model records its output distribution on the new-task images. Training then combines new-label supervision with a distillation loss that keeps those old responses stable while shared features change. It is old-data-free continual adaptation, not state-free learning: the pretrained backbone, old heads, and their responses remain essential. Official project · Official code Work: Compare fine-tuning, old-output distillation, replay, and joint training under identical task orders and retained-data access.
Exit evidence: Report old-task retention, new-task learning, task-order variance, model growth, stored data, and output-snapshot cost.
2018–2021
Peers, relations, and self-teachers
Mutual learning removes the fixed teacher, relational and contrastive methods preserve sample geometry, and born-again or momentum teachers make self-distillation a training strategy.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Deep Mutual Learning Deep Mutual Learning removes the fixed expert teacher. Two or more networks start from independent initializations and train together on the same mini-batches; every network minimizes its own supervised loss while matching the current class posterior of its peers. Each participant is simultaneously student and teacher, so useful secondary probabilities emerge during joint optimization instead of being copied from a pre-trained model. Official code 04.02 Born Again Neural Networks Born-Again Networks shows that distillation need not compress architecture: a freshly initialized student with the same architecture and capacity as its teacher can generalize better when trained on the teacher's output distribution. The student may become the teacher for another generation, and predictions from multiple generations may be ensembled, but the central claim concerns improved single descendants rather than compression alone. — 04.03 Relational Knowledge Distillation Relational Knowledge Distillation does not force a student to reproduce each teacher embedding independently. It transfers geometry among examples: normalized pairwise distances and triplet angles. A smaller student may therefore use a different coordinate system while preserving how samples are separated and oriented, which is especially natural for metric learning and retrieval. Official code 04.04 Contrastive Representation Distillation Contrastive Representation Distillation teaches a student to identify which teacher representation belongs to the same input. The matched teacher-student pair is positive; teacher representations from other examples are negatives. Optimizing this contrastive classification objective raises a lower bound on mutual information, encouraging the student to retain teacher information without reproducing every feature coordinate. Official code 04.05 Bootstrap Your Own Latent BYOL trains an online encoder–projector–predictor to match the normalized, stop-gradient projection made by a slowly moving target encoder–projector from another view of the same image. Only the online branch receives gradients; the target parameters are an exponential moving average of the online encoder/projector. The loss contains no explicit negatives, queue, prototype or reconstruction target, yet the asymmetric training dynamics empirically avoid constant representations. — 04.06 Emerging Properties in Self-Supervised Vision Transformers DINO is self-distillation with no labels: a student predicts the probability distribution produced by an exponential-moving-average teacher for another crop of the same image. Two large global crops go to both networks; additional small local crops go only to the student, creating local-to-global supervision. The teacher receives no gradient. Centering and sharpening its output counter opposing collapse modes, while the teacher's momentum update supplies a slowly evolving target. When applied to Vision Transformers, this recipe produces strong nearest-neighbor features and patch-token/attention structure without a segmentation objective. — 04.07 Training Data-Efficient Image Transformers & Distillation through Attention DeiT demonstrates that a ViT-B architecture can be trained competitively using ImageNet-1K alone through a carefully tuned augmentation, regularization, optimizer, and repeated-augmentation recipe. Its separate architectural contribution is a learned distillation token: it attends jointly with patch and class tokens, receives a teacher-derived target through its own head, and is fused with the class-token head at inference. — Work: Compare response matching with pairwise geometry and a moving-average teacher under identical views.
Exit evidence: Explain whether improvement comes from privileged targets, regularization, optimization, or an ensemble effect.
2020–2026
Capacity gaps, imbalance, and function-preserving CV transfer
Teacher assistants bridge harmful capacity gaps; ReviewKD, patient training, decoupled logits, masked generation, long-tail rebalancing, and streamlined variants reconsider what a student should inherit.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 Improved Knowledge Distillation via Teacher Assistant Teacher Assistant Knowledge Distillation addresses the capacity gap between a very large teacher and a small student. Instead of transferring directly, it trains an intermediate-capacity assistant from the teacher and then trains the student from that assistant. The assistant presents a function the student can approximate more faithfully, even though the assistant itself is weaker than the original teacher. Official code 05.02 Distilling Knowledge via Knowledge Review ReviewKD treats feature distillation as progressive knowledge review. A deeper student stage should not imitate only the teacher stage at the same depth; it can review several shallower teacher representations that contain reusable low- and mid-level knowledge. Attention-Based Fusion blocks aggregate student features recursively, and Hierarchical Context Loss compares fused student signals with teacher features at multiple spatial scales. Official code 05.03 Knowledge Distillation: A Good Teacher Is Patient and Consistent A Good Teacher Is Patient and Consistent argues that much of knowledge distillation's unrealized performance comes from protocol, not a new loss. The teacher and student must see the same strongly augmented input so they match the same function evaluation, and the student must train far longer than ordinary supervised schedules. With mixup-heavy augmentation and thousands of epochs, a compact ResNet approaches or exceeds stronger specialized KD recipes. Teacher model code and weights 05.04 Decoupled Knowledge Distillation Decoupled Knowledge Distillation rewrites classical logit KD into two interpretable signals. Target-Class KD matches the binary probability of the ground-truth class versus all other classes, conveying sample difficulty. Non-Target-Class KD renormalizes only the wrong-class probabilities and transfers their relative structure—the dark knowledge. DKD assigns these terms independent weights instead of letting teacher confidence couple them. Official code 05.05 Masked Generative Distillation Masked Generative Distillation transfers a teacher feature map by making the student reconstruct it from an intentionally incomplete student feature. A random spatial mask erases locations after channel alignment, and a small convolutional generator predicts the full teacher representation. The student must infer missing context rather than copy every activation pointwise, which supports classification, detection, and segmentation with the same basic objective. Official code 05.06 Streamlined Knowledge Distillation Streamlined Knowledge Distillation transfers two logit-level views: each example's softened class distribution and the directional relationships among examples in a batch. It L2-normalizes logits, forms a sample-by-sample Gramian of cosine similarities, and aligns teacher/student Gramian differences with a covariance-aware Mahalanobis loss. The method avoids intermediate feature adapters while retaining relational supervision. Official code 05.07 Distilling Balanced Knowledge from a Biased Teacher LTKD makes a crucial correction to the usual long-tailed distillation story: a teacher trained on imbalanced data is not automatically a balanced source of dark knowledge. Its probability mass is systematically larger for frequent Head classes and smaller for Tail classes, so ordinary KD can reproduce the bias. The paper decomposes KD into inter-group probability transfer and intra-group class relations for Head, Medium, and Tail groups. It then balances teacher group mass within each batch and assigns equal strength to within-group relational knowledge. The student still learns from labels, but its distillation target no longer lets frequency alone determine how much each group teaches. — Work: Sweep teacher size, class imbalance, and student capacity while holding data, augmentation, optimizer, and inference budget fixed.
Exit evidence: Separate gains from bridge teachers, transformations, loss decomposition, masking, bias correction, and extra optimization.
2019–2024
Data access, inversion, and rollout policy
Data-free student learning, Zero-Shot KD, and DeepInversion synthesize different transfer distributions from a frozen teacher; on-policy language distillation instead queries the teacher on prefixes visited by the evolving student. The source of training states is part of the algorithm.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 Data-Free Learning of Student NetworksPeer-reviewed ICCV 2019 proceedings paper. Data-Free Learning of Student Networks (DAFL) replaces unavailable training images with a generator optimized through a fixed teacher. The generator searches for inputs that make the teacher confident, strongly activated, and class-balanced; the student then matches teacher predictions on those synthetic inputs. The paper moves KD from 'teacher plus transfer data' to 'teacher as the source of a transfer set,' while still requiring internal teacher features for its full loss. Author manuscript 06.02 Zero-Shot Knowledge Distillation in Deep Networks Zero-Shot Knowledge Distillation handles a specific access constraint: the trained teacher is available, but neither its training examples nor metadata derived from them can be used. It synthesizes a transfer set by sampling plausible class-probability vectors from a teacher-derived Dirichlet model and optimizing random inputs until the teacher emits those vectors. The resulting Data Impressions then replace real examples in ordinary soft-target distillation. — 06.03 Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion DeepInversion synthesizes class-conditional images from a trained classifier when the original dataset is unavailable. It starts from noise and backpropagates through the frozen teacher, combining a desired class prediction with image priors and a feature-distribution prior derived from the running means and variances stored in batch-normalization layers. Adaptive DeepInversion additionally searches for examples on which teacher and student disagree. Official code 06.04 On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes Generalized Knowledge Distillation changes which prefixes receive teacher supervision. Instead of learning only on ground-truth or teacher-generated sequences, an autoregressive student samples complete outputs from its current policy; the teacher then supplies token distributions at those student-visited prefixes. A student-data fraction mixes these on-policy trajectories with a fixed dataset, while a selectable forward KL, reverse KL, or generalized Jensen-Shannon divergence controls how a capacity-limited student approximates the teacher. — Work: Compare original-data, generator-synthesized, teacher-inverted, fixed-output, and student-rollout training distributions.
Exit evidence: Record required model access, synthesis priors, sampling policy, and which privacy or shift claim is actually supported.
2016–2025
Sequence, pretraining, rationale, and generative trade-offs
Language students can inherit generated sequences, pretraining states, rationales, divergences, student trajectories, or a teacher-refined data distribution. DistilBERT and MiniPLM compress pretraining by different contracts; MiniLLM and DistiLLM make the student sampling distribution explicit.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 Sequence-Level Knowledge Distillation Sequence-Level Knowledge Distillation replaces a multimodal reference target with a complete sequence generated by a strong teacher. A compact sequence model is then trained by ordinary maximum likelihood on these teacher outputs. The central object is not a softened probability at each token but a decoded global hypothesis that exposes one coherent path through the teacher's sequence distribution. Official seq2seq implementation · Official mobile demonstration 07.02 DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and LighterWorkshop paper and arXiv preprint; not a main-track NeurIPS proceedings paper. DistilBERT compresses BERT during language-model pretraining rather than only after task fine-tuning. A six-layer student initialized from alternating BERT-base layers learns from three simultaneous signals: the teacher token distribution, masked-token ground truth, and cosine alignment of hidden states. Removing half the Transformer blocks, token-type embeddings, and the pooler yields a 66M-parameter encoder intended to preserve much of BERT's general English representation at lower inference cost. Official distillation code 07.03 Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Distilling Step-by-Step asks a large language model to annotate examples with both a task label and a natural-language rationale, then trains a much smaller model on two outputs. One prefixed task predicts the label; another predicts the rationale. The rationale supplies intermediate task knowledge during training, but the compact model only needs to generate the label at inference. Official code 07.04 MiniLLM: Knowledge Distillation of Large Language Models MiniLLM distills an autoregressive language model by minimizing reverse KL from the student sequence distribution to the teacher. The student generates its own responses, the teacher scores those trajectories, and policy-gradient machinery moves probability toward teacher-supported sequences. Reverse KL is mode seeking: it discourages the student from assigning mass to low-teacher-probability outputs rather than forcing it to cover every teacher mode. Official code 07.05 DistiLLM: Towards Streamlined Distillation for Large Language ModelsPeer-reviewed ICML 2024 proceedings paper. DistiLLM targets two coupled failures in autoregressive language-model distillation: unstable token-distribution matching and train/test exposure mismatch. It replaces ordinary forward or reverse KL with skew KL divergences whose mixture reference controls estimator behavior, then adaptively trains on student-generated sequences and reuses recent generations through an off-policy buffer. The student therefore learns both from fixed instruction data and from states its own policy is likely to visit. Official code 07.06 TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models TAID bridges a large language-model capacity gap by moving the student's target gradually from its own current distribution toward the teacher. At early steps, the interpolated target remains close to a detached student and is easy to fit; as learning stabilizes, the interpolation coefficient increases and exposes more teacher knowledge. The schedule adapts to objective progress instead of following only wall-clock training. Official code 07.07 MiniPLM: Knowledge Distillation for Pre-Training Language ModelsPeer-reviewed ICLR 2025 conference paper. MiniPLM moves language-model distillation out of the training loss and into pretraining-data selection. A large teacher and a much smaller reference model score each candidate sequence; examples with high log pteacher(x) − log preference(x) are selected, and any student is then trained from scratch with ordinary next-token prediction. Knowledge transfer becomes a reusable ranked corpus rather than stored token logits or a teacher loaded beside every student update. Official code, data, and models 07.08 DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning DeepSeek-R1 tests whether long-form reasoning can emerge from reinforcement learning driven chiefly by verifiable final-answer rewards rather than human-written chains of thought. R1-Zero applies Group Relative Policy Optimization (GRPO) directly to DeepSeek-V3 Base with accuracy and format rewards; it learns longer reflection, verification, and alternative-search behavior but remains hard to read and mixes languages. Final R1 wraps that experiment in a multistage pipeline—cold-start supervised data, reasoning RL, rejection-sampled supervised training, and broad preference RL—to retain reasoning while improving usability. The central distinction is R1-Zero discovers; R1 curates and aligns. — 07.09 Distillation Scaling Laws Distillation Scaling Laws predicts a language-model student's held-out cross-entropy from student size, student distillation tokens, and the teacher's achieved cross-entropy. Its fitted law includes a capacity-gap term: improving or enlarging a teacher can eventually stop helping a fixed student. The paper then combines this empirical law with explicit compute budgets to decide when distillation is preferable to supervised training. — 07.10 Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation The paper offers a minimal distribution-level explanation for why a small generative student can produce sharper samples after knowledge distillation. It separates three distributions: an underlying ground truth p-star, an imperfect teacher p-prime learned from it, and a lower-capacity student p-double-prime learned from teacher samples. Reducing teacher entropy concentrates its samples on high-density modes. Forward-KL or maximum-likelihood training then spends the student's limited capacity on those modes, increasing precision while decreasing recall. Gaussian mixtures expose the mechanism analytically; a SmolLM2 1.7B→360M→135M pipeline tests the same temperature-controlled trade-off in autoregressive language modeling. Official code Work: Separate token-, hidden-state-, data-distribution-, sequence-, rationale-, divergence-, rollout-, and coverage-level effects, then evaluate free-running generation.
Exit evidence: Report tokenizer compatibility, teacher policy, offline/online calls, divergence, filtering, precision/recall, shift, and compute-normalized quality.
2022–2026
Generative sampler and flow distillation
Progressive distillation repeatedly shortens diffusion sampling; LIFT/PLACE distills flow-based generation while preserving a useful trajectory. Teacher evaluations, repeated rounds, and final quality all belong to the compression contract.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 Progressive Distillation for Fast Sampling of Diffusion Models Progressive distillation teaches one deterministic diffusion step to reproduce two DDIM teacher steps, then promotes the student to teacher and repeats. Each stage halves the number of network evaluations, amortizing an increasingly long numerical-integration segment into the same-size model. A stable denoising parameterization is essential because the final stages operate at extremely low signal-to-noise ratios. Official code 08.02 LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models LIFT and PLACE stabilize knowledge distillation for aggressively compressed diffusion and flow models by separating easy statistical alignment from hard residual imitation. LIFT fits a global linear map between student and teacher denoising outputs, first driving its intercept and slope toward the identity relationship, then increasingly learning the remaining residual. PLACE repeats this fitting within error-ranked spatial groups so difficult regions receive local guidance. — Work: Compare cumulative training and teacher-evaluation cost against final sampling cost and matched quality.
Exit evidence: Report schedule, target construction, distillation rounds, function evaluations, trajectory error, and quality metrics.
2021–2026
Multimodal, multi-teacher, and foundation-model transfer
Compact VLMs transfer affinities, sparse-expert policies, preferences, uncertainties, or complementary teacher representations—not only class probabilities. LLaVA-MoD separates sparse student architecture, mimic distillation, and preference distillation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation ALBEF first aligns independently encoded image and text CLS vectors with a contrastive objective, then fuses their token sequences through cross-attention for masked-word prediction and pair matching. Contrastive similarity also selects hard mismatches for the fusion model. An exponential-moving-average copy supplies soft targets for ambiguous web captions and masked words. Its thesis is align the modalities before asking a multimodal encoder to reason jointly. — 09.02 TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance TinyCLIP compresses both towers of CLIP by transferring the teacher's full image-text affinity structure and inheriting a compact subset of teacher weights. Affinity mimicking matches bidirectional probability matrices over all image-text pairs in a batch, including non-diagonal relations. Weight inheritance initializes a thinner or shallower student from selected teacher parameters, reducing the optimization burden of learning vision-language alignment from scratch. Official code and checkpoints 09.03 MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training MobileCLIP attacks mobile vision-language inference and rapid architecture iteration together. It designs compact image and text encoders, but its larger idea is multi-modal dataset reinforcement: run an image captioner and an ensemble of strong CLIP teachers once, store synthetic captions, reproducible augmentation parameters, and teacher embeddings beside each original pair, then reuse that enriched record across many student runs. Teacher inference disappears from the training loop, so each student pays approximately ordinary CLIP training time while learning richer image-text affinities. — 09.04 LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge DistillationPeer-reviewed ICLR 2025 conference paper. LLaVA-MoD builds a compact multimodal language model by combining sparse mixture-of-experts capacity with progressive knowledge distillation. A dense small Qwen-family MLLM first mimics general teacher behavior, its feed-forward blocks are then upcycled into routed experts for specialized multi-task transfer, and preference distillation finally pushes positive responses above and negative responses below the large teacher's relative likelihood. Only a subset of experts activates per token, so total capacity and inference compute must be reported separately. Official code 09.05 Learning Task-Agnostic Representations through Multi-Teacher Distillation This paper distills several heterogeneous embedding teachers into one compact student without knowing the future task or requiring task labels. It begins with an ideal criterion: for any downstream task, the Bayes prediction made from the student embedding should agree with the predictions available from the teachers. A bound turns that inaccessible criterion into the average conditional entropy of each teacher embedding given the student embedding. The practical method learns teacher-specific Gaussian conditional decoders and minimizes their negative log-likelihood. This maximizes mutual information between the student and every teacher while allowing different architectures, embedding dimensions, modalities, and pretraining objectives. Official text-model code · Official vision-model code · Official molecular-model code 09.06 RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models RADIOv2.5 trains one vision transformer to imitate complementary feature spaces from language-aligned, semantic, and dense-prediction teachers. The paper's central lesson is that multi-teacher distillation is not just summing losses: each teacher has a native resolution, activation scale, spatial artifact, and useful output type. Multi-resolution matching prevents the student from changing personality with input size; mosaics make the expensive SAM teacher usable at lower student resolutions; PHI-S balances teacher feature energy; and output-level token merging preserves high-resolution information while capping the visual-token budget for an LLM. — 09.07 COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training COSMOS is a CLIP-compatible vision–language pretraining method that uses cross-modality self-distillation to improve both image and text representations. It makes global and local crops in both modalities, lets a student fuse image patches with text tokens through small cross-attention modules, and asks those fused student embeddings to predict global image and text embeddings from an exponential-moving-average teacher. The auxiliary fusion path is used for training; the released image and text encoders behave like ordinary two-tower CLIP encoders at inference. — 09.08 Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models Uncertainty-Aware KD interprets teacher-student discrepancy as an energy in a Gibbs prior and learns its precision beta instead of manually fixing loss weights. A high beta means confident, strong teacher alignment; a low beta weakens uncertain guidance. Beta-KD can learn one task-level scalar per objective or predict instance-level weights from each multimodal input, allowing CE, logit, and feature losses to balance themselves during MLLM compression. — Work: Inventory every teacher, modality, activated and total parameters, vote/alignment tensor, uncertainty signal, data source, and student head.
Exit evidence: Demonstrate retained alignment and broad downstream transfer under matched data and inference budgets.
2021–2026
Resolution transfer and evidence-budget design
Resolution distillation changes the information available to teacher and student. MSAD and resolution-asymmetric metric learning provide natural-image controls; pathology then separates four contracts: low-only deployment, retained multiscale inference, selective high-resolution acquisition, and subset learning after all high-resolution embeddings were already extracted.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 10.01 Multi-Scale Aligned Distillation for Low-Resolution DetectionPeer-reviewed CVPR 2021 proceedings paper. Multi-Scale Aligned Distillation (MSAD) trains a detector that receives only low-resolution images at deployment but learns from a teacher combining high- and low-resolution evidence. Its key engineering insight is that a factor-two input reduction maps naturally to a one-level shift in a feature pyramid. Spatially aligned pyramid maps can therefore be fused into a stronger teacher and compared directly with the low-resolution student's earlier FPN levels. Official code 10.02 Resolution-Based Distillation for Efficient Histology Image ClassificationPeer-reviewed journal article 119:102136. Resolution-based distillation trains a low-magnification histology classifier to imitate a same-architecture high-magnification teacher on spatially corresponding tissue. The deployed model receives only the downsampled image; it is therefore a genuine low-resolution inference method, not a multiscale model that still reads fine-detail tiles. Open-access article · Author manuscript 10.03 Large-to-Small Image Resolution Asymmetry in Deep Metric LearningPeer-reviewed WACV 2023 proceedings paper. Large-to-Small Resolution Asymmetry in Deep Metric Learning trains a low-resolution query encoder to remain geometrically compatible with a high-resolution database encoder. A frozen 448-pixel teacher embeds database images; a student initialized from it receives downsampled queries. Absolute embedding agreement and two relational similarity losses preserve both individual vectors and neighborhood geometry, enabling cheaper queries without rebuilding the high-quality database index. Official code 10.04 Attention to Detail: Inter-Resolution Knowledge DistillationPeer-reviewed EUSIPCO 2023 proceedings paper, pages 985–989. Attention to Detail teaches a lower-resolution prostate classifier to focus where a 10× teacher finds class-supporting morphology. ReLU-filtered Grad-CAM maps are normalized and matched with AT+; the teacher and map computation disappear after training, leaving one 5×, 2.5×, or 1.25× student at inference. Open proceedings paper · Official code 10.05 A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI ClassificationPeer-reviewed journal article 43(4):1412–1421; journal extension of DAS-MIL. DAS-MIL contextualizes DINO patch features with graphs within and across 5×, 10×, and 20×, then uses attention MIL at each scale. Prediction KL and criticality-map matching regularize agreement. Every chosen magnification remains active at inference; self-distillation does not create a 5×-only student. Official code 10.06 CEMIL: Contextual Attention Based Efficient Weakly Supervised Approach for Histopathology Image ClassificationPeer-reviewed WACV 2025 proceedings paper, pages 4248–4257. CEMIL trains an all-patch instructor and a top-k learner for weakly supervised WSI classification. Frozen PLIP features from every tile first pass through an instructor projection/attention map; the learner keeps the top 60% and matches selected representations plus class probabilities. It reduces the MIL-side workload, but still extracts and scores all tiles before selection. — 10.07 Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning HDMIL accelerates WSI classification by learning which physical regions can be rejected from a cheap 1.25× pyramid level before the corresponding 20× patches are cropped and encoded. A high-resolution Dynamic Multi-Instance Network, DMIN, first learns class-specific relevance masks by making a partial-instance student imitate an all-instance teacher. Those masks supervise a lightweight low-resolution Instance Pre-screening Network, LIPN. At inference, LIPN predicts masks from 16×16 low-resolution tiles, the system crops only spatially corresponding 256×256 high-resolution tiles, and DMIN classifies the retained features. Its systems insight is that the bottleneck occurs before the MIL head, in high-resolution cropping and feature extraction. Code 10.08 PatHME: Hierarchical Multi-Expert Knowledge Distillation for Whole Slide Image AnalysisPeer-reviewed journal article 13:163246–163257. PatHME builds regional WSI features from 5× and 20× pathology-foundation embeddings. Prompt-guided multiscale distillation aligns coarse/fine representations, multi-expert KD adds a second encoder's knowledge, and hierarchical multiscale attention fuses them. Both magnifications remain in the documented feature path, so PatHME is not 5×-only deployment. Official code 10.09 Streamline Pathology Foundation Model by Cross-Magnification DistillationJournal record verified through publisher metadata, Crossref and PubMed PMID 42874158 on 26 October 2026: electronic publication 10 October 2026, assigned to volume 115 (January 2027), article 104352. The stable ID retains the 2025 preprint year. Journal full text was unavailable; detailed scientific claims and visuals remain explicitly tied to the previously inspected arXiv v3, not asserted to match the journal version. The journal abstract describes seven tasks and encoder-level throughput, not an end-to-end speedup. XMAG distills one 896×896 tissue field at 20×—sixteen 224×224 UNI2-h teacher tiles—into one aligned 224×224 5× tile. A ViT-B/14 student's CLS token matches the mean of the sixteen teacher CLS embeddings, while 4×4 average pooling over its own 16×16 patch-token grid yields sixteen regional student features that match the sixteen teacher-tile CLS embeddings one to one. Only the 5× student is retained, making this reusable encoder-level low-only distillation. Inspected author manuscript (v3) · PubMed journal metadata and abstract · Official code · Official model 10.10 Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide ImagesPreprint; arXiv comments report submission to the AMIA 2026 Informatics Summit, not acceptance. Magnification-Aware Distillation replaces synthetic DINO local crops with physically aligned WSI pyramid views. An EMA teacher sees a low-magnification context tile, while the student sees four native higher-magnification children sampled from its 4×4 aligned grid. Pairwise 2.5×→10× and 10×→40× training aims to place all scales in one representation space; one shared encoder can later process any tested scale. Official model 10.11 A Magnification Alignment Framework Enables Computation- and Communication-Efficient Computational PathologyPeer-reviewed journal article 86(12):3060–3073. MAG-GLTrans teaches a low-magnification encoder to imitate frozen high-magnification feature maps over the same physical field, then classifies WSIs with a Global–Local Transformer. Training pairs 20× 256² with 10× 128² or 5× 64² views; inference stores, transfers, and processes only the chosen 5×/10× WSI. It directly targets compute and communication, not just model size. Author manuscript · Official code 10.12 LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image ClassificationOfficial MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02. Springer DOI and pagination remain pending; do not invent either. LRMIL asks whether a whole-slide classifier can learn from exhaustive 20× morphology but deploy using only 5× patches. It first distils matched high-resolution patch representations into a low-resolution encoder, then distils a trained high-resolution attention-MIL teacher into a low-resolution attention-MIL student at bag and instance levels. The high-resolution branch is discarded at inference. The scientific claim is therefore knowledge-preserving resolution reduction and measured within-study efficiency—not free high-resolution training, universal state-of-the-art accuracy, or a sparse high-resolution selector. Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF Work: For every method, draw the physical field of view, registration, teacher-only training path, student deployment path, and patch-count budget.
Exit evidence: Report accuracy with extraction, encoding, aggregation, storage, communication, and training costs—never parameter count alone.
2021–2026
Computational-pathology hierarchy, encoders, graphs, and experts
Pathology distillation moves evidence across tiles, MIL tiers, graph prototypes, magnifications, foundation encoders, and language-guided experts. H0-mini and Pathryoshka compress encoders; GraphTox uses teacher–student discrepancy with both branches present to score anomalies; PatHME and LaGuadia combine scale or expert teachers; LRMIL transfers both encoder and bag decisions into a 5×-only path.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 11.01 Resolution-Based Distillation for Efficient Histology Image ClassificationPeer-reviewed journal article 119:102136. Resolution-based distillation trains a low-magnification histology classifier to imitate a same-architecture high-magnification teacher on spatially corresponding tissue. The deployed model receives only the downsampled image; it is therefore a genuine low-resolution inference method, not a multiscale model that still reads fine-detail tiles. Open-access article · Author manuscript 11.02 DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification DTFD-MIL addresses the small-number-of-slides, huge-number-of-patches imbalance by randomly splitting every WSI bag into several pseudo-bags. Tier 1 learns slide-label-supervised attention MIL on each pseudo-bag; a derived class probability ranks its instances; a compact feature is distilled from every pseudo-bag; and Tier 2 learns the original slide from that short set of distilled features. The central bargain is deliberate: create more training bags even though some positive pseudo-bags inherit a false-positive label, then let the second tier combine complementary evidence and repair incomplete local views. — 11.03 Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification MHIM-MIL reverses the usual attention-MIL instinct during training. A slowly updated teacher sees every patch and ranks instances by attention; the most salient, apparently easy instances are removed, optionally together with low-attention and random subsets; and a student must classify the slide from the harder remainder. A consistency target from the full-input teacher and an exponential-moving-average update close the loop. At inference, masking and the teacher disappear: the student receives the complete bag. The method is therefore curriculum-like regularization over bag contents, not a new patch encoder. — 11.04 Attention to Detail: Inter-Resolution Knowledge DistillationPeer-reviewed EUSIPCO 2023 proceedings paper, pages 985–989. Attention to Detail teaches a lower-resolution prostate classifier to focus where a 10× teacher finds class-supporting morphology. ReLU-filtered Grad-CAM maps are normalized and matched with AT+; the teacher and map computation disappear after training, leaving one 5×, 2.5×, or 1.25× student at inference. Open proceedings paper · Official code 11.05 Knowledge Distillation in Histology Landscape by Multi-Layer Features SupervisionPeer-reviewed journal article 27(4):2037–2046. KDTP lets every shallow student layer learn from several deeper teacher layers. Projection modules reconcile feature shapes, learned attention weights the useful hints, and logit KD plus labels complete training. Teacher and student see the same 224×224 histology patch; deployment savings come from the compact CNN, not lower magnification. Open author manuscript 11.06 SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification SlideGCD augments any whole-slide MIL classifier with a second branch that learns relationships between slides. Each WSI becomes one node, a FIFO buffer supplies neighboring slide embeddings beyond the current mini-batch, and an adaptive k-nearest-neighbor hypergraph propagates cross-slide context. The original MIL classifier teaches the graph branch through response-level Jensen-Shannon distillation while both branches also learn labels. Official code 11.07 A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI ClassificationPeer-reviewed journal article 43(4):1412–1421; journal extension of DAS-MIL. DAS-MIL contextualizes DINO patch features with graphs within and across 5×, 10×, and 20×, then uses attention MIL at each scale. Prediction KL and criticality-map matching regularize agreement. Every chosen magnification remains active at inference; self-distillation does not create a 5×-only student. Official code 11.08 A generalizable pathology foundation model using a unified knowledge distillation pretraining framework GPFM is a ViT-L/14 pathology tile encoder that augments DINOv2-style self-distillation and masked patch prediction with feature distillation from three frozen pathology experts: UNI for WSI classification, Phikon for report-generation transfer, and CONCH for visual question answering. The central idea is unified expert and self knowledge distillation: a student should learn both the invariances available in its own 190-million-patch corpus and complementary representations already encoded by models trained on data it cannot directly access. — 11.09 Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning HDMIL accelerates WSI classification by learning which physical regions can be rejected from a cheap 1.25× pyramid level before the corresponding 20× patches are cropped and encoded. A high-resolution Dynamic Multi-Instance Network, DMIN, first learns class-specific relevance masks by making a partial-instance student imitate an all-instance teacher. Those masks supervise a lightweight low-resolution Instance Pre-screening Network, LIPN. At inference, LIPN predicts masks from 16×16 low-resolution tiles, the system crops only spatially corresponding 256×256 high-resolution tiles, and DMIN classifies the retained features. Its systems insight is that the bottleneck occurs before the MIL head, in high-resolution cropping and feature extraction. Code 11.10 Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathology MKD trains a pathology-only biomarker predictor with privileged genomics and multimodal information available only during training. Separate pathology, genomics, and fused teacher/student branches decompose modality-specific and shared knowledge. CORAL and orthogonality organize those representations; similarity-preserving KD transfers sample relationships; symmetric online distillation lets the three classifiers learn from one another. The pathology branch can then predict ER, PR, and HER2 from H&E alone. Official code 11.11 Distilling Foundation Models for Robust and Efficient Models in Digital PathologyPeer-reviewed MICCAI 2025 proceedings paper. H0-mini distills the frozen 1.1-billion-parameter H-Optimus-0 ViT-g/14 into an 86-million-parameter ViT-B/14 while keeping the same 224×224, 20× pathology-tile contract. It transfers teacher distributions for both the global class token and spatial patch tokens, then deploys only the compact student. Official model · Official robustness benchmark 11.12 CEMIL: Contextual Attention Based Efficient Weakly Supervised Approach for Histopathology Image ClassificationPeer-reviewed WACV 2025 proceedings paper, pages 4248–4257. CEMIL trains an all-patch instructor and a top-k learner for weakly supervised WSI classification. Frozen PLIP features from every tile first pass through an instructor projection/attention map; the learner keeps the top 60% and matches selected representations plus class probabilities. It reduces the MIL-side workload, but still extracts and scores all tiles before selection. — 11.13 Unsupervised Anomaly Detection on Preclinical Liver H&E Whole Slide Images Using Graph-Based Feature DistillationPeer-reviewed MICCAI 2025 proceedings paper; evaluated on preclinical rat-liver H&E WSIs. The author project page requests email contact for code access and showed code as coming soon when checked on 2026-08-23. GraphTox detects preclinical rat-liver abnormalities by distilling a resolution-aware teacher into a student on normal multi-resolution tissue graphs. Each local graph contains one 5× node, four aligned 10× nodes, and sixteen aligned 20× nodes; teacher–student graph discrepancy becomes the anomaly score. All three resolutions and both encoders are evaluated to score anomalies, so it is neither classifier KD nor low-only deployment. Author project and code-access page 11.14 PatHME: Hierarchical Multi-Expert Knowledge Distillation for Whole Slide Image AnalysisPeer-reviewed journal article 13:163246–163257. PatHME builds regional WSI features from 5× and 20× pathology-foundation embeddings. Prompt-guided multiscale distillation aligns coarse/fine representations, multi-expert KD adds a second encoder's knowledge, and hierarchical multiscale attention fuses them. Both magnifications remain in the documented feature path, so PatHME is not 5×-only deployment. Official code 11.15 Streamline Pathology Foundation Model by Cross-Magnification DistillationJournal record verified through publisher metadata, Crossref and PubMed PMID 42874158 on 26 October 2026: electronic publication 10 October 2026, assigned to volume 115 (January 2027), article 104352. The stable ID retains the 2025 preprint year. Journal full text was unavailable; detailed scientific claims and visuals remain explicitly tied to the previously inspected arXiv v3, not asserted to match the journal version. The journal abstract describes seven tasks and encoder-level throughput, not an end-to-end speedup. XMAG distills one 896×896 tissue field at 20×—sixteen 224×224 UNI2-h teacher tiles—into one aligned 224×224 5× tile. A ViT-B/14 student's CLS token matches the mean of the sixteen teacher CLS embeddings, while 4×4 average pooling over its own 16×16 patch-token grid yields sixteen regional student features that match the sixteen teacher-tile CLS embeddings one to one. Only the 5× student is retained, making this reusable encoder-level low-only distillation. Inspected author manuscript (v3) · PubMed journal metadata and abstract · Official code · Official model 11.16 Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide ImagesPreprint; arXiv comments report submission to the AMIA 2026 Informatics Summit, not acceptance. Magnification-Aware Distillation replaces synthetic DINO local crops with physically aligned WSI pyramid views. An EMA teacher sees a low-magnification context tile, while the student sees four native higher-magnification children sampled from its 4×4 aligned grid. Pairwise 2.5×→10× and 10×→40× training aims to place all scales in one representation space; one shared encoder can later process any tested scale. Official model 11.17 ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts ALICE—Agglomerative Learning via Integrated Computational pathology Embedding—distills eight existing pathology foundation models into one modular student. It first learns local morphology from UNI-2, Virchow-2, and H-Opt-1; freezes that vision backbone and learns language-aligned heads from CONCH, MUSK, and KEEP; then freezes tile extraction and trains a spatial slide transformer from TITAN and CARE. The result exposes three distinct interfaces for 3,840-D vision features, teacher-aligned vision–language features, and a 2,048-D slide feature. Its claim is capability consolidation by staged distillation, not learning eight tasks from new labels or replacing all teacher architectures with one identical output. Code and models 11.18 GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment AnalysisVersion 2 dated 2026-07-22; author-reported performance remains provisional. The paper compresses the GigaPath/GigaTIME family around one 22M-parameter ViT-S/16 tile encoder distilled from the frozen 1B-parameter GigaPath ViT-g teacher. GigaPath-Flash pairs its 384-dimensional tile features with a 21M, 12-layer LongNet slide encoder pretrained by masked representation prediction. GigaTIME-Flash instead adds LoRA and a roughly 2M-parameter convolutional decoder to predict 21 spatial-proteomics channels from H&E. Thus one compact tile encoder serves both slide representation and spatial proteomics, while the two downstream training stages and outputs remain distinct. — 11.19 HistoMILKD: A Multiple Instance Learning based Multi-Teacher Knowledge Distillation Framework for Whole Slide Image Classification HistoMILKD distils four pathology foundation-model views of each slide into one task-specific student pipeline. UNI is the default student encoder; H-Optimus-0, Hibou-L, Prov-GigaPath and Virchow2 are teachers. Every frozen encoder produces patch features, an independently pretrained ABMIL turns each view into a slide representation and prediction, and only the student ABMIL receives gradients from supervised, logit-distillation and feature-reconstruction losses. At deployment, discard all teacher branches and keep one frozen student foundation model plus its learned MIL adapter. The design exchanges expensive multi-model training for single-student inference. Code 11.20 Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation With Nested EmbeddingsECCV 2026 poster status reverified against the official complete program (poster 4454). No oral or spotlight designation is recorded. A Springer version of record was not verified on 2026-09-12; the existing review retains its declared manuscript revision. Pathryoshka distills three frozen pathology foundation models—Virchow2, UNI2-h, and H-optimus-1—into one smaller student while training every leading prefix of its embedding to remain useful. Per-teacher MLP heads align incompatible spaces; cosine loss transfers class-token semantics and standardized patch-token regression transfers local structure. Deployment keeps only Path-B or Path-S and a chosen nested output width. The input remains an ordinary pathology tile: this compresses encoder size and stored features, but does not infer 20× morphology from 5× pixels. Official ECCV record · Official model · Conference manuscript 11.21 A Magnification Alignment Framework Enables Computation- and Communication-Efficient Computational PathologyPeer-reviewed journal article 86(12):3060–3073. MAG-GLTrans teaches a low-magnification encoder to imitate frozen high-magnification feature maps over the same physical field, then classifies WSIs with a Global–Local Transformer. Training pairs 20× 256² with 10× 128² or 5× 64² views; inference stores, transfers, and processes only the chosen 5×/10× WSI. It directly targets compute and communication, not just model size. Author manuscript · Official code 11.22 LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation ModelsOfficial MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02; this supersedes the prior preprint classification. Springer DOI remains pending. LaGuadia compresses three pathology foundation encoders into an approximately 87M-parameter ViT-B student, but chooses teacher weights using report-derived H&E-visible concepts. GPT-5-mini extracts keywords, MedSigLIP assigns patch pseudo-keywords, and semantic consensus among GigaPath, UNI, and Virchow2 controls negative-cosine distillation. Only the student remains at inference; magnification stays 20×. Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF 11.23 LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image ClassificationOfficial MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02. Springer DOI and pagination remain pending; do not invent either. LRMIL asks whether a whole-slide classifier can learn from exhaustive 20× morphology but deploy using only 5× patches. It first distils matched high-resolution patch representations into a low-resolution encoder, then distils a trained high-resolution attention-MIL teacher into a low-resolution attention-MIL student at bag and instance levels. The high-resolution branch is discarded at inference. The scientific claim is therefore knowledge-preserving resolution reduction and measured within-study efficiency—not free high-resolution training, universal state-of-the-art accuracy, or a sparse high-resolution selector. Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF 11.24 Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast CancerOfficial MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02; Springer DOI and pagination remain pending. SmartStu customises a compact tile encoder for breast pathology rather than deploying a 300–681M-parameter general pathology foundation model unchanged. It regresses toward features from frozen UNI-v2, Phikon-v2 and Virchow-v2 teachers, while a separately trained nuisance encoder supplies site-predictive features that the student is taught to suppress through gradient reversal. Artefact-injected DINO-style self-supervision adds robustness pressure. The output remains a tile representation that requires a downstream MIL model; it is not an end-to-end breast diagnosis system. Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF Work: Trace every teacher/student input, magnification, field of view, aggregation level, frozen checkpoint, and patient-level label; mark which high-resolution paths and teacher branches remain at inference.
Exit evidence: Evaluate patient-disjoint internal and external cohorts while reporting latency, memory, calibration, subgroup effects, and low-only versus multiscale deployment.
2016–2026
Privileged, safety, capacity, and on-policy transfer
Generalized distillation formalizes training-only privileged information; Minitron couples structured pruning with KD; newer studies test safe soft-prompt transfer, conditional block matching, verifier-gated reasoning, and reward-guided diffusion/flow self-distillation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 12.01 Unifying Distillation and Privileged InformationICLR 2016 conference paper; canonical locator is the complete arXiv manuscript. Core mechanism: privileged information available only during training. The framework unifies learning using privileged information with teacher-student distillation: a teacher learns from richer training-only variables and transfers its soft predictions to a student that will deploy using ordinary inputs alone. This reading boundary matters: the framework explains when a training-only view can improve sample complexity under assumptions, not that every stronger modality or larger teacher will help every student. — 12.02 Compact Language Models via Pruning and Knowledge DistillationPublished in the NeurIPS 2024 main conference proceedings. Core mechanism: structured pruning followed by distillation. Minitron derives smaller language models by removing transformer depth, hidden dimensions, attention heads, or MLP channels from a pretrained base, initializing the child with retained weights, and recovering quality through knowledge distillation on a small fraction of pretraining data. This reading boundary matters: the recipe amortizes one expensive pretraining run into related compact checkpoints, not a claim that arbitrary architectures can be pruned without architecture- and data-specific recovery. Official implementation · Official model collection 12.03 Distilling Safe LLM Systems via Soft Prompts for On Device SettingsPublished in Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337. Core mechanism: guard behavior compressed into a soft prompt. A frozen on-device language model receives a learned continuous prompt whose parameters are optimized to reproduce a safer guard-model system. The transfer target is behavior at the output distribution, so the base model can remain unchanged. This reading boundary matters: the result is a parameter-efficient behavioral transfer mechanism, not proof that the student inherits every policy, adversarial robustness property, or hidden capability of the guard system. — 12.04 Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid itarXiv v1 preprint submitted 24 August 2026; conclusions are scoped to the reported homogeneous CNN image-classification study. Core mechanism: more distillation points can overconstrain the student. The study varies how many intermediate blocks of a convolutional student are forced to match a teacher and finds that dense multi-block supervision can promote overfitting. The best placement depends on data abundance and task granularity. This reading boundary matters: the result is a warning and selection heuristic for the tested homogeneous CNN classification setting, not a universal law that intermediate-feature distillation is harmful. — 12.05 On-policy Distillation with Verifiable RewardarXiv v1 preprint submitted 25 August 2026; evaluation is limited to the reported verifiable reasoning tasks. Core mechanism: verifier-gated token credit. The student samples its own reasoning trajectories, a teacher evaluates token-level preferences along those trajectories, and a verifiable final reward gates the sign of token updates so incorrect solutions are not reinforced as if they were correct. This reading boundary matters: the method provides correctness-aware credit for tasks with programmatic verification, not a general solution for open-ended truthfulness or unverifiable preferences. Official implementation 12.06 On-Policy Self-Distillation in Diffusion ModelsarXiv v1 technical report submitted 25 August 2026; reported reward gains rely on the chosen automated evaluators and two tested backbones. Core mechanism: on-policy self-distillation from generated trajectories. DiffusionOPSD samples trajectories from a frozen behavior copy of the current diffusion model, differentiates chosen rewards with respect to predicted clean outputs, converts those gradients into bounded positive/negative targets, and distills them into the trainable model with finite fitting and EMA updates. This reading boundary matters: the method is a reward-optimization procedure for the evaluated diffusion backbones and automated reward models, not proof of human-preference alignment or universal image-quality improvement. Official project · Official implementation 12.07 Self-OPD: On-Policy Distillation for Flow Matching Models without TeacherarXiv v1 preprint submitted 27 August 2026; teacher-free reward optimization remains benchmark- and reward-model-dependent. Core mechanism: teacher-free on-policy distillation for flow models. Self-OPD starts multiple stochastic branches from states visited by a flow-matching model, completes each branch with deterministic ODE integration, ranks their outcomes by reward, and trains the velocity field with normalized pull-push advantages without a separate teacher. This reading boundary matters: teacher-free means no larger teacher network; it does not mean reward-free, rollout-free, data-free, or guaranteed human alignment. Official implementation Work: Classify the transferred interface—training-only privileged variables, safe soft prompts, pruned capacity, block features, verifier rewards, diffusion endpoints, or flow velocities—then compare it with hard-label and no-distillation controls under the same student, data, and optimization budget.
Exit evidence: Report every teacher or auxiliary input available in training and deployment, teacher-query and rollout cost, student capacity, verifier coverage, negative block-matching evidence, and quality–latency trade-off.
2019–2026
Residual targets, correction paths, and residual audits
Residual KD is not one algorithm: work in this lineage transfers a feature error, a logit remainder, a missing shortcut path, an audio or LiDAR-only component, a frozen-teacher regularizer, a non-anchor probability mass, or a denoisable residual. The ICLR LLM method subtracts projected teacher state only on teacher-error tokens, while UniCAM measures what distillation changed instead of proposing another loss.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 13.01 Residual error based knowledge distillationPublished in Neurocomputing 433 (14 April 2021), pages 154–161, DOI 10.1016/j.neucom.2020.10.113. The differently authored arXiv:2002.09168 is recorded only as a related precursor, not as this journal version. Residual-error KD trains a student to imitate the teacher feature and a much smaller assistant to predict the remaining teacher-minus-student error. At inference, student and assistant features are summed. The journal record is canonical; the differently authored arXiv precursor is corroborative evidence, not a bibliographically merged version. — 13.02 ResKD: Residual-Guided Knowledge DistillationPublished in IEEE Transactions on Image Processing 30 (2021), pages 4735–4746. ResKD constructs a chain of progressively lighter res-students. The first student predicts class logits; every later model learns the teacher-to-cumulative-student logit residual. At inference, an energy rule can stop after an early stage, making residual depth a sample-adaptive accuracy-compute choice. Author manuscript 13.03 Residual Distillation: Towards Portable Deep Neural Networks without ShortcutsPublished in the NeurIPS 2020 main conference proceedings. Residual Distillation trains a shortcut-free plain CNN through a joint architecture containing a frozen pretrained residual teacher. Four plain/residual hybrid paths and final-feature matching transfer the teacher's optimization guidance, while inference retains only the plain network; the target is not a teacher-logit residual. Official implementation 13.04 Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge DistillationarXiv technical report submitted 26 May 2019; no peer-reviewed version or released implementation was located by 2026-09-02. RIM first hallucinates a high-resolution face from a 28 by 28 input, then recognizes it with a compact network whose assistant predicts blockwise feature residuals relative to a high-resolution teacher. The residual sum is the transferable representation; super-resolution and recognition are trained together rather than treated as independent modules. — 13.05 MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question AnsweringPublished in the IJCAI 2021 main track, pages 3985–3991, DOI 10.24963/ijcai.2021/549. MRD-Net distills a clean-text BERT teacher into an ASR-text student while its narrative intends an audio assistant to predict the remaining hidden-state residual. Speech-text attention aligns acoustic evidence with tokens, and progressive training combines the printed representation objective with answer-distribution KD. Equations 1 and 3 minimize a positive cosine term without a one-minus or negative sign, so the stated alignment intent and printed optimization direction are ambiguous. — 13.06 FROSTER: Frozen CLIP Is a Strong Teacher for Open-Vocabulary Action RecognitionPublished at ICLR 2024. FROSTER adapts CLIP for video action recognition while keeping a frozen image-CLIP copy as teacher. A residual feature-distillation adapter whose second fully connected layer W2 is zero-initialized refines the tuned video feature toward the frozen representation, seeking temporal task gains without erasing CLIP's open-vocabulary alignment. Official project · Official implementation 13.07 FOSTER: Feature Boosting and Compression for Class-Incremental LearningPublished in the ECCV 2022 main conference proceedings, pages 398–414. FOSTER expands a class-incremental model with a new feature module trained to correct residual classification error, then compresses the expanded teacher back to one backbone with class-balanced knowledge distillation. Residual fitting learns new capacity; distillation prevents deployment width from growing after every task. Official implementation 13.08 MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object DetectionPublished in the CVPR 2025 main conference proceedings, pages 22266–22275. MonoTAKD inserts a camera teaching assistant between a LiDAR BEV teacher and a monocular camera student. The assistant uses ground-truth depth during training to define an attainable camera-space BEV; cross-modal residual distillation then transfers only the LiDAR-teacher information not already represented by that assistant. Official implementation and models 13.09 CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge DistillationPublished in the ICCV 2025 main conference proceedings, pages 5990–6000. CleanPose improves category-level 6D pose estimation with two separable ideas: a causal front-door module intended to reduce spurious point-feature relations, and residual feature distillation from a frozen ULIP-2 PointBERT teacher. The zero-initialized residual adapter preserves the student's base feature while learning a teacher-aligned correction. Official implementation 13.10 Expandable Residual Approximation for Knowledge DistillationPublished in IEEE TNNLS 37(1), January 2026, pages 191–203; first available online in 2025. Expandable Residual Approximation treats the student as a zero-order teacher approximation and adds multiple lightweight MBR branches with ordered, cascaded teacher-feature residual targets. All branches are trained jointly in one stage rather than fitted sequentially. Teacher Weight Integration reuses the frozen teacher head, and the model can deploy student-only, teacher-head, or student-plus-residual modes according to the accuracy-cost budget. Author manuscript · Official implementation and models 13.11 Knowledge Distillation for Large Language Models through Residual LearningPublished at ICLR 2026; no official implementation was located by 2026-09-02. This ICLR 2026 method distills LLMs by emphasizing tokens where the teacher errs and representing the student's hidden state relative to a compressed teacher state. A pretrained teacher autoencoder, adaptive scaling, cross-tokenizer alignment, and mixture-of-experts fusion let residual supervision operate even when teacher and student vocabularies differ. OpenReview record · First-party Amazon Science page 13.12 ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual RegularizationThe arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026. ACTD distills reasoning models across different tokenizers by mapping teacher top-k tokens into student-space anchors. An anchor L1 term matches probability on aligned tokens, while residual regularization constrains the student's probability mass outside those anchors. With multiple teachers, ACTD computes a complete teacher-specific loss for each teacher and averages those losses; it does not average anchor signals or teacher distributions first. Official implementation 13.13 On Explaining Knowledge Distillation: Measuring and Visualising the Knowledge Transfer ProcessPublished in the WACV 2025 proceedings, pages 3467–3476. UniCAM explains what knowledge distillation changes by comparing a KD student with a same-architecture base model trained on raw labels. Partial distance correlation separates student-unique distilled features from base-unique residual features; gradient maps, feature similarity, and label-semantic relevance then characterize transfer layer by layer. Official implementation 13.14 On learning denoisable student logitsPeer-reviewed article 113684 in Pattern Recognition 179; available online 10 April 2026 and assigned to the November 2026 issue. Denoisable Student Logits treats the compact student's outputs as Gaussian-noised teacher logits and jointly learns the student, a compression path, and a diffusion recovery module. KD reduces the fitted residual standard deviation and cross-dimension correlation, moving the residual closer to the iid Gaussian assumption used by diffusion. At inference, selected and optionally quantized student logits can travel from an edge device to a server for decompression and DDIM recovery. — Work: For each paper, name the residual exactly, write its tensor shape and gradient recipients, and mark which assistants, projectors, branches, teachers, or recovery modules remain at inference.
Exit evidence: Reproduce a matched no-residual baseline and report total training cost, active inference path, calibration or stopping rule, and quality–latency curve; never use the bare acronym RKD.
2020–2026
Resolution, magnification, relation, and stain transfer
Resolution-aware KD moves signal from high- to low-resolution inputs; switchable networks distil an ensemble across discrete resolutions; pathology variants transfer same-patch relations, retain earlier magnification knowledge, or use an IHC teacher while deploying H&E. These contracts differ from LRMIL-style 5×-only inference.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 14.01 Human Visual Attention-Inspired Knowledge Distillation Underlying Interpretable Computational PathologyPeer-reviewed open-access journal article 15:42124. Low- and high-magnification language motivates the design, but HVisKD transfers sample- and region-relation features between teacher and student given the same pathology patches; it is not cross-magnification distillation. HVisKD transfers relations among pathology features at two levels: sample relations span patches in a batch, and region relations span a spatial pyramid inside each feature map. A large CNN teaches a lightweight CNN on the same input patches; the deployed student predicts patch tissue subtype and those predictions can be assembled into a WSI heatmap. Open-access article · Official code 14.02 Cross-Stain Knowledge Distillation for Low-Cost Lung Cancer Programmed Death Ligand-1 Assessment With Multi-Granularity Multiple Instance LearningPeer-reviewed journal article, volume 167, article 113762. IHC is privileged teacher information during training; deployment uses the H&E student, but its macro- and micro-granularity branches remain. No official code or model release was verified on 2026-09-02. CroSMuG treats paired immunohistochemistry as privileged training information for low-cost PD-L1 assessment from routine H&E. A cross-stain teacher guides an H&E student whose macro- and micro-granularity MIL branches are fused; IHC is removed at deployment, but both H&E granularity branches remain. — 14.03 KDTViT Knowledge Distillation, Transfer Learning and Transformer Based Deep Learning Framework for Efficient Histopathology Image ClassificationPeer-reviewed open-access journal article 16:20721. A MobileViT-XS is sequentially adapted from 40× through 400× using the previous checkpoint as teacher; BreakHis consists of biopsy images rather than WSIs, and the same-capacity teacher/student design targets retention rather than model compression. KDTViT trains one MobileViT-XS through a sequence of BreakHis magnifications: learn at 40×, freeze that checkpoint as teacher for 100×, then repeat for 200× and 400×. Each next student has the same architecture and learns labels plus soft predictions and final-feature matching, so the method targets progressive retention rather than compression. Open-access article 14.04 Resolution-Aware Knowledge Distillation for Efficient InferencePeer-reviewed journal article, volume 30, pages 6985–6996. The paper calls its framework RKD, meaning resolution-aware—not relational—knowledge distillation. Teacher and student use the same architecture but paired high- and low-resolution inputs; only the low-resolution student is needed at inference. Resolution-aware knowledge distillation trains a low-resolution student to match a same-architecture high-resolution teacher. A discriminator reduces the resolution-domain gap, while response, pairwise, and distribution constraints transfer structure; inference keeps only the student and its low-resolution image. PubMed record 14.05 Resolution Switchable Networks for Runtime Efficient Image RecognitionPeer-reviewed ECCV 2020 proceedings paper. One weight-shared CNN supports five trained input resolutions with private batch-normalization parameters; inference evaluates one chosen resolution and does not run the training-time ensemble. RS-Net trains one convolutional network to operate at five image resolutions. Convolutional weights are shared, batch-normalization statistics are resolution-specific, and a learned multi-resolution ensemble supplies distillation targets; at test time the user selects one resolution and executes one forward pass. Official code and released models · Authors' arXiv manuscript Work: Draw the teacher and student field of view, registration, stain, pixel size, and inference path before comparing objectives.
Exit evidence: Report information removed at deployment, every retained scale or stain, patient/slide grouping, acquisition and encoding cost, and external-site evidence.
2025–2026
Physical observability and task-aware polarization reconstruction
GenPolar predicts Stokes components from intensity using observability-aware supervision and one-step diffusion distillation. PolarAPP instead starts from measured polarization mosaics and couples demosaicking with downstream objectives through a learned metric. Keep generative priors separate from sensor-acquired polarization.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 15.01 Stokes-Informed Diffusion for Robust Linear Polarization EstimationECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5749. Publisher proceedings/version of record was not verified on 2026-09-12. GenPolar predicts RGB-channel linear Stokes components from an intensity image, then derives polarization magnitude and angle analytically. It combines observability-aware physical supervision with multi-step-to-one-step diffusion distillation and VAE-encoder LoRA. This targets stable polarimetric cues rather than merely plausible-looking angle maps. ECCV Spotlight program · Conference manuscript Work: Is the supervision a physically observable target, a synthetic reconstruction target, a downstream task signal or a teacher distribution?
Exit evidence: Weak-polarization angle masks change the evaluated population. Task-aware meta-learning is not automatically teacher–student KD, and metric improvements need not agree.
2025–2026
Physiology-aware cross-modal spectral distillation
RPM-Distill transfers peak position, off-peak suppression and spectral morphology from an RF teacher to an RGB student. A validation-driven policy chooses sample and loss weights; deployment removes the radar branch. Compare this privileged-sensor design with feature KD and reliability-weighted transfer.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 16.01 RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological MeasurementECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #10941. Publisher proceedings/version of record was not verified on 2026-09-12. RPM-Distill transfers rhythm information from a frozen radar teacher to a video-only physiological student. It compares band-limited spectra rather than forcing incompatible RF and RGB waveforms to match. A learned policy gates each sample and weights peak, off-peak and spectral-shape losses according to whether they improve held-out physiological recovery. ECCV Spotlight program · Conference manuscript · Official implementation Work: Which teacher signal is physiologically invariant across sensors, and what labeled validation budget trains the transfer policy?
Exit evidence: A supervised teacher and held-out labels are still required; the pipeline is not globally label-free. Policy residual blocks are not a residual-error KD target, and fusion methods retain different inference inputs.
2025–2026
Low-resolution student inference versus resolution-cascaded generation
GAP-Track removes teacher and residual reconstruction branches at inference. RMD instead retains high-resolution generative refinement. Both exploit cross-resolution supervision but solve different deployment problems.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 17.01 GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT TrackingECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Table 1 governs the recreated LasHeR values; the quarter-resolution CKD SR differs between Table 1 (35.3) and Table 4 (35.5). The PGS angle/radial equations need origin and quadrant safeguards not specified in the displayed formulas. The repository contains implementation directories despite a remaining release-soon notice; code was inspected, not executed. GAP-Track transfers full-resolution tracking supervision to a low-resolution RGB–thermal student. A masked residual feature generator learns to approximate teacher features during training, and a hierarchy of feature teachers eases extreme downsampling. The deployed model keeps only the low-resolution student and box head; it does not first reconstruct a high-resolution image at inference. Official ECCV poster record · Official code 17.02 Cross-Resolution Distribution Matching for Diffusion DistillationECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. RMD retains high-resolution refinement at inference; it is not low-resolution-only recognition. Reported speedups depend on the single-A100 baseline, CFG and NFE. The conference PDF author order differs from the arXiv record and is preserved here. RMD distills a diffusion generator into a few-step trajectory that begins at low resolution and ends at high resolution. It aligns resolution-specific noise states using logSNR, projects student distributions into the teacher’s resolution, and uses distribution matching plus noise reinjection to train the cascade. The output remains a high-resolution generated image or video. Official ECCV poster record Work: Is high resolution absent at deployment, or only used for fewer steps? What is the full-resolution reference?
Exit evidence: Synthetic downsampling does not reproduce all optical changes between microscope magnifications; lower-resolution gains need not equal full-resolution performance.
2025–2026
Teacher-class mixtures and privileged residual routing
MoE-KD reformulates classification through latent teacher-class experts and EM. MARS uses full-minus-partial features to teach a deployable router. Specify whether knowledge resides in label mixtures, prototypes, routing decisions or a residual target.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 18.01 MoE-KD: Your Teacher Model is Worth Mixture-of-Experts for Knowledge DistillationECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The EM convergence discussion assumes an improving M-step; it is not a guarantee of global optimum for stochastic neural training. Table 3 values take precedence over inconsistent prose deltas. No runnable official implementation was verified in this pass. MoE-KD treats the teacher’s class as a latent variable in the student classification model. Teacher-derived feature prototypes define class-conditioned experts, while a reused teacher classifier gates projected student features. Alternating posterior estimation and evidence-lower-bound optimization couples these pieces instead of imposing one fixed teacher soft-label distribution alongside the ground-truth target. Official ECCV poster record 18.02 Residual-Guided Expert Specialization for Incomplete Multimodal LearningECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. MARS distills a privileged residual router into an available-feature router; it is not a cascade of residual-error student networks. Full modalities are assumed available during training. Food-101 full-modality accuracy is below SimMLM despite higher missing-modality and average scores. MARS uses the difference between complete- and partial-modality representations to teach expert routing. A privileged residual router learns which experts address each missingness pattern; a second router learns to imitate those decisions from available features alone. Thus the residual is a training signal, while deployment uses a practical feature router without the missing modalities. Official ECCV poster record · Official code Work: Which privileged information is available only during training, and exactly what remains in the student?
Exit evidence: Residual-guided routing is not a residual-error student cascade; conditional optimization arguments do not certify best accuracy.
2025–2026
Spatial-frequency and response-token distillation targets
SFKD selectively aligns wavelet-localized Fourier amplitudes across backbone families. TRAG aligns response-to-vision attention using token-specific entropy-weighted KL. Both teach structure beyond final logits, but use different correspondence assumptions.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 19.01 SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral InteractionECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The linked official repository still says source code will be released soon. This is heterogeneous-architecture feature KD, not demonstrated cross-magnification transfer. The selected CIFAR-100 table includes a ViT-S→MobileNetV2 case where FBT is better. SFKD transfers knowledge between different backbone families by separating local spatial structure from global spectral information. It decomposes intermediate maps into Haar-wavelet subbands, refines the student bands through convolutional and Transformer streams, and matches Gaussian-filtered Fourier amplitudes to teacher bands. Reconstruction and logit supervision keep the transfer connected to the classification task. Official ECCV poster record · Official code 19.02 Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge DistillationECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Teacher and student share a frozen SigLIP vision encoder and resolution; this is not cross-resolution KD. The official code repository lists training/evaluation scripts as planned. Attention similarity correlations are not causal guarantees of faithful reasoning. TRAG distills how a multimodal teacher attends to visual tokens while producing a response. It aggregates middle-layer response-to-vision attention, renormalizes over visual keys and assigns each response token its own forward/reverse KL mixture using teacher attention entropy. This supplements language and unimodal distillation losses rather than replacing the full training recipe. Official ECCV poster record · Official code Work: Are the grids, semantic units and probability supports genuinely aligned before applying the loss?
Exit evidence: SFKD has architecture-specific regressions; TRAG shares a visual encoder and resolution. Neither establishes cross-magnification performance.
2025–2026
Perceptual experts, geometry and explicit grounding tokens
Continuous expert-supervised visual tokens, geometry-aware fusion and coarse-to-fine grounding alter how a language model accesses visual evidence. Compare the representation interface, auxiliary supervision and deployed computation before comparing answer scores.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 20.01 Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual TokensECCV 2026 Oral verified in the official program (event 6085); scientific claims use the conference-hosted manuscript #5952. Publisher proceedings/version of record was not verified on 2026-09-12. First preprint appeared in November 2025; stable ID preserves that origin. CoVT teaches a VLM to generate continuous visual thoughts alongside ordinary language. These latent tokens acquire perceptual meaning through supervision from segmentation, depth, edge, and self-supervised feature experts. The answer conditions on this compact visual reasoning sequence; dense predictions are optional explanatory outputs rather than mandatory inference-time tool calls. ECCV program · Conference manuscript · Author project · Code Work: Does a claimed reasoning gain arise from more visual evidence, better alignment, a changed output space or a larger inference budget?
Exit evidence: Expert distillation is not proof of faithful reasoning; stronger pointing or spatial QA need not improve every tracking metric.
2025–2026
Causal audiovisual generation and variable-step distillation
OmniForcing transfers a bidirectional audio-video generator to causal streaming; AnyFlow learns on-policy flow maps supporting different sampling budgets. Preserve training-stage, self-rollout, synchronization, step-count and hardware qualifiers.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 21.01 OmniForcing: Unleashing Real-time Joint Audio-Visual GenerationECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #9084. Publisher proceedings/version of record was not verified on 2026-09-12. Author list follows the conference manuscript and program; the current repository citation lists two additional authors. Review separates the paper's three-stage method from the later simplified two-stage code release. OmniForcing converts the 19B LTX-2 bidirectional audio–video generator into a streaming model through causal distillation. Its defining problem is not simply fewer diffusion evaluations: video and audio have different latent time scales, yet both must be generated causally and remain synchronized. The paper combines physically aligned one-second blocks, stabilized causal attention and joint self-forcing. It reports a 0.7-second first chunk and 25 FPS on one H200, while retaining explicit quality tradeoffs against the slower teacher. ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoint 21.02 AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map DistillationECCV 2026 Spotlight verified in the official program; scientific claims use the conference-hosted manuscript #9062. Publisher proceedings/version of record was not verified on 2026-09-12. AnyFlow distills a video generator that can use different sampling budgets without retraining. Instead of learning only a noisy-state-to-clean endpoint map, it learns transitions between two arbitrary diffusion times. A second, on-policy stage trains on the student's own trajectories using shortcut flow-map simulation, improving few-step quality while retaining useful refinement at more steps. ECCV Spotlight program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model collection Work: What teacher information disappears at inference, and does quality remain stable as the generated history or step budget changes?
Exit evidence: Real-time and any-step claims depend on resolution, duration, hardware and guidance; source-code revisions may change the training recipe.
2025–2026
Volume tokens, persistent geometry and registration
Compare learned volume representations, streaming geometric context, flow-based rigid registration and dense image matching. Each chooses a different correspondence unit and computation path; learned priors complement rather than abolish geometric constraints.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 22.01 Volume Transformer: Revisiting Vanilla Transformers for 3D Scene UnderstandingAccepted to ECCV 2026; oral presentation verified in the official program. This record and review use the conference-hosted manuscript checked 2026-09-12; a separate Springer proceedings record was not verified. Volt makes a plain global-attention Transformer useful for 3D scene segmentation by turning occupied cubic volumes into tokens. Its central lesson is that tokenization and training data determine whether a generic architecture succeeds: a convolutional teacher supplies a useful inductive bias even when that teacher is less accurate than the trained student. Official ECCV oral presentation · Official ECCV paper record · arXiv record · Official code · Official project Work: Which coordinates, features or correspondences are predicted, and how are rigidity, memory growth and distribution shift tested?
Exit evidence: Compact per-frame context still accumulates; multi-step integration is not one-pass inference, and improved average matching can hide benchmark regressions.
2025–2026
Historical adapter anchors and analytic continual classifiers
Historical LoRA anchors constrain encoder drift while text-guided accumulated statistics update a classifier without raw-example replay. This is a teacher-state and classifier-memory contract, not the residual-error cascade used by other KD methods.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 23.01 Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual LearningECCV 2026 Spotlight verified via the official related presentation event; the poster identity itself says Accept Poster. No Springer version of record or public runnable implementation was verified. Section 3.5's printed closed-form equation omits a term implied by its objective; the review flags this reproducibility issue without silently repairing the authors' formula. Multi-anchor distillation restrains accumulated CLIP feature drift while a text-guided analytic classifier avoids gradient-based overwriting of classifier weights. Historical LoRA states supply visual anchors; accumulated sufficient statistics and frozen text prototypes supply the classifier. This exemplar-free approach stores model state and statistics rather than old raw images. Official ECCV 2026 Spotlight Work: Which stored states prevent forgetting, and does the analytic argument still apply when the feature encoder changes?
Exit evidence: Stable-feature equivalence is conditional; check the printed objective/closed-form mismatch and separate average from final accuracy.
2025–2026
Risk-specific multi-teacher survival adapters
SurvMILKD combines frozen encoder features through trainable MIL adapters, teacher-space feature translation and soft ordinal risk distributions. It removes teacher branches at deployment while retaining one student feature pipeline.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 24.01 SurvMILKD: A Weakly Supervised Survival Analysis Framework for Multi-Teacher Knowledge Distillation using Pathology Foundation ModelsThe official ECCV 2026 poster and conference-hosted manuscript verify acceptance; a Springer version of record was not verified. The official repository was a code-coming-soon placeholder on 2026-09-12. All foundation encoders remain frozen: task-specific MIL adapters and translation modules learn, not the UNI backbone. Risk-aware multi-teacher distillation combines complementary slide-level survival signals while deploying only a student branch. SurvMILKD uses frozen pathology foundation encoders, trainable attention-MIL adapters, teacher-specific feature translators, and a four-bin representation of scalar risk. Its contribution is task-specific survival transfer rather than another general-purpose encoder pretraining recipe. Official ECCV 2026 Poster · Official repository — code pending Work: How does distillation represent censored survival risk, and which teacher and feature costs remain in training versus inference?
Exit evidence: Not every cohort benefits from more teachers; frozen encoders, external risk pooling and code-release status must remain explicit.
2026
Diffusion-boundary self-distillation and negative transfer
TRACE turns trajectory-derived attention boundaries into one-pass predictions through LoRA and an edge decoder. Read its natural-image gains alongside the reported MoNuSeg/TNBC degradation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 25.01 TRACE: Your Diffusion Model Is Secretly an Instance Edge DetectorOfficial ICLR 2026 oral and full proceedings manuscript checked. Diffusion-boundary self-distillation improves selected natural-image segmentation but degrades MoNuSeg/TNBC cell segmentation. One-pass speed is edge extraction only, with the large backbone retained. No WSI or cross-magnification validation; official code/checkpoints remain promised rather than released at the checked revision. TRACE extracts instance-boundary cues from self-attention inside a pretrained diffusion model, then self-distills a slower trajectory-based edge procedure into one backbone pass. Natural-image mask refinement improves, but the paper also reports worse cell segmentation on MoNuSeg and TNBC. OpenReview · Official project · Official repository — release pending Work: Does the teacher prior delineate the target objects before it is distilled, and what remains in the deployed model?
Exit evidence: A tiny decoder does not remove the large diffusion backbone. Edge-extraction speed is not end-to-end speed, and negative cell-level transfer is not WSI validation.
2026
Description-mediated transfer for image quality
RALI transfers teacher quality descriptions into a CLIP encoder, then fits compressed score bases. Read its image-only deployment path separately from the paper’s multi-domain RACT procedure.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 26.01 Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality AssessmentOfficial ICLR 2026 oral and full proceedings manuscript checked. Teacher-generated quality descriptions align a compact image encoder; deployed scoring uses PCA and fitted bases without an LLM. Keep RALI separate from multi-domain RACT. Dataset-specific trade-offs, causal-interpretation limits and the conflicting memory-reduction claim are qualified. No WSI or cross-magnification evaluation. RALI transfers quality-specific descriptions from a reasoning-based image quality assessment teacher into a CLIP image encoder. The deployed scorer does not generate a reasoning trace or load the teacher LLM: it compares a compressed image embedding with learned quality bases and combines their scalar scores. Official proceedings record · OpenReview · Official implementation Work: Which teacher information is transferred, which modules disappear at inference, and which domains lose performance?
Exit evidence: Comparable aggregate IQA performance is not universal replacement or WSI validation; account for offline description generation and the measured memory ratio.
2026
Cross-mode distillation into explicit 3D representations
FlashWorld combines a multi-view teacher, rendered-Gaussian student and learned fake-score network. Study distribution matching, consistency regularization and when OOD batches must omit adversarial targets.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 27.01 FlashWorld: High-quality 3D Scene Generation within SecondsOfficial ICLR 2026 oral and full proceedings manuscript checked. A frozen multi-view teacher guides a few-step rendered-Gaussian student using learned fake scores, consistency and OOD co-training. Shared-scene rendering is not guaranteed correct geometry. Preserve different benchmark timing protocols, proprietary-data and training-code access limitations. No WSI or cross-magnification evidence. FlashWorld distills a high-quality multi-view diffusion mode into a few-step generator that directly produces a 3D Gaussian scene. Its central idea is cross-mode distribution matching: transfer appearance quality from the multi-view teacher while rendering the student’s shared 3D representation during denoising. Official proceedings record · OpenReview · Official implementation Work: What does changing the student representation preserve, and which visual failures escape the numerical metrics?
Exit evidence: A shared rendered scene does not certify correct geometry, fewer parameters or cross-magnification transfer; preserve hardware and benchmark protocol differences.
2026
Real-data correction in generative distillation
RealUID unifies matching-model distillation and routes real examples through an auxiliary matching model. Read the alternating optimization and coefficient assumptions before interpreting the one-step generator results.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 28.01 Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)Official ICLR 2026 oral and full proceedings manuscript checked. Three-network training yields a one-step generator. No GAN discriminator does not mean no min–max optimization. Selected low-resolution image results include additional fine-tuning; teacher-error correction has restrictive assumptions. No cross-magnification or pathology evaluation. RealUID adds real-data supervision to a unified inverse-distillation formulation for matching models. A frozen teacher, a one-step generator and a learned fake matching model interact through matching losses. No GAN discriminator is added, but alternating min–max optimization remains: “No GANs” does not mean no adversarially structured optimization. Official proceedings record · OpenReview Work: Where does real data enter the teacher–student system, and which extra training stages are included in the comparison?
Exit evidence: No GAN discriminator does not mean no min–max problem; low-resolution generation does not establish pathology validity.
2026
Data-constrained ensembles and sequence distillation
Regularization and independent teachers extract more from a fixed real corpus; mixed real/synthetic sequence training transfers part of the ensemble benefit into one student.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 29.01 Pre-training under infinite computeOfficial ICLR 2026 oral and full proceedings manuscript checked. Fixed unique data, regularized teachers and real/synthetic sequence distillation. Infinite-compute claims are fitted asymptotes, not completed runs. Distinguish main mixing recipes from the matched-token ablation and inference parameter savings from total training cost. This study changes the optimization question from a fixed compute budget to a fixed supply of real data. It combines strong regularization, independently trained ensembles and sequence distillation to improve data-constrained language models. “Infinite compute” names a fitted limiting regime, not a completed infinite training run. Official proceedings record · OpenReview Work: What is held fixed: unique data, total tokens, parameters, generation cost or deployment cost?
Exit evidence: Fitted infinite-compute limits are not measured runs, and a smaller deployed student is not automatically cheaper to train.
2026
Conflicting supervision and adaptive distillation objectives
DTO-KD adapts the balance between task and teacher-transfer objectives using relative-loss optimization. Compare the exact log-gradient construction with the practical amortized update; control the transfer architecture before attributing gains to optimization.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 30.01 DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge DistillationOfficial ICLR 2026 oral and proceedings manuscript checked. Adaptive task/distillation optimization, not cross-magnification or residual-error KD. Separate the exact two-gradient construction from the amortized implementation; the printed simplex handling and lower-bound argument require qualification, documented in the source-checked review. DTO-KD treats task supervision and teacher imitation as a multi-objective optimization problem. Its central idea is to adapt their contribution according to relative loss improvement instead of choosing one fixed task/distillation weight. The paper distinguishes conflicting gradient directions from unequal gradient magnitudes, but its exact analytical construction and practical amortized algorithm must not be treated as identical. Official proceedings record · OpenReview · Theoretical supplement Work: Which objectives conflict, which update is actually implemented, and what costs and guarantees belong to that implementation?
Exit evidence: The printed closed form omits simplex boundary handling and the claimed non-collapse lower bound has a counterexample. Natural-image results do not establish cross-magnification or WSI validity.
2021–2026
Channel distributions and feature-pyramid transfer
CWD normalizes spatial activations inside each channel; CSCWD adds teacher P2 to student P3 alignment for tiny-object detection. Study what the spatial probability distribution preserves before changing feature resolution.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 31.01 Channel-Wise Knowledge Distillation for Dense PredictionICCV 2021 proceedings paper and author v4 manuscript checked; older missing prerequisite added during the October 2026 audit, not a new 2026 paper. Official repository records the semantic-segmentation code release on 20 August 2021. Channel-wise KD changes the axis on which dense-prediction activations become probabilities. Instead of normalizing classes separately at each pixel, it normalizes spatial positions separately within each channel. The student then matches teacher spatial distributions, encouraging it to preserve where each channel concentrates its evidence. Official implementation · Author manuscript 31.02 CSCWD: Cross-Scale Channel-wise Knowledge Distillation for Lightweight Tiny Object Detection on Edge DevicesarXiv v1 checked 2 October 2026; no official code release or acceptance verified. Cross-scale means teacher P2 to student P3 feature-map alignment at a shared input resolution, not independent high/low magnification acquisitions. MGD is stronger on reported mAP; external stricter mAP does not improve. CSCWD lets a compact detector learn from a teacher feature level that the student does not possess. The teacher’s fine P2 map supervises the student’s coarser P3 map through training-only alignment and channel-wise distribution matching. Only the original YOLO11n inference graph remains at deployment. — Work: Are the two models seeing different acquisitions, or the same image through different feature maps?
Exit evidence: CSCWD uses shared-resolution image inputs, not 20×/5× tissue pairs. Its recall gain coexists with MGD’s stronger mAP and a non-improving external strict mAP.
2026
Which feature directions should distillation preserve?
SpecMatch balances reconstruction along teacher principal components rather than letting high-variance directions dominate an L2 objective. Low variance need not mean low downstream value.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 32.01 The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation DistillationarXiv v1 checked 2 October 2026. No official code/model release or conference acceptance verified. Camelyon17 is patch-level evaluation, not whole-slide cross-magnification validation. PE teacher/student labels and parameter counts differ between main text/tables and Appendix C; this review uses unambiguous DINOv2-to-MobileNet Table 1 rows. SpecMatch changes how feature distillation allocates its learning signal. A fixed teacher PCA basis separates representation directions, and a sublinear function of each direction’s reconstruction error reduces the dominance of high-variance coordinates. The aim is to preserve useful low-variance information, not merely make the total feature MSE small. — Work: Does a smaller global feature error hide worse transfer on discriminative but low-variance directions?
Exit evidence: Patch-level Camelyon17 is not a WSI endpoint. Spectral balancing is not evidence that unobserved high-magnification information can be recovered.
2026
Cross-magnification transfer: utility, predictability and task benefit
After XMaG and LRMIL, test whether observed teacher targets help, whether native low-power features predict them, and whether MIL benefits from the predictions. Direct versus residual regression and native-feature retention are separate controls.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 33.01 What Makes High-Magnification Knowledge Transferable? A Study of Cross-Resolution Distillation in Whole-Slide ImagingPublic arXiv v2 (2026-10-01) verified 2026-10-02. The authors state ICLR 2027 under review; no acceptance inferred. Only public source material was used. A diagnostic study of target utility, predictability and downstream benefit, not a new universally superior distillation architecture. Separate teacher usefulness, student predictability and downstream benefit. Observed 20× regional means can help slide prediction when supplied with native 5× features, yet a 5× predictor may fail to reconstruct them usefully. Lower reconstruction error does not consistently translate into better slide-level performance. Reviewed manuscript revision Work: Can the proposed target beat a constant training-mean baseline and improve the final slide endpoint?
Exit evidence: Observed 20× access is a diagnostic reference, not low-power-only deployment. Lower reconstruction MSE is not guaranteed downstream improvement; the study is an ICLR 2027 submission-stage preprint.
2026
Distilling reusable MIL initializations
TITAN and CARE supervise nine lightweight MIL aggregators over regular ROI bags. Teacher-specific projections and dispersion-normalized alignment precede linear probing or supervised fine-tuning.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 34.01 Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation ModelsarXiv v1 (2026-07-16) checked 2026-10-02. Six cohorts cover 15 tasks, not 15 independent datasets. Official code and pretrained aggregator checkpoints are listed; experiments were not rerun. This transfers slide embeddings into MIL initialization, not 20×-to-5× acquisition. Pretrain the MIL aggregator, not only its tile encoder. A small, initially headless MIL network learns to reproduce complementary TITAN and CARE region embeddings. The resulting aggregator weights initialize multiple downstream slide tasks; this is reusable representation transfer, not a 20×-to-5× acquisition method. Versioned manuscript · Official implementation and pretrained MIL weights Work: What improves because of pretraining, and does that gain survive a matched architecture, encoder and tuning budget?
Exit evidence: Six cohorts contain 15 tasks; more teachers do not always win. Main-text angular loss and appendix pseudocode differ, and aggregator pretraining does not itself reduce acquisition magnification.
2026
Tissue distillation: masks, features, hierarchy and relations
Compare staged pixel/feature KD from Virchow2, PAGET’s hierarchical multi-teacher pseudo-labels, and ConStruct’s transfer of SegFormer structural affinities into adapted CONCH features for prototype-guided segmentation. Their annotation contracts and deployed components differ.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 35.01 Computationally Efficient Pathology Segmentation using Knowledge Distillation from Foundation ModelsarXiv v1 (2026-09-03), submitted to Medical Image Analysis according to the authors, not accepted. Verified methods and selected results on 2026-10-02. TIAToolbox is the stated release destination; availability of these exact weights was not independently confirmed. Patch-throughput measurements are not end-to-end WSI latency. Transfer Virchow2 segmentation knowledge into SegFormerB5 or EUNet-M using a staged combination of pixel predictions and global features. The deployed model is a compact segmentation student, not the foundation-model teacher. This changes model cost at fixed patch input, not optical magnification. Versioned manuscript · TIAToolbox — stated release destination 35.02 PAGET: Hierarchical Multi-Teacher Knowledge Distillation for Comprehensive Tumor Microenvironment SegmentationPeer-reviewed MIDL 2026 proceedings, PMLR 315:256–279; official record verified 2026-10-02. PAGET compresses specialized pathology teachers into a SegFormer student that segments 13 tumor-microenvironment components plus background. Its main intervention is taxonomy-aware pseudo-label construction: resolve tissue context, major cell categories, and fine immune subtypes in an explicit order before teaching the student. This is heterogeneous-teacher hard-label distillation, not averaging interchangeable classifiers or matching foundation-model feature vectors. Official proceedings PDF · Official implementation 35.03 ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology SegmentationPeer-reviewed MIDL 2026 proceedings, PMLR 315:2893–2905; official record verified 2026-10-02. ConStruct adds spatial organization to a pathology vision–language model for weakly supervised tissue segmentation. A frozen SegFormer teacher supplies token-to-token relations; trainable adapters refine frozen CONCH image features, and text-initialized prototypes turn the refined features into class activation maps. Training needs image-level tissue labels rather than dense pixel masks. The reported experiment is on BCSS-WSSS patches, not a whole-slide classification or cross-magnification benchmark. Official proceedings PDF · Official implementation Work: Which supervision remains observable at student resolution, and which teacher or post-processing components still run at inference?
Exit evidence: PAGET-H retains HoverNet. ConStruct’s residual adapters are not residual-error targets. Pixel segmentation, patch throughput and patient-level WSI outcomes are distinct evidence.
2026
Pathway-structured privileged supervision
MoPE uses training-only RNA to supervise WSI pathway experts through shared-memory distributions. It aligns a structured view of molecular information without requiring RNA at inference.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 36.01 Pathway-Structured Privileged Distillation for Deployable Computational PathologyarXiv v1 (2026-06-01) checked 2026-10-02; no conference acceptance verified. Paired RNA is training-only, while WSI is used at deployment. Pathway-memory readouts are not independently validated molecular measurements. MoPE learns pathway-structured histology predictors using paired bulk RNA during training, but deploys with WSI alone. Instead of forcing RNA and tissue embeddings to be identical, it aligns their distributions over a shared memory. This is privileged-modality distillation, not magnification reduction or direct RNA reconstruction. Versioned manuscript Work: Which part of the molecular target is transferable from morphology, and what independent evidence supports biological interpretation?
Exit evidence: Paired RNA is still needed for training. Model attention is not measured pathway activation; this is neither optical cross-magnification transfer nor prospective clinical validation.
2026
Distilling visual context and sensitivity to evidence
SCOPD teaches a sparse-context student from a full-context EMA teacher on student rollouts. CW-OPD instead tests how predictions change when answer-critical visual evidence is edited between two worlds.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 37.01 SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language ModelsPreprint checked 2026-10-02; project page still labels code and models as forthcoming. Visual-token retention is not an optical-magnification intervention. SCOPD treats visual-token pruning partly as a representation-utilization problem. A full-context teacher supervises responses generated by the sparse student; SCOPD+ concentrates the supervision on visually sensitive positions. Versioned full text · Author project page 37.02 Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language ModelsPreprint checked 2026-10-02. The paper calls the method CW-OPD; its linked code repository uses the name CWAD. No conference acceptance inferred. CW-OPD supervises how a VLM prediction changes when answer-critical visual evidence changes. It combines ordinary per-world distillation with a cross-world belief-transition target, rather than assuming a correct answer proves the student used the right evidence. Versioned full text · Author implementation (repository name CWAD) Work: Does the student learn visual evidence, or merely the teacher’s answer distribution?
Exit evidence: Token pruning and counterfactual image editing are not WSI acquisition changes. Track EMA updates, teacher-only hints, editor quality and training cost before comparing distillation methods.
2026
Token-level direction and teacher consensus
BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 38.01 Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation. BinaryOPD replaces the magnitude of a teacher-student log-probability reward with its sign. The paper then uses high-disagreement token analysis to motivate C-MOPD, which combines teachers while declining strongly conflicting updates. Versioned full text · Author implementation Work: Which disagreement carries useful information, and when should conflicting teachers produce no update?
Exit evidence: Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost.
2026
Distill visual grounding by withholding textual shortcuts
Hide to See masks parts of the student’s reasoning prefix while the teacher sees the complete prefix. The asymmetric distillation setup asks the student to rely more on the image rather than copy answers from preceding text.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 39.01 Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM DistillationNeurIPS 2026 acceptance is author-reported by NVIDIA Research and the official implementation, checked 8 October 2026. No exact title or author match was found in the dated official oral/poster snapshot; presentation tier and final proceedings are unverified. Review uses arXiv v4. Training-time textual-prefix masking is not WSI magnification transfer. Masking-KD distills a reasoning VLM while hiding selected textual shortcuts from the student. The teacher sees the complete image, question, and earlier response; the student loses access to its most attended earlier reasoning tokens and must predict the same next-token distribution with the image still available. The intervention is reasoning-prefix masking, not image masking or an inference-time pruning algorithm. Versioned full text · Official implementation · Author project · Author acceptance record Work: Does masking improve image-dependent reasoning, or only alter the difficulty of next-token prediction?
Exit evidence: Acceptance is author-reported; no exact match was located in the inspected official program snapshot. Prefix masking changes training context, not image resolution, and image-dependence proxies are not clinical grounding tests.
2026
Density-weighted residual alignment in hybrid VLMs
HEED aligns teacher and hybrid-student decoder residual streams, weighting visual positions by frozen-ViT local self-dissimilarity. This density proxy is not semantic saliency. Distinguish the hybrid architecture’s inference benefit from density weighting’s incremental benefit in a matched training recipe.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 40.01 HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model DistillationOfficial NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested. HEED protects locally distinctive visual positions when converting an attention-based VLM to a hybrid Mamba-2/attention model. Its contribution is density-weighted residual alignment: the student matches the teacher's residual stream more strongly at patches whose frozen visual features differ from their neighbors. The deployed architecture is unchanged by this loss; HEED does not delete visual tokens. Versioned full text · Official acceptance identity · Official NeurIPS program Work: Does density weighting outperform uniform residual alignment when data, trainable parameters and architecture are held fixed?
Exit evidence: The C3→C4 OCRBench v2 gain is 4.7 points; the larger headline change includes other training changes. The manuscript disagrees about Stage 3 trainable parameters. Neither the speedup nor this feature residual establishes WSI cross-magnification transfer.
2026
Numerical objectives: preserve the loss gradient, not just the loss value
A Fix for Softmax and Cross-Entropy separates hard-target absorption from near-equal teacher/student cancellation. Excluding the target from log-sum-exp and using softplus protects hard-label gradients; a logit-difference construction with log1p/expm1 repairs soft-target derivatives. The intended objective is unchanged.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 41.01 A Fix for Softmax and Cross-EntropyInspected arXiv:2610.31642v1, submitted 22 October 2026, and the author project page. Numerical-gradient and modular-arithmetic experiments are not evidence of improved WSI or large-scale foundation-model accuracy. No venue acceptance was verified. This paper repairs floating-point evaluation of familiar objectives rather than proposing a new classification target. Softmax Collapse loses the dominant class gradient at high confidence; Softmax Cancellation loses the small difference between nearly matching student and teacher distributions. Separate algebraic fixes preserve the intended cross-entropy gradient in these two regimes. Inspected manuscript (v1) · Author project and JAX implementations Work: Does the actual dtype and computation graph preserve high-margin and teacher-matched gradients?
Exit evidence: Evidence is numerical and small modular-arithmetic training, not WSI or large-scale foundation-model improvement. Higher precision and zero-sum projection are not automatically equivalent fixes. Check the real compiler and temperature convention before changing a training protocol.
2022–2026
Distillation projectors: dimension matching versus optimization interference
A projector can separate the student’s task representation from the space used to imitate a teacher even when both feature dimensions match. PEFD averages parallel nonlinear projector outputs before direction matching; the extended study adds logit, representation-similarity and calibration analyses.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 42.01 Understanding the Effects of Projectors in Knowledge DistillationIEEE publisher metadata identifies TPAMI 48(8), 9078–9092, August 2026. The accessible scientific text checked here is arXiv:2310.17183v1 (26 October 2023); final journal numerical identity was not established. This extends the authors’ NeurIPS 2022 Projector Ensemble Feature Distillation work, rather than originating PEFD in 2026. A distillation projector can do more than resize a feature vector: it can buffer the conflict between learning the label task and matching a teacher. This study tests projectors even when dimensions already agree, extends the idea to logits, and analyzes a parallel projector ensemble. The scientific summary follows the accessible 2023 manuscript; the catalog also identifies the 2026 journal publication. Inspected author manuscript (v1) · Author code and earlier PEFD implementation Work: Does the alignment head protect useful student features, or merely fit the teacher target more closely?
Exit evidence: PEFD originated in 2022. The accessible 2023 manuscript supports the reviewed scientific details; final 2026 journal numerical identity was not verified. More projectors or greater depth do not guarantee improvement.
2026
Distillation changes capabilities, not only average scores
Distilled Pretraining separates access to teacher data from the effect of soft targets. It contrasts context copying with multi-sample solution coverage and tests entropy-based token routing. A student can improve on pass@k yet become less responsive to supplied context.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 43.01 Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling Distilled pretraining transfers a teacher’s full next-token distribution, not only sampled text. The study finds a capability trade-off: soft targets can improve multi-sample solution coverage while weakening context-copying behavior in a matched-data regime. Entropy-based token routing partially mitigates the latter rather than making distillation uniformly beneficial. Proceedings PDF · arXiv Work: Which capability benefits from teacher smoothing, and which needs the original hard target?
Exit evidence: Matched token counts do not match total teacher–student compute. Context-copying experiments and pass@k are separate endpoints; routing results are not a universal recipe or demonstrated pathology transfer.
2026
Edge encoders: contrastive multi-teacher compression
MuCoDi caches fixed pathology-teacher vectors and trains compact students to match tile identity contrastively. MuCoEdge is evaluated with frozen-encoder MIL and single-tile CPU timing. Teacher and student both use 5×; capacity reduction and magnification reduction must remain separate hypotheses.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 44.01 Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation ModelsOfficial MICCAI satellite proceedings confirm the 2nd Efficient Medical AI workshop (EMA), not the main MICCAI track. LNCS volume 17265 is listed; DOI and pagination remain pending on the checked record. Full 11-page accepted manuscript inspected. Teachers and students use 2.0 µm/px (5×); this is capacity compression, not a demonstrated 20×→5× transfer method. Raspberry Pi timings concern individual encoder forward passes, not complete WSI processing. MuCoDi learns compact pathology encoders by multi-teacher contrastive distillation from fixed Virchow2, UNI2 and H-Optimus-1 tile embeddings. Its MuCoEdge students keep much of the teachers' downstream performance with small MobileOne or RepViT backbones. Crucially, the paper uses the same 5× sampling for teacher and student: it compresses model capacity, not a 20× evidence stream into a 5× one. Official workshop accepted manuscript · Author implementation and documented scope Work: Which performance survives after reducing capacity, at the same physical sampling and downstream readout?
Exit evidence: The 23 tasks are teacher-eligible; encoder timing excludes WSI IO and aggregation. Close average AUROC is not formal non-inferiority or cross-magnification validation.
2026
Route additional correction by teacher–student disagreement
DrOPD preserves reverse KL at every student-generated position and adds forward KL at the batch’s highest-JS positions. The candidate union retains tokens preferred by either model. Read it after on-policy KD to distinguish uncertainty, disagreement and teacher-supported modes the student under-covers.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 45.01 Divergence-Routed On-Policy Distillation for Language ModelsReview pins arXiv v1 (14 October 2026). The manuscript says under review at ICLR 2027; this is not acceptance. DrOPD changes where an additional teacher signal is applied. It keeps reverse-KL supervision on every student-generated response position and adds forward KL at the highest-disagreement positions. The motivation is under-coverage: reverse KL can barely push up a candidate to which the student already assigns almost no mass. This is a training-objective intervention, not a new language-model architecture. Version 1 paper and appendices Work: Which tokens need extra teacher-weighted correction, and is their teacher target reliable?
Exit evidence: Math-only preprint, two related Qwen pairs. Top-k approximate losses, eight-response mean accuracy and logged training-step cost are separate evidence contracts; no demonstrated WSI transfer.
2026
Audit teacher reliability before stronger on-policy imitation
The VLM teacher-inheritance study compares filtered SFT with full-vocabulary OPD on ChartQA and Geometry3K. It separates answer accuracy from completion, decomposes paired failures and shows OPD can erase completion already learned through SFT. This is an empirical warning, not a new KD objective.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 46.01 Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language ModelsVersion 1 empirical study. The current author README describes an anonymized code-only supplement and explicitly excludes evaluation records, logs, adapters, tests and figure scripts despite the manuscript release statement. This is an empirical boundary study, not a new distillation objective. On-policy distillation can transfer a teacher’s failure to finish an answer, even undoing completion learned through filtered supervised fine-tuning. The central distinction is between final-answer accuracy, presence of an answer line and actual termination. Agreement with a teacher is not evidence of correct or complete reasoning. Version 1 paper and appendices · Author code and current availability scope Work: Does the teacher provide a usable answer on the states the student actually visits?
Exit evidence: One model family and two main tasks; updates are fixed within channels, not compute-matched between them. Teacher probes are not a unique causal attribution. Current author code omits the evaluation records advertised in the manuscript.
2026
Global semantic anchors and local patch targets
Two Global Crops Suffice separates same-image global-view alignment from local crops and iBOT refinement in controlled DINO-family experiments. CODI instead changes the patch target: weakly mix the same teacher view’s class distribution into its local patch distribution. The diagnostic study and target-side method answer related questions, but neither independently validates the other.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 47.01 CODI: Improving Self-Supervised Vision Transformers with Cross DistillationarXiv:2610.30639v1, submitted 22 October 2026, inspected 26 October 2026. The manuscript has a NeurIPS 2026 footer, but no official acceptance record was independently located in this audit; retained as a preprint. The author repository contains only a Coming soon README, not released implementation code. CODI changes a masked patch’s teaching target, not the Vision Transformer architecture. It mixes 90% of the momentum teacher’s local patch distribution with 10% of that same view’s global class-token distribution. This gives low-attention patches a direct semantic training signal while retaining mostly local supervision. It is self-distillation during unlabeled image pretraining, not compression from a separately pretrained large teacher and not cross-magnification transfer. Inspected manuscript (v1) · Author repository — implementation pending at inspection Work: Which signal creates useful spatial semantics, which refines it, and do your global and dense evaluations agree?
Exit evidence: Both inspected sources are preprints. Crop ablations match iterations, not all computation; the depth-scale narrative has an appendix exception. CODI’s gradient statement is local/logit-level, its alpha is task-dependent, and its code was not released at inspection. Neither reports WSI or cross-magnification validation.
Research branches
Output, sequence, and reasoning transfer
Does the student match probabilities, complete sequences, rationales, reverse-divergence targets, or teacher distributions on its own trajectories?
Distilling the Knowledge in a Neural Network · Sequence-Level Knowledge Distillation · Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes · MiniLLM: Knowledge Distillation of Large Language Models · On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes · TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Feature, attention, relation, and capacity transfer
Which layer correspondence and geometry remain meaningful when architectures or capacities differ?
FitNets: Hints for Thin Deep Nets · Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer · Relational Knowledge Distillation · Contrastive Representation Distillation · TinyBERT: Distilling BERT for Natural Language Understanding · MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers · Improved Knowledge Distillation via Teacher Assistant · Distilling Knowledge via Knowledge Review · Decoupled Knowledge Distillation · Masked Generative Distillation
Self-, peer-, and online distillation
Where does the teacher advantage come from if teacher and student share data or capacity?
Deep Mutual Learning · Born Again Neural Networks · Bootstrap Your Own Latent · Emerging Properties in Self-Supervised Vision Transformers
Data-free and on-policy transfer
Are training states original examples, generated surrogates, teacher inversions, fixed traces, or samples from the current student policy?
Data-Free Learning of Student Networks · Zero-Shot Knowledge Distillation in Deep Networks · Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion · On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Continual function preservation
Is distillation preserving an earlier task function while the model adapts, and which old data or output snapshots remain available?
Diffusion and flow distillation
Which teacher trajectory or vector field is approximated, and how is quality traded against function evaluations?
Progressive Distillation for Fast Sampling of Diffusion Models · LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models
Multimodal and foundation-model distillation
Is alignment transferred at token, patch, embedding, preference, uncertainty, generated-data, or expert-ensemble level?
Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation · TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance · MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training · LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation · RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models · COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training · Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
Resolution and evidence-budget transfer
Is high-resolution evidence training-only, retained, selectively acquired, or fully extracted before subset selection—and is physical field of view registered?
Multi-Scale Aligned Distillation for Low-Resolution Detection · Resolution-Based Distillation for Efficient Histology Image Classification · Attention to Detail: Inter-Resolution Knowledge Distillation · Large-to-Small Image Resolution Asymmetry in Deep Metric Learning · A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI Classification · CEMIL: Contextual Attention Based Efficient Weakly Supervised Approach for Histopathology Image Classification · Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning · PatHME: Hierarchical Multi-Expert Knowledge Distillation for Whole Slide Image Analysis · Streamline Pathology Foundation Model by Cross-Magnification Distillation · A Magnification Alignment Framework Enables Computation- and Communication-Efficient Computational Pathology · LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification
Computational-pathology transfer
Does cross-resolution, graph, encoder, or multi-expert transfer preserve patient-level morphology under site and stain shift?
Resolution-Based Distillation for Efficient Histology Image Classification · DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification · Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification · Knowledge Distillation in Histology Landscape by Multi-Layer Features Supervision · SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification · A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI Classification · A generalizable pathology foundation model using a unified knowledge distillation pretraining framework · Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning · Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathology · Distilling Foundation Models for Robust and Efficient Models in Digital Pathology · CEMIL: Contextual Attention Based Efficient Weakly Supervised Approach for Histopathology Image Classification · Unsupervised Anomaly Detection on Preclinical Liver H&E Whole Slide Images Using Graph-Based Feature Distillation · PatHME: Hierarchical Multi-Expert Knowledge Distillation for Whole Slide Image Analysis · Streamline Pathology Foundation Model by Cross-Magnification Distillation · Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation With Nested Embeddings · A Magnification Alignment Framework Enables Computation- and Communication-Efficient Computational Pathology · LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models · HistoMILKD: A Multiple Instance Learning based Multi-Teacher Knowledge Distillation Framework for Whole Slide Image Classification · LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification · Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast Cancer
Privileged, safety, capacity, and on-policy transfer
What exact signal crosses the teacher–student boundary, and does that signal or its generator remain available after training?
Unifying Distillation and Privileged Information · Compact Language Models via Pruning and Knowledge Distillation · Distilling Safe LLM Systems via Soft Prompts for On Device Settings · Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it · On-policy Distillation with Verifiable Reward · On-Policy Self-Distillation in Diffusion Models · Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher
Residual correction and analysis
Is the residual an error target, an architectural path, probability left outside an alignment, or a post-hoc measurement—and who computes it at deployment?
Residual error based knowledge distillation · ResKD: Residual-Guided Knowledge Distillation · Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts · Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation · MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering · FROSTER: Frozen CLIP Is a Strong Teacher for Open-Vocabulary Action Recognition · FOSTER: Feature Boosting and Compression for Class-Incremental Learning · MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection · CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillation · Expandable Residual Approximation for Knowledge Distillation · Knowledge Distillation for Large Language Models through Residual Learning · ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization · On Explaining Knowledge Distillation: Measuring and Visualising the Knowledge Transfer Process · On learning denoisable student logits
Cross-resolution and pathology transfer
Does the student truly remove high-resolution or privileged evidence, retain several scales, or merely reuse one model across magnifications?
Human Visual Attention-Inspired Knowledge Distillation Underlying Interpretable Computational Pathology · Cross-Stain Knowledge Distillation for Low-Cost Lung Cancer Programmed Death Ligand-1 Assessment With Multi-Granularity Multiple Instance Learning · KDTViT Knowledge Distillation, Transfer Learning and Transformer Based Deep Learning Framework for Efficient Histopathology Image Classification · Resolution-Aware Knowledge Distillation for Efficient Inference · Resolution Switchable Networks for Runtime Efficient Image Recognition
Physical observability and task-aware polarization reconstruction
Is the supervision a physically observable target, a synthetic reconstruction target, a downstream task signal or a teacher distribution?
Stokes-Informed Diffusion for Robust Linear Polarization Estimation
Physiology-aware cross-modal spectral distillation
Which teacher signal is physiologically invariant across sensors, and what labeled validation budget trains the transfer policy?
Low-resolution student inference versus resolution-cascaded generation
Is high resolution absent at deployment, or only used for fewer steps? What is the full-resolution reference?
GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT Tracking · Cross-Resolution Distribution Matching for Diffusion Distillation
Teacher-class mixtures and privileged residual routing
Which privileged information is available only during training, and exactly what remains in the student?
MoE-KD: Your Teacher Model is Worth Mixture-of-Experts for Knowledge Distillation · Residual-Guided Expert Specialization for Incomplete Multimodal Learning
Spatial-frequency and response-token distillation targets
Are the grids, semantic units and probability supports genuinely aligned before applying the loss?
SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction · Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Perceptual experts, geometry and explicit grounding tokens
Does a claimed reasoning gain arise from more visual evidence, better alignment, a changed output space or a larger inference budget?
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Causal audiovisual generation and variable-step distillation
What teacher information disappears at inference, and does quality remain stable as the generated history or step budget changes?
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation · AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation
Volume tokens, persistent geometry and registration
Which coordinates, features or correspondences are predicted, and how are rigidity, memory growth and distribution shift tested?
Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding
Historical adapter anchors and analytic continual classifiers
Which stored states prevent forgetting, and does the analytic argument still apply when the feature encoder changes?
Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual Learning
Risk-specific multi-teacher survival adapters
How does distillation represent censored survival risk, and which teacher and feature costs remain in training versus inference?
Diffusion-boundary self-distillation and negative transfer
Does the teacher prior delineate the target objects before it is distilled, and what remains in the deployed model?
TRACE: Your Diffusion Model Is Secretly an Instance Edge Detector
Description-mediated transfer for image quality
Which teacher information is transferred, which modules disappear at inference, and which domains lose performance?
Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessment
Cross-mode distillation into explicit 3D representations
What does changing the student representation preserve, and which visual failures escape the numerical metrics?
Real-data correction in generative distillation
Where does real data enter the teacher–student system, and which extra training stages are included in the comparison?
Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
Data-constrained ensembles and sequence distillation
What is held fixed: unique data, total tokens, parameters, generation cost or deployment cost?
Conflicting supervision and adaptive distillation objectives
Which objectives conflict, which update is actually implemented, and what costs and guarantees belong to that implementation?
DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
Channel distributions and feature-pyramid transfer
Are the two models seeing different acquisitions, or the same image through different feature maps?
Channel-Wise Knowledge Distillation for Dense Prediction · CSCWD: Cross-Scale Channel-wise Knowledge Distillation for Lightweight Tiny Object Detection on Edge Devices
Which feature directions should distillation preserve?
Does a smaller global feature error hide worse transfer on discriminative but low-variance directions?
Cross-magnification transfer: utility, predictability and task benefit
Can the proposed target beat a constant training-mean baseline and improve the final slide endpoint?
Distilling reusable MIL initializations
What improves because of pretraining, and does that gain survive a matched architecture, encoder and tuning budget?
Tissue distillation: masks, features, hierarchy and relations
Which supervision remains observable at student resolution, and which teacher or post-processing components still run at inference?
Computationally Efficient Pathology Segmentation using Knowledge Distillation from Foundation Models · PAGET: Hierarchical Multi-Teacher Knowledge Distillation for Comprehensive Tumor Microenvironment Segmentation · ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology Segmentation
Pathway-structured privileged supervision
Which part of the molecular target is transferable from morphology, and what independent evidence supports biological interpretation?
Pathway-Structured Privileged Distillation for Deployable Computational Pathology
Distilling visual context and sensitivity to evidence
Does the student learn visual evidence, or merely the teacher’s answer distribution?
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models · Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
Token-level direction and teacher consensus
Which disagreement carries useful information, and when should conflicting teachers produce no update?
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Distill visual grounding by withholding textual shortcuts
Does masking improve image-dependent reasoning, or only alter the difficulty of next-token prediction?
Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation
Density-weighted residual alignment in hybrid VLMs
Does density weighting outperform uniform residual alignment when data, trainable parameters and architecture are held fixed?
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
Numerical objectives: preserve the loss gradient, not just the loss value
Does the actual dtype and computation graph preserve high-margin and teacher-matched gradients?
Distillation projectors: dimension matching versus optimization interference
Does the alignment head protect useful student features, or merely fit the teacher target more closely?
Understanding the Effects of Projectors in Knowledge Distillation
Distillation changes capabilities, not only average scores
Which capability benefits from teacher smoothing, and which needs the original hard target?
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Edge encoders: contrastive multi-teacher compression
Which performance survives after reducing capacity, at the same physical sampling and downstream readout?
Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
Route additional correction by teacher–student disagreement
Which tokens need extra teacher-weighted correction, and is their teacher target reliable?
Divergence-Routed On-Policy Distillation for Language Models
Audit teacher reliability before stronger on-policy imitation
Does the teacher provide a usable answer on the states the student actually visits?
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
Global semantic anchors and local patch targets
Which signal creates useful spatial semantics, which refines it, and do your global and dense evaluations agree?
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Response / sequence | Transfers teacher output distributions or generated targets. | Teacher errors, decoding bias, and temperature choices are inherited. | 2015 · 2016 |
| Feature / relation / capacity | Matches internal tensors, attention, distances, contrastive geometry, or capacity-adapted projections. | Forced layer correspondence or an over-capable teacher can constrain the student. | 2015 · 2017 · 2019 · 2020 · 2020 · 2021 · 2022 · 2022 |
| Online / self | Teacher targets evolve with peers, previous generations, or momentum weights. | Apparent transfer may be regularization or implicit ensembling rather than compression. | 2018 · 2018 · 2020 |
| Data access / rollout policy | Constructs surrogate inputs with a learned generator or teacher-derived priors, or asks the teacher about student-visited prefixes. | Synthetic inputs may cover the deployment distribution poorly or expose memorized structure; student rollouts add compute and require a competent initialization. | 2019 · 2019 · 2020 · 2024 |
| Continual retention | Uses an earlier model’s outputs to constrain adaptation on later-task inputs. | No old files does not mean no old information, and new-task inputs may poorly cover the old function. | 2016 |
| Generative trajectory | Compresses a diffusion or flow trajectory into fewer evaluations. | Fast sampling can hide repeated training rounds, teacher calls, or trajectory bias. | 2022 · 2026 |
| Multimodal / expert ensemble | Combines signals across modalities, encoders, resolutions, graphs, or domain experts. | Conflicting teachers, missing modalities, and cohort shift can produce a brittle compromise. | 2021 · 2023 · 2025 · 2024 · 2026 |
| Resolution / evidence budget | Transfers high-detail signals into low-only deployment, reconciles retained multiscale paths, or reduces bag processing after full feature extraction. | Same pixel size can cover different physical fields; multiscale accuracy may hide retained high-resolution cost; subset learning may still pay full extraction and instructor cost. | 2021 · 2024 · 2025 · 2026 · 2026 · 2026 |
| Privileged, safety, capacity, and on-policy transfer | Extends output matching to training-only side information, behavioral safety prompts, pruning recovery, conditional feature matching, verifier-gated reasoning, and reward-guided generative self-distillation. | A shared “distillation” label can hide incompatible access contracts; reward or verifier bias, extra rollouts, and dense feature matching can erase the claimed safety, generality, or efficiency gain. | 2016 · 2024 · 2026 · 2026 · 2026 · 2026 · 2026 |
| Residual targets, correction paths, and residual audits | Separates residual-feature assistants, additive logit cascades, shortcut removal, modality-exclusive transfer, residual projectors, continual boosting/compression, cross-tokenizer mass regularization, and residual diagnostics. | Shared terminology can conceal incompatible supervision and inference costs; an auxiliary branch may erase compression, and a residual regularizer need not be residual-error learning. | 2021 · 2021 · 2020 · 2019 · 2021 · 2024 · 2022 · 2025 · 2025 · 2026 · 2026 · 2026 · 2025 · 2026 |
| Resolution, magnification, relation, and stain transfer | Adds adversarial cross-resolution alignment, resolution-switchable ensemble teaching, pathology relation transfer, sequential magnification retention, and IHC-to-H&E privileged supervision. | Patch leakage, mismatched physical fields of view, retained multiscale branches, same-capacity teachers, or hidden registration can invalidate an efficiency claim. | 2025 · 2026 · 2026 · 2021 · 2020 |
| Physical observability and task-aware polarization reconstruction | GenPolar predicts Stokes components from intensity using observability-aware supervision and one-step diffusion distillation. PolarAPP instead starts from measured polarization mosaics and couples demosaicking with downstream objectives through a learned metric. Keep generative priors separate from sensor-acquired polarization. | Weak-polarization angle masks change the evaluated population. Task-aware meta-learning is not automatically teacher–student KD, and metric improvements need not agree. | 2026 |
| Physiology-aware cross-modal spectral distillation | RPM-Distill transfers peak position, off-peak suppression and spectral morphology from an RF teacher to an RGB student. A validation-driven policy chooses sample and loss weights; deployment removes the radar branch. Compare this privileged-sensor design with feature KD and reliability-weighted transfer. | A supervised teacher and held-out labels are still required; the pipeline is not globally label-free. Policy residual blocks are not a residual-error KD target, and fusion methods retain different inference inputs. | 2026 |
| Low-resolution student inference versus resolution-cascaded generation | GAP-Track removes teacher and residual reconstruction branches at inference. RMD instead retains high-resolution generative refinement. Both exploit cross-resolution supervision but solve different deployment problems. | Synthetic downsampling does not reproduce all optical changes between microscope magnifications; lower-resolution gains need not equal full-resolution performance. | 2026 · 2026 |
| Teacher-class mixtures and privileged residual routing | MoE-KD reformulates classification through latent teacher-class experts and EM. MARS uses full-minus-partial features to teach a deployable router. Specify whether knowledge resides in label mixtures, prototypes, routing decisions or a residual target. | Residual-guided routing is not a residual-error student cascade; conditional optimization arguments do not certify best accuracy. | 2026 · 2026 |
| Spatial-frequency and response-token distillation targets | SFKD selectively aligns wavelet-localized Fourier amplitudes across backbone families. TRAG aligns response-to-vision attention using token-specific entropy-weighted KL. Both teach structure beyond final logits, but use different correspondence assumptions. | SFKD has architecture-specific regressions; TRAG shares a visual encoder and resolution. Neither establishes cross-magnification performance. | 2026 · 2026 |
| Perceptual experts, geometry and explicit grounding tokens | Continuous expert-supervised visual tokens, geometry-aware fusion and coarse-to-fine grounding alter how a language model accesses visual evidence. Compare the representation interface, auxiliary supervision and deployed computation before comparing answer scores. | Expert distillation is not proof of faithful reasoning; stronger pointing or spatial QA need not improve every tracking metric. | 2026 |
| Causal audiovisual generation and variable-step distillation | OmniForcing transfers a bidirectional audio-video generator to causal streaming; AnyFlow learns on-policy flow maps supporting different sampling budgets. Preserve training-stage, self-rollout, synchronization, step-count and hardware qualifiers. | Real-time and any-step claims depend on resolution, duration, hardware and guidance; source-code revisions may change the training recipe. | 2026 · 2026 |
| Volume tokens, persistent geometry and registration | Compare learned volume representations, streaming geometric context, flow-based rigid registration and dense image matching. Each chooses a different correspondence unit and computation path; learned priors complement rather than abolish geometric constraints. | Compact per-frame context still accumulates; multi-step integration is not one-pass inference, and improved average matching can hide benchmark regressions. | 2026 |
| Historical adapter anchors and analytic continual classifiers | Historical LoRA anchors constrain encoder drift while text-guided accumulated statistics update a classifier without raw-example replay. This is a teacher-state and classifier-memory contract, not the residual-error cascade used by other KD methods. | Stable-feature equivalence is conditional; check the printed objective/closed-form mismatch and separate average from final accuracy. | 2026 |
| Risk-specific multi-teacher survival adapters | SurvMILKD combines frozen encoder features through trainable MIL adapters, teacher-space feature translation and soft ordinal risk distributions. It removes teacher branches at deployment while retaining one student feature pipeline. | Not every cohort benefits from more teachers; frozen encoders, external risk pooling and code-release status must remain explicit. | 2026 |
| Diffusion-boundary self-distillation and negative transfer | TRACE turns trajectory-derived attention boundaries into one-pass predictions through LoRA and an edge decoder. Read its natural-image gains alongside the reported MoNuSeg/TNBC degradation. | A tiny decoder does not remove the large diffusion backbone. Edge-extraction speed is not end-to-end speed, and negative cell-level transfer is not WSI validation. | 2026 |
| Description-mediated transfer for image quality | RALI transfers teacher quality descriptions into a CLIP encoder, then fits compressed score bases. Read its image-only deployment path separately from the paper’s multi-domain RACT procedure. | Comparable aggregate IQA performance is not universal replacement or WSI validation; account for offline description generation and the measured memory ratio. | 2026 |
| Cross-mode distillation into explicit 3D representations | FlashWorld combines a multi-view teacher, rendered-Gaussian student and learned fake-score network. Study distribution matching, consistency regularization and when OOD batches must omit adversarial targets. | A shared rendered scene does not certify correct geometry, fewer parameters or cross-magnification transfer; preserve hardware and benchmark protocol differences. | 2026 |
| Real-data correction in generative distillation | RealUID unifies matching-model distillation and routes real examples through an auxiliary matching model. Read the alternating optimization and coefficient assumptions before interpreting the one-step generator results. | No GAN discriminator does not mean no min–max problem; low-resolution generation does not establish pathology validity. | 2026 |
| Data-constrained ensembles and sequence distillation | Regularization and independent teachers extract more from a fixed real corpus; mixed real/synthetic sequence training transfers part of the ensemble benefit into one student. | Fitted infinite-compute limits are not measured runs, and a smaller deployed student is not automatically cheaper to train. | 2026 |
| Conflicting supervision and adaptive distillation objectives | DTO-KD adapts the balance between task and teacher-transfer objectives using relative-loss optimization. Compare the exact log-gradient construction with the practical amortized update; control the transfer architecture before attributing gains to optimization. | The printed closed form omits simplex boundary handling and the claimed non-collapse lower bound has a counterexample. Natural-image results do not establish cross-magnification or WSI validity. | 2026 |
| Channel distributions and feature-pyramid transfer | CWD normalizes spatial activations inside each channel; CSCWD adds teacher P2 to student P3 alignment for tiny-object detection. Study what the spatial probability distribution preserves before changing feature resolution. | CSCWD uses shared-resolution image inputs, not 20×/5× tissue pairs. Its recall gain coexists with MGD’s stronger mAP and a non-improving external strict mAP. | 2021 · 2026 |
| Which feature directions should distillation preserve? | SpecMatch balances reconstruction along teacher principal components rather than letting high-variance directions dominate an L2 objective. Low variance need not mean low downstream value. | Patch-level Camelyon17 is not a WSI endpoint. Spectral balancing is not evidence that unobserved high-magnification information can be recovered. | 2026 |
| Cross-magnification transfer: utility, predictability and task benefit | After XMaG and LRMIL, test whether observed teacher targets help, whether native low-power features predict them, and whether MIL benefits from the predictions. Direct versus residual regression and native-feature retention are separate controls. | Observed 20× access is a diagnostic reference, not low-power-only deployment. Lower reconstruction MSE is not guaranteed downstream improvement; the study is an ICLR 2027 submission-stage preprint. | 2026 |
| Distilling reusable MIL initializations | TITAN and CARE supervise nine lightweight MIL aggregators over regular ROI bags. Teacher-specific projections and dispersion-normalized alignment precede linear probing or supervised fine-tuning. | Six cohorts contain 15 tasks; more teachers do not always win. Main-text angular loss and appendix pseudocode differ, and aggregator pretraining does not itself reduce acquisition magnification. | 2026 |
| Tissue distillation: masks, features, hierarchy and relations | Compare staged pixel/feature KD from Virchow2, PAGET’s hierarchical multi-teacher pseudo-labels, and ConStruct’s transfer of SegFormer structural affinities into adapted CONCH features for prototype-guided segmentation. Their annotation contracts and deployed components differ. | PAGET-H retains HoverNet. ConStruct’s residual adapters are not residual-error targets. Pixel segmentation, patch throughput and patient-level WSI outcomes are distinct evidence. | 2026 · 2026 · 2026 |
| Pathway-structured privileged supervision | MoPE uses training-only RNA to supervise WSI pathway experts through shared-memory distributions. It aligns a structured view of molecular information without requiring RNA at inference. | Paired RNA is still needed for training. Model attention is not measured pathway activation; this is neither optical cross-magnification transfer nor prospective clinical validation. | 2026 |
| Distilling visual context and sensitivity to evidence | SCOPD teaches a sparse-context student from a full-context EMA teacher on student rollouts. CW-OPD instead tests how predictions change when answer-critical visual evidence is edited between two worlds. | Token pruning and counterfactual image editing are not WSI acquisition changes. Track EMA updates, teacher-only hints, editor quality and training cost before comparing distillation methods. | 2026 · 2026 |
| Token-level direction and teacher consensus | BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals. | Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost. | 2026 |
| Distill visual grounding by withholding textual shortcuts | Hide to See masks parts of the student’s reasoning prefix while the teacher sees the complete prefix. The asymmetric distillation setup asks the student to rely more on the image rather than copy answers from preceding text. | Acceptance is author-reported; no exact match was located in the inspected official program snapshot. Prefix masking changes training context, not image resolution, and image-dependence proxies are not clinical grounding tests. | 2026 |
| Density-weighted residual alignment in hybrid VLMs | HEED aligns teacher and hybrid-student decoder residual streams, weighting visual positions by frozen-ViT local self-dissimilarity. This density proxy is not semantic saliency. Distinguish the hybrid architecture’s inference benefit from density weighting’s incremental benefit in a matched training recipe. | The C3→C4 OCRBench v2 gain is 4.7 points; the larger headline change includes other training changes. The manuscript disagrees about Stage 3 trainable parameters. Neither the speedup nor this feature residual establishes WSI cross-magnification transfer. | 2026 |
| Numerical objectives: preserve the loss gradient, not just the loss value | A Fix for Softmax and Cross-Entropy separates hard-target absorption from near-equal teacher/student cancellation. Excluding the target from log-sum-exp and using softplus protects hard-label gradients; a logit-difference construction with log1p/expm1 repairs soft-target derivatives. The intended objective is unchanged. | Evidence is numerical and small modular-arithmetic training, not WSI or large-scale foundation-model improvement. Higher precision and zero-sum projection are not automatically equivalent fixes. Check the real compiler and temperature convention before changing a training protocol. | 2026 |
| Distillation projectors: dimension matching versus optimization interference | A projector can separate the student’s task representation from the space used to imitate a teacher even when both feature dimensions match. PEFD averages parallel nonlinear projector outputs before direction matching; the extended study adds logit, representation-similarity and calibration analyses. | PEFD originated in 2022. The accessible 2023 manuscript supports the reviewed scientific details; final 2026 journal numerical identity was not verified. More projectors or greater depth do not guarantee improvement. | 2026 |
| Distillation changes capabilities, not only average scores | Distilled Pretraining separates access to teacher data from the effect of soft targets. It contrasts context copying with multi-sample solution coverage and tests entropy-based token routing. A student can improve on pass@k yet become less responsive to supplied context. | Matched token counts do not match total teacher–student compute. Context-copying experiments and pass@k are separate endpoints; routing results are not a universal recipe or demonstrated pathology transfer. | 2026 |
| Edge encoders: contrastive multi-teacher compression | MuCoDi caches fixed pathology-teacher vectors and trains compact students to match tile identity contrastively. MuCoEdge is evaluated with frozen-encoder MIL and single-tile CPU timing. Teacher and student both use 5×; capacity reduction and magnification reduction must remain separate hypotheses. | The 23 tasks are teacher-eligible; encoder timing excludes WSI IO and aggregation. Close average AUROC is not formal non-inferiority or cross-magnification validation. | 2026 |
| Route additional correction by teacher–student disagreement | DrOPD preserves reverse KL at every student-generated position and adds forward KL at the batch’s highest-JS positions. The candidate union retains tokens preferred by either model. Read it after on-policy KD to distinguish uncertainty, disagreement and teacher-supported modes the student under-covers. | Math-only preprint, two related Qwen pairs. Top-k approximate losses, eight-response mean accuracy and logged training-step cost are separate evidence contracts; no demonstrated WSI transfer. | 2026 |
| Audit teacher reliability before stronger on-policy imitation | The VLM teacher-inheritance study compares filtered SFT with full-vocabulary OPD on ChartQA and Geometry3K. It separates answer accuracy from completion, decomposes paired failures and shows OPD can erase completion already learned through SFT. This is an empirical warning, not a new KD objective. | One model family and two main tasks; updates are fixed within channels, not compute-matched between them. Teacher probes are not a unique causal attribution. Current author code omits the evaluation records advertised in the manuscript. | 2026 |
| Global semantic anchors and local patch targets | Two Global Crops Suffice separates same-image global-view alignment from local crops and iBOT refinement in controlled DINO-family experiments. CODI instead changes the patch target: weakly mix the same teacher view’s class distribution into its local patch distribution. The diagnostic study and target-side method answer related questions, but neither independently validates the other. | Both inspected sources are preprints. Crop ablations match iterations, not all computation; the depth-scale narrative has an appendix exception. CODI’s gradient statement is local/logit-level, its alpha is task-dependent, and its code was not released at inspection. Neither reports WSI or cross-magnification validation. | 2026 |
Concept lineage
Model CompressionDo Deep Nets Really Need to Be Deep?
Large pseudo-labelled transfer sets become explicit function matching from deep ensembles into shallow students.
Do Deep Nets Really Need to Be Deep?Distilling the Knowledge in a Neural Network
Raw function regression is formalized as temperature-scaled soft-target training.
Distilling the Knowledge in a Neural NetworkFitNets: Hints for Thin Deep Nets
The transfer interface moves from final softened outputs to intermediate feature hints.
Distilling the Knowledge in a Neural NetworkLearning Without Forgetting
Teacher-output matching is repurposed from model compression to preserving an old task during adaptation.
Distilling the Knowledge in a Neural NetworkZero-Shot Knowledge Distillation in Deep Networks
Output matching is retained, but the original transfer set is replaced by teacher-inverted Data Impressions.
FitNets: Hints for Thin Deep NetsRelational Knowledge Distillation
Pointwise feature regression becomes preservation of distances and angles among examples.
Sequence-Level Knowledge DistillationDistilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Teacher-generated outputs expand from task sequences to explanatory rationales plus labels.
Sequence-Level Knowledge DistillationOn-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Training moves from fixed teacher-generated sequences to teacher feedback on prefixes sampled by the evolving student.
Bootstrap Your Own LatentEmerging Properties in Self-Supervised Vision Transformers
Asymmetric self-teachers move from global image embeddings to emergent ViT token structure.
Align Before Fuse: Vision and Language Representation Learning with Momentum DistillationA generalizable pathology foundation model using a unified knowledge distillation pretraining framework
Momentum cross-modal targets foreshadow multi-objective expert transfer in pathology foundation models.
Distilling the Knowledge in a Neural NetworkImproved Knowledge Distillation via Teacher Assistant
The classic response target is retained while an intermediate-capacity teacher repairs the teacher–student gap.
Zero-Shot Knowledge Distillation in Deep NetworksDreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Data-free transfer advances from class-conditioned impressions to high-fidelity images regularized by network statistics.
DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and LighterMiniPLM: Knowledge Distillation for Pre-Training Language Models
Pretraining transfer moves from matching teacher states and objectives online to offline teacher-guided refinement of the corpus distribution across model families.
Resolution-Based Distillation for Efficient Histology Image ClassificationStreamline Pathology Foundation Model by Cross-Magnification Distillation
Patch-classifier cross-resolution transfer scales into a compact pathology foundation encoder with global and spatial-token alignment.
Resolution-Based Distillation for Efficient Histology Image ClassificationA Magnification Alignment Framework Enables Computation- and Communication-Efficient Computational Pathology
The low-resolution student idea becomes registered same-field magnification alignment for whole-slide global–local modeling and communication cost.
Streamline Pathology Foundation Model by Cross-Magnification DistillationLRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification
Both deploy low magnification only, but LRMIL adds slide-level logit, soft-attention, and hard-instance transfer after encoder distillation.
A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI ClassificationLRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification
Graph multi-scale self-distillation retains all resolution branches at inference; LRMIL removes the 20× branch, so their efficiency contracts are not directly interchangeable.
Sequence-Level Knowledge DistillationMiniLLM: Knowledge Distillation of Large Language Models
Sequence targets evolve into reverse-KL language-model distillation designed around student generation.
Progressive Distillation for Fast Sampling of Diffusion ModelsLIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models
Sampler compression extends from iterative diffusion step halving to trajectory-aware flow distillation.
DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image ClassificationSlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification
Hierarchical MIL transfer is extended with slide-level graph contrast and distillation.
Failure modes to audit
- Comparing a distilled student with an under-trained non-distilled baseline.
- Calling every exponential-moving-average target “knowledge distillation” without naming the transferred tensor.
- Calling teacher inversion “privacy preserving” merely because the original files are absent.
- Evaluating an autoregressive student only on fixed prefixes after training it on fixed prefixes.
- Reporting parameter reduction without latency, memory, energy, and calibration.
- Calling a model “low-resolution deployment” while its inference path still extracts or encodes selected high-resolution tiles.
- Comparing magnifications without matching physical field of view, registration, interpolation, or patch count.
- Letting patient-, slide-, or site-correlated samples cross a pathology teacher–student evaluation split.
Questions for every paper
- What exact object is transferred?
- Where do the inputs or prefixes used for transfer come from?
- Why is the teacher privileged?
- Which gradients reach which components?
- At each magnification, what physical field of view and patch count does the model observe?
- Which teacher or high-resolution paths disappear at deployment?
- Does the student retain performance under the deployment shift and full systems budget?
Complete maintained paper route
Section titled “Complete maintained paper route”140 canonical readings · 2006–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered by an explicit prerequisite-aware teaching sequence. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 140 readings in maintained order
Model Compression
Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil
Peer-reviewed KDD 2006 proceedings paper, pages 535–541.
Metadata checked
Do Deep Nets Really Need to Be Deep?
Lei Jimmy Ba and Rich Caruana
Peer-reviewed NeurIPS 2014 proceedings paper.
Official artifacts: Author manuscript
Metadata checked
Distilling the Knowledge in a Neural Network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean
Metadata checked
A Fix for Softmax and Cross-Entropy
Jonathan T. Barron, Dan Gnanapragasam, and Michael Tschannen
Inspected arXiv:2610.31642v1, submitted 22 October 2026, and the author project page. Numerical-gradient and modular-arithmetic experiments are not evidence of improved WSI or large-scale foundation-model accuracy. No venue acceptance was verified.
Official artifacts: Inspected manuscript (v1) · Author project and JAX implementations
Metadata checked
Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessment
Shijie Zhao, Xuanyu Zhang, Weiqi Li, Junlin Li, Li Zhang, Tianfan Xue, Jian Zhang
Official ICLR 2026 oral and full proceedings manuscript checked. Teacher-generated quality descriptions align a compact image encoder; deployed scoring uses PCA and fitted bases without an LLM. Keep RALI separate from multi-domain RACT. Dataset-specific trade-offs, causal-interpretation limits and the conflicting memory-reduction claim are qualified. No WSI or cross-magnification evaluation.
Official artifacts: Official proceedings record · OpenReview · Official implementation
Metadata checked
MoE-KD: Your Teacher Model is Worth Mixture-of-Experts for Knowledge Distillation
Kuo Shi, Wenjie Zhu, Bo Peng
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The EM convergence discussion assumes an improving M-step; it is not a guarantee of global optimum for stochastic neural training. Table 3 values take precedence over inconsistent prose deltas. No runnable official implementation was verified in this pass.
Official artifacts: Official ECCV poster record
Metadata checked
Unifying Distillation and Privileged Information
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik
ICLR 2016 conference paper; canonical locator is the complete arXiv manuscript.
Metadata checked
Residual-Guided Expert Specialization for Incomplete Multimodal Learning
Seunghun Baek, Jihwan Park, Jaeyoon Sim, Minjae Jeong, Hoseok Lee, Won Hwa Kim
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. MARS distills a privileged residual router into an available-feature router; it is not a cascade of residual-error student networks. Full modalities are assumed available during training. Food-101 full-modality accuracy is below SimMLM despite higher missing-modality and average scores.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
FitNets: Hints for Thin Deep Nets
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio
Official artifacts: Official code
Metadata checked
Understanding the Effects of Projectors in Knowledge Distillation
Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, Brano Kusy, and Zi Huang
IEEE publisher metadata identifies TPAMI 48(8), 9078–9092, August 2026. The accessible scientific text checked here is arXiv:2310.17183v1 (26 October 2023); final journal numerical identity was not established. This extends the authors’ NeurIPS 2022 Projector Ensemble Feature Distillation work, rather than originating PEFD in 2026.
Official artifacts: Inspected author manuscript (v1) · Author code and earlier PEFD implementation
Metadata checked
Channel-Wise Knowledge Distillation for Dense Prediction
Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen
ICCV 2021 proceedings paper and author v4 manuscript checked; older missing prerequisite added during the October 2026 audit, not a new 2026 paper. Official repository records the semantic-segmentation code release on 20 August 2021.
Official artifacts: Official implementation · Author manuscript
Metadata checked
Computationally Efficient Pathology Segmentation using Knowledge Distillation from Foundation Models
Jiaqi Lv, Yijie Zhu, Saki Okada, Maxwell Stanley Renna, Robert Goldin, Stefan Antonowicz, and Shan E Ahmed Raza
arXiv v1 (2026-09-03), submitted to Medical Image Analysis according to the authors, not accepted. Verified methods and selected results on 2026-10-02. TIAToolbox is the stated release destination; availability of these exact weights was not independently confirmed. Patch-throughput measurements are not end-to-end WSI latency.
Official artifacts: Versioned manuscript · TIAToolbox — stated release destination
Metadata checked
RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurement
Jiyao Wang, Qingyong Hu, Duoxun Tang, Xiao Yang, Kaishun Wu, and Jiangbo Yu
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #10941. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript · Official implementation
Metadata checked
Sequence-Level Knowledge Distillation
Yoon Kim and Alexander M. Rush
Official artifacts: Official seq2seq implementation · Official mobile demonstration
Metadata checked
Pre-training under infinite compute
Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto
Official ICLR 2026 oral and full proceedings manuscript checked. Fixed unique data, regularized teachers and real/synthetic sequence distillation. Infinite-compute claims are fitted asymptotes, not completed runs. Distinguish main mixing recipes from the matched-token ablation and inference parameter savings from total training cost.
Official artifacts: Official proceedings record · OpenReview
Metadata checked
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Sachin Goyal, David Lopez-Paz, and Kartik Ahuja
Official artifacts: Proceedings PDF · arXiv
Metadata checked
Learning Without Forgetting
Zhizhong Li and Derek Hoiem
Peer-reviewed ECCV 2016 proceedings paper; the canonical locator is the open author manuscript.
Official artifacts: Official project · Official code
Metadata checked
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Sergey Zagoruyko and Nikos Komodakis
Official artifacts: Official code
Metadata checked
Deep Mutual Learning
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu
Official artifacts: Official code
Metadata checked
Born Again Neural Networks
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar
Metadata checked
Relational Knowledge Distillation
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho
Official artifacts: Official code
Metadata checked
Zero-Shot Knowledge Distillation in Deep Networks
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R. Venkatesh Babu, and Anirban Chakraborty
Metadata checked
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation
Hanyang Kong, Jian Zhao, Xiaoguang Tu, Junliang Xing, Shengmei Shen, and Jiashi Feng
arXiv technical report submitted 26 May 2019; no peer-reviewed version or released implementation was located by 2026-09-02.
Metadata checked
Data-Free Learning of Student Networks
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian
Peer-reviewed ICCV 2019 proceedings paper.
Official artifacts: Author manuscript
Metadata checked
DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf
Workshop paper and arXiv preprint; not a main-track NeurIPS proceedings paper.
Official artifacts: Official distillation code
Metadata checked
Improved Knowledge Distillation via Teacher Assistant
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh
Official artifacts: Official code
Metadata checked
GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT Tracking
Shiyu Zhang, Tianyang Xu, Zhangyong Tang, He Wang, Xiao-Jun Wu, and Josef Kittler
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Table 1 governs the recreated LasHeR values; the quarter-resolution CKD SR differs between Table 1 (35.3) and Table 4 (35.5). The PGS angle/radial equations need origin and quadrant safeguards not specified in the displayed formulas. The repository contains implementation directories despite a remaining release-soon notice; code was inspected, not executed.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts
Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin, Wei Zhang, Jiashi Feng, and Tong Zhang
Published in the NeurIPS 2020 main conference proceedings.
Official artifacts: Official implementation
Metadata checked
Resolution Switchable Networks for Runtime Efficient Image Recognition
Yikai Wang, Fuchun Sun, Duo Li, and Anbang Yao
Peer-reviewed ECCV 2020 proceedings paper. One weight-shared CNN supports five trained input resolutions with private batch-normalization parameters; inference evaluates one chosen resolution and does not run the training-time ensemble.
Official artifacts: Official code and released models · Authors' arXiv manuscript
Metadata checked
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz
Official artifacts: Official code
Metadata checked
Bootstrap Your Own Latent
Jean-Bastien Grill et al.
Metadata checked
Contrastive Representation Distillation
Yonglong Tian, Dilip Krishnan, and Phillip Isola
Official artifacts: Official code
Metadata checked
TinyBERT: Distilling BERT for Natural Language Understanding
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu
Official artifacts: Official implementation
Metadata checked
MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering
Chenyu You, Nuo Chen, and Yuexian Zou
Published in the IJCAI 2021 main track, pages 3985–3991, DOI 10.24963/ijcai.2021/549.
Metadata checked
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou
Official artifacts: Official implementation and checkpoints
Metadata checked
Emerging Properties in Self-Supervised Vision Transformers
Mathilde Caron et al.
Metadata checked
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, and Qingming Huang
arXiv:2610.30639v1, submitted 22 October 2026, inspected 26 October 2026. The manuscript has a NeurIPS 2026 footer, but no official acceptance record was independently located in this audit; retained as a preprint. The author repository contains only a Coming soon README, not released implementation code.
Official artifacts: Inspected manuscript (v1) · Author repository — implementation pending at inspection
Metadata checked
Training Data-Efficient Image Transformers & Distillation through Attention
Hugo Touvron et al.
Metadata checked
Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation
Junnan Li et al.
Metadata checked
Distilling Knowledge via Knowledge Review
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia
Official artifacts: Official code
Metadata checked
SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction
Cuipeng Wang, Haipeng Wang
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The linked official repository still says source code will be released soon. This is heterogeneous-architecture feature KD, not demonstrated cross-magnification transfer. The selected CIFAR-100 table includes a ViT-S→MobileNetV2 case where FBT is better.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
Residual error based knowledge distillation
Mengya Gao, Yujun Wang, and Liang Wan
Published in Neurocomputing 433 (14 April 2021), pages 154–161, DOI 10.1016/j.neucom.2020.10.113. The differently authored arXiv:2002.09168 is recorded only as a related precursor, not as this journal version.
Metadata checked
ResKD: Residual-Guided Knowledge Distillation
Xuewei Li, Songyuan Li, Bourahla Omar, Fei Wu, and Xi Li
Published in IEEE Transactions on Image Processing 30 (2021), pages 4735–4746.
Official artifacts: Author manuscript
Metadata checked
Resolution-Aware Knowledge Distillation for Efficient Inference
Zhanxiang Feng, Jianhuang Lai, and Xiaohua Xie
Peer-reviewed journal article, volume 30, pages 6985–6996. The paper calls its framework RKD, meaning resolution-aware—not relational—knowledge distillation. Teacher and student use the same architecture but paired high- and low-resolution inputs; only the low-resolution student is needed at inference.
Official artifacts: PubMed record
Metadata checked
Multi-Scale Aligned Distillation for Low-Resolution Detection
Lu Qi, Jason Kuen, Jiuxiang Gu, Zhe Lin, Yi Wang, Yukang Chen, Yanwei Li, and Jiaya Jia
Peer-reviewed CVPR 2021 proceedings paper.
Official artifacts: Official code
Metadata checked
Resolution-Based Distillation for Efficient Histology Image Classification
Joseph DiPalma, Arief A. Suriawinata, Laura J. Tafe, Lorenzo Torresani, and Saeed Hassanpour
Peer-reviewed journal article 119:102136.
Official artifacts: Open-access article · Author manuscript
Metadata checked
Progressive Distillation for Fast Sampling of Diffusion Models
Tim Salimans and Jonathan Ho
Official artifacts: Official code
Metadata checked
Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
Nikita Kornilov, David Li, Tikhon Mavrin, Aleksei Leonov, Nikita Gushchin, Evgeny Burnaev, Iaroslav Koshelev, Aleksandr Korotin
Official ICLR 2026 oral and full proceedings manuscript checked. Three-network training yields a one-step generator. No GAN discriminator does not mean no min–max optimization. Selected low-resolution image results include additional fine-tuning; teacher-error correction has restrictive assumptions. No cross-magnification or pathology evaluation.
Official artifacts: Official proceedings record · OpenReview
Metadata checked
FlashWorld: High-quality 3D Scene Generation within Seconds
Xinyang Li, Tengfei Wang, Zixiao Gu, Shengchuan Zhang, Chunchao Guo, Liujuan Cao
Official ICLR 2026 oral and full proceedings manuscript checked. A frozen multi-view teacher guides a few-step rendered-Gaussian student using learned fake scores, consistency and OOD co-training. Shared-scene rendering is not guaranteed correct geometry. Preserve different benchmark timing protocols, proprietary-data and training-code access limitations. No WSI or cross-magnification evidence.
Official artifacts: Official proceedings record · OpenReview · Official implementation
Metadata checked
Cross-Resolution Distribution Matching for Diffusion Distillation
Feiyang Chen, Hongpeng Pan, Haonan Xu, Xinyu Duan, Zhefeng Wang, and Yang Yang
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. RMD retains high-resolution refinement at inference; it is not low-resolution-only recognition. Reported speedups depend on the single-A100 baseline, CFG and NFE. The conference PDF author order differs from the arXiv record and is preserved here.
Official artifacts: Official ECCV poster record
Metadata checked
Stokes-Informed Diffusion for Robust Linear Polarization Estimation
Yidong Luo, Chenggong Li, Yuchao Feng, Boxin Shi, Junchao Zhang, and Xin Yuan
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5749. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript
Metadata checked
DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification
Hongrun Zhang et al.
Metadata checked
Knowledge Distillation: A Good Teacher Is Patient and Consistent
Lucas Beyer, Xiaohua Zhai, Amelie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov
Official artifacts: Teacher model code and weights
Metadata checked
FOSTER: Feature Boosting and Compression for Class-Incremental Learning
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan
Published in the ECCV 2022 main conference proceedings, pages 398–414.
Official artifacts: Official implementation
Metadata checked
Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual Learning
Qier Meng, De Cheng, Jiahao Li, and Cheng Deng
ECCV 2026 Spotlight verified via the official related presentation event; the poster identity itself says Accept Poster. No Springer version of record or public runnable implementation was verified. Section 3.5's printed closed-form equation omits a term implied by its objective; the review flags this reproducibility issue without silently repairing the authors' formula.
Official artifacts: Official ECCV 2026 Spotlight
Metadata checked
Decoupled Knowledge Distillation
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang
Official artifacts: Official code
Metadata checked
DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
Zeeshan Hayder, Ali Cheraghian, Lars Petersson, Mehrtash Harandi, Richard Hartley
Official ICLR 2026 oral and proceedings manuscript checked. Adaptive task/distillation optimization, not cross-magnification or residual-error KD. Separate the exact two-gradient construction from the amortized implementation; the printed simplex handling and lower-bound argument require qualification, documented in the source-checked review.
Official artifacts: Official proceedings record · OpenReview · Theoretical supplement
Metadata checked
Masked Generative Distillation
Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan
Official artifacts: Official code
Metadata checked
Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification
Wenhao Tang et al.
Metadata checked
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister
Official artifacts: Official code
Metadata checked
TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance
Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Chen, Xinggang Wang, Hongyang Chao, and Han Hu
Official artifacts: Official code and checkpoints
Metadata checked
Attention to Detail: Inter-Resolution Knowledge Distillation
Rocío del Amor, Julio Silva-Rodríguez, Adrián Colomer, and Valery Naranjo
Peer-reviewed EUSIPCO 2023 proceedings paper, pages 985–989.
Official artifacts: Open proceedings paper · Official code
Metadata checked
Large-to-Small Image Resolution Asymmetry in Deep Metric Learning
Pavel Suma and Giorgos Tolias
Peer-reviewed WACV 2023 proceedings paper.
Official artifacts: Official code
Metadata checked
Knowledge Distillation in Histology Landscape by Multi-Layer Features Supervision
Sajid Javed, Arif Mahmood, Talha Qaiser, and Naoufel Werghi
Peer-reviewed journal article 27(4):2037–2046.
Official artifacts: Open author manuscript
Metadata checked
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem
Metadata checked
Divergence-Routed On-Policy Distillation for Language Models
Tiantian Peng, Guannan Lv, Hongyang Li, Qiuhe Hong, Hang Cheng, Hongjian Dou, Xiangyu Wu, and Yuyang Liu
Review pins arXiv v1 (14 October 2026). The manuscript says under review at ICLR 2027; this is not acceptance.
Official artifacts: Version 1 paper and appendices
Metadata checked
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
Frank F. Yang, Kefei Yan, Yiren Wang, and Yiyang Wang
Version 1 empirical study. The current author README describes an anonymized code-only supplement and explicitly excludes evaluation records, logs, adapters, tests and figure scripts despite the manuscript release statement.
Official artifacts: Version 1 paper and appendices · Author code and current availability scope
Metadata checked
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, and Babak Taati
Preprint checked 2026-10-02; project page still labels code and models as forthcoming. Visual-token retention is not an optical-magnification intervention.
Official artifacts: Versioned full text · Author project page
Metadata checked
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, and Xinbing Wang
Preprint checked 2026-10-02. The paper calls the method CW-OPD; its linked code repository uses the name CWAD. No conference acceptance inferred.
Official artifacts: Versioned full text · Author implementation (repository name CWAD)
Metadata checked
Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation
Seonghoon Yu, Dongjun Nam, Byung-Kwan Lee, and Jeany Son
NeurIPS 2026 acceptance is author-reported by NVIDIA Research and the official implementation, checked 8 October 2026. No exact title or author match was found in the dated official oral/poster snapshot; presentation tier and final proceedings are unverified. Review uses arXiv v4. Training-time textual-prefix masking is not WSI magnification transfer.
Official artifacts: Versioned full text · Official implementation · Author project · Author acceptance record
Metadata checked
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
Yihao Liang and Niraj K. Jha
Official NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested.
Official artifacts: Versioned full text · Official acceptance identity · Official NeurIPS program
Metadata checked
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, and Gao Huang
Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation.
Official artifacts: Versioned full text · Author implementation
Metadata checked
MiniLLM: Knowledge Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang
Official artifacts: Official code
Metadata checked
DistiLLM: Towards Streamlined Distillation for Large Language Models
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun
Peer-reviewed ICML 2024 proceedings paper.
Official artifacts: Official code
Metadata checked
Compact Language Models via Pruning and Knowledge Distillation
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov
Published in the NeurIPS 2024 main conference proceedings.
Official artifacts: Official implementation · Official model collection
Metadata checked
Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Jiaming Lv, Haoyuan Yang, and Peihua Li
Official artifacts: Official project page · Official code
Metadata checked
SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification
Tong Shu, Jun Shi, Dongdong Sun, Zhiguo Jiang, and Yushan Zheng
Official artifacts: Official code
Metadata checked
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
Pavan Kumar Anasosalu Vasu et al.
Metadata checked
FROSTER: Frozen CLIP Is a Strong Teacher for Open-Vocabulary Action Recognition
Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han
Published at ICLR 2024.
Official artifacts: Official project · Official implementation
Metadata checked
A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI Classification
Gianpaolo Bontempo, Federico Bolelli, Angelo Porrello, Simone Calderara, and Elisa Ficarra
Peer-reviewed journal article 43(4):1412–1421; journal extension of DAS-MIL.
Official artifacts: Official code
Metadata checked
RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models
Greg Heinrich et al.
Metadata checked
Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding
Kadir Yilmaz, Adrian Kruse, Tristan Höfer, Daan de Geus, and Bastian Leibe
Accepted to ECCV 2026; oral presentation verified in the official program. This record and review use the conference-hosted manuscript checked 2026-09-12; a separate Springer proceedings record was not verified.
Official artifacts: Official ECCV oral presentation · Official ECCV paper record · arXiv record · Official code · Official project
Metadata checked
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba
Official artifacts: Official code
Metadata checked
MiniPLM: Knowledge Distillation for Pre-Training Language Models
Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang
Peer-reviewed ICLR 2025 conference paper.
Official artifacts: Official code, data, and models
Metadata checked
LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation
Fangxun Shu, Yue Liao, Lei Zhang, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Zhelun Yu, Si Liu, Hongsheng Li, and Hao Jiang
Peer-reviewed ICLR 2025 conference paper.
Official artifacts: Official code
Metadata checked
Learning Task-Agnostic Representations through Multi-Teacher Distillation
Philippe Formont, Maxime Darrin, Banafsheh Karimian, Eric Granger, Jackie CK Cheung, Ismail Ben Ayed, Mohammadhadi Shateri, and Pablo Piantanida
Official artifacts: Official text-model code · Official vision-model code · Official molecular-model code
Metadata checked
Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Sungmin Cha and Kyunghyun Cho
Official artifacts: Official code
Metadata checked
Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathology
Qibin Zhang, Xinyu Hao, Qiao Chen, Rui Xu, Fengyu Cong, Cheng Lu, and Hongming Xu
Official artifacts: Official code
Metadata checked
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Sanghwan Kim et al.
Metadata checked
A generalizable pathology foundation model using a unified knowledge distillation pretraining framework
Ma et al.
Metadata checked
Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
Tim Lenz, Maurice Heide, Marco Gustav, Nic G. Reitsam, and Jakob Nikolas Kather
Official MICCAI satellite proceedings confirm the 2nd Efficient Medical AI workshop (EMA), not the main MICCAI track. LNCS volume 17265 is listed; DOI and pagination remain pending on the checked record. Full 11-page accepted manuscript inspected. Teachers and students use 2.0 µm/px (5×); this is capacity compression, not a demonstrated 20×→5× transfer method. Raspberry Pi timings concern individual encoder forward passes, not complete WSI processing.
Official artifacts: Official workshop accepted manuscript · Author implementation and documented scope
Metadata checked
Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning
Jiuyang Dong et al.
Official artifacts: Code
Metadata checked
Distillation Scaling Laws
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb
Metadata checked
Distilling Foundation Models for Robust and Efficient Models in Digital Pathology
Alexandre Filiot, Nicolas Dop, Oussama Tchita, Auriane Riou, Rémy Dubois, Thomas Peeters, Daria Valter, Marin Scalbert, Charlie Saillard, Geneviève Robin, and Antoine Olivier
Peer-reviewed MICCAI 2025 proceedings paper.
Official artifacts: Official model · Official robustness benchmark
Metadata checked
CEMIL: Contextual Attention Based Efficient Weakly Supervised Approach for Histopathology Image Classification
Tawsifur Rahman, Alexander S. Baras, and Rama Chellappa
Peer-reviewed WACV 2025 proceedings paper, pages 4248–4257.
Metadata checked
Unsupervised Anomaly Detection on Preclinical Liver H&E Whole Slide Images Using Graph-Based Feature Distillation
Lin Li, Lillie Shelton, Thomas Forest, Kyathanahalli Janardhan, Tiffany Jenkins, Michael J. Napolitano, Roujia Wang, David Leigh, Tosha Shah, Grady Earl Carlson, Rajath Soans, and Antong Chen
Peer-reviewed MICCAI 2025 proceedings paper; evaluated on preclinical rat-liver H&E WSIs. The author project page requests email contact for code access and showed code as coming soon when checked on 2026-08-23.
Official artifacts: Author project and code-access page
Metadata checked
PatHME: Hierarchical Multi-Expert Knowledge Distillation for Whole Slide Image Analysis
Jing Wei Tan, Gangsu Kim, and Won-Ki Jeong
Peer-reviewed journal article 13:163246–163257.
Official artifacts: Official code
Metadata checked
Human Visual Attention-Inspired Knowledge Distillation Underlying Interpretable Computational Pathology
Muzhou Yu, Zihan Zhong, Xingang Zhou, Yuekun Wang, Tingyu Liang, Jiamin Chen, Hongmin Huang, Junhan Zhou, Dachun Zhao, Bo Lei, Yu Wang, Wenbin Ma, and Kaisheng Ma
Peer-reviewed open-access journal article 15:42124. Low- and high-magnification language motivates the design, but HVisKD transfers sample- and region-relation features between teacher and student given the same pathology patches; it is not cross-magnification distillation.
Official artifacts: Open-access article · Official code
Metadata checked
MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection
Hou-I Liu, Christine Wu, Jen-Hao Cheng, Wenhao Chai, Shian-Yun Wang, Gaowen Liu, Hugo Latapie, Jhih-Ciang Wu, Jenq-Neng Hwang, Hong-Han Shuai, and Wen-Huang Cheng
Published in the CVPR 2025 main conference proceedings, pages 22266–22275.
Official artifacts: Official implementation and models
Metadata checked
CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillation
Xiao Lin, Yun Peng, Liuyi Wang, Xianyou Zhong, Minghao Zhu, Yi Feng, Jingwei Yang, Chengju Liu, and Qijun Chen
Published in the ICCV 2025 main conference proceedings, pages 5990–6000.
Official artifacts: Official implementation
Metadata checked
On Explaining Knowledge Distillation: Measuring and Visualising the Knowledge Transfer Process
Gereziher Adhane, Mohammad Mahdi Dehshibi, Dennis Vetter, David Masip, and Gemma Roig
Published in the WACV 2025 proceedings, pages 3467–3476.
Official artifacts: Official implementation
Metadata checked
Streamline Pathology Foundation Model by Cross-Magnification Distillation
Ziyu Su, Abdul Rehman Akbar, Usama Sajjad, Anil V. Parwani, and Muhammad Khalid Khan Niazi
Journal record verified through publisher metadata, Crossref and PubMed PMID 42874158 on 26 October 2026: electronic publication 10 October 2026, assigned to volume 115 (January 2027), article 104352. The stable ID retains the 2025 preprint year. Journal full text was unavailable; detailed scientific claims and visuals remain explicitly tied to the previously inspected arXiv v3, not asserted to match the journal version. The journal abstract describes seven tasks and encoder-level throughput, not an end-to-end speedup.
Official artifacts: Inspected author manuscript (v3) · PubMed journal metadata and abstract · Official code · Official model
Metadata checked
Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide Images
Mahmut S. Gokmen, Mitchell A. Klusty, Peter T. Nelson, Allison M. Neltner, Sen-Ching Samson Cheung, Thomas M. Pearce, David A. Gutman, Brittany N. Dugger, Devavrat S. Bisht, Margaret E. Flanagan, and V. K. Cody Bumgardner
Preprint; arXiv comments report submission to the AMIA 2026 Informatics Summit, not acceptance.
Official artifacts: Official model
Metadata checked
DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning
Daya Guo et al.
Metadata checked
ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts
Jiawen Li, Tian Guan, Huijuan Shi, Xitong Ling, Mingxi Fu, Anjia Han, Chao He, and Yonghong He
Official artifacts: Code and models
Metadata checked
GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis
Naoto Usuyama et al.
Version 2 dated 2026-07-22; author-reported performance remains provisional.
Metadata checked
HistoMILKD: A Multiple Instance Learning based Multi-Teacher Knowledge Distillation Framework for Whole Slide Image Classification
Mayur Mallya, Ali Khajegili Mirabadi, Hossein Farahani, and Ali Bashashati
Official artifacts: Code
Metadata checked
Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation With Nested Embeddings
Christian Grashei, Christian Brechenmacher, Rao Muhammad Umer, Jingsong Liu, Carsten Marr, Peter J. Schüffler, and Ewa Szczurek
ECCV 2026 poster status reverified against the official complete program (poster 4454). No oral or spotlight designation is recorded. A Springer version of record was not verified on 2026-09-12; the existing review retains its declared manuscript revision.
Official artifacts: Official ECCV record · Official model · Conference manuscript
Metadata checked
SurvMILKD: A Weakly Supervised Survival Analysis Framework for Multi-Teacher Knowledge Distillation using Pathology Foundation Models
Mayur Mallya, Ali Khajegili Mirabadi, Hossein Farahani, and Ali Bashashati
The official ECCV 2026 poster and conference-hosted manuscript verify acceptance; a Springer version of record was not verified. The official repository was a code-coming-soon placeholder on 2026-09-12. All foundation encoders remain frozen: task-specific MIL adapters and translation modules learn, not the UNI backbone.
Official artifacts: Official ECCV 2026 Poster · Official repository — code pending
Metadata checked
A Magnification Alignment Framework Enables Computation- and Communication-Efficient Computational Pathology
Chu Han, Bingchao Zhao, Tianpeng Deng, Jingqi Huang, Fangfang Liu, Jiatai Lin, Shanshan Lyu, Longfei Wang, Cheng Lu, Changhong Liang, Hannah Y. Wen, Zhenwei Shi, Xiaojing Guo, and Zaiyi Liu
Peer-reviewed journal article 86(12):3060–3073.
Official artifacts: Author manuscript · Official code
Metadata checked
LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models
Gangsu Kim and Won-Ki Jeong
Official MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02; this supersedes the prior preprint classification. Springer DOI remains pending.
Official artifacts: Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF
Metadata checked
LRMIL: Efficient Low-Resolution Multiple Instance Learning via High-Resolution Knowledge Distillation for Whole Slide Image Classification
Yonghan Shin and Won-Ki Jeong
Official MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02. Springer DOI and pagination remain pending; do not invent either.
Official artifacts: Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF
Metadata checked
What Makes High-Magnification Knowledge Transferable? A Study of Cross-Resolution Distillation in Whole-Slide Imaging
Zhiyuan Yang, Jiahao Cheng, and Mahdi S. Hosseini
Public arXiv v2 (2026-10-01) verified 2026-10-02. The authors state ICLR 2027 under review; no acceptance inferred. Only public source material was used. A diagnostic study of target utility, predictability and downstream benefit, not a new universally superior distillation architecture.
Official artifacts: Reviewed manuscript revision
Metadata checked
Cross-Stain Knowledge Distillation for Low-Cost Lung Cancer Programmed Death Ligand-1 Assessment With Multi-Granularity Multiple Instance Learning
Yi Shi, Chong Ge, Fang Zhao, Anli Zhang, Ao Li, Haibo Wu, and Minghui Wang
Peer-reviewed journal article, volume 167, article 113762. IHC is privileged teacher information during training; deployment uses the H&E student, but its macro- and micro-granularity branches remain. No official code or model release was verified on 2026-09-02.
Metadata checked
KDTViT Knowledge Distillation, Transfer Learning and Transformer Based Deep Learning Framework for Efficient Histopathology Image Classification
Aarv Mankodi, Praveen Kumar Shukla, and Hitesh Tekchandani
Peer-reviewed open-access journal article 16:20721. A MobileViT-XS is sequentially adapted from 40× through 400× using the previous checkpoint as teacher; BreakHis consists of biopsy images rather than WSIs, and the same-capacity teacher/student design targets retention rather than model compression.
Official artifacts: Open-access article
Metadata checked
Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast Cancer
Zhiwei Chen, Yang Hu, Yuxiang Xiao, Yakun Ju, Tianyang Zhang, Yingxue Xu, Wei Li, Hao Chen, Jens Rittscher, and Kaixiang Yang
Official MICCAI 2026 proceedings record and accepted-version open-access PDF verified 2026-10-02; Springer DOI and pagination remain pending.
Official artifacts: Official code · Official MICCAI 2026 proceedings · Official accepted-version PDF
Metadata checked
Streamlined Knowledge Distillation
Hyeon-Jin Jeong, Han-Jin Lee, and Seok-Hwan Choi
Official artifacts: Official code
Metadata checked
LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models
Hyunsoo Han, Sangyeop Yeo, and Jaejun Yoo
Metadata checked
Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
Jingchen Sun, Shaobo Han, Deep Patel, Wataru Kohno, Can Jin, and Changyou Chen
Metadata checked
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Mark A. Hasegawa-Johnson, and Chang D. Yoo
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Teacher and student share a frozen SigLIP vision encoder and resolution; this is not cross-resolution KD. The official code repository lists training/evaluation scripts as planned. Attention similarity correlations are not causal guarantees of faithful reasoning.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, and XuDong Wang
ECCV 2026 Oral verified in the official program (event 6085); scientific claims use the conference-hosted manuscript #5952. Publisher proceedings/version of record was not verified on 2026-09-12. First preprint appeared in November 2025; stable ID preserves that origin.
Official artifacts: ECCV program · Conference manuscript · Author project · Code
Metadata checked
Distilling Balanced Knowledge from a Biased Teacher
Seonghak Kim
Metadata checked
Distilling Safe LLM Systems via Soft Prompts for On Device Settings
Motasem Alfarra, Cristina Pinneri, Dana Kianfar, Mohammed Almousa, and Christos Louizos
Published in Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337.
Metadata checked
On-policy Distillation with Verifiable Reward
Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang
arXiv v1 preprint submitted 25 August 2026; evaluation is limited to the reported verifiable reasoning tasks.
Official artifacts: Official implementation
Metadata checked
Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it
Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner, Isaac Triguero, and Daniel Molina
arXiv v1 preprint submitted 24 August 2026; conclusions are scoped to the reported homogeneous CNN image-classification study.
Metadata checked
On-Policy Self-Distillation in Diffusion Models
Wei Zhou, Xiongwei Zhu, Lingdong Kong, Bo Chen, Lei Zhang, Yongyuan Liang, Xiaoxia Hou, Ye Tian, Xian Sun, Yingshuo Wang, Linfeng Li, Shengqiong Wu, Leigang Qu, Feng Li, Wei Liu, Julian McAuley, and Tat-Seng Chua
arXiv v1 technical report submitted 25 August 2026; reported reward gains rely on the chosen automated evaluators and two tested backbones.
Official artifacts: Official project · Official implementation
Metadata checked
Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher
Shiyi Zhang, Mushui Liu, Yunze Tong, Wanggui He, Siyu Zou, Jinlong Liu, Yunlong Yu, Jian Song, Hao Jiang, Pipei Huang, and Bo Zheng
arXiv v1 preprint submitted 27 August 2026; teacher-free reward optimization remains benchmark- and reward-model-dependent.
Official artifacts: Official implementation
Metadata checked
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Haoyang Huang, and Nan Duan
ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #9084. Publisher proceedings/version of record was not verified on 2026-09-12. Author list follows the conference manuscript and program; the current repository citation lists two additional authors. Review separates the paper's three-stage method from the later simplified two-stage code release.
Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model checkpoint
Metadata checked
AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation
Yuchao Gu, Guian Fang, Yuxin Jiang, Weijia Mao, Song Han, Han Cai, and Mike Zheng Shou
ECCV 2026 Spotlight verified in the official program; scientific claims use the conference-hosted manuscript #9062. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript · Official implementation · arXiv manuscript · Author project · Model collection
Metadata checked
Expandable Residual Approximation for Knowledge Distillation
Zhaoyi Yan, Binghui Chen, Yunfan Liu, and Qixiang Ye
Published in IEEE TNNLS 37(1), January 2026, pages 191–203; first available online in 2025.
Official artifacts: Author manuscript · Official implementation and models
Metadata checked
On learning denoisable student logits
Diqi Chen, Yang Li, Jiajun Liu, Brano Kusy, Jun Zhou, and Yongsheng Gao
Peer-reviewed article 113684 in Pattern Recognition 179; available online 10 April 2026 and assigned to the November 2026 issue.
Metadata checked
Knowledge Distillation for Large Language Models through Residual Learning
Thinh On, Hengzhi Pei, Leonard Lausen, and George Karypis
Published at ICLR 2026; no official implementation was located by 2026-09-02.
Official artifacts: OpenReview record · First-party Amazon Science page
Metadata checked
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, and Bing Qin
The arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026.
Official artifacts: Official implementation
Metadata checked
TRACE: Your Diffusion Model Is Secretly an Instance Edge Detector
Sanghyun Jo, Ziseok Lee, Wooyeol Lee, Jonghyun Choi, Jaesik Park, Kyungsu Kim
Official ICLR 2026 oral and full proceedings manuscript checked. Diffusion-boundary self-distillation improves selected natural-image segmentation but degrades MoNuSeg/TNBC cell segmentation. One-pass speed is edge extraction only, with the large backbone retained. No WSI or cross-magnification validation; official code/checkpoints remain promised rather than released at the checked revision.
Official artifacts: OpenReview · Official project · Official repository — release pending
Metadata checked
The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation
Kuniaki Saito and Yoshitaka Ushiku
arXiv v1 checked 2 October 2026. No official code/model release or conference acceptance verified. Camelyon17 is patch-level evaluation, not whole-slide cross-magnification validation. PE teacher/student labels and parameter counts differ between main text/tables and Appendix C; this review uses unambiguous DINOv2-to-MobileNet Table 1 rows.
Metadata checked
CSCWD: Cross-Scale Channel-wise Knowledge Distillation for Lightweight Tiny Object Detection on Edge Devices
Amir Zamani and Zeinab Ghasemi-Naraghi
arXiv v1 checked 2 October 2026; no official code release or acceptance verified. Cross-scale means teacher P2 to student P3 feature-map alignment at a shared input resolution, not independent high/low magnification acquisitions. MGD is stronger on reported mAP; external stricter mAP does not improve.
Metadata checked
PAGET: Hierarchical Multi-Teacher Knowledge Distillation for Comprehensive Tumor Microenvironment Segmentation
Daisuke Komura, Maki Takao, Mieko Ochi, Takumi Onoyama, Hiroto Katoh, Hiroyuki Abe, Hiroyuki Sano, Teppei Konishi, Toshio Kumasaka, Tomoyuki Yokose, Yohei Miyagi, Tetsuo Ushiku, and Shumpei Ishikawa
Peer-reviewed MIDL 2026 proceedings, PMLR 315:256–279; official record verified 2026-10-02.
Official artifacts: Official proceedings PDF · Official implementation
Metadata checked
ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology Segmentation
Khang Le, Ha Thach, Anh M. Vu, Trang T. K. Vo, Han H. Huynh, David Yang, Minh H. N. Le, Thanh-Huy Nguyen, Akash Awasthi, Chandra Mohan, Zhu Han, and Hien Van Nguyen
Peer-reviewed MIDL 2026 proceedings, PMLR 315:2893–2905; official record verified 2026-10-02.
Official artifacts: Official proceedings PDF · Official implementation
Metadata checked
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Mingxi Fu, Jiawen Li, Renao Yan, Jiali Hu, Qiehe Sun, Tian Guan, and Yonghong He
arXiv v1 (2026-07-16) checked 2026-10-02. Six cohorts cover 15 tasks, not 15 independent datasets. Official code and pretrained aggregator checkpoints are listed; experiments were not rerun. This transfers slide embeddings into MIL initialization, not 20×-to-5× acquisition.
Official artifacts: Versioned manuscript · Official implementation and pretrained MIL weights
Metadata checked
Pathway-Structured Privileged Distillation for Deployable Computational Pathology
Yongxin Guo, Hao Lu, Onur C. Koyun, Zhengjie Zhu, Muhammet F. Demir, and Metin N. Gurcan
arXiv v1 (2026-06-01) checked 2026-10-02; no conference acceptance verified. Paired RNA is training-only, while WSI is used at deployment. Pathway-memory readouts are not independently validated molecular measurements.
Official artifacts: Versioned manuscript
Metadata checked