Probability, risk, and proper scoring
A differentiable surrogate may optimize ranking while degrading probability meaning.
Loss & objective design
A loss is a scientific contract between a target, a prediction, and the errors the training procedure will emphasize. This chapter organizes objective design by the failure it addresses—rather than collecting fashionable loss names—and keeps probability quality, ranking, geometry, overlap, robustness, and cross-modal alignment as distinct claims. For dense prediction it also separates region overlap from signed-distance boundary objectives, because near-contour and far-away errors are not the same training target.
The implemented gradient is part of that contract. In A Fix for Softmax and Cross-Entropy, algebraically equivalent calculations behave differently at finite precision: high-confidence hard labels can lose the target gradient, while nearly matching teacher and student logits can acquire a spurious distillation gradient. The numerical-objectives reading below separates these failure modes and their repairs. Its small algorithmic experiments motivate implementation checks, not an assumption of improved pathology or large-scale model accuracy.
61 maintained papers · literature reviewed through 2 October 2026
An objective chooses which errors receive gradient and how strongly. Proper probability estimation, ranking, class imbalance, representation geometry, region overlap, label noise, preference learning, and multimodal negatives require different contracts. A new loss is credible only when its target, assumptions, gradients, baselines, metric alignment, calibration, and deployment prevalence are audited separately.
Prerequisite contract
Metrics, splits, and calibration
Training prevalence and target evaluation prevalence must not be silently conflated.
How the field developed
1995–2019
Margins, metric geometry, and angular classifiers
Maximum-margin classification, pairwise contrast, visual-semantic ranking, triplet embedding, prototypes, and additive angular margins decide which examples separate, attract, repel, or define a class centre.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Support-Vector Networks A support-vector network maps inputs implicitly into a feature space through a kernel, then fits a maximum-margin linear separator there. Slack variables extend the construction to non-separable data, while the final decision depends only on training points with active dual weights—the support vectors. — 01.02 Dimensionality Reduction by Learning an Invariant Mapping DrLIM learns a nonlinear embedding from pair relations: similar pairs are pulled together, while dissimilar pairs are pushed apart only until their Euclidean distance reaches a margin. The same network processes both inputs, so the resulting mapping can embed unseen examples rather than merely arranging the training set. — 01.03 DeViSE: A Deep Visual-Semantic Embedding Model DeViSE replaces a closed 1,000-way visual classifier with a learned map from an image representation into a fixed language-semantic space. A pretrained vision network supplies a 4,096-dimensional image vector; a linear projection maps it beside unit-normalized skip-gram label vectors. Nearest-label retrieval can then score any vocabulary term that already has a text embedding, including categories with no visual training examples. Its central claim is therefore not merely better classification, but an output space whose geometry transfers linguistic relatedness into visual prediction. — 01.04 FaceNet: A Unified Embedding for Face Recognition and Clustering FaceNet learns one compact Euclidean space in which distance itself answers face verification, identification, and clustering queries. A convolutional network maps each crop to a 128-dimensional L2-normalized embedding; training does not classify a fixed roster of identities. Instead, online triplets require an anchor to lie closer to a same-identity positive than to a different-identity negative by a margin. The decisive engineering detail is semi-hard mining: useful negatives remain farther than the positive but still violate the margin, avoiding many collapsed early-training states caused by the absolute hardest examples. — 01.05 Prototypical Networks for Few-shot Learning A prototypical network turns each few-shot episode into a temporary classifier. It embeds the labeled support examples, averages embeddings within each class to form one prototype, embeds each query, and predicts with a softmax over negative distances to those prototypes. The learned object is therefore the embedding geometry; the test-time class representatives are computed, not stored classifier weights. — 01.06 ArcFace: Additive Angular Margin Loss for Deep Face Recognition ArcFace converts closed-set softmax training into hyperspherical metric learning by normalizing both embeddings and classifier weights, then adding a fixed angular margin to the target class before cross-entropy. The target logit becomes s cos(theta_y + m), while non-target logits remain s cos(theta_j). Because m is an additive angle, it corresponds directly to geodesic separation on the unit sphere and yields a constant angular decision margin rather than an input-dependent surrogate. Official implementation family Work: Write the score, distance, margin, mining rule, positive/negative construction, and reduction unit before implementation.
Exit evidence: Explain how scale, mining, normalization, and class or batch composition change geometry and open-set behaviour.
2018–2026
Contrastive and multimodal representation objectives
Predictive coding, augmented views, supervised multi-positive learning, and global, token-level, generative, sigmoid, or false-negative-aware image–text alignment make the negative pool and batch context part of the loss.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Representation Learning with Contrastive Predictive Coding Contrastive Predictive Coding encodes local observations into latents z, summarizes the available past or spatial context as c, and identifies the true future latent among N−1 samples from a proposal marginal. A step-specific predictor scores exp(zᵀWₖc), and InfoNCE trains that score as a conditional-to-marginal density ratio. CPC therefore learns information shared across distant signal parts without decoding every future pixel, waveform sample or word; either local z, contextual c, or a pooled version can transfer. — 02.02 A Simple Framework for Contrastive Learning of Visual Representations SimCLR samples two independent augmentation pipelines for every image, encodes all 2N views with one shared ResNet, maps the representations through a nonlinear projection head, and symmetrically identifies each view's same-source companion among the other batch views. The contrastive loss acts on normalized projected z, but downstream tasks use the pre-projection h after g is discarded. Its central empirical lesson is that augmentation composition, the separation of loss space from representation space, temperature, model scale and optimization duration matter as much as the nominal contrastive equation. — 02.03 Supervised Contrastive Learning Supervised contrastive learning uses every same-class view in a multiview batch as a positive for an anchor, while all different-class views remain contrastive alternatives. Rather than asking only an augmented copy of one image to match itself, the loss shapes a label-aware embedding in which multiple instances of a class align. A projection head receives the contrastive loss and is discarded; a linear classifier is then trained on frozen encoder representations. Code 02.04 Learning Transferable Visual Models From Natural Language Supervision CLIP jointly trains an image encoder and a text encoder to recognize which image and text belong together among all pairings in a large batch. The shared embedding space is then reused without retraining the encoders: descriptions of downstream classes are encoded into text-generated classifier weights, and a new image is assigned to the most similar class embedding. This turns natural language into a flexible classification interface while retaining a separate frozen image representation for supervised linear-probe evaluation. — 02.05 FILIP: Fine-grained Interactive Language-Image Pre-Training FILIP keeps the retrieval-friendly dual-encoder structure of CLIP but replaces one global image-text dot product with cross-modal late interaction. Every image patch searches for its most similar non-padding text token and every text token searches for its most similar image patch; the two directional scores average those token-wise maxima. Contrastive learning then aligns matched pairs and separates in-batch mismatches while retaining independently encodable image and text towers. — 02.06 CoCa: Contrastive Captioners are Image-Text Foundation Models CoCa turns one image-to-text encoder-decoder into both a global alignment model and a conditional generator. Its text decoder is split: lower causal layers never cross-attend and yield a text-only sentence embedding for a CLIP-like contrastive loss; upper causal layers cross-attend to image tokens and predict the caption autoregressively. Two task-specific attentional poolers simultaneously expose one global image vector and a 256-token visual sequence. The central idea is therefore one shared forward graph with two representation granularities, not two separately pretrained models. — 02.07 Sigmoid Loss for Language Image Pre-Training SigLIP retains the CLIP-style two-tower representation space but changes the training question. CLIP uses two globally normalized softmax classifications—each image selects its paired text and each text selects its paired image from the whole batch. SigLIP instead treats every image–text combination as an independent binary decision: the observed diagonal pair is positive and every off-diagonal combination is negative. A learned sigmoid loss is applied once to the signed pair logit. This removes the loss's need for a global softmax denominator, enables chunk-by-chunk distributed evaluation, and performs especially well at modest batch sizes; it does not remove the encoders, the negative pairs, or the quadratic number of pair interactions. — 02.08 FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment FALCON is a training-time mini-batch constructor for vision–language pretraining. Instead of always placing the most similar candidate beside an anchor, it learns a probability distribution over similarity quantiles for every anchor and uses the sampled quantile to choose the next pair. The scheduler is rewarded when the resulting batch produces a larger reduction in a generative cross-modal loss. Its purpose is therefore to preserve the learning value of hard negatives without repeatedly treating semantically valid alternatives as negatives. — Work: Compare objectives under identical encoders, views, examples, negative pools, and budgets.
Exit evidence: Identify false negatives, token/global alignment, temperature, normalization, captioning, and retrieval effects.
2024
Distribution discrepancies for distillation
Wasserstein KD asks whether transport geometry can replace the conventional KL discrepancy when teacher and student distributions differ. It is a loss-design question, not merely a compression implementation detail.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation WKD asks whether knowledge distillation must compare teacher and student with Kullback-Leibler divergence or pointwise feature regression. Its answer is a two-level optimal-transport design: WKD-L transports probability mass between semantically related classes instead of charging every class mismatch equally, while WKD-F compares the distributions of spatial feature vectors with the closed-form 2-Wasserstein distance between Gaussians. The two losses can be used separately or together. The essential reading rule is therefore: logits are treated as structured categorical distributions; features are treated as geometric distributions, not coordinate-by-coordinate targets. Official project page · Official code Work: Match logits, temperature, teacher/student pair, and optimizer, then compare KL and Wasserstein gradients and cost.
Exit evidence: Explain the ground metric, transport approximation, computational overhead, and where the discrepancy improves transfer.
2010–2026
Proper losses, calibration, imbalance, and prior shift
Composite-loss theory separates link and scoring rule; focal, margin, effective-number, logit, polynomial, and bias-corrected distillation terms alter which classes or confidence regions dominate.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Composite Binary Losses A composite binary loss separates two choices that are often conflated: a proper loss that judges a probability estimate and an invertible link that maps that probability to the real-valued score emitted by a model. Reid and Williamson show that a single weight function characterizes the proper loss, while the ratio between that weight and the link derivative controls the composite score-space loss. — 04.02 On Calibration of Modern Neural Networks Modern neural networks can be accurate yet overconfident; a single positive temperature fitted on held-out validation negative log likelihood often substantially improves in-distribution confidence calibration while preserving every predicted class. — 04.03 Focal Loss for Dense Object Detection Focal loss makes dense one-stage detection trainable at two-stage accuracy by multiplying cross entropy with a confidence-dependent factor. When an example is already classified correctly with high probability, its contribution shrinks toward zero; rare mistakes and ambiguous foreground/background locations retain weight. RetinaNet demonstrates the idea in a deliberately simple ResNet-FPN detector that learns from roughly 100,000 anchors per image without sampling most easy negatives away. — 04.04 Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss LDAM gives minority classes larger classification margins instead of only multiplying their losses. For class j with n_j training examples, the target-logit margin scales as C/n_j^(1/4); rarer labels therefore need more separation before softmax considers them correct. The paper pairs this objective with Deferred Re-Weighting (DRW): first learn representations without class re-weighting, then introduce inverse-frequency weights only after the learning-rate decay. The loss and schedule address different failure modes and are strongest together. Official code 04.05 Class-Balanced Loss Based on Effective Number of Samples Class-balanced loss replaces a class's raw count with its effective number of samples, a saturating quantity derived from a simplified random-coverage model. Reweighting by the inverse effective number treats early examples as highly informative but discounts later examples that are increasingly likely to overlap already covered class content. The weight can multiply softmax, sigmoid, or focal losses without changing their inner form. Official code 04.06 Long-Tail Learning via Logit Adjustment Logit adjustment corrects a classifier trained on a long-tailed label prior when the target metric is balanced error. The post-hoc form subtracts a scaled log training prior from each learned logit before argmax. The training-time form adds the log prior inside the softmax loss so the network learns scores whose ordinary argmax is already aligned with the balanced decision rule. — 04.07 PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions PolyLoss rewrites classification objectives as polynomial series in the target error q=1-p_t. Cross-entropy has coefficients 1/j; focal loss shifts polynomial powers. This basis reveals another design axis: directly perturb coefficients. The practical Poly-1 objective changes only the leading coefficient by adding epsilon_1(1-p_t) to cross-entropy. One scalar can steepen or flatten the gradient near confident predictions, but its useful sign is task-dependent rather than universally positive. Author-linked EfficientNetV2 implementation 04.08 Distilling Balanced Knowledge from a Biased Teacher LTKD makes a crucial correction to the usual long-tailed distillation story: a teacher trained on imbalanced data is not automatically a balanced source of dark knowledge. Its probability mass is systematically larger for frequent Head classes and smaller for Tail classes, so ordinary KD can reproduce the bias. The paper decomposes KD into inter-group probability transfer and intra-group class relations for Head, Medium, and Tail groups. It then balances teacher group mass within each batch and assigns equal strength to within-group relational knowledge. The student still learns from labels, but its distillation target no longer lets frequency alone determine how much each group teaches. — Work: Compare proper loss, calibration, reweighting, resampling, margins, polynomial terms, teacher-bias correction, and thresholding at fixed prevalence.
Exit evidence: Report discrimination, head/medium/tail error, calibration, and decision utility separately.
2016–2019
Dense prediction: regions, overlap, boxes, and boundaries
Dice-family losses target region overlap, Lovász targets IoU, GIoU extends box overlap when boxes do not intersect, and boundary loss weights predictions by signed contour distance.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation V-Net couples a fully convolutional three-dimensional encoder-decoder with an overlap objective computed directly from soft predictions and the target volume. Instead of reducing segmentation to independently classifying heavily imbalanced voxels, the Dice formulation aggregates foreground agreement over the whole volume, while residual stages and learned down/up-sampling preserve an end-to-end volumetric path. Official code 05.02 Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations Generalised Dice Loss turns a multi-class overlap score into a differentiable training objective and counteracts class-size imbalance with inverse-squared reference volumes. Every class contributes to one global numerator and denominator, so small foreground structures receive much larger weights than the background. The paper's central evidence is a controlled robustness study: across 2D and 3D lesion segmentation, learning rates, patch sizes, sampling ratios, and four architectures, the volume-weighted variant more reliably avoids failed training when foreground occupancy becomes extreme. Open author manuscript 05.03 The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks Lovasz-Softmax converts the discrete per-class Jaccard loss into a piecewise-linear surrogate over softmax errors. It sorts voxel or pixel errors from largest to smallest and weights them by the incremental change in Jaccard loss, directing gradient toward the mistakes that most affect intersection-over-union rather than treating every pixel independently. Official code 05.04 Generalized Intersection Over Union: A Metric and A Loss for Bounding Box Regression Generalized IoU repairs the zero-signal region of ordinary IoU. It subtracts the fraction of the smallest enclosing region left outside the union from IoU, producing a scale-invariant similarity that remains informative when two boxes do not overlap. The training loss is 1 minus GIoU. When intersection is empty, shrinking the enclosing box while expanding the useful union moves prediction toward the target, so localization receives a gradient where IoU loss is flat. Official project and implementations 05.05 Boundary loss for highly unbalanced segmentation Boundary loss changes the geometric object optimized in highly imbalanced segmentation. Rather than summing a loss uniformly over foreground and background regions, it precomputes a signed distance to the ground-truth contour and weights each soft foreground prediction by that distance. Predictions inside the reference object receive negative weights and far-away false positives receive larger positive penalties. The paper combines this contour term with generalized Dice instead of treating it as a stand-alone replacement. Official code Work: Compare cross-entropy, Dice-family, IoU/GIoU, and contour-distance surrogates at matched preprocessing and resolution.
Exit evidence: State class-presence rules, empty-mask behaviour, smoothing, distance convention, reduction unit, schedule, and post-processing.
2018–2026
Noisy supervision and robust geometry
Generalized and symmetric cross-entropy, adaptive robust penalties, and geometry-aware objectives make different assumptions about corrupted labels, residual distributions, and shifted representations.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels Generalized cross entropy replaces negative log probability with a q-shaped target-probability loss. As q approaches zero it becomes categorical cross-entropy, whose strong gradients learn quickly but can chase mislabeled examples; at q=1 it becomes the mean-absolute-error form, whose equal weighting is noise-tolerant but difficult to optimize. Intermediate q trades between those behaviors, and a truncated variant caps the influence of low-confidence labels. — 06.02 Symmetric Cross Entropy for Robust Learning with Noisy Labels Symmetric Cross Entropy combines ordinary cross-entropy, H(q,p), with Reverse Cross Entropy, H(p,q). CE provides fast convergence and learns hard classes but can memorize wrong labels; RCE weights the log target distribution by model probabilities and is noise-tolerant under stated conditions. The practical Symmetric Learning loss alpha CE + beta RCE balances these behaviors. Because one-hot q contains zeros, the method defines log 0 as a finite negative constant, making RCE computable and controlling its gradient scale. Official code 06.03 A General and Adaptive Robust Loss Function Barron's general robust loss uses a continuous shape parameter alpha to span L2, Charbonnier, Cauchy, Geman–McClure, and Welsch-like behavior, with scale c controlling the quadratic basin. Smaller alpha reduces the influence of large residuals; alpha can be fixed, annealed, or learned. To learn alpha without the trivial solution of making every residual maximally robust, the method interprets the loss as a negative log likelihood and includes the alpha-dependent partition function. Official PyTorch implementation 06.04 The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models GRACE targets a three-way fine-tuning tension in CLIP-style vision-language models: preserve clean in-distribution accuracy, retain generalization to natural or unseen-domain shifts, and resist bounded adversarial inputs. Its central claim is geometric, not merely architectural. A fine-tuned solution should remain near the pretrained parameters, avoid sharp directions in the weight-loss landscape, and keep the same sample's representation coherent when either the pixels or trainable weights are perturbed. GRACE therefore freezes the pretrained CLIP backbone, learns rank-64 LoRA updates, constructs layer-adaptive low-rank adversarial weight perturbations, and minimizes a Gram-volume surrogate over clean, adversarial-input, and adversarial-weight feature vectors. The published evidence supports a more balanced measured trade-off on the named ImageNet protocols; it does not establish universal VLM robustness. — Work: Stress each objective under controlled label noise, outliers, subgroup shift, and clean-data conditions.
Exit evidence: Show where robustness helps, where calibration or geometry changes, and which assumption fails first.
2023–2024
Reference-based and reference-free preference objectives
DPO, KTO, and SimPO replace reward-model-plus-RL pipelines with distinct pairwise or prospect-inspired likelihood contracts. Their reference support, length effects, and preference noise are not interchangeable.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 Direct Preference Optimization: Your Language Model is Secretly a Reward Model Direct Preference Optimization (DPO) turns the usual reward-model-plus-reinforcement-learning stage of RLHF into one binary classification loss on preferred and rejected responses. Its key algebra says that, under a Bradley-Terry preference model and a KL-regularized reward objective, the optimal policy's log-ratio to a fixed reference policy is itself a reward representation. Reward differences remove the prompt-dependent partition function, so the language model can be trained directly from offline preference pairs. The memory line is policy log-ratio becomes the implicit reward. — 07.02 Model Alignment as Prospect Theoretic Optimization Kahneman–Tversky Optimization aligns a language model from unpaired binary feedback: each response is labeled desirable or undesirable, without requiring a winner and loser for the same prompt. KTO measures the policy/reference log-probability ratio relative to an average KL-derived reference point, then applies opposite saturating sigmoid value functions to desirable and undesirable examples. Separate lambda weights handle feedback imbalance. The objective is framed as a human-aware loss inspired by prospect-theory gain/loss asymmetry, not as a literal psychological model of every user. Official HALOs code 07.03 SimPO: Simple Preference Optimization with a Reference-Free Reward SimPO trains on paired preferred and dispreferred responses with no frozen reference model. Its implicit reward is beta times the average token log probability under the trainable policy, matching the length-normalized likelihood commonly used to rank generated sequences. A positive target margin gamma inside the Bradley–Terry objective requires the preferred response to exceed the rejected response by more than zero. Length normalization, reference removal, and the reward margin are separate design choices, and ablations show the first two objective terms matter. Official code and models Work: Compare preference objectives on the same pairs, reference policy, tokenizer, length distribution, and evaluation suite.
Exit evidence: Audit implicit reward, reference dependence, margin, length bias, calibration, and judge sensitivity.
2018
Competing-risk and time-to-event objectives
DeepHit jointly models discrete-time competing-risk incidence and a ranking term, illustrating how a loss encodes clinical event structure rather than ordinary class error.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 DeepHit: A Deep Learning Approach to Survival Analysis with Competing Risks DeepHit directly predicts a normalized joint distribution over discrete event times and competing causes. A shared network captures covariate structure, cause-specific branches retain event-specific effects, and one global softmax makes all cause-time cells sum to one. Training combines a censoring-aware likelihood with a cause-specific ranking objective built from acceptable patient pairs. This avoids proportional-hazards and linear-risk assumptions while yielding cumulative incidence functions for each event. Official code Work: Reconstruct risk set, censoring convention, time bins, competing events, and ranking pairs.
Exit evidence: Report discrimination, calibration, cumulative incidence, and external time-shift behaviour separately.
2025
Pathology MIL attention regularization
AEM treats attention collapse in whole-slide MIL as an objective-design problem by maximizing attention entropy, connecting slide-level weak supervision to explicit evidence-spread control.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 AEM: Attention Entropy Maximization for Multiple Instance Learning based Whole Slide Image Classification AEM regularizes the attention distribution of a whole-slide multiple-instance learner instead of adding another aggregator. For attention weights A={a_n}, it minimizes negative entropy sum a_n log a_n alongside slide-level cross-entropy, discouraging early concentration on only a few patches. Cosine Weight Annealing gradually reduces the regularizer so training starts with broad instance exploration and later permits focused evidence. The method is a plug-in objective for WSI classification, not a survival-analysis loss. Official code Work: Compare entropy regularization with matched attention-MIL baselines under patient-level splits.
Exit evidence: Report attention entropy, classification, localization faithfulness, calibration, and external-cohort behaviour.
2026
Rank-aware ordinal evidence
RACR-MIL couples ordered grade structure to contextual WSI attention; the objective belongs beside calibration and ranking losses, while its reader pilot remains separate from clinical validation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 10.01 RACR-MIL: Rank-aware contextual reasoning for weakly supervised grading of squamous cell carcinoma using whole slide imagesThe complete arXiv v3 manuscript identifies Medical Image Analysis as the journal and the abstract record reports acceptance; a final DOI or journal issue record was not yet available when checked on 30 August 2026. RACR-MIL grades squamous-cell carcinoma from slide labels by combining two contextual graphs with ordinal, rank-aware attention. A spatial graph models neighboring tissue and a latent graph connects phenotypically similar patches, including distant regions; inter-grade and intra-grade constraints then teach attention to prioritize the most aggressive reliable tumor evidence rather than merely the most class-discriminative patch. — Work: Compare nominal cross-entropy, ordinal regression or ranking baselines, and RACR-MIL with the same WSI encoder, graph/context module, sampling, patient split, and tuning budget.
Exit evidence: Report per-grade prevalence and confusion, ordinal distance errors, weighted agreement, calibration, anatomy/site transfer, graph ablations, uncertainty, and the reader-pilot denominator separately.
2026
Residual probability mass and representation-aligned flow objectives
ACTD penalizes probability left outside cross-tokenizer anchors; ReFlowSET adds clean-target representation alignment to conditional flow matching. Both require stating exactly which space is aligned.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 11.01 ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual RegularizationThe arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026. ACTD distills reasoning models across different tokenizers by mapping teacher top-k tokens into student-space anchors. An anchor L1 term matches probability on aligned tokens, while residual regularization constrains the student's probability mass outside those anchors. With multiple teachers, ACTD computes a complete teacher-specific loss for each teacher and averages those losses; it does not average anchor signals or teacher distributions first. Official implementation 11.02 ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image TranslationarXiv v1 submitted 1 September 2026. The exact title follows the current arXiv record, superseding an earlier feed title. Code, weights, and an official project page were available; no peer-reviewed record was verified. Core idea: audit the latent codec on both radar conditions and optical targets before training the generator, then learn conditional rectified flow in the selected latent space. ReFlowSET uses a from-scratch conditional DiT with separate SAR and noisy-EO streams, delayed channel fusion, and a frozen vision-foundation-model alignment loss that guides training but is removed at inference. Official implementation and checkpoints · Official project page Work: Trace every normalized distribution, token map, latent codec, teacher feature, loss weight, and gradient recipient.
Exit evidence: Demonstrate gains against matched objective ablations and quantify alignment error, compute, and failure on unmapped tokens or modality-specific detail.
2025–2026
Physical observability and task-aware polarization reconstruction
GenPolar predicts Stokes components from intensity using observability-aware supervision and one-step diffusion distillation. PolarAPP instead starts from measured polarization mosaics and couples demosaicking with downstream objectives through a learned metric. Keep generative priors separate from sensor-acquired polarization.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 12.01 Stokes-Informed Diffusion for Robust Linear Polarization EstimationECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5749. Publisher proceedings/version of record was not verified on 2026-09-12. GenPolar predicts RGB-channel linear Stokes components from an intensity image, then derives polarization magnitude and angle analytically. It combines observability-aware physical supervision with multi-step-to-one-step diffusion distillation and VAE-encoder LoRA. This targets stable polarimetric cues rather than merely plausible-looking angle maps. ECCV Spotlight program · Conference manuscript 12.02 PolarAPP: Beyond Polarization Demosaicking for Polarimetric ApplicationsECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5713. Publisher proceedings/version of record was not verified on 2026-09-12. PolarAPP jointly optimizes polarization demosaicking and a downstream task using a meta-learned feature-alignment metric. An equivalent-imaging self-consistency constraint improves reconstruction without treating rearranged mosaic pixels as perfect targets. Training-time alignment modules disappear at inference, leaving a demosaicker, analytic polarization cues and the task network. ECCV Spotlight program · Conference manuscript Work: Is the supervision a physically observable target, a synthetic reconstruction target, a downstream task signal or a teacher distribution?
Exit evidence: Weak-polarization angle masks change the evaluated population. Task-aware meta-learning is not automatically teacher–student KD, and metric improvements need not agree.
2025–2026
Anatomical token selection and staged CT report generation
A coarse abnormality language model guides structure-specific evidence selection for a second report model. Training-only anatomical masks and a downstream report reward align the stages. Inspect which component GRPO updates and which signals come from segmentation supervision.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 13.01 Mimicking Radiologists: A Coarse-to-Fine Framework with Structural Sparse Tokens for Dual-LLM Computed Tomography Report GenerationECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #9339. Publisher proceedings/version of record was not verified on 2026-09-12. The paper-linked CTRG repository README still describes the earlier Structure Observation Driven Image-Text Contrastive Learning work and LLaMA-2-7B. A matching dual-LLM/GRPO release was not verified; do not treat this link as reproduction evidence. This dual-LLM CT reporting system first proposes abnormalities from coarse structural tokens, then selects local evidence to generate the full report. AP-LTF combines text–token similarity, learned scoring and attention weights, while training-only anatomical masks discourage attention outside the intended structure. GRPO refines the proposal policy using the downstream report reward. ECCV Spotlight program · Conference manuscript · Paper-linked repository (earlier method; matching release unverified) Work: Can an early abnormality proposal remove evidence needed by the final report, and how is that failure measured?
Exit evidence: Labeler-based clinical efficacy scores are proxy metrics, not clinical validation. The linked repository documents an earlier method; a matching release is not yet verified.
2025–2026
Physiology-aware cross-modal spectral distillation
RPM-Distill transfers peak position, off-peak suppression and spectral morphology from an RF teacher to an RGB student. A validation-driven policy chooses sample and loss weights; deployment removes the radar branch. Compare this privileged-sensor design with feature KD and reliability-weighted transfer.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 14.01 RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological MeasurementECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #10941. Publisher proceedings/version of record was not verified on 2026-09-12. RPM-Distill transfers rhythm information from a frozen radar teacher to a video-only physiological student. It compares band-limited spectra rather than forcing incompatible RF and RGB waveforms to match. A learned policy gates each sample and weights peak, off-peak and spectral-shape losses according to whether they improve held-out physiological recovery. ECCV Spotlight program · Conference manuscript · Official implementation Work: Which teacher signal is physiologically invariant across sensors, and what labeled validation budget trains the transfer policy?
Exit evidence: A supervised teacher and held-out labels are still required; the pipeline is not globally label-free. Policy residual blocks are not a residual-error KD target, and fusion methods retain different inference inputs.
2025–2026
Low-resolution student inference versus resolution-cascaded generation
GAP-Track removes teacher and residual reconstruction branches at inference. RMD instead retains high-resolution generative refinement. Both exploit cross-resolution supervision but solve different deployment problems.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 15.01 GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT TrackingECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Table 1 governs the recreated LasHeR values; the quarter-resolution CKD SR differs between Table 1 (35.3) and Table 4 (35.5). The PGS angle/radial equations need origin and quadrant safeguards not specified in the displayed formulas. The repository contains implementation directories despite a remaining release-soon notice; code was inspected, not executed. GAP-Track transfers full-resolution tracking supervision to a low-resolution RGB–thermal student. A masked residual feature generator learns to approximate teacher features during training, and a hierarchy of feature teachers eases extreme downsampling. The deployed model keeps only the low-resolution student and box head; it does not first reconstruct a high-resolution image at inference. Official ECCV poster record · Official code Work: Is high resolution absent at deployment, or only used for fewer steps? What is the full-resolution reference?
Exit evidence: Synthetic downsampling does not reproduce all optical changes between microscope magnifications; lower-resolution gains need not equal full-resolution performance.
2025–2026
Teacher-class mixtures and privileged residual routing
MoE-KD reformulates classification through latent teacher-class experts and EM. MARS uses full-minus-partial features to teach a deployable router. Specify whether knowledge resides in label mixtures, prototypes, routing decisions or a residual target.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 16.01 MoE-KD: Your Teacher Model is Worth Mixture-of-Experts for Knowledge DistillationECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The EM convergence discussion assumes an improving M-step; it is not a guarantee of global optimum for stochastic neural training. Table 3 values take precedence over inconsistent prose deltas. No runnable official implementation was verified in this pass. MoE-KD treats the teacher’s class as a latent variable in the student classification model. Teacher-derived feature prototypes define class-conditioned experts, while a reused teacher classifier gates projected student features. Alternating posterior estimation and evidence-lower-bound optimization couples these pieces instead of imposing one fixed teacher soft-label distribution alongside the ground-truth target. Official ECCV poster record Work: Which privileged information is available only during training, and exactly what remains in the student?
Exit evidence: Residual-guided routing is not a residual-error student cascade; conditional optimization arguments do not certify best accuracy.
2025–2026
Spatial-frequency and response-token distillation targets
SFKD selectively aligns wavelet-localized Fourier amplitudes across backbone families. TRAG aligns response-to-vision attention using token-specific entropy-weighted KL. Both teach structure beyond final logits, but use different correspondence assumptions.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 17.01 SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral InteractionECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The linked official repository still says source code will be released soon. This is heterogeneous-architecture feature KD, not demonstrated cross-magnification transfer. The selected CIFAR-100 table includes a ViT-S→MobileNetV2 case where FBT is better. SFKD transfers knowledge between different backbone families by separating local spatial structure from global spectral information. It decomposes intermediate maps into Haar-wavelet subbands, refines the student bands through convolutional and Transformer streams, and matches Gaussian-filtered Fourier amplitudes to teacher bands. Reconstruction and logit supervision keep the transfer connected to the classification task. Official ECCV poster record · Official code 17.02 Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge DistillationECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Teacher and student share a frozen SigLIP vision encoder and resolution; this is not cross-resolution KD. The official code repository lists training/evaluation scripts as planned. Attention similarity correlations are not causal guarantees of faithful reasoning. TRAG distills how a multimodal teacher attends to visual tokens while producing a response. It aggregates middle-layer response-to-vision attention, renormalizes over visual keys and assigns each response token its own forward/reverse KL mixture using teacher attention entropy. This supplements language and unimodal distillation losses rather than replacing the full training recipe. Official ECCV poster record · Official code Work: Are the grids, semantic units and probability supports genuinely aligned before applying the loss?
Exit evidence: SFKD has architecture-specific regressions; TRAG shares a visual encoder and resolution. Neither establishes cross-magnification performance.
2025–2026
Masked generation and diffusion-friendly latent spectra
MIMFlow combines masked image modeling with normalizing flows; SSVAE changes the spectral structure of video latents to improve downstream diffusion. Separate tokenizer reconstruction quality, likelihood or generation objectives, and final sample quality.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 18.01 Delving into Latent Spectral Biasing of Video VAEs for Superior DiffusabilityECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #7577. Publisher proceedings/version of record was not verified on 2026-09-12. The initial arXiv manuscript appeared in December 2025; year follows the 2026 conference. SSVAE trains video latents to be easier for a diffusion model to learn, not merely easy for a decoder to reconstruct. It promotes two separate spectral biases: a few dominant channel modes and greater low-frequency spatio-temporal energy. Latent Masked Reconstruction (LMR) and Local Correlation Regularization (LCR) implement those targets without requiring a new downstream diffusion objective. ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Model checkpoints Work: Which representation is easier to model, and what fidelity or sampling cost is exchanged for that advantage?
Exit evidence: Good reconstruction alone does not establish good diffusability, and normalizing flows are not interchangeable with flow matching.
2026
Real-data correction in generative distillation
RealUID unifies matching-model distillation and routes real examples through an auxiliary matching model. Read the alternating optimization and coefficient assumptions before interpreting the one-step generator results.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 19.01 Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)Official ICLR 2026 oral and full proceedings manuscript checked. Three-network training yields a one-step generator. No GAN discriminator does not mean no min–max optimization. Selected low-resolution image results include additional fine-tuning; teacher-error correction has restrictive assumptions. No cross-magnification or pathology evaluation. RealUID adds real-data supervision to a unified inverse-distillation formulation for matching models. A frozen teacher, a one-step generator and a learned fake matching model interact through matching losses. No GAN discriminator is added, but alternating min–max optimization remains: “No GANs” does not mean no adversarially structured optimization. Official proceedings record · OpenReview Work: Where does real data enter the teacher–student system, and which extra training stages are included in the comparison?
Exit evidence: No GAN discriminator does not mean no min–max problem; low-resolution generation does not establish pathology validity.
2026
Distributional geometry of contrastive representations
InfoNCE Induces Gaussian Distribution separates a plateau/projection argument from an explicitly regularized surrogate. Its diagnostic table is a lesson in testing, rather than assuming, representation distributions.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 20.01 InfoNCE Induces Gaussian DistributionOfficial ICLR 2026 oral and full proceedings manuscript checked. Conditional theory and representation diagnostics, not proof that every InfoNCE encoder is jointly Gaussian. Separate fixed-dimensional projection limits from a regularized population surrogate and coordinate-wise non-rejection from joint normality. This paper investigates when contrastive representations have approximately Gaussian low-dimensional marginals. It gives two different analytical routes: an alignment/uniformity argument under plateau and concentration assumptions, and an explicitly regularized population surrogate. The title is broader than a theorem that arbitrary InfoNCE training always produces jointly Gaussian embeddings. Official proceedings record · OpenReview Work: Which assumptions make the distributional claim possible, and what does a univariate normality test actually establish?
Exit evidence: Coordinate-wise non-rejection is not joint Gaussianity; the paper does not prove that arbitrary SGD reaches the described optimum.
2026
Conflicting supervision and adaptive distillation objectives
DTO-KD adapts the balance between task and teacher-transfer objectives using relative-loss optimization. Compare the exact log-gradient construction with the practical amortized update; control the transfer architecture before attributing gains to optimization.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 21.01 DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge DistillationOfficial ICLR 2026 oral and proceedings manuscript checked. Adaptive task/distillation optimization, not cross-magnification or residual-error KD. Separate the exact two-gradient construction from the amortized implementation; the printed simplex handling and lower-bound argument require qualification, documented in the source-checked review. DTO-KD treats task supervision and teacher imitation as a multi-objective optimization problem. Its central idea is to adapt their contribution according to relative loss improvement instead of choosing one fixed task/distillation weight. The paper distinguishes conflicting gradient directions from unequal gradient magnitudes, but its exact analytical construction and practical amortized algorithm must not be treated as identical. Official proceedings record · OpenReview · Theoretical supplement Work: Which objectives conflict, which update is actually implemented, and what costs and guarantees belong to that implementation?
Exit evidence: The printed closed form omits simplex boundary handling and the claimed non-collapse lower bound has a counterexample. Natural-image results do not establish cross-magnification or WSI validity.
2021–2026
Channel distributions and feature-pyramid transfer
CWD normalizes spatial activations inside each channel; CSCWD adds teacher P2 to student P3 alignment for tiny-object detection. Study what the spatial probability distribution preserves before changing feature resolution.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 22.01 Channel-Wise Knowledge Distillation for Dense PredictionICCV 2021 proceedings paper and author v4 manuscript checked; older missing prerequisite added during the October 2026 audit, not a new 2026 paper. Official repository records the semantic-segmentation code release on 20 August 2021. Channel-wise KD changes the axis on which dense-prediction activations become probabilities. Instead of normalizing classes separately at each pixel, it normalizes spatial positions separately within each channel. The student then matches teacher spatial distributions, encouraging it to preserve where each channel concentrates its evidence. Official implementation · Author manuscript Work: Are the two models seeing different acquisitions, or the same image through different feature maps?
Exit evidence: CSCWD uses shared-resolution image inputs, not 20×/5× tissue pairs. Its recall gain coexists with MGD’s stronger mAP and a non-improving external strict mAP.
2026
Which feature directions should distillation preserve?
SpecMatch balances reconstruction along teacher principal components rather than letting high-variance directions dominate an L2 objective. Low variance need not mean low downstream value.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 23.01 The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation DistillationarXiv v1 checked 2 October 2026. No official code/model release or conference acceptance verified. Camelyon17 is patch-level evaluation, not whole-slide cross-magnification validation. PE teacher/student labels and parameter counts differ between main text/tables and Appendix C; this review uses unambiguous DINOv2-to-MobileNet Table 1 rows. SpecMatch changes how feature distillation allocates its learning signal. A fixed teacher PCA basis separates representation directions, and a sublinear function of each direction’s reconstruction error reduces the dominance of high-variance coordinates. The aim is to preserve useful low-variance information, not merely make the total feature MSE small. — Work: Does a smaller global feature error hide worse transfer on discriminative but low-variance directions?
Exit evidence: Patch-level Camelyon17 is not a WSI endpoint. Spectral balancing is not evidence that unobserved high-magnification information can be recovered.
2026
Distilling reusable MIL initializations
TITAN and CARE supervise nine lightweight MIL aggregators over regular ROI bags. Teacher-specific projections and dispersion-normalized alignment precede linear probing or supervised fine-tuning.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 24.01 Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation ModelsarXiv v1 (2026-07-16) checked 2026-10-02. Six cohorts cover 15 tasks, not 15 independent datasets. Official code and pretrained aggregator checkpoints are listed; experiments were not rerun. This transfers slide embeddings into MIL initialization, not 20×-to-5× acquisition. Pretrain the MIL aggregator, not only its tile encoder. A small, initially headless MIL network learns to reproduce complementary TITAN and CARE region embeddings. The resulting aggregator weights initialize multiple downstream slide tasks; this is reusable representation transfer, not a 20×-to-5× acquisition method. Versioned manuscript · Official implementation and pretrained MIL weights Work: What improves because of pretraining, and does that gain survive a matched architecture, encoder and tuning budget?
Exit evidence: Six cohorts contain 15 tasks; more teachers do not always win. Main-text angular loss and appendix pseudocode differ, and aggregator pretraining does not itself reduce acquisition magnification.
2026
Distilling visual context and sensitivity to evidence
SCOPD teaches a sparse-context student from a full-context EMA teacher on student rollouts. CW-OPD instead tests how predictions change when answer-critical visual evidence is edited between two worlds.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 25.01 Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language ModelsPreprint checked 2026-10-02. The paper calls the method CW-OPD; its linked code repository uses the name CWAD. No conference acceptance inferred. CW-OPD supervises how a VLM prediction changes when answer-critical visual evidence changes. It combines ordinary per-world distillation with a cross-world belief-transition target, rather than assuming a correct answer proves the student used the right evidence. Versioned full text · Author implementation (repository name CWAD) Work: Does the student learn visual evidence, or merely the teacher’s answer distribution?
Exit evidence: Token pruning and counterfactual image editing are not WSI acquisition changes. Track EMA updates, teacher-only hints, editor quality and training cost before comparing distillation methods.
2026
Token-level direction and teacher consensus
BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 26.01 Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation. BinaryOPD replaces the magnitude of a teacher-student log-probability reward with its sign. The paper then uses high-disagreement token analysis to motivate C-MOPD, which combines teachers while declining strongly conflicting updates. Versioned full text · Author implementation Work: Which disagreement carries useful information, and when should conflicting teachers produce no update?
Exit evidence: Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost.
2026
Density-weighted residual alignment in hybrid VLMs
HEED aligns teacher and hybrid-student decoder residual streams, weighting visual positions by frozen-ViT local self-dissimilarity. This density proxy is not semantic saliency. Distinguish the hybrid architecture’s inference benefit from density weighting’s incremental benefit in a matched training recipe.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 27.01 HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model DistillationOfficial NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested. HEED protects locally distinctive visual positions when converting an attention-based VLM to a hybrid Mamba-2/attention model. Its contribution is density-weighted residual alignment: the student matches the teacher's residual stream more strongly at patches whose frozen visual features differ from their neighbors. The deployed architecture is unchanged by this loss; HEED does not delete visual tokens. Versioned full text · Official acceptance identity · Official NeurIPS program Work: Does density weighting outperform uniform residual alignment when data, trainable parameters and architecture are held fixed?
Exit evidence: The C3→C4 OCRBench v2 gain is 4.7 points; the larger headline change includes other training changes. The manuscript disagrees about Stage 3 trainable parameters. Neither the speedup nor this feature residual establishes WSI cross-magnification transfer.
2026
Numerical objectives: preserve the loss gradient, not just the loss value
A Fix for Softmax and Cross-Entropy separates hard-target absorption from near-equal teacher/student cancellation. Excluding the target from log-sum-exp and using softplus protects hard-label gradients; a logit-difference construction with log1p/expm1 repairs soft-target derivatives. The intended objective is unchanged.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 28.01 A Fix for Softmax and Cross-EntropyInspected arXiv:2610.31642v1, submitted 22 October 2026, and the author project page. Numerical-gradient and modular-arithmetic experiments are not evidence of improved WSI or large-scale foundation-model accuracy. No venue acceptance was verified. This paper repairs floating-point evaluation of familiar objectives rather than proposing a new classification target. Softmax Collapse loses the dominant class gradient at high confidence; Softmax Cancellation loses the small difference between nearly matching student and teacher distributions. Separate algebraic fixes preserve the intended cross-entropy gradient in these two regimes. Inspected manuscript (v1) · Author project and JAX implementations Work: Does the actual dtype and computation graph preserve high-margin and teacher-matched gradients?
Exit evidence: Evidence is numerical and small modular-arithmetic training, not WSI or large-scale foundation-model improvement. Higher precision and zero-sum projection are not automatically equivalent fixes. Check the real compiler and temperature convention before changing a training protocol.
2022–2026
Distillation projectors: dimension matching versus optimization interference
A projector can separate the student’s task representation from the space used to imitate a teacher even when both feature dimensions match. PEFD averages parallel nonlinear projector outputs before direction matching; the extended study adds logit, representation-similarity and calibration analyses.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 29.01 Understanding the Effects of Projectors in Knowledge DistillationIEEE publisher metadata identifies TPAMI 48(8), 9078–9092, August 2026. The accessible scientific text checked here is arXiv:2310.17183v1 (26 October 2023); final journal numerical identity was not established. This extends the authors’ NeurIPS 2022 Projector Ensemble Feature Distillation work, rather than originating PEFD in 2026. A distillation projector can do more than resize a feature vector: it can buffer the conflict between learning the label task and matching a teacher. This study tests projectors even when dimensions already agree, extends the idea to logits, and analyzes a parallel projector ensemble. The scientific summary follows the accessible 2023 manuscript; the catalog also identifies the 2026 journal publication. Inspected author manuscript (v1) · Author code and earlier PEFD implementation Work: Does the alignment head protect useful student features, or merely fit the teacher target more closely?
Exit evidence: PEFD originated in 2022. The accessible 2023 manuscript supports the reviewed scientific details; final 2026 journal numerical identity was not verified. More projectors or greater depth do not guarantee improvement.
2026
Global semantic anchors and local patch targets
Two Global Crops Suffice separates same-image global-view alignment from local crops and iBOT refinement in controlled DINO-family experiments. CODI instead changes the patch target: weakly mix the same teacher view’s class distribution into its local patch distribution. The diagnostic study and target-side method answer related questions, but neither independently validates the other.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 30.01 CODI: Improving Self-Supervised Vision Transformers with Cross DistillationarXiv:2610.30639v1, submitted 22 October 2026, inspected 26 October 2026. The manuscript has a NeurIPS 2026 footer, but no official acceptance record was independently located in this audit; retained as a preprint. The author repository contains only a Coming soon README, not released implementation code. CODI changes a masked patch’s teaching target, not the Vision Transformer architecture. It mixes 90% of the momentum teacher’s local patch distribution with 10% of that same view’s global class-token distribution. This gives low-attention patches a direct semantic training signal while retaining mostly local supervision. It is self-distillation during unlabeled image pretraining, not compression from a separately pretrained large teacher and not cross-magnification transfer. Inspected manuscript (v1) · Author repository — implementation pending at inspection Work: Which signal creates useful spatial semantics, which refines it, and do your global and dense evaluations agree?
Exit evidence: Both inspected sources are preprints. Crop ablations match iterations, not all computation; the depth-scale narrative has an appendix exception. CODI’s gradient statement is local/logit-level, its alpha is task-dependent, and its code was not released at inspection. Neither reports WSI or cross-magnification validation.
Research branches
Probability and calibration
Does the loss elicit a calibrated probability, a score, or only a decision boundary?
Composite Binary Losses · On Calibration of Modern Neural Networks · PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions
Imbalance and prior shift
Is the intervention changing gradients, class margins, teacher bias, the learned prior, or the final threshold?
Focal Loss for Dense Object Detection · Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss · Class-Balanced Loss Based on Effective Number of Samples · Long-Tail Learning via Logit Adjustment · Distilling Balanced Knowledge from a Biased Teacher
Distribution discrepancy
What probability geometry is assumed by KL, Wasserstein, or another divergence?
Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Metric and contrastive learning
What defines positive, negative, distance, margin, temperature, and batch context?
Dimensionality Reduction by Learning an Invariant Mapping · FaceNet: A Unified Embedding for Face Recognition and Clustering · ArcFace: Additive Angular Margin Loss for Deep Face Recognition · Supervised Contrastive Learning · FILIP: Fine-grained Interactive Language-Image Pre-Training · CoCa: Contrastive Captioners are Image-Text Foundation Models · Sigmoid Loss for Language Image Pre-Training
Regions, boxes, and boundaries
Does the surrogate match pixel probability, region overlap, box overlap, contour distance, or instance utility?
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation · Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations · The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks · Generalized Intersection Over Union: A Metric and A Loss for Bounding Box Regression · Boundary loss for highly unbalanced segmentation
Noisy supervision and robust residuals
Which noise or residual model and clean-data trade-off support the robustness claim?
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels · Symmetric Cross Entropy for Robust Learning with Noisy Labels · A General and Adaptive Robust Loss Function · The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
Multimodal and preference alignment
Which pairs or preferences are assumed correct, independent, representative, and supported by a reference?
Learning Transferable Visual Models From Natural Language Supervision · Sigmoid Loss for Language Image Pre-Training · Direct Preference Optimization: Your Language Model is Secretly a Reward Model · Model Alignment as Prospect Theoretic Optimization · SimPO: Simple Preference Optimization with a Reference-Free Reward · FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment
Structured clinical and weak supervision
What hierarchy, censoring process, competing event, or evidence-spread constraint is encoded?
DeepHit: A Deep Learning Approach to Survival Analysis with Competing Risks · AEM: Attention Entropy Maximization for Multiple Instance Learning based Whole Slide Image Classification
Rank-aware ordinal WSI objectives
Does the objective use the ordering between grades, and is any gain separable from added local/non-local WSI context?
Cross-space alignment objectives
Which probability or representation space makes the compared objects commensurate?
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization · ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation
Physical observability and task-aware polarization reconstruction
Is the supervision a physically observable target, a synthetic reconstruction target, a downstream task signal or a teacher distribution?
Stokes-Informed Diffusion for Robust Linear Polarization Estimation · PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications
Anatomical token selection and staged CT report generation
Can an early abnormality proposal remove evidence needed by the final report, and how is that failure measured?
Physiology-aware cross-modal spectral distillation
Which teacher signal is physiologically invariant across sensors, and what labeled validation budget trains the transfer policy?
Low-resolution student inference versus resolution-cascaded generation
Is high resolution absent at deployment, or only used for fewer steps? What is the full-resolution reference?
GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT Tracking
Teacher-class mixtures and privileged residual routing
Which privileged information is available only during training, and exactly what remains in the student?
MoE-KD: Your Teacher Model is Worth Mixture-of-Experts for Knowledge Distillation
Spatial-frequency and response-token distillation targets
Are the grids, semantic units and probability supports genuinely aligned before applying the loss?
SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction · Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Masked generation and diffusion-friendly latent spectra
Which representation is easier to model, and what fidelity or sampling cost is exchanged for that advantage?
Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
Real-data correction in generative distillation
Where does real data enter the teacher–student system, and which extra training stages are included in the comparison?
Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
Distributional geometry of contrastive representations
Which assumptions make the distributional claim possible, and what does a univariate normality test actually establish?
Conflicting supervision and adaptive distillation objectives
Which objectives conflict, which update is actually implemented, and what costs and guarantees belong to that implementation?
DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
Channel distributions and feature-pyramid transfer
Are the two models seeing different acquisitions, or the same image through different feature maps?
Which feature directions should distillation preserve?
Does a smaller global feature error hide worse transfer on discriminative but low-variance directions?
Distilling reusable MIL initializations
What improves because of pretraining, and does that gain survive a matched architecture, encoder and tuning budget?
Distilling visual context and sensitivity to evidence
Does the student learn visual evidence, or merely the teacher’s answer distribution?
Token-level direction and teacher consensus
Which disagreement carries useful information, and when should conflicting teachers produce no update?
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Density-weighted residual alignment in hybrid VLMs
Does density weighting outperform uniform residual alignment when data, trainable parameters and architecture are held fixed?
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
Numerical objectives: preserve the loss gradient, not just the loss value
Does the actual dtype and computation graph preserve high-margin and teacher-matched gradients?
Distillation projectors: dimension matching versus optimization interference
Does the alignment head protect useful student features, or merely fit the teacher target more closely?
Understanding the Effects of Projectors in Knowledge Distillation
Global semantic anchors and local patch targets
Which signal creates useful spatial semantics, which refines it, and do your global and dense evaluations agree?
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Proper probability loss | Targets truthful conditional probabilities under stated sampling assumptions. | Reweighting or sampling can change the implied target prior. | 2010 |
| Margin / metric / contrastive | Shapes relative geometry, ranking, separation, or angular class margins. | Good embeddings need not yield calibrated probabilities or robust open-set thresholds. | 2006 · 2015 · 2019 · 2020 |
| Region / box / boundary surrogate | Optimizes region overlap, box IoU, or distance-weighted contour agreement. | Empty classes, smoothing, distance convention, scale, scheduling, and thresholding can dominate. | 2016 · 2017 · 2018 · 2019 · 2019 |
| Robust / rebalanced | Changes sample, class, or teacher influence under label noise, outliers, or long tails. | A gain at one synthetic corruption rate or prior may not transfer. | 2018 · 2019 · 2019 · 2019 · 2019 · 2021 · 2026 |
| Probability discrepancy | Chooses the geometry used to compare teacher and student distributions. | A richer discrepancy can add approximation assumptions and computational cost. | 2024 |
| Pair / preference alignment | Optimizes relative compatibility, token alignment, or preference likelihood. | False negatives, annotator bias, length effects, and reference support are hidden supervision. | 2022 · 2023 · 2023 · 2024 · 2024 |
| Structured outcome / attention regularizer | Encodes censoring, competing risks, time ranking, or the spread of weakly supervised evidence. | A structurally plausible loss can still misalign with clinical calibration or localization faithfulness. | 2018 · 2025 |
| Rank-aware ordinal evidence | RACR-MIL couples rank-aware ordinal attention with hybrid local and non-local WSI context for squamous-cell-carcinoma grading. | Architecture changes, pooled anatomy, class imbalance, or a small reader pilot can be credited to the ordinal loss without an isolated objective ablation. | 2026 |
| Residual probability mass and representation-aligned flow objectives | Uses anchor mass for incompatible vocabularies and frozen visual semantics for conditional generation. | Loss improvements can reflect a convenient but lossy mapping rather than better task evidence. | 2026 · 2026 |
| Physical observability and task-aware polarization reconstruction | GenPolar predicts Stokes components from intensity using observability-aware supervision and one-step diffusion distillation. PolarAPP instead starts from measured polarization mosaics and couples demosaicking with downstream objectives through a learned metric. Keep generative priors separate from sensor-acquired polarization. | Weak-polarization angle masks change the evaluated population. Task-aware meta-learning is not automatically teacher–student KD, and metric improvements need not agree. | 2026 · 2026 |
| Anatomical token selection and staged CT report generation | A coarse abnormality language model guides structure-specific evidence selection for a second report model. Training-only anatomical masks and a downstream report reward align the stages. Inspect which component GRPO updates and which signals come from segmentation supervision. | Labeler-based clinical efficacy scores are proxy metrics, not clinical validation. The linked repository documents an earlier method; a matching release is not yet verified. | 2026 |
| Physiology-aware cross-modal spectral distillation | RPM-Distill transfers peak position, off-peak suppression and spectral morphology from an RF teacher to an RGB student. A validation-driven policy chooses sample and loss weights; deployment removes the radar branch. Compare this privileged-sensor design with feature KD and reliability-weighted transfer. | A supervised teacher and held-out labels are still required; the pipeline is not globally label-free. Policy residual blocks are not a residual-error KD target, and fusion methods retain different inference inputs. | 2026 |
| Low-resolution student inference versus resolution-cascaded generation | GAP-Track removes teacher and residual reconstruction branches at inference. RMD instead retains high-resolution generative refinement. Both exploit cross-resolution supervision but solve different deployment problems. | Synthetic downsampling does not reproduce all optical changes between microscope magnifications; lower-resolution gains need not equal full-resolution performance. | 2026 |
| Teacher-class mixtures and privileged residual routing | MoE-KD reformulates classification through latent teacher-class experts and EM. MARS uses full-minus-partial features to teach a deployable router. Specify whether knowledge resides in label mixtures, prototypes, routing decisions or a residual target. | Residual-guided routing is not a residual-error student cascade; conditional optimization arguments do not certify best accuracy. | 2026 |
| Spatial-frequency and response-token distillation targets | SFKD selectively aligns wavelet-localized Fourier amplitudes across backbone families. TRAG aligns response-to-vision attention using token-specific entropy-weighted KL. Both teach structure beyond final logits, but use different correspondence assumptions. | SFKD has architecture-specific regressions; TRAG shares a visual encoder and resolution. Neither establishes cross-magnification performance. | 2026 · 2026 |
| Masked generation and diffusion-friendly latent spectra | MIMFlow combines masked image modeling with normalizing flows; SSVAE changes the spectral structure of video latents to improve downstream diffusion. Separate tokenizer reconstruction quality, likelihood or generation objectives, and final sample quality. | Good reconstruction alone does not establish good diffusability, and normalizing flows are not interchangeable with flow matching. | 2026 |
| Real-data correction in generative distillation | RealUID unifies matching-model distillation and routes real examples through an auxiliary matching model. Read the alternating optimization and coefficient assumptions before interpreting the one-step generator results. | No GAN discriminator does not mean no min–max problem; low-resolution generation does not establish pathology validity. | 2026 |
| Distributional geometry of contrastive representations | InfoNCE Induces Gaussian Distribution separates a plateau/projection argument from an explicitly regularized surrogate. Its diagnostic table is a lesson in testing, rather than assuming, representation distributions. | Coordinate-wise non-rejection is not joint Gaussianity; the paper does not prove that arbitrary SGD reaches the described optimum. | 2026 |
| Conflicting supervision and adaptive distillation objectives | DTO-KD adapts the balance between task and teacher-transfer objectives using relative-loss optimization. Compare the exact log-gradient construction with the practical amortized update; control the transfer architecture before attributing gains to optimization. | The printed closed form omits simplex boundary handling and the claimed non-collapse lower bound has a counterexample. Natural-image results do not establish cross-magnification or WSI validity. | 2026 |
| Channel distributions and feature-pyramid transfer | CWD normalizes spatial activations inside each channel; CSCWD adds teacher P2 to student P3 alignment for tiny-object detection. Study what the spatial probability distribution preserves before changing feature resolution. | CSCWD uses shared-resolution image inputs, not 20×/5× tissue pairs. Its recall gain coexists with MGD’s stronger mAP and a non-improving external strict mAP. | 2021 |
| Which feature directions should distillation preserve? | SpecMatch balances reconstruction along teacher principal components rather than letting high-variance directions dominate an L2 objective. Low variance need not mean low downstream value. | Patch-level Camelyon17 is not a WSI endpoint. Spectral balancing is not evidence that unobserved high-magnification information can be recovered. | 2026 |
| Distilling reusable MIL initializations | TITAN and CARE supervise nine lightweight MIL aggregators over regular ROI bags. Teacher-specific projections and dispersion-normalized alignment precede linear probing or supervised fine-tuning. | Six cohorts contain 15 tasks; more teachers do not always win. Main-text angular loss and appendix pseudocode differ, and aggregator pretraining does not itself reduce acquisition magnification. | 2026 |
| Distilling visual context and sensitivity to evidence | SCOPD teaches a sparse-context student from a full-context EMA teacher on student rollouts. CW-OPD instead tests how predictions change when answer-critical visual evidence is edited between two worlds. | Token pruning and counterfactual image editing are not WSI acquisition changes. Track EMA updates, teacher-only hints, editor quality and training cost before comparing distillation methods. | 2026 |
| Token-level direction and teacher consensus | BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals. | Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost. | 2026 |
| Density-weighted residual alignment in hybrid VLMs | HEED aligns teacher and hybrid-student decoder residual streams, weighting visual positions by frozen-ViT local self-dissimilarity. This density proxy is not semantic saliency. Distinguish the hybrid architecture’s inference benefit from density weighting’s incremental benefit in a matched training recipe. | The C3→C4 OCRBench v2 gain is 4.7 points; the larger headline change includes other training changes. The manuscript disagrees about Stage 3 trainable parameters. Neither the speedup nor this feature residual establishes WSI cross-magnification transfer. | 2026 |
| Numerical objectives: preserve the loss gradient, not just the loss value | A Fix for Softmax and Cross-Entropy separates hard-target absorption from near-equal teacher/student cancellation. Excluding the target from log-sum-exp and using softplus protects hard-label gradients; a logit-difference construction with log1p/expm1 repairs soft-target derivatives. The intended objective is unchanged. | Evidence is numerical and small modular-arithmetic training, not WSI or large-scale foundation-model improvement. Higher precision and zero-sum projection are not automatically equivalent fixes. Check the real compiler and temperature convention before changing a training protocol. | 2026 |
| Distillation projectors: dimension matching versus optimization interference | A projector can separate the student’s task representation from the space used to imitate a teacher even when both feature dimensions match. PEFD averages parallel nonlinear projector outputs before direction matching; the extended study adds logit, representation-similarity and calibration analyses. | PEFD originated in 2022. The accessible 2023 manuscript supports the reviewed scientific details; final 2026 journal numerical identity was not verified. More projectors or greater depth do not guarantee improvement. | 2026 |
| Global semantic anchors and local patch targets | Two Global Crops Suffice separates same-image global-view alignment from local crops and iBOT refinement in controlled DINO-family experiments. CODI instead changes the patch target: weakly mix the same teacher view’s class distribution into its local patch distribution. The diagnostic study and target-side method answer related questions, but neither independently validates the other. | Both inspected sources are preprints. Crop ablations match iterations, not all computation; the depth-scale narrative has an appendix exception. CODI’s gradient statement is local/logit-level, its alpha is task-dependent, and its code was not released at inspection. Neither reports WSI or cross-magnification validation. | 2026 |
Concept lineage
Dimensionality Reduction by Learning an Invariant MappingSupervised Contrastive Learning
Pairwise metric learning becomes a multi-positive batch objective for class-conditioned representations.
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image SegmentationThe Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks
Overlap optimization moves from a soft Dice ratio to a convex-extension surrogate for IoU.
The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural NetworksBoundary loss for highly unbalanced segmentation
Set-overlap optimization is complemented by a signed-distance term that distinguishes near-boundary from far-away errors.
Focal Loss for Dense Object DetectionClass-Balanced Loss Based on Effective Number of Samples
Hard-example modulation and class-frequency reweighting address different sources of imbalance.
Learning Transferable Visual Models From Natural Language SupervisionSigmoid Loss for Language Image Pre-Training
Global softmax coupling over image–text pairs is replaced by independent sigmoid pair classification.
Sigmoid Loss for Language Image Pre-TrainingFALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment
Scaling the pairwise loss exposes false-negative structure that must be modeled explicitly.
Dimensionality Reduction by Learning an Invariant MappingFaceNet: A Unified Embedding for Face Recognition and Clustering
Pairwise attraction and repulsion become mined triplets for identity embeddings.
FaceNet: A Unified Embedding for Face Recognition and ClusteringArcFace: Additive Angular Margin Loss for Deep Face Recognition
Sample-triplet geometry is replaced by a normalized classifier with an explicit additive angular margin.
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image SegmentationGeneralised Dice overlap as a deep learning loss function for highly unbalanced segmentations
Dice overlap is reweighted to reduce region-size bias in highly unbalanced segmentation.
Focal Loss for Dense Object DetectionLearning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
Example-difficulty modulation is contrasted with class-frequency-dependent decision margins.
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy LabelsSymmetric Cross Entropy for Robust Learning with Noisy Labels
Noise tolerance moves from an interpolation between CE and MAE to an explicit bidirectional entropy construction.
Direct Preference Optimization: Your Language Model is Secretly a Reward ModelModel Alignment as Prospect Theoretic Optimization
Pairwise reference-relative preferences are reformulated around prospect-theoretic desirable and undesirable outcomes.
Direct Preference Optimization: Your Language Model is Secretly a Reward ModelSimPO: Simple Preference Optimization with a Reference-Free Reward
Reference-policy dependence is removed in favour of a length-normalized implicit reward and target margin.
On Calibration of Modern Neural NetworksDistilling Balanced Knowledge from a Biased Teacher
Calibration concerns become explicit when a biased teacher redistributes knowledge across head, medium, and tail groups.
Failure modes to audit
- Selecting a loss on the test metric without a validation protocol.
- Treating class reweighting as calibrated prior correction.
- Comparing objectives with different augmentations, batch sizes, or negative pools.
- Using a boundary term alone without checking the empty-foreground stationary failure.
- Optimizing slide patches while reporting patient outcomes without hierarchical uncertainty.
Questions for every paper
- What statistical object does the loss elicit?
- Which examples receive the largest gradients?
- Does the training target match the deployment metric and prevalence?
- Which calibration, subgroup, and shift checks could falsify the improvement?
Complete maintained paper route
Section titled “Complete maintained paper route”61 canonical readings · 1995–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered by an explicit prerequisite-aware teaching sequence. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 61 readings in maintained order
Support-Vector Networks
Corinna Cortes and Vladimir Vapnik
Metadata checked
Dimensionality Reduction by Learning an Invariant Mapping
Raia Hadsell, Sumit Chopra, and Yann LeCun
Metadata checked
Composite Binary Losses
Mark D. Reid and Robert C. Williamson
Metadata checked
A Fix for Softmax and Cross-Entropy
Jonathan T. Barron, Dan Gnanapragasam, and Michael Tschannen
Inspected arXiv:2610.31642v1, submitted 22 October 2026, and the author project page. Numerical-gradient and modular-arithmetic experiments are not evidence of improved WSI or large-scale foundation-model accuracy. No venue acceptance was verified.
Official artifacts: Inspected manuscript (v1) · Author project and JAX implementations
Metadata checked
DeViSE: A Deep Visual-Semantic Embedding Model
Andrea Frome et al.
Metadata checked
FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, and James Philbin
Metadata checked
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi
Official artifacts: Official code
Metadata checked
Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M. Jorge Cardoso
Official artifacts: Open author manuscript
Metadata checked
Prototypical Networks for Few-shot Learning
Jake Snell, Kevin Swersky, and Richard Zemel
Metadata checked
MoE-KD: Your Teacher Model is Worth Mixture-of-Experts for Knowledge Distillation
Kuo Shi, Wenjie Zhu, Bo Peng
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The EM convergence discussion assumes an improving M-step; it is not a guarantee of global optimum for stochastic neural training. Table 3 values take precedence over inconsistent prose deltas. No runnable official implementation was verified in this pass.
Official artifacts: Official ECCV poster record
Metadata checked
Focal Loss for Dense Object Detection
Tsung-Yi Lin et al.
Metadata checked
On Calibration of Modern Neural Networks
Chuan Guo et al.
Metadata checked
Representation Learning with Contrastive Predictive Coding
Aaron van den Oord, Yazhe Li, and Oriol Vinyals
Metadata checked
SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction
Cuipeng Wang, Haipeng Wang
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. The linked official repository still says source code will be released soon. This is heterogeneous-architecture feature KD, not demonstrated cross-magnification transfer. The selected CIFAR-100 table includes a ViT-S→MobileNetV2 case where FBT is better.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
Zhilu Zhang and Mert R. Sabuncu
Metadata checked
DeepHit: A Deep Learning Approach to Survival Analysis with Competing Risks
Changhee Lee, William R. Zame, Jinsung Yoon, and Mihaela van der Schaar
Official artifacts: Official code
Metadata checked
The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks
Maxim Berman, Amal Rannen Triki, and Matthew B. Blaschko
Official artifacts: Official code
Metadata checked
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou
Official artifacts: Official implementation family
Metadata checked
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma
Official artifacts: Official code
Metadata checked
Generalized Intersection Over Union: A Metric and A Loss for Bounding Box Regression
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese
Official artifacts: Official project and implementations
Metadata checked
GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT Tracking
Shiyu Zhang, Tianyang Xu, Zhangyong Tang, He Wang, Xiao-Jun Wu, and Josef Kittler
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Table 1 governs the recreated LasHeR values; the quarter-resolution CKD SR differs between Table 1 (35.3) and Table 4 (35.5). The PGS angle/radial equations need origin and quadrant safeguards not specified in the displayed formulas. The repository contains implementation directories despite a remaining release-soon notice; code was inspected, not executed.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
Symmetric Cross Entropy for Robust Learning with Noisy Labels
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey
Official artifacts: Official code
Metadata checked
A General and Adaptive Robust Loss Function
Jonathan T. Barron
Official artifacts: Official PyTorch implementation
Metadata checked
Class-Balanced Loss Based on Effective Number of Samples
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie
Official artifacts: Official code
Metadata checked
Boundary loss for highly unbalanced segmentation
Hoel Kervadec, Jihene Bouchtiba, Christian Desrosiers, Eric Granger, Jose Dolz, and Ismail Ben Ayed
Official artifacts: Official code
Metadata checked
A Simple Framework for Contrastive Learning of Visual Representations
Ting Chen et al.
Metadata checked
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, and Qingming Huang
arXiv:2610.30639v1, submitted 22 October 2026, inspected 26 October 2026. The manuscript has a NeurIPS 2026 footer, but no official acceptance record was independently located in this audit; retained as a preprint. The author repository contains only a Coming soon README, not released implementation code.
Official artifacts: Inspected manuscript (v1) · Author repository — implementation pending at inspection
Metadata checked
InfoNCE Induces Gaussian Distribution
Roy Betser, Eyal Gofer, Meir Yossef Levi, Guy Gilboa
Official ICLR 2026 oral and full proceedings manuscript checked. Conditional theory and representation diagnostics, not proof that every InfoNCE encoder is jointly Gaussian. Separate fixed-dimensional projection limits from a regularized population surrogate and coordinate-wise non-rejection from joint normality.
Official artifacts: Official proceedings record · OpenReview
Metadata checked
Supervised Contrastive Learning
Prannay Khosla et al.
Official artifacts: Code
Metadata checked
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford et al.
Metadata checked
Long-Tail Learning via Logit Adjustment
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar
Metadata checked
FILIP: Fine-grained Interactive Language-Image Pre-Training
Lewei Yao et al.
Metadata checked
CoCa: Contrastive Captioners are Image-Text Foundation Models
Jiahui Yu et al.
Metadata checked
PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, and Dragomir Anguelov
Official artifacts: Author-linked EfficientNetV2 implementation
Metadata checked
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov et al.
Metadata checked
Mimicking Radiologists: A Coarse-to-Fine Framework with Structural Sparse Tokens for Dual-LLM Computed Tomography Report Generation
Hong Liu, Dong Wei, Yefeng Zheng, Xian Wu, and Liansheng Wang
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #9339. Publisher proceedings/version of record was not verified on 2026-09-12. The paper-linked CTRG repository README still describes the earlier Structure Observation Driven Image-Text Contrastive Learning work and LLaMA-2-7B. A matching dual-LLM/GRPO release was not verified; do not treat this link as reproduction evidence.
Official artifacts: ECCV Spotlight program · Conference manuscript · Paper-linked repository (earlier method; matching release unverified)
Metadata checked
Sigmoid Loss for Language Image Pre-Training
Xiaohua Zhai et al.
Metadata checked
Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Jiaming Lv, Haoyuan Yang, and Peihua Li
Official artifacts: Official project page · Official code
Metadata checked
RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurement
Jiyao Wang, Qingyong Hu, Duoxun Tang, Xiao Yang, Kaishun Wu, and Jiangbo Yu
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #10941. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript · Official implementation
Metadata checked
Model Alignment as Prospect Theoretic Optimization
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela
Official artifacts: Official HALOs code
Metadata checked
SimPO: Simple Preference Optimization with a Reference-Free Reward
Yu Meng, Mengzhou Xia, and Danqi Chen
Official artifacts: Official code and models
Metadata checked
AEM: Attention Entropy Maximization for Multiple Instance Learning based Whole Slide Image Classification
Yunlong Zhang, Honglin Li, Yuxuan Sun, Zhongyi Shui, Jingxiong Li, Chenglu Zhu, and Lin Yang
Official artifacts: Official code
Metadata checked
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Mark A. Hasegawa-Johnson, and Chang D. Yoo
ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Teacher and student share a frozen SigLIP vision encoder and resolution; this is not cross-resolution KD. The official code repository lists training/evaluation scripts as planned. Attention similarity correlations are not causal guarantees of faithful reasoning.
Official artifacts: Official ECCV poster record · Official code
Metadata checked
FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment
Myunsoo Kim, Seongwoong Shim, and Byung-Jun Lee
Metadata checked
The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
Shivang Chopra, Shaunak Halbe, Chengyue Huang, Brisa Maneechotesuwan, and Zsolt Kira
Metadata checked
ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation
Jeonghyeok Do, Seungchul Lee, and Munchurl Kim
arXiv v1 submitted 1 September 2026. The exact title follows the current arXiv record, superseding an earlier feed title. Code, weights, and an official project page were available; no peer-reviewed record was verified.
Official artifacts: Official implementation and checkpoints · Official project page
Metadata checked
Stokes-Informed Diffusion for Robust Linear Polarization Estimation
Yidong Luo, Chenggong Li, Yuchao Feng, Boxin Shi, Junchao Zhang, and Xin Yuan
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5749. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript
Metadata checked
PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications
Yidong Luo, Chenggong Li, Yunfeng Song, Ping Wang, Boxin Shi, Junchao Zhang, and Xin Yuan
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5713. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript
Metadata checked
Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
Shizhan Liu, Xinran Deng, Zhuoyi Yang, Jiayan Teng, Xiaotao Gu, and Jie Tang
ECCV 2026 Oral verified in the official program; scientific claims use the conference-hosted manuscript #7577. Publisher proceedings/version of record was not verified on 2026-09-12. The initial arXiv manuscript appeared in December 2025; year follows the 2026 conference.
Official artifacts: ECCV Oral program · Conference manuscript · Official implementation · arXiv manuscript · Model checkpoints
Metadata checked
Distilling Balanced Knowledge from a Biased Teacher
Seonghak Kim
Metadata checked
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, and Bing Qin
The arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026.
Official artifacts: Official implementation
Metadata checked
RACR-MIL: Rank-aware contextual reasoning for weakly supervised grading of squamous cell carcinoma using whole slide images
Anirudh Choudhary, Mosbah Aouad, Krishnakant Saboo, Angelina Hwang, Jacob Kechter, Blake Bordeaux, Puneet Bhullar, David DiCaudo, Steven Nelson, Nneka Comfere, Emma Johnson, Olayemi Sokumbi, Jason Sluzevich, Leah Swanson, Dennis Murphree, Aaron Mangold, and Ravishankar Iyer
The complete arXiv v3 manuscript identifies Medical Image Analysis as the journal and the abstract record reports acceptance; a final DOI or journal issue record was not yet available when checked on 30 August 2026.
Metadata checked
DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
Zeeshan Hayder, Ali Cheraghian, Lars Petersson, Mehrtash Harandi, Richard Hartley
Official ICLR 2026 oral and proceedings manuscript checked. Adaptive task/distillation optimization, not cross-magnification or residual-error KD. Separate the exact two-gradient construction from the amortized implementation; the printed simplex handling and lower-bound argument require qualification, documented in the source-checked review.
Official artifacts: Official proceedings record · OpenReview · Theoretical supplement
Metadata checked
Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
Nikita Kornilov, David Li, Tikhon Mavrin, Aleksei Leonov, Nikita Gushchin, Evgeny Burnaev, Iaroslav Koshelev, Aleksandr Korotin
Official ICLR 2026 oral and full proceedings manuscript checked. Three-network training yields a one-step generator. No GAN discriminator does not mean no min–max optimization. Selected low-resolution image results include additional fine-tuning; teacher-error correction has restrictive assumptions. No cross-magnification or pathology evaluation.
Official artifacts: Official proceedings record · OpenReview
Metadata checked
The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation
Kuniaki Saito and Yoshitaka Ushiku
arXiv v1 checked 2 October 2026. No official code/model release or conference acceptance verified. Camelyon17 is patch-level evaluation, not whole-slide cross-magnification validation. PE teacher/student labels and parameter counts differ between main text/tables and Appendix C; this review uses unambiguous DINOv2-to-MobileNet Table 1 rows.
Metadata checked
Channel-Wise Knowledge Distillation for Dense Prediction
Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen
ICCV 2021 proceedings paper and author v4 manuscript checked; older missing prerequisite added during the October 2026 audit, not a new 2026 paper. Official repository records the semantic-segmentation code release on 20 August 2021.
Official artifacts: Official implementation · Author manuscript
Metadata checked
Understanding the Effects of Projectors in Knowledge Distillation
Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, Brano Kusy, and Zi Huang
IEEE publisher metadata identifies TPAMI 48(8), 9078–9092, August 2026. The accessible scientific text checked here is arXiv:2310.17183v1 (26 October 2023); final journal numerical identity was not established. This extends the authors’ NeurIPS 2022 Projector Ensemble Feature Distillation work, rather than originating PEFD in 2026.
Official artifacts: Inspected author manuscript (v1) · Author code and earlier PEFD implementation
Metadata checked
HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
Yihao Liang and Niraj K. Jha
Official NeurIPS 2026 program snapshot lists Accept (poster), event 150667 and forum gNV0DzH5iz, checked 8 October 2026. Review uses arXiv v1; final proceedings and a public author implementation were not verified. Section 4.1 and Appendix C.3 disagree about Stage 3 trainable parameters. Inference speed belongs to the hybrid architecture, not HEED weighting; no WSI magnification transfer was tested.
Official artifacts: Versioned full text · Official acceptance identity · Official NeurIPS program
Metadata checked
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, and Xinbing Wang
Preprint checked 2026-10-02. The paper calls the method CW-OPD; its linked code repository uses the name CWAD. No conference acceptance inferred.
Official artifacts: Versioned full text · Author implementation (repository name CWAD)
Metadata checked
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, and Gao Huang
Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation.
Official artifacts: Versioned full text · Author implementation
Metadata checked
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Mingxi Fu, Jiawen Li, Renao Yan, Jiali Hu, Qiehe Sun, Tian Guan, and Yonghong He
arXiv v1 (2026-07-16) checked 2026-10-02. Six cohorts cover 15 tasks, not 15 independent datasets. Official code and pretrained aggregator checkpoints are listed; experiments were not rerun. This transfers slide embeddings into MIL initialization, not 20×-to-5× acquisition.
Official artifacts: Versioned manuscript · Official implementation and pretrained MIL weights
Metadata checked