Optimization, regularization, and representation learning
Freezing, fine-tuning, inner-loop updates, and stability penalties change which gradients can alter a representation.
Transfer, adaptation, and continual learning
This chapter asks a question that “fine-tuning” alone does not answer: what is allowed to change between a source model and its use in a new setting? The answer may involve the data distribution, available labels, task family, trainable parameters, adaptation compute, or sequence of experiences. Each choice defines a different scientific problem and a different fair baseline.
Start every experiment with a source-to-target contract:
source data + source objective + source checkpoint ↓target examples and labels available during adaptation ↓trainable state + update budget + selection rule ↓held-out target performance + retained competence + resource costDistinguish the learning settings
Section titled “Distinguish the learning settings”| Setting | What changes | Minimum evidence |
|---|---|---|
| representation transfer | task or domain after pretraining | frozen, partial, and full-tuning baselines under one split |
| domain adaptation | source and target distributions | explicit target-label access and source-versus-target evaluation |
| few-shot / meta-learning | new task with a small support set | held-out task families and a fixed adaptation budget |
| semi-supervised learning | label availability within a population | labelled-set size, unlabelled distribution, and pseudo-label audit |
| parameter-efficient fine-tuning | trainable parameterization | quality plus memory, storage, latency, and optimization cost |
| continual learning | ordered tasks or distributions | forgetting, order variation, forward transfer, and future plasticity |
These labels are not synonyms. LoRA restricts a parameter update; it does not by itself solve domain shift. DANN aligns domains; it does not establish few-shot generalization. A model that retains old accuracy can still lose the capacity to learn later tasks.
Reading and experiment contract
Section titled “Reading and experiment contract”For every paper, record:
- source and target populations, inputs, tasks, and label availability;
- checkpoint, frozen state, trainable state, and whether optimizer state is reused;
- examples, updates, hyperparameter trials, and target validation labels consumed;
- full fine-tuning, frozen representation, and same-budget adaptation baselines;
- shift, calibration, task order, forgetting, and resource measurements;
- which information will still be available when the system is deployed.
The detailed route below separates the branches, connects their landmark papers, and supplies an exit task for each stage.
31 maintained papers · literature reviewed through 2 October 2026
Transfer learning reuses parameters or representations; domain adaptation addresses a distribution shift; few-shot and meta-learning specify how rapidly a learner must specialize; semi-supervised learning changes label availability; parameter-efficient fine-tuning restricts which parameters move; and continual learning asks whether later training destroys earlier competence or future plasticity. These are related but not interchangeable claims. A valid study states the source and target distributions, labels available in each, adaptation budget, parameter access, task order, retained data, and evaluation protocol.
Prerequisite contract
Evaluation under shift and repeated selection
Target labels, validation reuse, task order, and adaptation compute can silently change the comparison.
How the field developed
2014–2016
Reusable features and domain-invariant representations
The transferability study asks how feature reuse changes across network depth and target distance. DANN then adds a gradient-reversal domain classifier so features remain predictive for the source task while becoming less predictive of source versus target membership. The second objective is domain adaptation, not a universal guarantee that every useful target factor should be invariant.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 How Transferable Are Features in Deep Neural Networks? Feature generality is measured operationally: copy the first n layers of a network trained on task A into a network for task B, freeze or fine-tune those layers, retrain the remaining layers, and compare target accuracy with matched controls. Early ImageNet features transfer well; later ones become task-specific. Middle-layer freezing can also fail because a cut breaks fragile co-adaptation, not because the features themselves are unusable. — 01.02 Domain-Adversarial Training of Neural Networks DANN learns features that predict source task labels while preventing a domain classifier from telling source from target. A gradient reversal layer is identity in the forward pass but multiplies the returning domain gradient by a negative factor. Ordinary backpropagation therefore minimizes source label loss and domain-classifier loss for its own head, while making the shared feature extractor maximize domain confusion. — Work: Compare frozen, partial, and full fine-tuning, then add adversarial alignment while holding target-label access fixed.
Exit evidence: State which layers transfer, which domain variable is suppressed, and whether target performance improves without hidden target supervision.
2017
Metric-based few-shot learning and optimization-based meta-learning
Prototypical Networks classify a query by distance to class means in an episodically learned embedding. MAML instead optimizes an initialization whose parameters can reach a task-specific solution with a few gradient steps. One learns a metric-and-prototype rule; the other learns for fast optimization, so their episode construction and adaptation budgets must be compared explicitly.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Prototypical Networks for Few-shot Learning A prototypical network turns each few-shot episode into a temporary classifier. It embeds the labeled support examples, averages embeddings within each class to form one prototype, embeds each query, and predicts with a softmax over negative distances to those prototypes. The learned object is therefore the embedding geometry; the test-time class representatives are computed, not stored classifier weights. — 02.02 Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks MAML does not directly learn one predictor for all tasks. It learns an initialization whose parameters can become a good task-specific predictor after one or a few ordinary gradient steps on a small support set. Meta-training differentiates query performance through that adaptation step, so the initialization is optimized for what happens after learning rather than only for its pre-update loss. — Work: Build identical train, validation, and held-out task families and compare nearest-prototype inference with fixed-step inner-loop adaptation.
Exit evidence: Separate examples per class, number of tasks, meta-validation choices, gradient steps, and transductive information.
2020
Semi-supervised adaptation through confident consistency
FixMatch creates a pseudo-label from a weakly augmented unlabelled example only when confidence exceeds a threshold, then trains the strongly augmented view toward that label. Its gain depends on calibration, augmentation validity, class balance, and compatibility between labelled and unlabelled populations.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence FixMatch combines pseudo-labeling and consistency regularization in one asymmetric operation: predict an unlabeled image under weak augmentation, retain its hard argmax only when the maximum probability crosses a threshold, and train the same model to reproduce that pseudo-label under strong augmentation. Labeled images still use ordinary cross-entropy. Confidence gates which unlabeled examples can teach the model. — Work: Sweep the confidence threshold and labelled-set size while auditing pseudo-label accuracy by class and data source.
Exit evidence: Report labelled and unlabelled distributions, augmentation operators, accepted pseudo-label rate, and per-class error.
2018–2023
From controlled fine-tuning to modular and low-rank adaptation
ULMFiT combines staged language-model fine-tuning, discriminative learning rates, and a slanted schedule. Adapters insert small trainable modules into a frozen backbone; MAD-X composes language and task adapters for cross-lingual transfer. LoRA represents selected weight updates with low-rank factors, while QLoRA backpropagates through a frozen 4-bit quantized base into LoRA parameters. Parameter count, optimizer state, activation memory, storage, throughput, and quality are distinct efficiency axes.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Universal Language Model Fine-tuning for Text Classification ULMFiT turns one pretrained recurrent language model into a general text-classification transfer recipe with three stages: general-domain language-model pretraining, target-domain language-model fine-tuning on unlabeled task text, and supervised classifier fine-tuning. Its central engineering insight is that transfer quality depends on controlling how quickly different layers change, not merely loading pretrained weights. — 04.02 Parameter-Efficient Transfer Learning for NLP Parameter-efficient adapters freeze one shared pretrained Transformer and learn small task-specific bottleneck modules inside every layer, plus task heads and layer-normalization parameters. A user can therefore store one BERT backbone and a thin parameter set per task instead of a complete fine-tuned copy, while retaining nearly the same benchmark quality in the paper's experiments. — 04.03 MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer MAD-X adapts a frozen multilingual Transformer by composing small language adapters with a task adapter. A language adapter learns unsupervised masked-language modeling for one language; a task adapter learns labeled supervision in a source language; inference swaps only the language component for the target. Invertible adapters around the embedding interface reduce mismatch between a shared multilingual vocabulary and languages that were rare or absent in pretraining. Official AdapterHub resource · Official adapters implementation 04.04 LoRA: Low-Rank Adaptation of Large Language Models LoRA keeps a pretrained dense weight W₀ frozen and learns only a low-rank update ΔW = BA. The base transformation and update see the same input, their outputs are added during training, and BA can be merged into W₀ for ordinary dense inference. The central claim is therefore parameter-efficient adaptation of selected weights—not compression of the frozen base model and not a new pretraining architecture. — 04.05 QLoRA: Efficient Finetuning of Quantized LLMs QLoRA stores a pretrained base model in 4-bit NormalFloat, dequantizes each needed weight block to BF16 for computation, keeps the base frozen, and backpropagates only into BF16 LoRA adapters attached throughout the transformer. Double quantization shrinks scale metadata and paged optimizers absorb transient optimizer-memory spikes. It is a memory-efficient fine-tuning procedure—not 4-bit gradient descent on all base weights. — Work: Adapt one backbone with full fine-tuning, adapters, LoRA, and quantized LoRA under matched data and selection budgets.
Exit evidence: Report trainable and total parameters, peak memory, checkpoint size, tokens or examples processed, latency, and target quality.
2016–2024
Retaining prior competence and preserving future plasticity
Learning without Forgetting preserves old-model outputs on new-task images without retaining the old files; EWC instead penalizes movement of parameters estimated important to previous tasks. Loss of Plasticity shows a complementary failure: prolonged training can also make a network progressively less able to learn new targets. Output preservation, parameter preservation, replay, retention, and future learnability are distinct.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 Learning Without ForgettingPeer-reviewed ECCV 2016 proceedings paper; the canonical locator is the open author manuscript. Learning without Forgetting adds a new task to a pretrained classifier without storing old task images. Before updating, the old model records its output distribution on the new-task images. Training then combines new-label supervision with a distillation loss that keeps those old responses stable while shared features change. It is old-data-free continual adaptation, not state-free learning: the pretrained backbone, old heads, and their responses remain essential. Official project · Official code 05.02 Overcoming Catastrophic Forgetting in Neural Networks Elastic Weight Consolidation learns a new task with its ordinary loss while adding parameter-wise springs back to an earlier solution. Each spring's stiffness is the estimated diagonal Fisher information for the old task: parameters judged important move slowly, while weakly constrained parameters remain plastic. It is selective regularization, not a frozen network and not replay of old examples. — 05.03 Loss of Plasticity in Deep Continual Learning Deep networks can retain access to current data yet progressively lose the ability to learn it well after long sequences of updates. The paper calls this loss of plasticity and separates it from catastrophic forgetting: forgetting asks whether old-task performance survives; plasticity asks whether the already-trained network can still acquire the present task as effectively as an appropriate fresh reference. Across supervised and reinforcement-learning settings, standard methods accumulate dormant units, reduced representational diversity, and large weights alongside declining learning ability. — Work: Run multiple task orders with fine-tuning, old-output distillation, replay, and parameter-importance penalties, then probe fresh-task learning after each stage.
Exit evidence: Plot average accuracy, forgetting, forward transfer, order variance, and fresh-task learning curves.
2016–2026
Privileged information, test-time adaptation, and selective transfer
Generalized distillation and safe prompts use training-only teachers; Tent updates normalization through target entropy; continual restoration isolates degradation adapters; LanGuSTE and PANDA remove or selectively acquire modalities at deployment, so availability contracts must remain explicit.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 Unifying Distillation and Privileged InformationICLR 2016 conference paper; canonical locator is the complete arXiv manuscript. Core mechanism: privileged information available only during training. The framework unifies learning using privileged information with teacher-student distillation: a teacher learns from richer training-only variables and transfers its soft predictions to a student that will deploy using ordinary inputs alone. This reading boundary matters: the framework explains when a training-only view can improve sample complexity under assumptions, not that every stronger modality or larger teacher will help every student. — 06.02 Distilling Safe LLM Systems via Soft Prompts for On Device SettingsPublished in Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337. Core mechanism: guard behavior compressed into a soft prompt. A frozen on-device language model receives a learned continuous prompt whose parameters are optimized to reproduce a safer guard-model system. The transfer target is behavior at the output distribution, so the base model can remain unchanged. This reading boundary matters: the result is a parameter-efficient behavioral transfer mechanism, not proof that the student inherits every policy, adversarial robustness property, or hidden capability of the guard system. — 06.03 Tent: Fully Test-Time Adaptation by Entropy MinimizationPublished at ICLR 2021; OpenReview is the canonical conference record. Core mechanism: entropy minimization at test time. TENT adapts a pretrained classifier during deployment by minimizing prediction entropy on unlabeled target batches while updating only normalization statistics and affine scale/shift parameters. Source examples are not revisited. This reading boundary matters: the method is an online adaptation rule for supported normalization-based classifiers, not a calibration guarantee or a source-free proof of robustness under arbitrary shifts. Official implementation 06.04 Restoring Without Forgetting: Continual Learning Across Image DegradationsarXiv v1 preprint submitted 24 August 2026; validation covers the reported five degradation domains rather than arbitrary composite corruption. Core mechanism: frozen restoration backbone with isolated low-rank adapters. The method freezes a shared restoration backbone, learns a separate low-rank adapter for each arriving degradation, stores feature prototypes for degradation recognition, and routes an input to the corresponding adapter so earlier domains are not overwritten. This reading boundary matters: the architecture avoids parameter overwrite across the five studied degradation domains, not automatic handling of unseen, mixed, continuously varying, or misrouted corruptions. Official implementation 06.05 LanGuSTE: Language-Guided Coarse-to-Fine Patch Selection for Efficient Whole Slide Image AnalysisarXiv preprint first submitted in 2025 and substantially revised as v2 on 24 August 2026; no peer-reviewed version or official implementation was verified by the review cutoff. LanGuSTE learns prompts that make 5× patches semantically resemble their aligned 20× children, then uses LLM-ranked pathology concepts and a pathology VLM to screen coarse regions before encoding only selected 20× patches. It is an acquisition-and-selection pipeline: low resolution guides where to spend high-resolution compute; it does not synthesize 20× features or eliminate 20× at deployment. — 06.06 PANDA: Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimer's MRI and TCGA PathologyarXiv v1 preprint submitted 26 August 2026. The manuscript promises code upon acceptance, but no public implementation was available at the review cutoff. PANDA converts incomplete auxiliary modalities into frozen class-level anchors, then aligns every primary-modality sample to those anchors even when that individual has no paired auxiliary data. In the pathology instantiation, class/event prototypes derived from bulk RNA guide a UNI2-h plus ABMIL WSI model during training; RNA encoders are discarded, so survival prediction uses histology alone at inference. — Work: For each method, write the source and target distributions, labels, auxiliary modalities, target-time access, parameters updated, task order, retained data, and high-resolution evidence acquired; compare source-only, matched adaptation, replay, and oracle controls where valid.
Exit evidence: Separate training-only privileged transfer, online target adaptation, continual degradation adapters, selective high-resolution acquisition, and missing-modality deployment; report leakage, forgetting, plasticity, acquisition cost, and external shift.
2022–2026
Residual boosting, fast weights, normalized adapters, and frozen transport
FOSTER boosts a class-incremental residual before compressing the expanded model; Fast Weight Attention makes causal online updates explicit; NoRA regularizes LoRA down-projections; CASTER transports frozen source statistics without gradients; KDTViT sequentially retains magnification knowledge.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 FOSTER: Feature Boosting and Compression for Class-Incremental LearningPublished in the ECCV 2022 main conference proceedings, pages 398–414. FOSTER expands a class-incremental model with a new feature module trained to correct residual classification error, then compresses the expanded teacher back to one backbone with class-balanced knowledge distillation. Residual fitting learns new capacity; distillation prevents deployment width from growing after every task. Official implementation 07.02 Fast Weight Attention for Continual LearningarXiv v1 submitted 27 August 2026. The official implementation was available when checked on 2 September 2026; no peer-reviewed record was verified. Core idea: treat a recurrent fast-weight state as an online learner and align each write with the prefix that actually predicts the newly revealed value. For the paper's autoregressive objective, the correct local example pairs the previous key feature with the current value; this read-after-write alignment leads to normalized Falcon update families that separate plasticity, forgetting, and bounded rehearsal. Official implementation 07.03 Normalized Low-Rank AdaptationarXiv v1 submitted 31 August 2026. The authors released an implementation; no peer-reviewed record was verified on 2 September 2026. Core idea: normalize LoRA's down-projection along the rank dimension so the input-to-latent map begins, and optionally remains, at a controlled scale. Because the up-projection starts at zero, early gradients depend strongly on this down-projection geometry. NoRA changes optimization conditioning without adding trainable parameters or inference computation, and NoRA-init applies the same constraint only once at initialization. Official implementation 07.04 Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell YouarXiv v1 submitted 31 August 2026. The official CASTER repository was available when checked on 2 September 2026; no peer-reviewed record was verified. Core idea: adapt a completely frozen classifier by transporting its stored source class distributions through a single target-batch affine map. CASTER estimates that map from unlabeled target moments in a discriminative subspace, classifies under the transported Gaussian model, and abstains from transport when a residual-to-margin diagnostic says the observed batch does not support the mechanism. Official implementation 07.05 KDTViT Knowledge Distillation, Transfer Learning and Transformer Based Deep Learning Framework for Efficient Histopathology Image ClassificationPeer-reviewed open-access journal article 16:20721. A MobileViT-XS is sequentially adapted from 40× through 400× using the previous checkpoint as teacher; BreakHis consists of biopsy images rather than WSIs, and the same-capacity teacher/student design targets retention rather than model compression. KDTViT trains one MobileViT-XS through a sequence of BreakHis magnifications: learn at 40×, freeze that checkpoint as teacher for 100×, then repeat for 200× and 400×. Each next student has the same architecture and learns labels plus soft predictions and final-feature matching, so the method targets progressive retention rather than compression. Open-access article Work: Declare task order, online observations, labels, replay, state size, trainable parameters, stopping or gating rules, and what remains frozen.
Exit evidence: Report forward and backward cost, forgetting, plasticity, state growth, calibration, and shift-specific failures under matched streams.
2026
Few-shot transduction over frozen WSI features
SlideCRF turns sparse pathologist clicks or scribbles into a graph-transductive labeling problem over frozen vision-language features, including protocols where some classes are absent from the annotations.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 Whole-Slide Image Analysis under Realistic Few-Shot Annotation ProtocolsarXiv v1 submitted 31 August 2026; the manuscript title page says it was submitted on 29 July 2026. Code and data instructions were released, but no peer-reviewed record was verified on 2 September 2026. Core idea: convert every WSI into a patch graph, use a pathology VLM for zero-shot unary evidence, and let a few class-localized clicks or scribbles steer conditional-random-field inference over all patches on that same slide. SlideCRF combines annotation similarity, label diversity, spatial continuity, texture and color affinity, plus a cautious absent-class penalty. It is transductive refinement of one slide—not training a reusable slide classifier from a handful of labels. Official implementation and data instructions Work: Fix the frozen encoder, annotation unit and budget, graph construction, class-presence assumptions, and WSI split before comparing few-shot adaptation.
Exit evidence: Report performance versus annotation effort, absent-class failure, slide/site shift, and end-to-end interaction cost.
2025–2026
Physical observability and task-aware polarization reconstruction
GenPolar predicts Stokes components from intensity using observability-aware supervision and one-step diffusion distillation. PolarAPP instead starts from measured polarization mosaics and couples demosaicking with downstream objectives through a learned metric. Keep generative priors separate from sensor-acquired polarization.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 PolarAPP: Beyond Polarization Demosaicking for Polarimetric ApplicationsECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5713. Publisher proceedings/version of record was not verified on 2026-09-12. PolarAPP jointly optimizes polarization demosaicking and a downstream task using a meta-learned feature-alignment metric. An equivalent-imaging self-consistency constraint improves reconstruction without treating rearranged mosaic pixels as perfect targets. Training-time alignment modules disappear at inference, leaving a demosaicker, analytic polarization cues and the task network. ECCV Spotlight program · Conference manuscript Work: Is the supervision a physically observable target, a synthetic reconstruction target, a downstream task signal or a teacher distribution?
Exit evidence: Weak-polarization angle masks change the evaluated population. Task-aware meta-learning is not automatically teacher–student KD, and metric improvements need not agree.
2025–2026
Atlas-conditioned foundation segmentation and reliability fusion
A single annotated volume can generate spatial prompts and an independent registered-mask prediction. AtlasSegFM learns a voxel-wise fusion gate through support-space consistency while leaving the foundation backbone frozen. Read it as a one-shot supervision budget with substantial test-time adaptation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 10.01 Atlas is Your Perfect Context: One-Shot Customization for Generalizable Foundational Medical Image SegmentationECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #2028. Publisher proceedings/version of record was not verified on 2026-09-12. AtlasSegFM customizes a frozen segmentation foundation model using one annotated atlas. It registers atlas anatomy to the query, converts the warped mask into prompts, and combines atlas and foundation-model predictions with a voxel-wise reliability gate. One-shot refers to the support annotation budget, not absence of test-time optimization. ECCV Spotlight program · Conference manuscript Work: What is labeled, what is optimized for each query, and can the gate be trained without using query ground truth?
Exit evidence: Anatomical mismatch can corrupt both branches; interactive methods receiving ground-truth-derived clicks have a different information budget. CT/MRI anatomy is not WSI validation.
2025–2026
Readable prompt ensembles, local concepts and selective adaptation
TPB reweights mistakes across discrete prompt classifiers, CFM exposes spatially grounded sparse concepts, and Robotic Steering selects action-predictive attention heads. Compare what is selected or trained, how frozen features are reused, and whether the final intervention preserves unrelated capabilities.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 11.01 AdaBoosting Text Prompts for Vision-Language ModelsECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #5256 checked on 2026-09-12. Publisher version of record was not independently verified. Text Prompt Boosting (TPB) adapts a frozen CLIP-style classifier by sequentially constructing readable prompt banks that focus on previously misclassified images. The weak learners are entire class-wise text-prompt classifiers; their confidence outputs are combined through SAMME.R boosting. The result is an error-correcting ensemble in language space, not a smaller student trained to copy teacher features. ECCV Spotlight program · Conference manuscript · Official project or implementation 11.02 Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot DemonstrationsECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #13378 checked on 2026-09-12. Publisher version of record was not independently verified. Robotic Steering chooses which VLA attention heads to adapt using a few demonstrations. Head activations are scored by how well cross-trajectory nearest-neighbor retrieval predicts demonstrated actions. Low-rank updates then target selected query and output projections. The central contribution is data-driven adaptation location, not a new distillation loss or an inference-time steering vector. ECCV Spotlight program · Conference manuscript · Official project or implementation Work: Is the adaptation unit a prompt bank, a concept dictionary, an attention head, or another trainable branch?
Exit evidence: Frozen backbones do not imply label-free or zero-cost adaptation. Probe predictability is not causal proof, and Robotic Steering also trains FFN LoRA and its action head.
2025–2026
Historical adapter anchors and analytic continual classifiers
Historical LoRA anchors constrain encoder drift while text-guided accumulated statistics update a classifier without raw-example replay. This is a teacher-state and classifier-memory contract, not the residual-error cascade used by other KD methods.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 12.01 Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual LearningECCV 2026 Spotlight verified via the official related presentation event; the poster identity itself says Accept Poster. No Springer version of record or public runnable implementation was verified. Section 3.5's printed closed-form equation omits a term implied by its objective; the review flags this reproducibility issue without silently repairing the authors' formula. Multi-anchor distillation restrains accumulated CLIP feature drift while a text-guided analytic classifier avoids gradient-based overwriting of classifier weights. Historical LoRA states supply visual anchors; accumulated sufficient statistics and frozen text prototypes supply the classifier. This exemplar-free approach stores model state and statistics rather than old raw images. Official ECCV 2026 Spotlight Work: Which stored states prevent forgetting, and does the analytic argument still apply when the feature encoder changes?
Exit evidence: Stable-feature equivalence is conditional; check the printed objective/closed-form mismatch and separate average from final accuracy.
2025
Locate visual concepts and align representations across models
PatchSAE examines spatial visual features and how adaptation changes their use. Universal SAE learns cross-model sparse correspondences from representations of shared images. The first audits adaptation; the second aligns feature spaces, rather than training a distilled student.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 13.01 Sparse autoencoders reveal selective remapping of visual concepts during adaptation PatchSAE turns intermediate CLIP image-token activations into a sparse concept dictionary with spatial locations. It then uses the same dictionary to compare CLIP with prompt-adapted MaPLe: much of the observed adaptation benefit comes from remapping already recognized concepts to task labels, rather than discovering an entirely new visual vocabulary. Proceedings PDF · Code and demo Work: Which concepts persist across image locations, adaptation settings, and encoders?
Exit evidence: Readable examples and cross-decoding do not prove a complete or unique concept basis. Preserve the source-specific activation unit and the actual downstream intervention.
Research branches
Representation and domain transfer
Which source factors remain useful, and which shift factors should be removed?
How Transferable Are Features in Deep Neural Networks? · Domain-Adversarial Training of Neural Networks
Few-shot and meta-learning
Is adaptation a distance rule, a learned optimizer initialization, or another task-conditioned mechanism?
Prototypical Networks for Few-shot Learning · Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Semi-supervised learning
When are model confidence and augmentation invariance reliable enough to create targets?
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Parameter-efficient adaptation
Which parameters move, how are updates represented, and which resource is actually saved?
Universal Language Model Fine-tuning for Text Classification · Parameter-Efficient Transfer Learning for NLP · MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer · LoRA: Low-Rank Adaptation of Large Language Models · QLoRA: Efficient Finetuning of Quantized LLMs
Continual retention and plasticity
Does the method preserve old outputs, parameters, or examples—and can it still acquire genuinely new skills?
Learning Without Forgetting · Overcoming Catastrophic Forgetting in Neural Networks · Loss of Plasticity in Deep Continual Learning
Privileged, test-time, continual, and selective transfer
Which information and gradients exist at each phase, and which deployment resource is actually removed, updated, retained, or selectively requested?
Unifying Distillation and Privileged Information · Distilling Safe LLM Systems via Soft Prompts for On Device Settings · Tent: Fully Test-Time Adaptation by Entropy Minimization · Restoring Without Forgetting: Continual Learning Across Image Degradations · LanGuSTE: Language-Guided Coarse-to-Fine Patch Selection for Efficient Whole Slide Image Analysis · PANDA: Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimer's MRI and TCGA Pathology
State-efficient adaptation and retention
What state changes online, and what certificate or control says that the change is safe or useful?
FOSTER: Feature Boosting and Compression for Class-Incremental Learning · Fast Weight Attention for Continual Learning · Normalized Low-Rank Adaptation · Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You · KDTViT Knowledge Distillation, Transfer Learning and Transformer Based Deep Learning Framework for Efficient Histopathology Image Classification
Transductive few-shot WSI adaptation
How far can sparse slide-specific supervision propagate without updating the feature encoder?
Whole-Slide Image Analysis under Realistic Few-Shot Annotation Protocols
Physical observability and task-aware polarization reconstruction
Is the supervision a physically observable target, a synthetic reconstruction target, a downstream task signal or a teacher distribution?
PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications
Atlas-conditioned foundation segmentation and reliability fusion
What is labeled, what is optimized for each query, and can the gate be trained without using query ground truth?
Readable prompt ensembles, local concepts and selective adaptation
Is the adaptation unit a prompt bank, a concept dictionary, an attention head, or another trainable branch?
AdaBoosting Text Prompts for Vision-Language Models · Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Historical adapter anchors and analytic continual classifiers
Which stored states prevent forgetting, and does the analytic argument still apply when the feature encoder changes?
Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual Learning
Locate visual concepts and align representations across models
Which concepts persist across image locations, adaptation settings, and encoders?
Sparse autoencoders reveal selective remapping of visual concepts during adaptation
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Domain adaptation | Changes the input or joint distribution between source and target while controlling target supervision. | Alignment can erase target-predictive domain-specific information. | 2014 · 2016 |
| Few-shot / meta-learning | Learns across tasks so a new task can be solved from a small support set. | Episode construction or transductive access can make the test task easier than deployment. | 2017 · 2017 |
| Semi-supervised learning | Uses unlabelled target-distribution examples to augment a smaller labelled set. | Confirmation bias amplifies confident errors and class imbalance. | 2020 |
| Parameter-efficient fine-tuning | Constrains trainable updates or their numerical representation. | Fewer trainable parameters do not automatically mean lower inference cost or better transfer. | 2019 · 2022 · 2023 |
| Continual learning | Constrains old outputs or important parameters while measuring retention and future learnability across a task sequence. | New-task images may not cover the old function; a single task order or final average hides forgetting and loss of plasticity. | 2016 · 2017 · 2024 |
| Privileged information, test-time adaptation, and selective transfer | Covers training-only teachers or RNA, entropy-minimizing target updates, degradation-specific low-rank adapters, and language-guided selection of retained 20× pathology evidence. | Target leakage, non-random modality missingness, hidden replay, adapter growth, or omitted 20× acquisition can make an apparently deployable transfer result incomparable. | 2016 · 2026 · 2021 · 2026 · 2026 · 2026 |
| Residual boosting, fast weights, normalized adapters, and frozen transport | Separates model expansion/compression, learned fast-memory updates, low-rank parameter normalization, analytic statistic transport, and sequential scale retention. | Unmatched streams, hidden replay, batch-composition dependence, or a weak transport certificate can overstate continual or test-time gains. | 2022 · 2026 · 2026 · 2026 · 2026 |
| Few-shot transduction over frozen WSI features | Separates graph transduction on frozen features from parameter-efficient fine-tuning and continual model updates. | Annotation topology, missing classes, frozen-feature bias, and within-slide transduction can overstate transfer to a new slide or site. | 2026 |
| Physical observability and task-aware polarization reconstruction | GenPolar predicts Stokes components from intensity using observability-aware supervision and one-step diffusion distillation. PolarAPP instead starts from measured polarization mosaics and couples demosaicking with downstream objectives through a learned metric. Keep generative priors separate from sensor-acquired polarization. | Weak-polarization angle masks change the evaluated population. Task-aware meta-learning is not automatically teacher–student KD, and metric improvements need not agree. | 2026 |
| Atlas-conditioned foundation segmentation and reliability fusion | A single annotated volume can generate spatial prompts and an independent registered-mask prediction. AtlasSegFM learns a voxel-wise fusion gate through support-space consistency while leaving the foundation backbone frozen. Read it as a one-shot supervision budget with substantial test-time adaptation. | Anatomical mismatch can corrupt both branches; interactive methods receiving ground-truth-derived clicks have a different information budget. CT/MRI anatomy is not WSI validation. | 2026 |
| Readable prompt ensembles, local concepts and selective adaptation | TPB reweights mistakes across discrete prompt classifiers, CFM exposes spatially grounded sparse concepts, and Robotic Steering selects action-predictive attention heads. Compare what is selected or trained, how frozen features are reused, and whether the final intervention preserves unrelated capabilities. | Frozen backbones do not imply label-free or zero-cost adaptation. Probe predictability is not causal proof, and Robotic Steering also trains FFN LoRA and its action head. | 2026 · 2026 |
| Historical adapter anchors and analytic continual classifiers | Historical LoRA anchors constrain encoder drift while text-guided accumulated statistics update a classifier without raw-example replay. This is a teacher-state and classifier-memory contract, not the residual-error cascade used by other KD methods. | Stable-feature equivalence is conditional; check the printed objective/closed-form mismatch and separate average from final accuracy. | 2026 |
| Locate visual concepts and align representations across models | PatchSAE examines spatial visual features and how adaptation changes their use. Universal SAE learns cross-model sparse correspondences from representations of shared images. The first audits adaptation; the second aligns feature spaces, rather than training a distilled student. | Readable examples and cross-decoding do not prove a complete or unique concept basis. Preserve the source-specific activation unit and the actual downstream intervention. | 2025 |
Concept lineage
How Transferable Are Features in Deep Neural Networks?Domain-Adversarial Training of Neural Networks
Feature reuse becomes an explicit optimization problem when source and target domains differ.
Prototypical Networks for Few-shot LearningModel-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Both train across episodes, but one fixes adaptation as prototype construction while the other differentiates through gradient-based adaptation.
Parameter-Efficient Transfer Learning for NLPLoRA: Low-Rank Adaptation of Large Language Models
Parameter-efficient transfer moves from inserted bottleneck modules to low-rank reparameterizations of selected weight updates.
LoRA: Low-Rank Adaptation of Large Language ModelsQLoRA: Efficient Finetuning of Quantized LLMs
Low-rank updates are retained while the frozen base model is quantized to reduce fine-tuning memory.
Learning Without ForgettingOvercoming Catastrophic Forgetting in Neural Networks
Continual retention moves from matching the old model function on new-task inputs to constraining parameters according to estimated old-task importance.
Overcoming Catastrophic Forgetting in Neural NetworksLoss of Plasticity in Deep Continual Learning
Protecting old-task parameters addresses forgetting; the later diagnosis shows that capacity to learn future tasks is a separate outcome.
Failure modes to audit
- Calling any fine-tuning experiment domain adaptation without measuring a source–target shift.
- Using target test labels to choose layers, thresholds, ranks, or checkpoints.
- Comparing adaptation methods with different data, augmentation, search, or compute budgets.
- Equating trainable-parameter count with memory, latency, storage, or total energy.
- Reporting only final average accuracy in continual learning, hiding order effects, forgetting, and plasticity.
Questions for every paper
- What exactly changes between source, adaptation, and deployment distributions?
- Which labels, examples, model states, and gradients are available at adaptation time?
- Is the main constraint labelled data, trainable parameters, memory, time, storage, or retained competence?
- Which frozen, full-tuning, and same-budget baselines isolate the proposed mechanism?
- How are target shift, task order, forgetting, and future plasticity measured separately?
Complete maintained paper route
Section titled “Complete maintained paper route”31 canonical readings · 2014–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered by an explicit prerequisite-aware teaching sequence. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 31 readings in maintained order
How Transferable Are Features in Deep Neural Networks?
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson
Metadata checked
Unifying Distillation and Privileged Information
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik
ICLR 2016 conference paper; canonical locator is the complete arXiv manuscript.
Metadata checked
Domain-Adversarial Training of Neural Networks
Yaroslav Ganin et al.
Metadata checked
Tent: Fully Test-Time Adaptation by Entropy Minimization
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell
Published at ICLR 2021; OpenReview is the canonical conference record.
Official artifacts: Official implementation
Metadata checked
Atlas is Your Perfect Context: One-Shot Customization for Generalizable Foundational Medical Image Segmentation
Ziyu Zhang, Yi Yu, Simeng Zhu, Ahmed Aly, Yunhe Gao, Ning Gu, and Yuan Xue
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #2028. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript
Metadata checked
Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You
Salim Khazem and Ibrahim Mohamed Serouis
arXiv v1 submitted 31 August 2026. The official CASTER repository was available when checked on 2 September 2026; no peer-reviewed record was verified.
Official artifacts: Official implementation
Metadata checked
Prototypical Networks for Few-shot Learning
Jake Snell, Kevin Swersky, and Richard Zemel
Metadata checked
AdaBoosting Text Prompts for Vision-Language Models
Seokhee Jin, Changhwan Sung, Sunung Mun, Hoyoung Kim, and Jungseul Ok
ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #5256 checked on 2026-09-12. Publisher version of record was not independently verified.
Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation
Metadata checked
Whole-Slide Image Analysis under Realistic Few-Shot Annotation Protocols
Tiffanie Godelaine, Maxime Zanella, Karim El Khoury, Benoit Macq, and Christophe De Vleeschouwer
arXiv v1 submitted 31 August 2026; the manuscript title page says it was submitted on 29 July 2026. Code and data instructions were released, but no peer-reviewed record was verified on 2 September 2026.
Official artifacts: Official implementation and data instructions
Metadata checked
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine
Metadata checked
PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications
Yidong Luo, Chenggong Li, Yunfeng Song, Ping Wang, Boxin Shi, Junchao Zhang, and Xin Yuan
ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #5713. Publisher proceedings/version of record was not verified on 2026-09-12.
Official artifacts: ECCV Spotlight program · Conference manuscript
Metadata checked
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Kihyuk Sohn et al.
Metadata checked
Universal Language Model Fine-tuning for Text Classification
Jeremy Howard and Sebastian Ruder
Metadata checked
Parameter-Efficient Transfer Learning for NLP
Neil Houlsby et al.
Metadata checked
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder
Official artifacts: Official AdapterHub resource · Official adapters implementation
Metadata checked
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu et al.
Metadata checked
Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Chancharik Mitra, Yusen Luo, Raj Saravanan, Dantong Niu, Anirudh Pai, Jesse Thomason, Trevor Darrell, Abrar Anwar, Deva Ramanan, and Roei Herzig
ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #13378 checked on 2026-09-12. Publisher version of record was not independently verified.
Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation
Metadata checked
QLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers et al.
Metadata checked
Normalized Low-Rank Adaptation
Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, and Weiyang Liu
arXiv v1 submitted 31 August 2026. The authors released an implementation; no peer-reviewed record was verified on 2 September 2026.
Official artifacts: Official implementation
Metadata checked
Learning Without Forgetting
Zhizhong Li and Derek Hoiem
Peer-reviewed ECCV 2016 proceedings paper; the canonical locator is the open author manuscript.
Official artifacts: Official project · Official code
Metadata checked
Overcoming Catastrophic Forgetting in Neural Networks
James Kirkpatrick et al.
Metadata checked
FOSTER: Feature Boosting and Compression for Class-Incremental Learning
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan
Published in the ECCV 2022 main conference proceedings, pages 398–414.
Official artifacts: Official implementation
Metadata checked
Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual Learning
Qier Meng, De Cheng, Jiahao Li, and Cheng Deng
ECCV 2026 Spotlight verified via the official related presentation event; the poster identity itself says Accept Poster. No Springer version of record or public runnable implementation was verified. Section 3.5's printed closed-form equation omits a term implied by its objective; the review flags this reproducibility issue without silently repairing the authors' formula.
Official artifacts: Official ECCV 2026 Spotlight
Metadata checked
Fast Weight Attention for Continual Learning
Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, and Andrew Chi-Chih Yao
arXiv v1 submitted 27 August 2026. The official implementation was available when checked on 2 September 2026; no peer-reviewed record was verified.
Official artifacts: Official implementation
Metadata checked
Loss of Plasticity in Deep Continual Learning
Shibhansh Dohare et al.
Metadata checked
Distilling Safe LLM Systems via Soft Prompts for On Device Settings
Motasem Alfarra, Cristina Pinneri, Dana Kianfar, Mohammed Almousa, and Christos Louizos
Published in Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337.
Metadata checked
Sparse autoencoders reveal selective remapping of visual concepts during adaptation
Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider
Official artifacts: Proceedings PDF · Code and demo
Metadata checked
Restoring Without Forgetting: Continual Learning Across Image Degradations
Alif Ashrafee and Bartosz Krawczyk
arXiv v1 preprint submitted 24 August 2026; validation covers the reported five degradation domains rather than arbitrary composite corruption.
Official artifacts: Official implementation
Metadata checked
KDTViT Knowledge Distillation, Transfer Learning and Transformer Based Deep Learning Framework for Efficient Histopathology Image Classification
Aarv Mankodi, Praveen Kumar Shukla, and Hitesh Tekchandani
Peer-reviewed open-access journal article 16:20721. A MobileViT-XS is sequentially adapted from 40× through 400× using the previous checkpoint as teacher; BreakHis consists of biopsy images rather than WSIs, and the same-capacity teacher/student design targets retention rather than model compression.
Official artifacts: Open-access article
Metadata checked
LanGuSTE: Language-Guided Coarse-to-Fine Patch Selection for Efficient Whole Slide Image Analysis
Yonghan Shin, Gangsu Kim, and Won-Ki Jeong
arXiv preprint first submitted in 2025 and substantially revised as v2 on 24 August 2026; no peer-reviewed version or official implementation was verified by the review cutoff.
Metadata checked
PANDA: Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimer's MRI and TCGA Pathology
Sheethal Bhat, Mahfuzur Rahman Chowdhury, Paula Andrea Perez-Toro, Stephan Wunderlich, Rose Dawn Bharat, Siming Bayer, and Andreas Maier
arXiv v1 preprint submitted 26 August 2026. The manuscript promises code upon acceptance, but no public implementation was available at the review cutoff.
Metadata checked