Skip to content
Computational Pathology Reading Roadmap

Classical machine learning

Classical machine learning is not a historical warm-up. Linear models, kernels, nearest neighbours, probabilistic models, and tree ensembles remain strong baselines when samples are limited, features are structured, or the cost of an error must be explained. They also expose assumptions that a deep model can hide.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Prerequisites

Section titled “Prerequisites”

Complete the shape, likelihood, estimation, optimization, and uncertainty sections of mathematics, probability, and statistics. You should be able to keep a test set untouched, standardize a feature using training data only, and explain the difference between a fitted parameter and a population quantity.

Learning objectives

Section titled “Learning objectives”

After this chapter, you should be able to:

  • translate a prediction problem into a loss, hypothesis class, and regularizer;
  • explain the inductive bias of a linear, local, kernel, probabilistic, tree, or ensemble method;
  • distinguish prediction, probability estimation, ranking, and causal interpretation;
  • select preprocessing, hyperparameters, and thresholds without leaking the test set;
  • fit and diagnose clustering, mixture, and low-dimensional representation methods;
  • compare models under an equal information and model-selection budget;
  • recognize when a modern tabular foundation model is applicable and when it is not.

The common objective

Section titled “The common objective”

For examples (x_i, y_i), empirical risk minimization fits

θ̂ = arg minθ  (1/n) Σᵢ L(fθ(xᵢ), yᵢ) + λ Ω(θ).

The loss defines the fitted target, the model class restricts possible functions, and the penalty encodes a preference such as small coefficients or sparsity. The optimization algorithm finds a solution to this finite-sample problem. None of those choices alone guarantees performance on the population.

Before selecting an algorithm, specify:

Contract Questions
observation Is one independent unit a patient, slide, document, image, or event?
target Is it continuous, categorical, ordinal, censored, multilabel, or missing not at random?
information Which features are available at the actual prediction time?
loss Which errors should training penalize?
decision Is a probability, ranking, class, or action ultimately required?
generalization Is the target new examples, new groups, a later period, or a new domain?

Supervised learning

Section titled “Supervised learning”

Linear regression and generalized linear models

Section titled “Linear regression and generalized linear models”

Ordinary least squares estimates a conditional mean using a linear function. Geometry explains projection onto the column space; probability explains the conditions under which standard errors and intervals have their advertised meaning. Inspect residuals, nonlinearity, heteroscedasticity, dependence, collinearity, and influential points.

Logistic regression models a linear log-odds:

log[p(y=1|x) / (1-p(y=1|x))] = wᵀx + b.

It can produce useful probabilities, but calibration still depends on sampling, regularization, specification, prevalence, and shift. Multinomial, ordinal, count, and survival problems require target-appropriate likelihoods rather than forcing every task into binary classification.

Regularization changes the fitted problem:

  • ridge (L2) shrinks correlated coefficients and improves conditioning;
  • lasso (L1) can set coefficients to zero, but selected variables may be unstable;
  • elastic net combines sparsity and grouped shrinkage;
  • structured penalties can encode groups, smoothness, or known relationships.

A non-zero coefficient is not automatically a causal effect or a reproducible biomarker. Selection and inference use different guarantees.

Local and probabilistic classifiers

Section titled “Local and probabilistic classifiers”

k-nearest neighbours predicts from nearby training examples. Its useful bias is local smoothness; its weaknesses are scale sensitivity, high-dimensional distance, memory cost, and uneven density. Choose the distance, representation, and k inside the training procedure.

Naive Bayes assumes conditional independence of features given a class. That assumption is often false, yet the classifier can work because accurate joint-density estimation is not required for every decision boundary. Linear and quadratic discriminant analysis model class-conditional distributions with shared or separate covariances. Estimate covariance cautiously when features approach or exceed samples.

Margins, kernels, and support vector machines

Section titled “Margins, kernels, and support vector machines”

A maximum-margin classifier chooses a separating boundary with a large geometric margin. Slack variables permit violations, and the regularization parameter trades margin against training error. The kernel trick replaces inner products with a positive-semidefinite similarity, allowing nonlinear boundaries without explicitly constructing the feature map.

Kernel choice is an assumption about similarity. Scale every input using training statistics, tune kernel and regularization parameters in inner validation, and calibrate scores separately if probabilities are needed. Exact kernel methods become expensive as sample size grows.

Trees and ensembles

Section titled “Trees and ensembles”

A decision tree recursively partitions feature space. Trees naturally express interactions and nonlinear thresholds, but an unconstrained tree is unstable and can memorize. Pruning, depth, leaf size, and feature handling are part of the model.

Ensembles reduce or redirect that instability:

  • bagging averages fits on resampled data;
  • random forests also randomize candidate features to decorrelate trees;
  • AdaBoost reweights examples through a sequence of weak classifiers and exposes how margins and sensitivity to hard or noisy examples interact;
  • gradient boosting fits weak learners to functional loss gradients, extending boosting beyond the exponential loss;
  • XGBoost and related systems regularize, subsample, handle sparsity, and implement boosting efficiently.

Feature importance is method-dependent. Impurity importance favours some feature types; permutation importance can be distorted by correlation; SHAP values explain a chosen model under a chosen background distribution, not the causal effect of a feature. Report predictive evidence separately from explanation claims.

Unsupervised and latent-variable learning

Section titled “Unsupervised and latent-variable learning”

Clustering

Section titled “Clustering”

k-means minimizes within-cluster squared Euclidean distance. It favours roughly spherical, similarly scaled groups and requires k. Initialization can change the solution. Lloyd’s alternating assignment/update algorithm monotonically decreases that objective but reaches only a local optimum, so initialization count, convergence tolerance, scaling, empty-cluster handling, and seed are part of the method. Hierarchical clustering additionally requires a distance and linkage rule; DBSCAN-like density methods require a meaningful neighbourhood scale.

Clusters are properties of a representation, metric, sample, and algorithm. Validate stability, sensitivity, external usefulness, and known nuisance variables before giving them biological names.

Mixtures and expectation–maximization

Section titled “Mixtures and expectation–maximization”

A finite mixture represents a density as weighted component distributions. The expectation–maximization algorithm alternates:

  1. computing expected latent assignments under current parameters;
  2. maximizing the expected complete-data log likelihood.

EM monotonically improves its objective under its conditions, but can converge to a local optimum or degenerate covariance. Multiple starts, constraints, and held-out predictive checks are necessary. A mixture component is not inherently a real-world class.

Dimensionality reduction

Section titled “Dimensionality reduction”

PCA gives a global linear projection that preserves variance under squared error. Independent component analysis seeks statistically independent sources under stronger assumptions. Nonlinear visualization methods answer different questions:

  • t-SNE emphasizes local neighbour probabilities and can distort global scale;
  • UMAP constructs a neighbour graph and optimizes a low-dimensional fuzzy representation, with results depending on metric and hyperparameters.

Use a two-dimensional embedding for exploration, not as standalone evidence of separation. Fit transforms within the training boundary for predictive evaluation.

Anomaly and one-class learning

Section titled “Anomaly and one-class learning”

Anomaly scores can measure low density, distance, isolation, reconstruction error, or deviation from a learned boundary. “Anomalous” is relative to the reference distribution. Evaluate against operationally relevant anomalies and shifts, not only synthetic corruptions generated by the same assumptions as the detector.

Statistical learning theory

Section titled “Statistical learning theory”

Population risk is the expected loss under a target distribution; empirical risk is its finite-sample average. Generalization asks when performance on the sample controls performance on new draws from that distribution. A learning guarantee must state:

  • the hypothesis class and capacity measure;
  • whether examples are independent and identically distributed;
  • whether the loss is bounded, Lipschitz, or otherwise controlled;
  • the probability and sample-size statement;
  • whether the result is uniform over models or conditional on a fixed model;
  • the optimization error separating the fitted solution from the empirical optimum.

Probably Approximately Correct (PAC) learning formalizes sample complexity for achieving an error tolerance with high probability. VC dimension measures the largest set a binary hypothesis class can shatter. Rademacher complexity adapts capacity to a sample, while stability analyzes sensitivity to changing observations. Margin and norm bounds can be more informative than parameter count for some models.

These tools explain why fitting and generalization are different, but a loose worst-case bound is not an estimate of deployed error. Cross-validation, an external test, and shift analysis answer empirical questions that a distributional theorem may not cover. “No free lunch” means performance requires assumptions about tasks; it does not mean algorithms cannot be compared on a specified population.

Model selection is part of the algorithm

Section titled “Model selection is part of the algorithm”

Keep three roles distinct:

  1. training data fit parameters;
  2. validation or inner cross-validation selects preprocessing, features, hyperparameters, and thresholds;
  3. test data estimates performance after those choices are fixed.

When data are scarce, nested cross-validation places the complete selection pipeline inside each outer training fold. Grouped, temporal, or site-level splitting must preserve the true independence unit. Imputation, normalization, feature selection, oversampling, dimensionality reduction, and representation learning all belong inside that boundary.

Compare algorithms with the same inputs, outer folds, primary metric, tuning trials, early-stopping access, and compute envelope. A default linear model versus a heavily tuned ensemble is not a controlled comparison.

Modern tabular foundation models

Section titled “Modern tabular foundation models”

TabPFN frames small tabular problems as inference under a learned prior over synthetic tasks. The 2025 Nature work demonstrates broad performance across the evaluated small-to-medium classification and regression setting. It does not establish universal superiority for arbitrary sample sizes, feature semantics, censoring, groups, domain shift, or clinical use. Treat task-size limits, preprocessing, transductive access, licence, uncertainty, and compute as part of the applicability check, and retain transparent classical baselines.

Staged primary-source reading ladder

Section titled “Staged primary-source reading ladder”
Order Stage Primary paper or resource · year · venue/status Read it for
1 foundations Stanford CS229 notes — official course notes connected derivations of linear models, kernels, learning theory, mixtures, and EM
2 foundations An Introduction to Statistical Learning — 2023 · second edition, Springer; author-hosted labs applied modelling, resampling, regularization, trees, SVMs, and unsupervised learning
3 latent variables Maximum Likelihood from Incomplete Data via the EM Algorithm — 1977 · JRSS Series B the EM decomposition, monotonic likelihood improvement, and scope
4 clustering Least Squares Quantization in PCM — 1982 · IEEE Transactions on Information Theory Lloyd iteration, squared-distance assignments, centroids, and local convergence
5 learnability A Theory of the Learnable — 1984 · Communications of the ACM PAC learning, sample complexity, and computational learnability
6 margin methods Support-Vector Networks — 1995 · Machine Learning maximum margins, kernels, and capacity control
7 sparse models Regression Shrinkage and Selection via the Lasso — 1996 · JRSS Series B coefficient shrinkage and sparse selection
8 adaptive boosting A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting — 1997 · Journal of Computer and System Sciences weak-to-strong learning, adaptive example weights, and the exponential-loss view
9 ensembles Random Forests — 2001 · Machine Learning randomized tree ensembles, margins, and variable importance
10 gradient boosting Greedy Function Approximation: A Gradient Boosting Machine — 2001 · Annals of Statistics stagewise function fitting as optimization in function space
11 visualization Visualizing Data using t-SNE — 2008 · JMLR neighbour-probability matching and visualization caveats
12 scalable boosting XGBoost: A Scalable Tree Boosting System — 2016 · KDD regularized boosting objectives and systems design
13 manifold visualization UMAP: Uniform Manifold Approximation and Projection — 2018 · JOSS graph construction, low-dimensional optimization, and assumptions
14 learned tabular prior Accurate predictions on small data with a tabular foundation model — 2025 · Nature prior-data fitted networks, benchmark scope, and current small-tabular endpoint

Read the original work for its actual assumptions and experiment. Use the books and course notes for a coherent first explanation.

Controlled lab

Section titled “Controlled lab”

Choose one grouped tabular dataset; if it has repeated measurements, make the group identifier explicit. Freeze an outer train/validation/test or nested group-split protocol before fitting.

  1. Build a majority or mean predictor and a one-feature baseline.
  2. Fit regularized linear or logistic regression, an RBF SVM, a random forest, and gradient-boosted trees in pipelines.
  3. Give every method the same outer folds and tuning-trial budget.
  4. Plot learning curves and calibration curves; report discrimination, log loss or Brier score, and a task-relevant threshold metric with paired intervals.
  5. Compare raw features with a PCA transform fitted inside each training fold.
  6. If the data fall within its documented scope, evaluate TabPFN under the same outer protocol and record any transductive or resource assumptions.
  7. Deliberately preprocess before splitting once, then quantify the optimistic bias.

Save the split identifiers, pipeline, search space, random seeds, package versions, fit time, prediction time, and failed runs.

Common failure modes

Section titled “Common failure modes”
  • Splitting rows when several rows belong to the same patient, slide, speaker, or case.
  • Standardizing, imputing, selecting features, or running PCA before cross-validation.
  • Choosing a threshold or the “best model” on the final test set.
  • Reporting accuracy alone on an imbalanced task.
  • Interpreting association, coefficient size, feature importance, or SHAP values as causation.
  • Treating an attractive cluster or embedding as a discovered biological entity.
  • Comparing models with unequal feature access or hyperparameter budgets.
  • Applying SMOTE or another sampler before the split rather than inside training folds.
  • Reporting only the best seed or fold.
  • Calling a foundation model universally superior outside the data regime it evaluated.

Self-check

Section titled “Self-check”

Without notes, answer:

  1. Which quantity does logistic regression make linear?
  2. How do ridge and lasso alter the geometry of the solution?
  3. Why can a naive Bayes classifier work when conditional independence is false?
  4. What makes a function a valid kernel?
  5. How do bagging and boosting address different errors?
  6. Why is a mixture component not necessarily a true class?
  7. What information can t-SNE or UMAP safely support?
  8. Why must feature selection occur inside an inner fold?
  9. Under what data regime is a tabular foundation model a relevant comparator?

Exit task

Section titled “Exit task”

For one real dataset, write the observation unit, target, available information, and generalization target. Compare a regularized linear model, kernel method, and tree ensemble with nested model selection. Report paired uncertainty, calibration, learning curves, computation, and one failed assumption for each. Then explain whether the evidence justifies moving to deep learning or whether the classical model is the better endpoint.

Readable prompt ensembles, local concepts and selective adaptation

TPB reweights mistakes across discrete prompt classifiers, CFM exposes spatially grounded sparse concepts, and Robotic Steering selects action-predictive attention heads. Compare what is selected or trained, how frozen features are reused, and whether the final intervention preserves unrelated capabilities.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
AdaBoosting Text Prompts for Vision-Language ModelsSeokhee Jin, Changhwan Sung, Sunung Mun, Hoyoung Kim, and Jungseul Ok2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #5256 checked on 2026-09-12. Publisher version of record was not independently verified.Text Prompt Boosting (TPB) adapts a frozen CLIP-style classifier by sequentially constructing readable prompt banks that focus on previously misclassified images. The weak learners are entire class-wise text-prompt classifiers; their confidence outputs are combined through SAMME.R boosting. The result is an error-correcting ensemble in language space, not a smaller student trained to copy teacher features.ECCV Spotlight program · Conference manuscript · Official project or implementation

Reading question: Is the adaptation unit a prompt bank, a concept dictionary, an attention head, or another trainable branch?

Evidence boundary: Frozen backbones do not imply label-free or zero-cost adaptation. Probe predictability is not causal proof, and Robotic Steering also trains FFN LoRA and its action head.

Complete maintained paper route

Section titled “Complete maintained paper route”

14 canonical readings · 1977–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 14 readings in maintained order

  4. #41995Machine LearningPeer reviewedL1–L2

    Support-Vector Networks

    Corinna Cortes and Vladimir Vapnik

    Metadata checked

  7. #72001Machine LearningPeer reviewedL1–L2

    Random Forests

    Leo Breiman

    Metadata checked

  9. #92006KDDPeer reviewedL2–L3

    Model Compression

    Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil

    Peer-reviewed KDD 2006 proceedings paper, pages 535–541.

    Metadata checked