Skip to content
Computational Pathology Reading Roadmap

Mathematics, probability, and statistics for ML

Machine-learning notation becomes useful when every symbol has a geometric, probabilistic, or statistical meaning. This chapter develops that language around the operations that recur in models and papers; it is not a requirement to finish an entire mathematics degree before training a model.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Prerequisites

Section titled “Prerequisites”

You should be comfortable with high-school algebra, functions, summation notation, and basic Python. Use the official NumPy documentation to practice shapes, broadcasting, indexing, and numerical data types alongside this chapter.

Learning objectives

Section titled “Learning objectives”

After this chapter, you should be able to:

  • annotate the shape and meaning of every tensor in a small model;
  • interpret dot products, norms, projections, rank, eigenvectors, and singular vectors;
  • distinguish a scalar derivative, gradient, Jacobian, Hessian, and vector–Jacobian product;
  • apply the chain rule to a computation graph and verify a derivative numerically;
  • form a likelihood, posterior, expectation, covariance, and Monte Carlo estimate;
  • distinguish an estimator’s bias, variance, standard error, and confidence interval;
  • explain entropy, cross-entropy, KL divergence, and mutual information without treating them as interchangeable;
  • recognize a convex problem, a constrained problem, and a non-convex neural objective;
  • state which population quantity an experiment is trying to estimate.

Notation and the data-generating view

Section titled “Notation and the data-generating view”

Let X ∈ R^(n×d) contain n observations and d measured features. A supervised target may be y ∈ R^n, and model parameters may be collected in θ. Keep four levels separate:

  1. population: the cases about which the scientific claim is made;
  2. random variables: a mathematical model of possible observations;
  3. sample: the particular finite data that were collected;
  4. estimate: a function of the sample, such as fitted parameters or test accuracy.

Writing E[X] for a population expectation and x̄ for a sample mean prevents many conceptual errors. A dataset is not the population, and a network probability is not automatically a calibrated probability of correctness.

Linear algebra and geometry

Section titled “Linear algebra and geometry”

Vectors, linear maps, and shape contracts

Section titled “Vectors, linear maps, and shape contracts”

A vector is both an ordered list and a point or direction in a vector space. A matrix A ∈ R^(m×d) represents a linear map from R^d to R^m. Matrix multiplication is composition: if B ∈ R^(k×m), then BA maps R^d → R^k.

Track:

  • basis and coordinates: coordinates change when the basis changes, although the underlying vector does not;
  • span, rank, and null space: rank is the number of independent output directions;
  • inner product: xᵀy measures signed alignment in Euclidean space;
  • norm: ||x||₂ measures Euclidean length, while ||x||₁ and matrix norms impose different geometry;
  • projection: the closest point in a subspace under a chosen norm;
  • conditioning: nearly dependent directions can make estimates numerically and statistically unstable.

Tensor dimensions are semantic contracts. In attention, for example, a product that is algebraically legal can still be scientifically wrong if it mixes patients instead of tokens.

Eigenstructure, SVD, and PCA

Section titled “Eigenstructure, SVD, and PCA”

For a square matrix, Av = λv identifies a direction preserved by the transformation. The singular-value decomposition works for any matrix:

X = U Σ Vᵀ.

The right singular vectors give feature-space directions, singular values give their strength, and left singular vectors give observation coordinates. For centred data, the first k principal components solve a reconstruction problem:

min over rank-k X̂  ||X - X̂||²_F.

PCA is a linear variance-preserving projection, not a guarantee that visible clusters are biological classes. Standardization, outliers, batch effects, and the observation unit all change its result.

Calculus and automatic differentiation

Section titled “Calculus and automatic differentiation”

Local change

Section titled “Local change”

For a scalar function f: R^d → R, the gradient ∇f points in the direction of steepest local increase under Euclidean geometry. A Jacobian contains partial derivatives for a vector-valued function; a Hessian contains second derivatives of a scalar function.

For a composition z = f(g(x)), the chain rule composes local derivatives. Backward automatic differentiation evaluates vector–Jacobian products from a scalar loss toward the inputs and parameters. It is an efficient application of the chain rule, not a numerical approximation.

Forward mode propagates Jacobian–vector products and is efficient when there are few input directions; reverse mode propagates vector–Jacobian products and is efficient for a scalar loss with many parameters. Both operate on a recorded computation graph and differ from symbolic differentiation and finite differences. The automatic-differentiation survey connects these modes to backpropagation and differentiable programming.

A linear-classifier example

Section titled “A linear-classifier example”

For logits z = Wx + b, softmax probabilities

p_j = exp(z_j) / Σ_k exp(z_k)

and one-hot target y, cross-entropy is L = -Σ_j y_j log p_j. The derivative with respect to the logits is p - y; gradients for W follow by the chain rule. In code, subtract max(z) before exponentiation to avoid overflow without changing the result.

Gradient checking

Section titled “Gradient checking”

For one parameter direction v, compare the analytic directional derivative with

[f(θ + εv) - f(θ - εv)] / (2ε).

Large ε has truncation error; tiny ε has floating-point cancellation. Gradient checking diagnoses an implementation at selected points—it does not prove the model or objective is correct.

Probability

Section titled “Probability”

Conditioning and Bayes’ rule

Section titled “Conditioning and Bayes’ rule”

Probability represents uncertainty under a model. Joint, marginal, and conditional distributions answer different questions:

p(a | b) = p(b | a) p(a) / p(b).

Reversing p(disease | test) and p(test | disease) ignores the base rate. In ML, the same error appears when a class-conditional score is interpreted as a posterior without specifying class prevalence.

Expectation, variance, and covariance

Section titled “Expectation, variance, and covariance”

Expectation is a distribution-weighted average. Variance measures squared deviation from the mean. Covariance records linear co-variation, and the covariance matrix must be positive semidefinite. Independence implies zero covariance when moments exist, but zero covariance does not generally imply independence.

The law of total expectation and law of total variance are useful for separating sources of uncertainty across sites, patients, and repeated measurements.

Likelihood, priors, and prediction

Section titled “Likelihood, priors, and prediction”

Given observations D, likelihood treats p(D | θ) as a function of θ. Maximum likelihood estimates

θ̂_ML = arg maxθ log p(D | θ),

while maximum a posteriori estimation adds log p(θ). A Bayesian posterior p(θ | D) represents uncertainty over parameters under the model and prior. A posterior predictive integrates predictions over that posterior.

Regularization may correspond to a prior in a particular probabilistic formulation, but the optimization penalty, prior belief, and resulting uncertainty statement should not be conflated automatically.

Sampling and Monte Carlo estimation

Section titled “Sampling and Monte Carlo estimation”

If an expectation is analytically unavailable, samples can approximate it. Record the sampling distribution, dependence between samples, effective sample size, and Monte Carlo error. Resampling data with the wrong unit—tiles instead of patients, for example—produces falsely narrow uncertainty.

Statistical estimation and inference

Section titled “Statistical estimation and inference”

Bias, variance, and generalization

Section titled “Bias, variance, and generalization”

An estimator’s bias is the difference between its expected estimate and the target quantity. Its variance measures change across hypothetical repeated samples. Model approximation, finite-sample estimation, label noise, and irreducible outcome variation are distinct sources of error.

The familiar bias–variance picture is a guide, not a complete theory of modern overparameterized networks. Always return to the population, estimator, sampling process, and loss being analyzed.

Confidence intervals and tests

Section titled “Confidence intervals and tests”

A frequentist confidence procedure has long-run coverage under repeated sampling and its assumptions; it is not the posterior probability that this realized interval contains a fixed parameter.

For ML comparisons:

  • pair predictions on the same held-out cases when possible;
  • resample at the independent unit;
  • report the effect estimate and interval, not only a p-value;
  • pre-specify the primary comparison or account for multiplicity;
  • distinguish statistical detectability from practical importance.

Bootstrap intervals are flexible but are not assumption-free. Small samples, rare events, clustering, and site structure may require stratified, clustered, hierarchical, or exact procedures.

Information theory

Section titled “Information theory”

For a discrete distribution p, entropy is

H(p) = -Σ_x p(x) log p(x).

Cross-entropy H(p, q) = -Σ p log q measures coding or predictive loss when q approximates p. KL divergence

KL(p || q) = Σ_x p(x) log[p(x)/q(x)]

is asymmetric and is not a distance. The identity H(p, q) = H(p) + KL(p || q) explains why minimizing expected cross-entropy recovers the target distribution when the model family and optimization allow it.

Mutual information measures departure from independence:

I(X;Y) = KL(p(x,y) || p(x)p(y)).

Estimating mutual information in high dimensions is difficult. A lower bound or proxy objective should not be reported as the true information shared by representations.

Optimization language

Section titled “Optimization language”

An unconstrained problem minimizes f(θ); a constrained problem also requires g_i(θ) ≤ 0 or h_j(θ) = 0. Convexity makes every local minimum global, but most neural-network parameterizations are non-convex. Lagrangians and duality turn constraints into analyzable objectives and underpin SVMs, entropy-constrained models, and many fairness or resource constraints.

Gradient descent uses local first-order information. Newton and quasi-Newton methods use curvature. Stochastic optimization uses noisy estimates of the full gradient. Convergence of the optimization objective and generalization to new data remain different questions.

Staged resources and primary readings

Section titled “Staged resources and primary readings”
Order Level Resource or primary reading · year · status Read it for
1 L0–L1 MIT 18.06 Linear Algebra — official MIT OpenCourseWare vector spaces, systems, eigenvalues, and positive-definite matrices
2 L1 Mathematics for Machine Learning — 2020 · Cambridge University Press book, author-hosted PDF a direct bridge from linear algebra, geometry, calculus, probability, and optimization into ML
3 L1–L2 Convex Optimization — 2004 · Cambridge University Press book, author-hosted convex sets/functions, duality, constrained optimization, and algorithms
4 L1–L2 Automatic Differentiation in Machine Learning: a Survey — 2018 · JMLR, peer reviewed forward/reverse accumulation, computation graphs, and the relation to backpropagation
5 L1–L2 Maximum Likelihood from Incomplete Data via the EM Algorithm — 1977 · JRSS Series B (journal) latent variables, expected complete-data objectives, and iterative estimation
6 L2 A Mathematical Theory of Communication — 1948 · Bell System Technical Journal (journal) entropy and information as formal quantities rather than metaphors
7 L1–L3 Probabilistic Machine Learning: An Introduction — 2022 · MIT Press book a common notation for probability, prediction, inference, and decision making
8 L2–L3 Probabilistic Machine Learning: Advanced Topics — 2023 · MIT Press book; author-maintained draft variational inference, Monte Carlo methods, deep probabilistic models, and decisions

Read textbooks for connected explanation and primary papers for the original claim, assumptions, and evidence. A primary paper is not always the best first tutorial.

Controlled lab

Section titled “Controlled lab”

Use a small synthetic dataset whose generating process you know.

  1. Create correlated features with a known covariance and one nearly redundant column.
  2. Compute the mean, covariance, eigendecomposition, and SVD; explain why the numerical rank changes with tolerance.
  3. Project onto principal components and report reconstruction error as k changes.
  4. Implement stable softmax cross-entropy and its gradient without autograd.
  5. Compare the analytic gradient with central finite differences across several ε.
  6. Fit logistic regression and bootstrap its held-out log loss by resampling complete observations.
  7. Change the resampling unit incorrectly by duplicating measurements per subject; quantify how the interval becomes misleading.

Record the random-number generator, dtype, tolerance, sample unit, and every shape.

Common failure modes

Section titled “Common failure modes”
  • Treating a tensor’s shape as an implementation detail rather than part of the model.
  • Interpreting PCA or a two-dimensional embedding as proof of discrete natural groups.
  • Applying Bayes’ rule without the relevant prevalence or sampling design.
  • Calling a softmax value “confidence” without checking calibration.
  • Confusing likelihood with probability of the parameter.
  • Reporting standard deviation across examples as uncertainty of a mean.
  • Bootstrapping correlated tiles or visits as if they were independent patients.
  • Assuming a low p-value implies a large, useful, or reproducible effect.
  • Treating KL divergence as symmetric or as a metric.
  • Assuming convergence of training loss establishes convergence to a useful predictor.

Self-check

Section titled “Self-check”

Without notes, answer:

  1. What geometric operation does a dot product perform?
  2. How do eigendecomposition and SVD differ?
  3. Why is the backward pass a sequence of vector–Jacobian products?
  4. What changes when cross-entropy targets are soft rather than one-hot?
  5. When does zero covariance fail to imply independence?
  6. What is the difference between likelihood, posterior, and posterior predictive?
  7. At which unit should a confidence interval be resampled?
  8. Why is KL(p || q) different from KL(q || p)?

Exit task

Section titled “Exit task”

Take one loss function from a current paper. Define every random variable and tensor shape, derive its gradient with respect to the model output, state the estimator and population target, and identify the assumptions behind its reported uncertainty. Then continue to classical machine learning or deep learning.

Complete maintained paper route

Section titled “Complete maintained paper route”

6 canonical readings · 1901–2025· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 6 readings in maintained order