Skip to content
Computational Pathology Reading Roadmap

Deep learning

Deep learning composes differentiable transformations and learns their parameters from data. The central skill is not memorizing model names: it is tracing shapes, information, gradients, normalization, residual paths, computational cost, and experimental evidence through a system.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Prerequisites

Section titled “Prerequisites”

Complete gradients, probability, information theory, and optimization in mathematics, probability, and statistics. Complete at least one controlled comparison in classical machine learning so that “use a neural network” is a tested modelling choice rather than a default. Basic Python, NumPy, and the official PyTorch beginner tutorials are sufficient programming prerequisites.

Learning objectives

Section titled “Learning objectives”

After this chapter, you should be able to:

  • derive and gradient-check a multilayer perceptron;
  • explain activation, initialization, normalization, residual, and regularization choices mechanistically;
  • distinguish an optimizer, learning-rate schedule, and generalization intervention;
  • trace tensor shapes and receptive fields through convolutional, recurrent, attention, graph, and state-space layers;
  • estimate parameter, activation, optimizer-state, and attention-memory costs;
  • design an equal-budget architecture or optimizer ablation;
  • train with mixed precision and distributed execution without silently changing the scientific comparison;
  • read modern architecture papers as claims with assumptions, not rankings.

Neural networks as computation graphs

Section titled “Neural networks as computation graphs”

Affine maps, activations, and depth

Section titled “Affine maps, activations, and depth”

A dense layer maps x to h = φ(Wx + b). Without a nonlinear activation, composing affine layers still gives one affine map. Depth becomes expressive through nonlinear transformations and parameter sharing, but expressivity does not imply trainability or generalization.

For a classification network with logits z, softmax cross-entropy couples the final activation and likelihood. Keep logits for numerically stable library losses. Applying softmax twice or taking log of rounded probabilities can destroy the calculation.

Common activations impose different behaviour:

Activation Useful property Failure or cost to inspect
sigmoid / tanh bounded gates and smooth state updates saturation and vanishing gradients
ReLU simple positive-region derivative and sparse activation dead units and unbounded activation scale
GELU / SiLU smooth input-dependent gating more compute; does not repair poor scaling
GLU / SwiGLU learned multiplicative gate wider projections and parameter-accounting differences

GELU (2016) defines x Φ(x): the input is smoothly weighted by a standard-Gaussian cumulative probability rather than hard-thresholded at zero. The authors motivate it as the expectation of an input-dependent stochastic gate and report experiments on several small image, speech, and language tasks. GELU is a scalar nonlinearity; it is not the two-projection gate used by GLU.

GLU (2017) computes a content path multiplied by a sigmoid-gated path inside a causal convolutional language model. The untransformed linear path through the product was intended to ease gradient propagation. GLU Variants Improve Transformer (2020) then replaces the sigmoid with ReLU, GELU, or Swish and places the gate inside the Transformer feed-forward network. SwiGLU means Swish on one projection multiplied by a second linear projection; compare it only after reducing hidden width so total parameters and compute remain matched. Both GELU and SwiGLU are 2020-or-earlier preprints in the cited records; their later adoption does not change those source statuses.

Backpropagation

Section titled “Backpropagation”

Backpropagation applies the chain rule from a scalar loss toward every reused parameter. Modern autodiff systems accumulate vector–Jacobian products over a dynamic or compiled graph. For a parameter used on several paths, its gradient is the sum of all path contributions.

Audit a backward pass by recording:

  1. each tensor’s shape and semantic axes;
  2. values cached for the local derivative;
  3. broadcasting and reduction axes;
  4. parameter reuse and gradient accumulation;
  5. nondifferentiable or stopped-gradient operations.

Numerical gradient checking on a tiny, double-precision network remains one of the best tests of a from-scratch implementation.

The automatic-differentiation survey separates forward- and reverse-mode accumulation from symbolic differentiation and finite differences. Backpropagation is reverse-mode differentiation specialized to a scalar objective; understanding that distinction helps diagnose graph retention, in-place mutation, stopped gradients, and unintended gradient accumulation.

Initialization, signal propagation, and normalization

Section titled “Initialization, signal propagation, and normalization”

An initialization should keep activations and gradients in a useful range as depth grows. Xavier/Glorot scaling balances variance for approximately symmetric activations; He-style scaling adapts the argument for rectifiers. These derivations depend on independence and variance approximations that training soon violates, so inspect empirical activation and gradient distributions.

Normalization changes parameterization and information flow:

  • batch normalization uses batch statistics during training and running statistics for inference; small, grouped, or shifted batches can be problematic;
  • layer normalization normalizes features within an example and is common in sequence models;
  • group and instance normalization choose other axes suited to vision or style;
  • RMSNorm normalizes root-mean-square magnitude without mean centring.

Pre-normalized and post-normalized residual blocks have different gradient paths. Always state axes, placement, learned scale/bias, epsilon, and train/evaluation behaviour rather than saying only “we use normalization.”

Group Normalization (2018) partitions channels into groups and computes statistics within each example across the group’s channels and spatial positions. It therefore has no batch-statistic train/evaluation mismatch. On the paper’s ResNet-50 ImageNet protocol at two images per GPU, GN reports 24.1% top-1 error versus 34.7% for BN—a 10.6 percentage-point gap—while BN is slightly better at the regular batch size of 32. This is evidence for small-batch robustness in that setup, not universal superiority. Group count, channels per group, image resolution, and frozen-versus-updated BN statistics belong in every comparison.

Residual connections learn an update x + F(x) and provide a direct signal path. Projection shortcuts change dimensions. Dense connections concatenate earlier features. Both affect representation reuse, memory, and parameter count.

Optimization and regularization

Section titled “Optimization and regularization”

First-order optimization

Section titled “First-order optimization”

Mini-batch stochastic gradient descent uses a noisy gradient estimate. Momentum accumulates a velocity; Nesterov-style methods evaluate a look-ahead direction. Adaptive methods scale updates using gradient moments. Adam is convenient, but its effective step depends on learning rate, moments, epsilon, parameter scale, and implementation details.

AdaGrad (2011) changes the proximal geometry using accumulated gradient outer products. Its practical diagonal form scales coordinate i roughly by the inverse square root of the sum of its past squared gradients; a full-matrix form adapts directions but is substantially more expensive. This can give rare sparse features relatively larger effective steps and comes with online-regret analyses. Because the accumulator only grows, later updates can become very small. AdaGrad is an important ancestor of adaptive optimization, but it is not Adam: Adam adds exponentially weighted first and second moments and different bias/decay behaviour.

Decoupled weight decay, as in AdamW, applies shrinkage separately from the adaptive gradient update. It is not generally equivalent to adding an L2 term inside Adam. Biases and normalization scales are often excluded, so report parameter groups.

Learning-rate warmup, cosine decay, step decay, and schedule-free methods alter the trajectory. Compare them under a fixed number of examples or optimizer steps and a matched tuning budget. A method that trains longer or sweeps more settings has not isolated an optimizer effect.

Second-order or curvature-aware approximations seek better-scaled directions. Sophia uses inexpensive curvature estimates for language-model pretraining. Muon orthogonalizes momentum-like updates for matrix parameters and remains an active technical-report method without a verified peer-reviewed venue as of the review boundary. Neither should be promoted from a benchmark result to a universal default without task-, scale-, and implementation- matched evidence.

Regularization

Section titled “Regularization”

Regularization can enter through data, objective, architecture, or optimization:

  • weight decay and explicit penalties;
  • dropout or stochastic depth;
  • label smoothing and target regularization;
  • data augmentation, mixup, or masking;
  • early stopping;
  • parameter sharing and architectural constraints;
  • transfer learning or freezing;
  • noise introduced by batches and precision.

Each changes more than “overfitting.” For example, augmentation asserts invariance, label smoothing changes the optimum target distribution, and early stopping couples the result to validation access.

Core architecture families

Section titled “Core architecture families”

Convolutional networks

Section titled “Convolutional networks”

A convolution shares a local kernel across spatial positions. Its inductive bias is translation equivariance, while padding, stride, dilation, pooling, and boundary handling determine the actual receptive field and resolution. Stacking layers grows the theoretical receptive field, but the effective influence may remain concentrated.

Modern convolutional networks combine convolution with residual paths, normalization, multi-scale features, and augmentation. Use the detailed computer-vision route for classification, detection, segmentation, and natural-image architectures.

Recurrent and gated networks

Section titled “Recurrent and gated networks”

A recurrent network reuses a state transition across positions. Backpropagation through time multiplies many Jacobians, producing exploding or vanishing gradients. Gradient clipping controls magnitude but does not recover lost long-range information.

LSTM and GRU gates create additive state paths that regulate remembering, writing, and forgetting. Recurrent models remain useful for streaming and bounded-state settings, even though attention dominates many offline sequence tasks.

Attention and Transformers

Section titled “Attention and Transformers”

Scaled dot-product attention computes

Attention(Q,K,V) = softmax(QKᵀ / √d_k) V.

Queries select, keys address, and values carry content. Multi-head attention learns several projected interactions; concatenating heads does not guarantee distinct functions. Causal masks prevent access to future positions. Padding masks and cross-attention masks encode different constraints.

A Transformer block combines attention, a token-wise feed-forward network, normalization, residual paths, and position information. Absolute embeddings, relative bias, rotary position embeddings, and learned interpolation make different extrapolation assumptions. Multi-query and grouped-query attention share key/value heads to reduce cache memory.

Full attention over sequence length n materializes an interaction with quadratic size in n. FlashAttention computes exact attention with an IO-aware tiled algorithm; it changes memory traffic and kernels, not the mathematical attention function.

Use the NLP route and the natural-image vision-language route for tokenization, language modelling, multimodal alignment, and current applications.

Graph neural networks

Section titled “Graph neural networks”

A graph network updates a node from its features and a permutation-invariant aggregation of neighbours. Spectral graph convolutions motivate localized filters; the message-passing framework separates message construction, neighbourhood aggregation, node update, and graph-level readout and thereby unifies many spatial variants. GraphSAGE samples and aggregates neighbours for inductive use, while graph attention learns data-dependent neighbour weights.

Depth can cause over-smoothing, over-squashing, and unstable propagation. The graph itself may leak labels, encode measurement bias, or be unavailable for new examples. State whether edges are observed, constructed, learned, directed, typed, or temporal.

State-space and recurrent alternatives

Section titled “State-space and recurrent alternatives”

Structured state-space models express a sequence through a latent dynamical system and can be evaluated using recurrences or convolutions. Selective state-space models such as Mamba make transition behaviour input-dependent while retaining linear-time sequence scaling. Mamba-2 relates selective state spaces and structured masked attention. xLSTM revisits exponential gates and matrix-valued memory.

TTT Layers (2025) makes a recurrent hidden state the weights of an inner learner updated by a self-supervised objective at test time. It is a sequence-layer mechanism, not ordinary downstream fine-tuning: identify the mutable weights, update loss, reset boundary, causal information, and per-token update cost. Modular TTT (2026) then generalizes that inner learner into a graph of fast-weight modules and update rules.

These methods trade explicit pairwise token interaction for compressed state and hardware-aware scans. Evaluate quality, length scaling, state capacity, streaming latency, kernel maturity, and matched parameter/compute budgets. Linear asymptotic complexity alone does not establish faster end-to-end training.

Kolmogorov–Arnold Networks place learned univariate functions on edges rather than fixed activations at nodes. They are an interesting alternative parameterization, not a replacement established across all deep-learning workloads. Check approximation, optimization, memory, and equal-budget results.

Generalization in overparameterized networks

Section titled “Generalization in overparameterized networks”

Modern networks can interpolate random labels, so parameter count alone does not explain generalization. Relevant perspectives include implicit bias of optimization, margin, norm, data geometry, augmentation, stability, neural tangent limits, feature learning, scaling, and double descent. These are partial theories with different assumptions.

Sharpness-aware minimization optimizes a local worst-case parameter perturbation, but raw sharpness depends on parameterization. Neural tangent kernel results describe particular width and training limits; they do not imply every finite network operates in a fixed-feature regime. Report the regime each explanation addresses.

Training systems are part of the experiment

Section titled “Training systems are part of the experiment”

Precision and memory

Section titled “Precision and memory”

Training memory includes parameters, gradients, optimizer states, activations, and temporary kernels. Mixed-precision training uses lower-precision compute/storage while retaining selected higher-precision operations and loss scaling. bfloat16 has a wider exponent range than IEEE float16, but neither is numerically interchangeable with float32.

Activation checkpointing recomputes selected forward operations during backward to trade compute for memory. Gradient accumulation changes the optimizer update interval; it is not always equivalent to a large batch when normalization, dropout, schedules, or optimizer state depend on micro-batches.

Throughput and distribution

Section titled “Throughput and distribution”

Measure examples or tokens per second only after specifying sequence/image size, precision, batch, accumulation, hardware, compiler, and data pipeline. Profile input loading, host-to-device transfer, kernels, synchronization, and checkpoint writing.

Data parallelism replicates the model and partitions examples. Fully sharded data parallel methods partition parameters, gradients, and optimizer state. Tensor and pipeline parallelism partition operations or layers. Communication, failure recovery, and numerical order can alter both speed and reproducibility.

PyTorch compilation, fused kernels, and optimized attention may improve throughput, but verify numerical tolerance and preserve an unfused reference test. The μnit scaling work studies stable FP8 training recipes at scale; adoption requires hardware and model-regime validation rather than changing precision silently.

Staged primary-source reading ladders

Section titled “Staged primary-source reading ladders”

Mechanics, optimization, and generalization

Section titled “Mechanics, optimization, and generalization”
Order Stage Primary paper · year · venue/status Read it for
1 computational neuron A Logical Calculus of the Ideas Immanent in Nervous Activity — 1943 · Bulletin of Mathematical Biophysics a threshold-unit abstraction and its historical limits
2 learned classifier The Perceptron — 1958 · Psychological Review an early trainable linear threshold system
3 gradients Learning representations by back-propagating errors — 1986 · Nature efficient credit assignment in multilayer networks
4 approximation Approximation by superpositions of a sigmoidal function — 1989 · Mathematics of Control, Signals and Systems a universal approximation result and what it does not say about learning
5 initialization Understanding the difficulty of training deep feedforward neural networks — 2010 · AISTATS variance propagation and Xavier initialization
6 adaptive geometry Adaptive Subgradient Methods for Online Learning and Stochastic Optimization — 2011 · JMLR diagonal versus full-matrix accumulated-gradient geometry, sparse coordinates, and regret bounds
7 activations Deep Sparse Rectifier Neural Networks — 2011 · AISTATS rectifiers and sparse activation
8 regularization Dropout: A Simple Way to Prevent Neural Networks from Overfitting — 2014 · JMLR stochastic unit masking and ensemble interpretation
9 optimization Adam: A Method for Stochastic Optimization — 2015 · ICLR adaptive first and second moments, contrasted with AdaGrad’s cumulative second moment
10 normalization Batch Normalization — 2015 · ICML batch-statistic normalization and training behaviour
11 activation Gaussian Error Linear Units — 2016 · arXiv preprint smooth probabilistic input weighting and the distinction from a learned two-path gate
12 normalization Layer Normalization — 2016 · arXiv; widely adopted method within-example normalization for sequence models
13 gated block Language Modeling with Gated Convolutional Networks — 2017 · ICML GLU’s multiplicative content/gate paths and parallel causal convolution
14 evidence Understanding deep learning requires rethinking generalization — 2017 · ICLR interpolation of random labels and limits of simple capacity stories
15 normalization Group Normalization — 2018 · ECCV per-example channel groups and controlled small-batch comparisons
16 theory Neural Tangent Kernel — 2018 · NeurIPS an infinite-width kernel regime
17 regularization Decoupled Weight Decay Regularization — 2019 · ICLR why AdamW differs from an L2 term in Adam
18 gated Transformer FFN GLU Variants Improve Transformer — 2020 · arXiv preprint parameter-matched GEGLU/SwiGLU variants and the boundary of their empirical evidence
19 robust objective Sharpness-Aware Minimization — 2021 · ICLR neighbourhood-aware training and its empirical claim
20 optimizer Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — 2024 · ICLR lightweight curvature estimates at language-model scale
21 schedule The Road Less Scheduled — 2024 · NeurIPS schedule-free optimization and evaluation protocol
22 optimizer frontier Muon is Scalable for LLM Training — 2025 · Moonshot AI technical report matrix update orthogonalization and evidence still requiring independent replication

Architecture progression

Section titled “Architecture progression”
Order Family Primary paper · year · venue/status Mechanism to reconstruct
1 recurrence Long Short-Term Memory — 1997 · Neural Computation gated additive memory and gradient paths
2 convolution Gradient-Based Learning Applied to Document Recognition — 1998 · Proceedings of the IEEE convolution, subsampling, shared weights, and end-to-end recognition
3 convolution at scale ImageNet Classification with Deep Convolutional Neural Networks — 2012 · NeurIPS convolutional hierarchy, rectifiers, dropout, and GPU training
4 residual Deep Residual Learning for Image Recognition — 2016 · CVPR identity shortcuts and residual optimization
5 attention Attention Is All You Need — 2017 · NeurIPS scaled attention, multi-head projections, position, and residual blocks
6 message passing Neural Message Passing for Quantum Chemistry — 2017 · ICML messages, aggregation, node updates, and graph readout as a common GNN framework
7 graph convolution Semi-Supervised Classification with Graph Convolutional Networks — 2017 · ICLR localized propagation over a normalized graph
8 inductive graph Inductive Representation Learning on Large Graphs — 2017 · NeurIPS neighbour sampling and aggregation
9 graph attention Graph Attention Networks — 2018 · ICLR learned neighbour weighting
10 selective state space Mamba: Linear-Time Sequence Modeling with Selective State Spaces — 2024 · COLM input-dependent selection and hardware-aware scan
11 state-space duality Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — 2024 · ICML Mamba-2 and the relation between SSMs and structured attention
12 recurrent memory xLSTM: Extended Long Short-Term Memory — 2024 · NeurIPS exponential gating and scalar/matrix memories
13 online-learned recurrent state Learning to (Learn at Test Time): RNNs with Expressive Hidden States — 2025 · ICML fast-weight state, self-supervised update, reset boundary, and update cost
14 alternative parameterization KAN: Kolmogorov–Arnold Networks — 2025 · ICLR learned edge functions and the scope of comparative evidence

Efficient training systems

Section titled “Efficient training systems”
Order System idea Primary paper · year · venue/status Read it for
1 mixed precision Mixed Precision Training — 2018 · ICLR master weights, loss scaling, range, and speed
2 exact efficient attention FlashAttention — 2022 · NeurIPS IO-aware tiling for exact attention
3 FP8 scaling μnit Scaling: Simple and Scalable FP8 LLM Training — 2025 · ICML numerical scaling, stability, and system constraints

Controlled labs

Section titled “Controlled labs”

Lab A: derive before scaling

Section titled “Lab A: derive before scaling”
  1. Implement a two-layer MLP in NumPy with stable softmax cross-entropy.
  2. Implement backward propagation and verify random parameter directions with central differences in float64.
  3. Compare sigmoid, tanh, ReLU, and GELU under matched width and initialization.
  4. Log activation means/variances and gradient norms at every layer.
  5. Overfit a tiny batch, then fit a fixed train/validation split.

Lab B: isolate training interventions

Section titled “Lab B: isolate training interventions”

On one small image or tabular task, freeze data, augmentations, model, steps, seeds, checkpoint rule, and tuning trials. Change one factor at a time:

  • Xavier versus He initialization;
  • no normalization versus batch or layer normalization where semantically valid;
  • SGD with momentum versus AdamW;
  • no residual path versus a shape-matched residual path;
  • full precision versus automatic mixed precision.

Report learning curves, final paired estimates, calibration, maximum memory, wall time, and examples processed. A speed comparison without accuracy and numerical checks is incomplete.

Lab C: audit one modern block

Section titled “Lab C: audit one modern block”

Choose a Transformer, graph layer, Mamba-style state-space block, or xLSTM block. Write every input/output shape, parameter matrix, normalization, residual path, asymptotic time/memory term, and state carried between positions. Reimplement a tiny reference and compare outputs/gradients with the library version.

Common failure modes

Section titled “Common failure modes”
  • Omitting tensor axes and accidentally mixing examples, tokens, heads, or channels.
  • Reporting a deeper or wider model without matching parameter, data, or compute budget.
  • Using a training-only batch statistic during inference.
  • Applying weight decay twice or to unintended normalization and bias parameters.
  • Comparing optimizers with one shared learning rate or unequal search budgets.
  • Calling a schedule, precision change, longer run, or fused kernel an architecture gain.
  • Selecting the best seed and hiding unstable or failed runs.
  • Assuming lower asymptotic complexity means lower measured latency at the relevant size.
  • Treating attention weights or graph messages as faithful causal explanations.
  • Claiming state-space, recurrent, KAN, or other new blocks replace Transformers from a narrow benchmark.
  • Ignoring data-loader and synchronization bottlenecks in throughput results.
  • Changing effective batch, update count, or validation frequency during a systems comparison.

Self-check

Section titled “Self-check”

Without notes, answer:

  1. Why does a stack of affine layers need nonlinearities?
  2. How does reverse-mode autodiff reuse local vector–Jacobian products?
  3. What assumptions motivate Xavier and He initialization?
  4. Which axes do batch normalization and layer normalization use?
  5. Why does AdamW differ from adding an L2 penalty to Adam?
  6. What does a residual connection change in the backward path?
  7. Which tensors make full attention quadratic in sequence length?
  8. How does message passing preserve node-order equivariance?
  9. What information can a selective state-space model compress or lose?
  10. Which memory terms are missing if you count only parameters?

Exit task

Section titled “Exit task”

Select one current neural architecture. Reconstruct one block from equations to tensor shapes and implement a tiny gradient-checked version. Compare it with the strongest simple baseline under matched data, selection trials, steps, seeds, and precision. Report predictive uncertainty, parameters, peak memory, throughput, and a failure case. Then continue to generative and representation learning or a domain-specific route.

Second-order attention and lightweight image denoising

NLformer augments non-local first-order patch averaging with low-rank intra-patch statistics and a Woodbury-inspired inverse update. The resulting attention and feed-forward blocks offer a concrete example of turning a classical estimator into a trainable architecture.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Bayesian Self-Attention with Intra-Token Modeling for Lightweight Denoising TransformersRunyang He, Zuowei Shen, and Hui Ji2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #7178. Publisher proceedings/version of record was not verified on 2026-09-12. Conference program and code use 'Local Pixel Correlations'; checked conference PDF uses 'Intra-Token Modeling'. Both titles describe submission 7178; PDF title is canonical here.NLformer treats denoising attention as a combination of non-local patch averaging and a local second-order correction. Its dual attention branches model first-order inter-token relations and low-rank intra-token statistics; a Woodbury-based feed-forward module makes the associated matrix inversion affordable. The result is a compact denoiser, not a distilled student.ECCV Spotlight program · Conference manuscript · Official implementation

Reading question: Which part estimates a mean, which part corrects local covariance, and how is the matrix inverse made tractable?

Evidence boundary: A simplified Bayesian derivation is not an exact guarantee for real camera noise. Fewer parameters need not mean fewer FLOPs than every lightweight competitor.

Teacher-class mixtures and privileged residual routing

MoE-KD reformulates classification through latent teacher-class experts and EM. MARS uses full-minus-partial features to teach a deployable router. Specify whether knowledge resides in label mixtures, prototypes, routing decisions or a residual target.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Residual-Guided Expert Specialization for Incomplete Multimodal LearningSeunghun Baek, Jihwan Park, Jaeyoon Sim, Minjae Jeong, Hoseok Lee, Won Hwa Kim2026 · ECCV · acceptedECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. MARS distills a privileged residual router into an available-feature router; it is not a cascade of residual-error student networks. Full modalities are assumed available during training. Food-101 full-modality accuracy is below SimMLM despite higher missing-modality and average scores.MARS uses the difference between complete- and partial-modality representations to teach expert routing. A privileged residual router learns which experts address each missingness pattern; a second router learns to imitate those decisions from available features alone. Thus the residual is a training signal, while deployment uses a practical feature router without the missing modalities.Official ECCV poster record · Official code

Reading question: Which privileged information is available only during training, and exactly what remains in the student?

Evidence boundary: Residual-guided routing is not a residual-error student cascade; conditional optimization arguments do not certify best accuracy.

Numerical objectives: preserve the loss gradient, not just the loss value

A Fix for Softmax and Cross-Entropy separates hard-target absorption from near-equal teacher/student cancellation. Excluding the target from log-sum-exp and using softplus protects hard-label gradients; a logit-difference construction with log1p/expm1 repairs soft-target derivatives. The intended objective is unchanged.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
A Fix for Softmax and Cross-EntropyJonathan T. Barron, Dan Gnanapragasam, and Michael Tschannen2026 · arXiv · preprintInspected arXiv:2610.31642v1, submitted 22 October 2026, and the author project page. Numerical-gradient and modular-arithmetic experiments are not evidence of improved WSI or large-scale foundation-model accuracy. No venue acceptance was verified.This paper repairs floating-point evaluation of familiar objectives rather than proposing a new classification target. Softmax Collapse loses the dominant class gradient at high confidence; Softmax Cancellation loses the small difference between nearly matching student and teacher distributions. Separate algebraic fixes preserve the intended cross-entropy gradient in these two regimes.Inspected manuscript (v1) · Author project and JAX implementations

Reading question: Does the actual dtype and computation graph preserve high-margin and teacher-matched gradients?

Evidence boundary: Evidence is numerical and small modular-arithmetic training, not WSI or large-scale foundation-model improvement. Higher precision and zero-sum projection are not automatically equivalent fixes. Check the real compiler and temperature convention before changing a training protocol.

Complete maintained paper route

Section titled “Complete maintained paper route”

54 canonical readings · 1943–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 54 readings in maintained order

  5. #51997Neural ComputationPeer reviewedL1–L2

    Long Short-Term Memory

    Sepp Hochreiter and Jürgen Schmidhuber

    Metadata checked

  15. #152016arXivPreprintL1–L2

    Layer Normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton

    Metadata checked

  46. #462025Moonshot AI technical reportTechnical reportL3

    Muon is Scalable for LLM Training

    Jingyuan Liu et al.

    The manuscript self-identifies as a technical report; no peer-reviewed venue was verified through 2026-07-28.

    Metadata checked

  52. #522026ECCVAccepted · proceedings pendingL3

    Bayesian Self-Attention with Intra-Token Modeling for Lightweight Denoising Transformers

    Runyang He, Zuowei Shen, and Hui Ji

    ECCV 2026 Spotlight verified in the official program, session Computational, Medical & Scientific Imaging; scientific claims use conference-hosted manuscript #7178. Publisher proceedings/version of record was not verified on 2026-09-12. Conference program and code use 'Local Pixel Correlations'; checked conference PDF uses 'Intra-Token Modeling'. Both titles describe submission 7178; PDF title is canonical here.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official implementation

    Metadata checked

  53. #532026ECCVAccepted · proceedings pendingL3

    Residual-Guided Expert Specialization for Incomplete Multimodal Learning

    Seunghun Baek, Jihwan Park, Jaeyoon Sim, Minjae Jeong, Hoseok Lee, Won Hwa Kim

    ECCV 2026 poster verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. MARS distills a privileged residual router into an available-feature router; it is not a cascade of residual-error student networks. Full modalities are assumed available during training. Food-101 full-modality accuracy is below SimMLM despite higher missing-modality and average scores.

    Official artifacts: Official ECCV poster record · Official code

    Metadata checked