Skip to content
Computational Pathology Reading Roadmap

Transformers and language pretraining

This chapter explains the transformer as a computation graph and then separates architecture, objective, data, scale, and adaptation. Those variables often change together in papers but should not be treated as one idea.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Individual ICLR oral-paper studies retain their 19 September 2026 source-check dates; the later discovery audit does not re-date that evidence.

Prerequisites and objectives

Section titled “Prerequisites and objectives”

Complete text representations and sequence models. You should be able to:

  • write Q/K/V tensor shapes and the attention/masking equations;
  • distinguish encoder, causal decoder, and encoder–decoder interfaces;
  • explain absolute, relative, rotary, and linear-bias position strategies;
  • compare dense, sparse, recurrent-memory, state-space, and IO-aware sequence methods;
  • map MLM, causal, denoising, and replaced-token objectives to model behavior;
  • choose frozen features, adapters, LoRA, or full fine-tuning under an explicit budget.

1. One transformer block

Section titled “1. One transformer block”

For input X ∈ R^(T×d) and h heads:

Q = XW_Q,  K = XW_K,  V = XW_V
Attention(Q,K,V) = softmax((QKᵀ / √d_k) + mask)V

Multi-head outputs are concatenated and projected, followed by a position-wise MLP. Residual connections and normalization form the optimization pathway. Record whether normalization is pre- or post-sublayer and whether the MLP uses ReLU, GELU, gated linear units, or another activation. RMSNorm (2019) normalizes by root-mean-square magnitude without mean centering; its efficiency claim must be tested inside the complete training stack.

Masks define the task:

  • bidirectional encoder: all non-padding tokens may interact;
  • causal decoder: position t cannot see future positions;
  • encoder–decoder: bidirectional source encoding plus causal target decoding and source cross-attention;
  • prefix or block masks: selected regions have different visibility.

The quadratic attention matrix costs O(T²) elements. Kernel choice, activation storage, KV caching, batch shape, precision, and hardware determine observed memory and latency.

2. Position and long sequences

Section titled “2. Position and long sequences”

Attention without position is permutation equivariant. Learned or sinusoidal absolute vectors add position; relative biases modify pair scores; RoPE rotates Q/K features; ALiBi adds a distance-dependent bias.

Transformer-XL (2019) reuses hidden states across segments. Longformer and BigBird (2020) use sparse attention patterns. FlashAttention (2022) computes exact dense attention with an IO-aware algorithm; it changes implementation cost, not the mathematical attention pattern. Mamba (2024) is a selective state-space alternative, while Mamba-2 (2024) develops state-space duality with attention-like structured algorithms. Compare quality and measured systems behavior, not asymptotic notation alone.

Native Sparse Attention (2025) combines compressed block-level context, selected fine-grained blocks, and a local sliding window in a sparsity pattern intended to be trained natively and executed with hardware-aligned kernels. Separate three claims: which evidence the sparse pattern can reach, whether optimization learns to use it, and whether the implementation is faster on declared hardware. A byte hierarchy changes the sequence units instead: MEGABYTE (2023) makes the global model operate over fixed byte patches with local byte models; Byte Latent Transformer (2025) replaces those fixed boundaries with entropy-defined dynamic patches. Sparse interaction and lower-rate units can be combined conceptually, but they are not the same efficiency mechanism.

3. Pretraining objective families

Section titled “3. Pretraining objective families”
Interface Representative objective What it naturally supports Important boundary
encoder masked-token or replaced-token prediction contextual representations, classification, extraction [MASK] mismatch and no native free-form decoder
causal decoder next-token likelihood continuation, prompting, open-ended generation left-to-right conditioning and exposure to generated prefixes
encoder–decoder span corruption or text-to-text denoising conditional generation and transformation two stacks and task formatting choices

BERT (2019) establishes masked encoder pretraining. ELECTRA (2020) learns to identify replaced tokens. BART and T5 develop denoising/text-to-text encoder–decoder interfaces. GPT-3 (2020) demonstrates scale and in-context task formatting for causal models.

Objective, corpus, tokenizer, parameter count, optimization, and inference procedure all affect a result. A fair ablation changes one controlled factor.

Multilingual models add another axis: language sampling, script coverage, tokenizer fertility, parallel versus monolingual data, and transfer direction. XLM-R (2020) studies multilingual masked pretraining at scale; mBART (2020) pretrains a multilingual denoising encoder–decoder; MAD-X (2020) composes language and task adapters for modular transfer; mT5 (2021) extends text-to-text transfer; and ByT5 (2022) tests byte-level modelling. Report per-language results and negative transfer rather than only a macro average.

4. Transfer and parameter-efficient adaptation

Section titled “4. Transfer and parameter-efficient adaptation”
  • Frozen representation: train only a probe; tests representation accessibility.
  • Full fine-tuning: update all weights; higher capacity and storage/forgetting risk.
  • Adapters: insert small trainable modules.
  • Prompt/prefix tuning: learn continuous task tokens or KV prefixes.
  • LoRA: learn low-rank weight updates.
  • QLoRA: backpropagate through a quantized frozen base into LoRA parameters.

Compare adaptation methods with the same base checkpoint, data order, tokens, tuning budget, stopping rule, precision, and evaluation. Trainable-parameter count alone does not capture optimizer states, activations, quantization kernels, or inference cost.

Staged primary-paper route

Section titled “Staged primary-paper route”
Level Reading · year · venue/status Concept to reconstruct
L1–L2 Attention Is All You Need — 2017 · NeurIPS multi-head self/cross-attention, mask, residual stream, position
L2 RMSNorm — 2019 · NeurIPS normalization statistics and optimization pathway
L2 GPT-1 — 2018 · technical report; BERT — 2019 · NAACL-HLT causal versus masked pretraining and task adaptation
L2 RoBERTa — 2019 · preprint; ELECTRA — 2020 · ICLR recipe versus objective changes
L2 BART — 2020 · ACL; T5 — 2020 · JMLR denoising and unified text-to-text transfer
L2 XLM-R — 2020 · ACL; mBART — 2020 · TACL; mT5 — 2021 · NAACL-HLT; ByT5 — 2022 · TACL masked versus denoising multilingual transfer, language sampling, and byte/subword interfaces
L2–L3 MAD-X — 2020 · EMNLP compose language and task adapters; test unseen-language transfer against full fine-tuning under matched capacity
L2–L3 MiniLM — 2020 · NeurIPS task-agnostic compression by matching self-attention and value relations; record the transferred matrices and student serving gain
L2–L3 Transformer-XL — 2019 · ACL; BigBird — 2020 · NeurIPS memory reuse and sparse long-context patterns
L2–L3 RoPE — 2024 · Neurocomputing; ALiBi — 2022 · ICLR relative geometry and length extrapolation assumptions
L2–L3 FlashAttention — 2022 · NeurIPS; GQA — 2023 · EMNLP training IO and decoder KV-cache efficiency
L2–L3 MEGABYTE — 2023 · NeurIPS; Byte Latent Transformer — 2025 · ACL fixed then entropy-defined byte patches, with local/global compute accounting
L2 Adapters — 2019 · ICML; LoRA — 2022 · ICLR; QLoRA — 2023 · NeurIPS where adaptation capacity and memory actually reside
L3 Mamba — 2024 · COLM; Mamba-2 — 2024 · ICML selective state spaces and structured state-space duality

Controlled labs

Section titled “Controlled labs”
  1. Implement single-head then multi-head attention with padding and causal masks; test that masked logits cannot affect the output.
  2. Measure peak memory and wall time over sequence length for a reference attention implementation and an optimized kernel.
  3. Fine-tune one encoder using a linear probe, LoRA, and full updating under a common token/tuning budget. Report accuracy, calibration, memory, wall time, and variance.
  4. For long context, move the answer while holding token count constant; add distractors and report retrieval-by-position, not only average accuracy.

Failure modes and exit check

Section titled “Failure modes and exit check”

Common errors are omitting the attention mask, comparing different token budgets, calling an optimized exact kernel “linear attention,” claiming long-context ability from accepted length alone, and tuning one adaptation method more heavily.

Exit task: draw one transformer block with tensor shapes; state which weights are trained; calculate the attention-logit matrix size; and explain what changes between MLM, causal, and span-corruption runs. Then continue to tasks, retrieval, and evaluation.

Boundary tokens and camera-aware relative position

PuzLM converts regular puzzle borders into discrete sequences for a language model, whereas URoPE transports calibrated camera geometry into relative positional features. Both change the representation interface, but neither is a general claim that language models discover geometry without suitable inputs.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
URoPE: Universal Relative Position Embedding across Geometric SpacesYichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, and Wei Zhan2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #11480 checked on 2026-09-12. Publisher version of record was not independently verified.URoPE extends rotary position embedding to tokens from different cameras or geometric spaces. A key image patch is lifted along its camera ray at fixed candidate depths, then projected into the query camera. Ordinary two-dimensional RoPE operates on the resulting positions. Geometry changes the positional interface without adding a learned depth predictor.ECCV Spotlight program · Conference manuscript · Official project or implementation

Reading question: What geometric information is preserved in the interface, and which calibration or discretization assumptions make that possible?

Evidence boundary: Square-piece puzzle reconstruction is not arbitrary fracture recovery. Parameter-free positional encoding still needs trained host models and known camera parameters.

State-space discretization, rotations and update rank

Mamba-3 separates numerical discretization, complex-state expressivity and MIMO hardware utilization. Follow each mechanism into its own quality or efficiency experiment.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Mamba-3: Improved Sequence Modeling using State Space PrinciplesAakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, Albert Gu2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Three selective-SSM changes evaluated on language and synthetic sequences. Distinguish conditional discretization theory from learned gates, parameter matching from compute matching, and fixed-state decoding from information-retrieval capability. No direct pathology or natural-image evaluation.Mamba-3 changes a recurrent state-space language model along three axes: exponential-trapezoidal discretization, complex-valued state dynamics expressed through input-dependent rotations, and multi-input multi-output (MIMO) updates. The aim is better modeling and state tracking while using memory-bound decoding hardware more effectively.Official proceedings record · OpenReview · Official implementation

Reading question: Which assumptions are enforced, which costs are matched, and where does a fixed state lose information?

Evidence boundary: Parameter matching is not compute matching; language and synthetic-sequence results do not establish WSI performance.

Complete maintained paper route

Section titled “Complete maintained paper route”

45 canonical readings · 2017–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 45 readings in maintained order

  40. #402026arXivPreprintL3

    Sliding-window beats linear attention

    Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais

    arXiv v1 submitted 28 August 2026. No peer-reviewed record, code release, or model artifact was verified on 2 September 2026; it is retained for its unusually important negative baseline evidence rather than publication status.

    Metadata checked

  41. #412026arXivPreprintL3

    Fast Weight Attention for Continual Learning

    Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, and Andrew Chi-Chih Yao

    arXiv v1 submitted 27 August 2026. The official implementation was available when checked on 2 September 2026; no peer-reviewed record was verified.

    Official artifacts: Official implementation

    Metadata checked

  42. #422026arXivPreprintL3

    Normalized Low-Rank Adaptation

    Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, and Weiyang Liu

    arXiv v1 submitted 31 August 2026. The authors released an implementation; no peer-reviewed record was verified on 2 September 2026.

    Official artifacts: Official implementation

    Metadata checked

  44. #442026ECCVAccepted · proceedings pendingL3

    URoPE: Universal Relative Position Embedding across Geometric Spaces

    Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu, Quentin Herau, Masayoshi Tomizuka, and Wei Zhan

    ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #11480 checked on 2026-09-12. Publisher version of record was not independently verified.

    Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation

    Metadata checked

  45. #452026ICLRPeer reviewedL3

    Mamba-3: Improved Sequence Modeling using State Space Principles

    Aakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, Albert Gu

    Official ICLR 2026 oral and full proceedings manuscript checked. Three selective-SSM changes evaluated on language and synthetic sequences. Distinguish conditional discretization theory from learned gates, parameter matching from compute matching, and fixed-state decoding from information-retrieval capability. No direct pathology or natural-image evaluation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked