Skip to content
Computational Pathology Reading Roadmap

Text representations and sequence models

This chapter turns raw text into explicit modelling choices. Its endpoint is the attention-based encoder–decoder; transformers are treated in the next chapter.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Individual ICLR oral-paper studies retain their 19 September 2026 source-check dates; the later discovery audit does not re-date that evidence.

Prerequisites and objectives

Section titled “Prerequisites and objectives”

Prerequisites are conditional probability, matrix multiplication, cross-entropy, and gradient descent. By the end, you should be able to:

  • specify a reversible normalization and tokenization pipeline;
  • derive an n-gram or neural language-model objective;
  • distinguish count, static, and contextual representations;
  • trace hidden-state and gradient flow through an RNN/LSTM;
  • explain teacher forcing, exposure bias, and beam-search assumptions;
  • derive additive attention and identify what an attention weight does—and does not— establish.

1. Text is not already a sequence of model tokens

Section titled “1. Text is not already a sequence of model tokens”

Start with Unicode code points, normalization forms, whitespace, punctuation, case, numbers, markup, sentence boundaries, and language-specific segmentation. A lossy normalizer can merge clinically or legally distinct strings. Keep the raw source and make every transform versioned.

Common token units are words, characters, bytes, byte pairs, and unigram subwords. Given a vocabulary V, tokenization controls:

  • sequence length and therefore compute;
  • how rare names, morphology, code, and multilingual scripts are represented;
  • whether copying a substring is easy;
  • which probabilities are comparable across models.

Read BPE for neural machine translation (2016) and SentencePiece (2018). SentencePiece learns from raw sentences and makes detokenization explicit; it does not eliminate corpus or normalization bias.

ByT5 (2022) removes a learned subword vocabulary and models UTF-8 bytes directly. This avoids unknown tokens and can improve robustness to spelling and morphology, but usually creates longer sequences. Compare byte and subword systems at matched compute, not only matched example count.

MEGABYTE (2023) reduces the global sequence length by dividing bytes into fixed patches. A global model passes patch representations to local models that predict bytes inside each patch. Patch size, global/local capacity, and compute matching are therefore part of the method—not free consequences of using bytes.

Byte Latent Transformer (2025) retains byte-level coverage but groups bytes into dynamic patches using a next-byte entropy signal. A local encoder forms patch representations, a lower-rate latent Transformer handles global interaction, and a local decoder returns to bytes. This is not simply a new tokenizer: the patch boundary rule and the local/global compute split are learned-system variables. Compare bytes processed, patch lengths, FLOPs, latency, and downstream quality rather than reporting token counts as if units were identical.

2. Sparse representations and language models

Section titled “2. Sparse representations and language models”

A document–term matrix can contain binary indicators, counts, or TF–IDF values. Linear models over these features remain essential controls because they reveal lexical shortcuts and are cheap to inspect.

Spärck Jones (1972) gives the statistical origin of inverse document frequency: a term occurring in few documents is more specific than one spread through the collection. IDF is a collection-dependent weight, not an intrinsic word meaning; corpus size, document unit, smoothing, and time window change it. This concept later enters BM25 and CIDEr.

An n-gram language model factorizes:

p(x_1:T) ≈ ∏_t p(x_t | x_(t-n+1):x_(t-1))

Smoothing reserves probability for unseen events. Perplexity is the exponentiated mean negative log-likelihood over the chosen tokenization; changing token units changes its denominator.

3. Distributional and static embeddings

Section titled “3. Distributional and static embeddings”

The distributional hypothesis motivates representing a word by its contexts. A Neural Probabilistic Language Model (2003) learns a distributed input representation jointly with a language model. word2vec (2013) uses predictive local-context objectives, while GloVe (2014) factorizes information derived from global co-occurrence.

For skip-gram with negative sampling, the positive pair score is increased and sampled noise-pair scores are reduced. Nearest-neighbour geometry depends on corpus, subsampling, negative distribution, dimension, and similarity metric. Analogies do not demonstrate general semantic understanding, and embeddings can encode social biases in the corpus.

fastText subword embeddings (2017) compose character n-gram features, improving morphology and unseen-word handling without making meanings context-dependent.

4. Recurrent sequence models

Section titled “4. Recurrent sequence models”

A simple recurrent network updates:

h_t = φ(W_x x_t + W_h h_(t-1) + b)

Backpropagation through time multiplies many Jacobians. Depending on their singular values, gradients can vanish or explode. LSTM gates create an additive cell-state path that can preserve information, but recurrence remains sequential and a finite state still compresses history.

For sequence labelling, independent token decisions ignore dependencies among output labels. BiLSTM–CRF NER (2016) combines contextual recurrent features with a globally normalized linear-chain CRF. Reconstruct the emission scores, transition scores, forward algorithm, and Viterbi decode; a CRF constrains label sequences but does not solve annotation ambiguity or nested entities.

Read the original CRF paper (2001) before the neural application. A conditional random field scores an entire label sequence given the observation sequence and normalizes globally with a partition function. Forward–backward computes marginals and Viterbi returns the best sequence. Global normalization avoids the local-normalization label bias identified for maximum-entropy Markov models; it does not make the chosen feature set, label ontology, or linear-chain assumption correct.

Encoder–decoder learning

Section titled “Encoder–decoder learning”

Sequence to Sequence Learning (2014) maps an input sequence to an encoder state and trains a decoder with teacher-forced previous targets. Training sees correct prefixes; free-running inference sees its own errors. This mismatch is exposure bias.

Greedy decoding selects one next token. Beam search keeps several prefixes, but a larger beam is not guaranteed to improve task utility. Length normalization, constraints, stopping, and temperature are part of the evaluated method.

5. Neural attention

Section titled “5. Neural attention”

Neural machine-translation attention (2015) lets a decoder form a step-specific context:

e_(t,s) = vᵀ tanh(W_h h_s + W_d d_(t-1))
α_(t,s) = softmax_s(e_(t,s))
c_t = Σ_s α_(t,s) h_s

Attention removes the single-vector bottleneck and can help alignment. An attention weight is a learned routing coefficient—not automatically a causal explanation or a validated word alignment.

Staged primary-paper route

Section titled “Staged primary-paper route”
Level Primary reading · year · venue/status Read it for Do not conclude
L1–L2 LSTM — 1997 · Neural Computation gated recurrent memory gates remove all long-range or optimization limits
L1 A Neural Probabilistic Language Model — 2003 · JMLR distributed representations learned with prediction modern scale is necessary to understand the objective
L1 word2vec — 2013 · arXiv preprint; GloVe — 2014 · EMNLP predictive versus count-derived geometry nearest neighbours establish fairness or compositional meaning
L1–L2 Sequence to Sequence Learning — 2014 · NeurIPS encoder–decoder factorization and teacher forcing a fixed state preserves every source detail
L2 Neural alignment and attention — 2015 · ICLR content-dependent source routing attention alone is faithful explanation
L1–L2 BiLSTM–CRF NER — 2016 · NAACL-HLT contextual emissions, structured normalization, and constrained decoding a flat BIO tagger represents nested or discontinuous entities
L1 BPE for NMT — 2016 · ACL; SentencePiece — 2018 · EMNLP System Demonstrations open-vocabulary subword interfaces a tokenizer is language-neutral because its algorithm is
L2 ELMo — 2018 · NAACL-HLT contextual representations before transformers context guarantees factual or task-relevant meaning
L2 ByT5 — 2022 · TACL byte-level modelling and tokenizer-free trade-offs bytes make sequence length or multilingual coverage free
L2–L3 MEGABYTE — 2023 · NeurIPS fixed byte patches with separate global and local models the hierarchy makes patch size or compute allocation irrelevant
L3 Byte Latent Transformer — 2025 · ACL entropy-defined dynamic byte patches and local/global processing token count can be compared directly across subword, byte, and patch units

Controlled lab

Section titled “Controlled lab”

Use one fixed corpus and split:

  1. train a TF–IDF linear classifier;
  2. compare word, unigram-subword, and byte tokenization by fertility, unknown handling, sequence length, latency, and subgroup/language;
  3. train a small LSTM classifier and attention encoder–decoder;
  4. report parameter count, tokens per example, wall time, task metrics, calibration, and errors on rare strings;
  5. shuffle or mask diagnostic keywords to expose shortcuts.

For generation, compare greedy and beam decoding while keeping the checkpoint fixed. Inspect omissions, repetitions, unsupported details, and length—not only BLEU.

Failure modes and self-check

Section titled “Failure modes and self-check”
  • A random sentence split can leak authors, templates, documents, or conversations.
  • Lowercasing or stripping punctuation can alter labels and negation.
  • Static embeddings conflate senses.
  • Recurrent final states favor recent information.
  • Teacher-forced loss can improve while free generation degrades.
  • Attention heatmaps can be stable-looking but non-causal.

Without notes, explain why two tokenizers change compute, derive one LSTM gate’s role, write the sequence likelihood, and distinguish hidden-state attention from evidence attribution.

Exit task: produce a data card for the corpus and a tensor-shape trace for one encoder–decoder batch. Then continue to transformers and pretraining.

Boundary tokens and camera-aware relative position

PuzLM converts regular puzzle borders into discrete sequences for a language model, whereas URoPE transports calibrated camera geometry into relative positional features. Both change the representation interface, but neither is a general claim that language models discover geometry without suitable inputs.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
PuzLM: Solving Jigsaw Puzzles with Sequence-to-Sequence Language ModelsGur Elkin, Ofir Itzhak Shahar, and Ohad Ben-Shahar2026 · ECCV · acceptedECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #3026 checked on 2026-09-12. Publisher version of record was not independently verified.PuzLM turns square jigsaw assembly into discrete sequence transduction. A lightweight PCA and k-means tokenizer summarizes each piece's border; a standard encoder-decoder predicts the grid position of each piece. The solver receives symbols rather than original pixels, but those symbols still originate from visual measurements and are not evidence-free reasoning.ECCV Spotlight program · Conference manuscript

Reading question: What geometric information is preserved in the interface, and which calibration or discretization assumptions make that possible?

Evidence boundary: Square-piece puzzle reconstruction is not arbitrary fracture recovery. Parameter-free positional encoding still needs trained host models and known camera parameters.

State-space discretization, rotations and update rank

Mamba-3 separates numerical discretization, complex-state expressivity and MIMO hardware utilization. Follow each mechanism into its own quality or efficiency experiment.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Mamba-3: Improved Sequence Modeling using State Space PrinciplesAakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, Albert Gu2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Three selective-SSM changes evaluated on language and synthetic sequences. Distinguish conditional discretization theory from learned gates, parameter matching from compute matching, and fixed-state decoding from information-retrieval capability. No direct pathology or natural-image evaluation.Mamba-3 changes a recurrent state-space language model along three axes: exponential-trapezoidal discretization, complex-valued state dynamics expressed through input-dependent rotations, and multi-input multi-output (MIMO) updates. The aim is better modeling and state tracking while using memory-bound decoding hardware more effectively.Official proceedings record · OpenReview · Official implementation

Reading question: Which assumptions are enforced, which costs are matched, and where does a fixed state lose information?

Evidence boundary: Parameter matching is not compute matching; language and synthetic-sequence results do not establish WSI performance.

Complete maintained paper route

Section titled “Complete maintained paper route”

21 canonical readings · 1972–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 21 readings in maintained order

  2. #21997Neural ComputationPeer reviewedL1–L2

    Long Short-Term Memory

    Sepp Hochreiter and Jürgen Schmidhuber

    Metadata checked

  19. #192026arXivPreprintL3

    Fast Weight Attention for Continual Learning

    Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, and Andrew Chi-Chih Yao

    arXiv v1 submitted 27 August 2026. The official implementation was available when checked on 2 September 2026; no peer-reviewed record was verified.

    Official artifacts: Official implementation

    Metadata checked

  21. #212026ICLRPeer reviewedL3

    Mamba-3: Improved Sequence Modeling using State Space Principles

    Aakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, Albert Gu

    Official ICLR 2026 oral and full proceedings manuscript checked. Three selective-SSM changes evaluated on language and synthetic sequences. Distinguish conditional discretization theory from learned gates, parameter matching from compute matching, and fixed-state decoding from information-retrieval capability. No direct pathology or natural-image evaluation.

    Official artifacts: Official proceedings record · OpenReview · Official implementation

    Metadata checked