Skip to content
Computational Pathology Reading Roadmap

LLM pretraining, data, and scaling

This chapter explains what a base LLM is and how to compare pretraining claims without reducing them to parameter count.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Individual ICLR oral-paper studies retain their 19 September 2026 source-check dates; the later discovery audit does not re-date that evidence.

Objectives

Section titled “Objectives”

You should be able to derive causal language modelling, interpret scaling-law plots within their fitted regime, distinguish total from active MoE parameters, audit a training-data statement, estimate major memory terms, and design an equal-compute comparison.

1. Causal modelling and token accounting

Section titled “1. Causal modelling and token accounting”

A causal model minimizes:

L = - Σ_t log p_θ(x_t | x_<t)

The effective examples are token prefixes, not documents alone. Report tokenizer, unique and repeated tokens, sequence packing, context length, batch tokens, epochs or reuse, and data mixture. A lower token-level loss is not directly comparable across tokenizers.

The corpus pipeline should document sources, collection dates, licenses/terms, language and domain mixture, quality filters, personally identifiable information, toxicity filters, exact and near deduplication, benchmark exclusion, and removals. Filtering encodes values and can erase minority language or domain material.

Deduplicating Training Data (2022) connects duplicate removal to memorization and evaluation leakage. RefinedWeb (2023) and Dolma (2024) expose different web-corpus filtering and documentation choices. DoReMi (2023) optimizes domain weights against a reference model. None establishes a universally optimal mixture: inspect domains, exclusions, licenses, and downstream selection.

2. Scaling laws and compute allocation

Section titled “2. Scaling laws and compute allocation”

Scaling Laws for Neural Language Models (2020) fits power-law relationships between loss and model/data/compute in measured regimes. Chinchilla (2022) revisits compute-optimal parameter/token allocation. These empirical fits do not prove downstream usefulness, truthfulness, safe behavior, or permanent exponents.

For a fixed training compute budget, compare:

  • parameter count and active parameter count;
  • unique versus repeated tokens and mixture;
  • optimizer and schedule;
  • achieved training loss and downstream curves;
  • inference memory, throughput, and latency;
  • data rights and reproducibility.

3. Architecture

Section titled “3. Architecture”

A decoder block combines pre-normalized self-attention, residual paths, and a feed forward network. Modern variants may use RoPE, RMSNorm (2019), SwiGLU, grouped-query or multi-query attention, and tied or untied embeddings.

Mixture-of-experts (MoE) layers route each token to a subset of experts. Record total and active parameters, router objective, expert capacity, dropped tokens, load-balancing, communication, and inference placement. Switch Transformers (2022) is a useful anchor. Sparse activation does not make communication or memory free. Mamba-2 (2024) connects selective state-space models to structured attention-like algorithms. Treat it as an alternative sequence architecture, not evidence that one family dominates across data, scale, and hardware.

Byte hierarchy and attention sparsity change different axes. MEGABYTE (2023) uses fixed byte patches, a global patch model, and local byte models. Byte Latent Transformer (2025) retains the local/global hierarchy but makes patch boundaries data-dependent through next-byte entropy. Native Sparse Attention retains a Transformer-style stream but restricts interaction through compressed, selected, and local attention branches co-designed with kernels. Match training data, processed bytes, loss definition, hardware, and wall-clock budget before attributing a gain to patching or sparsity.

TTT Layers (2025) treats the hidden state of a recurrent sequence model as the weights of an inner learner, updated online by a self-supervised loss. This changes the state contract: report which weights mutate, the update/reset boundary, update cost, and causal information available to the loss. Modular TTT (2026) represents the inner learner as a modular directed acyclic graph. Its fast-weight network, loss, learning rate, weight decay, and normalization are explicit design choices rather than one fixed TTT block. Record that graph, the initial checkpoint and mutable fast weights, reset boundary, sequence order, and update cost; a nominal parameter count does not describe the effective stateful system.

4. Context and training systems

Section titled “4. Context and training systems”

Accepted context length is not demonstrated effective context. Position method, training length distribution, data packing, attention kernel, KV cache, and evaluation position matter.

Major training-memory terms include parameters, gradients, optimizer states, and activations. Mixed precision, sharding, rematerialization, fused kernels, parallelism, and communication overlap change systems cost. Report hardware, precision, utilization, tokens/second, failures, and carbon/energy estimates when available.

FlashAttention (2022) reduces memory traffic for exact attention. Grouped-query attention reduces inference KV storage. Quantization can reduce weight memory, but kernel support and accuracy must be measured for the actual hardware.

Long-context training also depends on whether examples contain useful distant dependencies. LongFilter (2026) proposes an information-gain criterion for selecting such data. It is frontier evidence: reproduce selection cost and compare against length-, domain-, and quality-matched controls before adopting the recipe.

Staged reading route

Section titled “Staged reading route”
Level Reading · year · status Read it for Boundary
L2 RMSNorm — 2019 · NeurIPS normalization statistics and computational claim a component benchmark predicts end-to-end training speed
L1–L2 GPT-3 — 2020 · NeurIPS causal scaling and in-context task interface few-shot prompts do not remove contamination
L2 Scaling Laws — 2020 · preprint; Chinchilla — 2022 · NeurIPS fitted loss/compute/data relationships extrapolation is not a law of task utility
L2–L3 Deduplication — 2022 · ACL; DoReMi — 2023 · NeurIPS duplicate control and optimized domain mixtures one filtering or weighting recipe transfers unchanged
L2 RefinedWeb — 2023 · NeurIPS Datasets & Benchmarks; Dolma — 2024 · ACL inspectable corpus construction and disclosure “open” removes licensing, representation, or contamination questions
L2–L3 Switch Transformers — 2022 · JMLR sparse expert routing and scaling active parameters capture communication
L2–L3 PaLM — 2023 · JMLR; LLaMA — 2023 · technical report scale, training recipes, data/compute disclosure model-family comparisons are controlled ablations
L2–L3 MEGABYTE — 2023 · NeurIPS fixed byte patches and global/local sequence modeling lower global sequence length makes total compute automatically matched
L2–L3 OLMo — 2024 · ACL an inspectable data/code/checkpoint training stack open artifacts guarantee unbiased data
L3 Mixtral — 2024 · technical report; DeepSeek-V3 — 2024 · technical report current MoE system designs report status or self-reported efficiency equals independent evidence
L3 Mamba-2 — 2024 · ICML structured state-space duality and efficient algorithms asymptotic form determines measured quality or systems cost
L3 TTT Layers — 2025 · ICML online self-supervised updates as expressive recurrent hidden state mutable state and per-token updates are free at inference
L3 OLMo 2 — 2025 · COLM; Qwen3 — 2025 · technical report transparent recipe iteration and hybrid thinking modes latest checkpoint is universally preferable
L3 LongFilter — 2026 · ICLR information-based selection for long-context pretraining one accepted frontier result is a settled data recipe

Controlled lab

Section titled “Controlled lab”

Train small decoder models under one fixed token/compute budget while varying only:

  1. parameter count versus tokens;
  2. tokenizer;
  3. context length distribution;
  4. dense versus small MoE, if resources permit.

Plot validation loss against tokens and compute. Measure downstream accuracy, calibration, memorization, tokens/second, peak memory, and inference KV-cache size. Do not tune each variant on the test set.

Exit check

Section titled “Exit check”

For one base model, state exact checkpoint, tokenizer, data mixture/cutoff, objective, tokens, total/active parameters, context method, optimizer, compute/hardware, license, contamination controls, and measured inference cost. Unknowns remain unknowns.

Data-constrained ensembles and sequence distillation

Regularization and independent teachers extract more from a fixed real corpus; mixed real/synthetic sequence training transfers part of the ensemble benefit into one student.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Pre-training under infinite computeKonwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto2026 · ICLR · peer-reviewedOfficial ICLR 2026 oral and full proceedings manuscript checked. Fixed unique data, regularized teachers and real/synthetic sequence distillation. Infinite-compute claims are fitted asymptotes, not completed runs. Distinguish main mixing recipes from the matched-token ablation and inference parameter savings from total training cost.This study changes the optimization question from a fixed compute budget to a fixed supply of real data. It combines strong regularization, independently trained ensembles and sequence distillation to improve data-constrained language models. “Infinite compute” names a fitted limiting regime, not a completed infinite training run.Official proceedings record · OpenReview

Reading question: What is held fixed: unique data, total tokens, parameters, generation cost or deployment cost?

Evidence boundary: Fitted infinite-compute limits are not measured runs, and a smaller deployed student is not automatically cheaper to train.

Distillation changes capabilities, not only average scores

Distilled Pretraining separates access to teacher data from the effect of soft targets. It contrasts context copying with multi-sample solution coverage and tests entropy-based token routing. A student can improve on pass@k yet become less responsive to supplied context.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time ScalingSachin Goyal, David Lopez-Paz, and Kartik Ahuja2026 · ICLR · peer-reviewedDistilled pretraining transfers a teacher’s full next-token distribution, not only sampled text. The study finds a capability trade-off: soft targets can improve multi-sample solution coverage while weakening context-copying behavior in a matched-data regime. Entropy-based token routing partially mitigates the latter rather than making distillation uniformly beneficial.Proceedings PDF · arXiv

Reading question: Which capability benefits from teacher smoothing, and which needs the original hard target?

Evidence boundary: Matched token counts do not match total teacher–student compute. Context-copying experiments and pass@k are separate endpoints; routing results are not a universal recipe or demonstrated pathology transfer.

Complete maintained paper route

Section titled “Complete maintained paper route”

40 canonical readings · 2017–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 40 readings in maintained order

  19. #192024arXivTechnical reportL2–L3

    Mixtral of Experts

    Albert Q. Jiang et al.

    Metadata checked

  23. #232024NeurIPSPeer reviewedL2–L3

    Discrete Flow Matching

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman

    Metadata checked

  25. #252025COLMPeer reviewedL3

    2 OLMo 2 Furious (COLM’s Version)

    OLMo Team

    The COLM 2025 paper identifies OLMo Team as the corporate author and provides the full contributor list in the paper.

    Metadata checked

  27. #272025ICMLPeer reviewedL2–L3

    Distillation Scaling Laws

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb

    Metadata checked

  36. #362026arXivPreprintL3

    Sliding-window beats linear attention

    Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais

    arXiv v1 submitted 28 August 2026. No peer-reviewed record, code release, or model artifact was verified on 2 September 2026; it is retained for its unusually important negative baseline evidence rather than publication status.

    Metadata checked

  37. #372026arXivPreprintL3

    Fast Weight Attention for Continual Learning

    Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, and Andrew Chi-Chih Yao

    arXiv v1 submitted 27 August 2026. The official implementation was available when checked on 2 September 2026; no peer-reviewed record was verified.

    Official artifacts: Official implementation

    Metadata checked

  38. #382026arXivPreprintL3

    Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

    Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Mingzhe Zhang, Kaiyue Wen, Kaifeng Lyu, and Wenguang Chen

    arXiv v1 submitted 27 August 2026. Catalog authors follow the current arXiv metadata; the PDF title page additionally names Chengxia Li, an unresolved source-metadata discrepancy that should be rechecked on revision. Code, data, and checkpoints were released, but no peer-reviewed record was verified.

    Official artifacts: Official training implementation · Official models and data collection

    Metadata checked

  39. #392026ICLRPeer reviewedL3

    Pre-training under infinite compute

    Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto

    Official ICLR 2026 oral and full proceedings manuscript checked. Fixed unique data, regularized teachers and real/synthetic sequence distillation. Infinite-compute claims are fitted asymptotes, not completed runs. Distinguish main mixing recipes from the matched-token ablation and inference parameter savings from total training cost.

    Official artifacts: Official proceedings record · OpenReview

    Metadata checked