Transformer computation and scaling
Systems methods partition concrete parameters, activations, attention, and states.
LLM training and serving systems
The effective model includes its precision, partition, kernel, cache, scheduler, and request distribution. This route connects systems claims to measured evidence.
31 maintained papers · literature reviewed through 2 October 2026
Precision, sharding, tensor and pipeline parallelism, attention kernels, KV-cache management, speculative decoding, and quantization alter feasible batch, sequence, checkpoint, and latency regimes. Quality, memory, throughput, latency, communication, and reliability must be measured at a declared system point.
Prerequisite contract
Reproducible benchmarking
Hardware, load, sequence, batch, and quality define the comparison.
How the field developed
2018–2021
Precision and distributed state partitioning
Mixed precision reduces arithmetic and memory cost; ZeRO partitions optimizer, gradient, and parameter states; Megatron combines tensor and pipeline parallelism.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Mixed Precision Training Mixed-precision training uses FP16 tensors for most forward and backward computation, accumulates vulnerable arithmetic in FP32, and updates an FP32 master copy of weights before recasting them to FP16. Loss scaling protects small gradients from FP16 underflow. Together these safeguards preserve FP32-like accuracy while enabling faster, lower-memory arithmetic on supporting hardware. — 01.02 ZeRO: Memory Optimizations Toward Training Trillion Parameter Models ZeRO removes the replicated model-state memory hidden inside data parallel training. Its cumulative stages partition optimizer states, then gradients, then parameters across data-parallel workers while dynamically communicating each state when needed. ZeRO-R separately attacks residual memory from activation checkpoints, temporary buffers, and fragmentation. The result preserves large compute kernels while making aggregate cluster memory usable. DeepSpeed 01.03 Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Megatron-LM composes three parallelism axes for large Transformer training: tensor parallelism splits operations inside a layer, pipeline parallelism assigns layer groups to stages, and data parallelism replicates whole model partitions across batches. The PTD-P arrangement keeps high-communication tensor collectives within fast multi-GPU servers, pipelines across servers, and uses an interleaved 1F1B schedule to shrink idle bubbles. Megatron-LM Work: Account for every parameter, gradient, optimizer state, activation, and communication edge.
Exit evidence: Predict per-device memory and compare it with measured peak allocation.
2019–2025
Distilled language- and multimodal-model deployment
DistilBERT and MiniLM compress encoder pretraining states; MiniLLM and DistiLLM change the divergence and sampling policy for generation; MiniPLM refines the pretraining corpus offline; TAID adapts targets over time; LLaVA-MoD adds a sparse MoE student and preference distillation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and LighterWorkshop paper and arXiv preprint; not a main-track NeurIPS proceedings paper. DistilBERT compresses BERT during language-model pretraining rather than only after task fine-tuning. A six-layer student initialized from alternating BERT-base layers learns from three simultaneous signals: the teacher token distribution, masked-token ground truth, and cosine alignment of hidden states. Removing half the Transformer blocks, token-type embeddings, and the pooler yields a 66M-parameter encoder intended to preserve much of BERT's general English representation at lower inference cost. Official distillation code 02.02 MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers MiniLM compresses a pretrained Transformer before task-specific fine-tuning by matching relations inside the teacher's final self-attention layer. It distills both attention distributions and a newly defined value-relation matrix. Because supervision is taken from only the last layer, teacher and student may differ in depth and hidden size without hand-designed layer mapping. Official implementation and checkpoints 02.03 MiniLLM: Knowledge Distillation of Large Language Models MiniLLM distills an autoregressive language model by minimizing reverse KL from the student sequence distribution to the teacher. The student generates its own responses, the teacher scores those trajectories, and policy-gradient machinery moves probability toward teacher-supported sequences. Reverse KL is mode seeking: it discourages the student from assigning mass to low-teacher-probability outputs rather than forcing it to cover every teacher mode. Official code 02.04 DistiLLM: Towards Streamlined Distillation for Large Language ModelsPeer-reviewed ICML 2024 proceedings paper. DistiLLM targets two coupled failures in autoregressive language-model distillation: unstable token-distribution matching and train/test exposure mismatch. It replaces ordinary forward or reverse KL with skew KL divergences whose mixture reference controls estimator behavior, then adaptively trains on student-generated sequences and reuses recent generations through an off-policy buffer. The student therefore learns both from fixed instruction data and from states its own policy is likely to visit. Official code 02.05 TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models TAID bridges a large language-model capacity gap by moving the student's target gradually from its own current distribution toward the teacher. At early steps, the interpolated target remains close to a detached student and is easy to fit; as learning stabilizes, the interpolation coefficient increases and exposes more teacher knowledge. The schedule adapts to objective progress instead of following only wall-clock training. Official code 02.06 MiniPLM: Knowledge Distillation for Pre-Training Language ModelsPeer-reviewed ICLR 2025 conference paper. MiniPLM moves language-model distillation out of the training loss and into pretraining-data selection. A large teacher and a much smaller reference model score each candidate sequence; examples with high log pteacher(x) − log preference(x) are selected, and any student is then trained from scratch with ordinary next-token prediction. Knowledge transfer becomes a reusable ranked corpus rather than stored token logits or a teacher loaded beside every student update. Official code, data, and models 02.07 LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge DistillationPeer-reviewed ICLR 2025 conference paper. LLaVA-MoD builds a compact multimodal language model by combining sparse mixture-of-experts capacity with progressive knowledge distillation. A dense small Qwen-family MLLM first mimics general teacher behavior, its feed-forward blocks are then upcycled into routed experts for specialized multi-task transfer, and preference distillation finally pushes positive responses above and negative responses below the large teacher's relative likelihood. Only a subset of experts activates per token, so total capacity and inference compute must be reported separately. Official code Work: Deploy teacher and student checkpoints on one stack; separate online/offline teacher cost, pretraining/post-training transfer, activated/total parameters, and steady-state inference cost.
Exit evidence: Report parameters, activated experts, memory, quality, calibration, prefill/decode latency, throughput, teacher amortization, and tokenizer compatibility.
2022–2023
Attention kernels and serving memory
FlashAttention reduces memory traffic with tiled exact attention; vLLM manages fragmented KV caches with PagedAttention.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness FlashAttention computes the same softmax attention output as standard attention but changes the execution schedule: it tiles Q, K, and V into on-chip SRAM, maintains online softmax statistics, and never materializes the full N×N score or probability matrix in high-bandwidth memory (HBM). It trades some recomputation for far fewer slow-memory transfers, making this an IO-aware exact algorithm rather than an approximate attention rule. — 03.02 Efficient Memory Management for Large Language Model Serving with PagedAttention vLLM applies a virtual-memory analogy to autoregressive serving: PagedAttention divides each request's growing key-value cache into fixed-size logical blocks and maps them to non-contiguous GPU physical blocks. Memory is allocated only when tokens arrive, so a request no longer reserves one maximum-length contiguous region. A central scheduler can fill the recovered memory with more concurrent requests, share prompt blocks through copy-on-write, and preempt under pressure. The 2023 claim is a systems result—higher serving throughput at comparable latency and unchanged model outputs—not a new language model or attention approximation. Serving system Work: Profile compute, memory bandwidth, cache occupancy, prefill, decode, and queueing separately.
Exit evidence: Report latency distributions and throughput under an explicit request trace.
2023–2026
Drafting and low-precision inference
Speculative decoding verifies draft tokens; SmoothQuant moves activation difficulty into weights; AWQ protects salient weight channels without full reconstruction; SliderQuant adapts precision further.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Fast Inference from Transformers via Speculative Decoding Speculative decoding uses a cheap draft model to propose several autoregressive tokens, evaluates all corresponding prefixes in one parallel target-model call, accepts each proposal with a probability determined by the target-to-draft probability ratio, and samples a correction from the positive residual at the first rejection. If every draft is accepted, it draws one additional target token. This draft-many, verify-once pattern can turn one serial target step into as many as gamma plus one output tokens while preserving the target model's standardized output distribution exactly. — 04.02 SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models SmoothQuant makes full W8A8 post-training quantization practical for large transformers by applying an offline, mathematically equivalent channel scaling: it suppresses persistent activation outliers and moves part of their dynamic-range burden into weights, which are empirically easier to quantize. The transformed linear layer computes the same floating-point function before rounding, so runtime can use regular INT8 matrix multiplications rather than a mixed FP16 outlier path. Code 04.03 AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration AWQ is a post-training, weight-only quantization method that protects weight channels identified as salient by activation magnitude. Rather than keeping those channels in higher precision, it rescales weights and inversely rescales activations before low-bit quantization so the exact floating-point layer is unchanged but quantization error is redistributed. TinyChat then realizes the memory reduction as speed through fused dequantization and device-aware kernels. Official project page · Official implementation and model zoo 04.04 SliderQuant: Accurate Post-Training Quantization for LLMsPeer-reviewed ICLR 2026 poster; retain as a frontier quantization method after systems and quantization foundations. SliderQuant replaces uniform sequential post-training quantization with an adaptive overlapping-window schedule. It repeatedly includes the sensitive first layer while expanding through shallow layers, uses fixed overlapping windows through the middle, contracts toward the sensitive last layer, and incrementally quantizes parameter slices inside each window. The goal is to optimize cross-layer error where the paper observes it matters most, especially under W4A4 and two-bit weight settings. Code Work: Hold outputs or quality tolerance fixed while measuring calibration, memory, kernels, and end-to-end serving cost.
Exit evidence: Publish acceptance, quantization error, fallback, memory, and quality-by-latency curves.
2024
Disaggregated serving and structured generation runtimes
DistServe separates prefill and decode to satisfy per-phase latency objectives; SGLang exposes a structured generation language and RadixAttention runtime for prefix sharing and scheduling.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving DistServe places LLM prefill and autoregressive decoding on separate GPU pools so compute-heavy prompt processing cannot interrupt bandwidth-heavy token generation. It independently chooses parallelism and replica counts for the two phases, transfers the first-token state and KV cache between them, and places matching model stages near one another. The optimization target is per-GPU goodput: maximum request arrival rate that meets both time-to-first-token and time-per-output-token SLOs at a required attainment fraction. Official implementation 05.02 SGLang: Efficient Execution of Structured Language Model Programs SGLang couples a Python-embedded language for structured language-model programs with a specialized runtime that exploits structure across requests and program steps. The frontend expresses generation, parallel branches, control flow, external functions, and multimodal inputs. The runtime interprets those operations, reuses shared KV-cache prefixes through RadixAttention, accelerates constrained decoding with compressed finite-state machines, and can speculatively execute API calls. Official implementation Work: Replay the same request trace while varying prefill/decode placement, prefix reuse, batching, and admission control.
Exit evidence: Report time-to-first-token, inter-token latency, throughput, cache hit rate, utilization, and SLO attainment.
2025–2026
Native sparsity and stateful sequence modules
Native Sparse Attention co-designs trainable sparse attention with hardware-aligned kernels. TTT Layers makes the recurrent hidden state a fast-weight model updated by a learned self-supervised rule; Modular TTT then expresses that inner learner as a configurable DAG of fast-weight network, loss, learning rate, weight decay, and normalization.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionACL 2025 Best Paper. Native Sparse Attention replaces one full causal attention map with three jointly trained views of the prefix. A compression branch summarizes overlapping blocks for global coverage; those compression scores nominate a small set of contiguous fine-grained blocks; and a sliding-window branch preserves recent detail. Separate key/value projections and learned gates combine the three outputs. Crucially, the sparse pattern is used during pretraining, prefilling, decoding, and backpropagation, and the selection granularity is chosen for contiguous Tensor-Core-friendly memory access rather than theoretical sparsity alone. — 06.02 Learning to (Learn at Test Time): RNNs with Expressive Hidden States A TTT layer makes its recurrent hidden state a small machine-learning model. For every input sequence, that inner model is updated online with a learned self-supervised objective and then queried to emit token representations. The slow network learns how to construct the inner task; the fast weights learn from the current sequence. TTT-Linear uses a linear inner model and TTT-MLP uses a two-layer MLP, retaining constant state size and linear sequence complexity. Official code 06.03 Modular TTT: Rethinking Test-Time Training as Composable ModulesPreprint status retained; no peer-reviewed proceedings record was available when checked on 2026-08-10. Modular TTT represents a test-time-training memory as a directed acyclic graph rather than a separately hand-derived layer. Each node registers three local rules—train-view forward, train-view backward, and causal query-view forward—and the framework composes them into a chunk-level online learner whose fast weights change while a sequence is processed. This makes the fast network, loss, learned inner learning rate, forgetting rule, activation, and normalization independently swappable design axes. Official code Work: Match quality, context length, training tokens, hardware, and precision before comparing dense attention, native sparsity, and stateful adaptation.
Exit evidence: Report both algorithmic complexity and measured training/prefill/decode curves.
2024–2026
Prune, distil, and audit causal serving
Minitron couples structured pruning with KD; GISP builds nested globally ranked subnetworks; prefix-invariance auditing detects information paths that a nominal causal mask may miss. Structural sparsity still requires backend timing.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 Compact Language Models via Pruning and Knowledge DistillationPublished in the NeurIPS 2024 main conference proceedings. Core mechanism: structured pruning followed by distillation. Minitron derives smaller language models by removing transformer depth, hidden dimensions, attention heads, or MLP channels from a pretrained base, initializing the child with retained weights, and recovering quality through knowledge distillation on a small fraction of pretraining data. This reading boundary matters: the recipe amortizes one expensive pretraining run into related compact checkpoints, not a claim that arbitrary architectures can be pruned without architecture- and data-specific recovery. Official implementation · Official model collection 07.02 From Local to Global: Revisiting Structured Pruning Paradigms for Large Language ModelsPeer-reviewed ACL 2026 long paper and Outstanding Paper award recipient. GISP prunes structured units by comparing their importance globally across the whole language model, then removes them gradually rather than in one destructive step. First-order loss sensitivity is aggregated into attention-head and MLP-channel scores, normalized within structure types, and globally ranked. Recomputing those scores after each small pruning step yields a sequence of nested subnetworks at many usable sparsity levels without parameter fine-tuning. Author manuscript 07.03 The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence ModelsarXiv preprint first submitted 24 August 2026; not peer reviewed as of the audit date. The audit tests prefix invariance: in a correct causal sequence model, outputs for an existing prefix must not change when unseen suffix tokens are appended. AX-RAY runs only two forward passes—one on a prefix and one on prefix plus suffix—then compares overlapping positions layer by layer. This catches information leaks outside the visible attention mask, including state-space scans, normalization, and hybrid residual paths, without gradients or a reference model. Official AX-RAY demo Work: Prune and distil one model into fixed and nested capacity points, then audit prefix invariance at each checkpoint and measure end-to-end prefill/decode latency on the same hardware and software stack.
Exit evidence: Report total and active parameters, pruning rule, KD data and teacher cost, subnet switching, memory, throughput, latency distribution, output quality, and causal-mask violations.
2026
Residual distillation, bounded attention, and low-cost pretraining
Residual-learning LLM KD and ACTD address hidden-state or vocabulary mismatch; sliding-window supplies a cheap retrofit baseline; Puro-2B makes the pretraining hardware and cost ledger inspectable.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 Knowledge Distillation for Large Language Models through Residual LearningPublished at ICLR 2026; no official implementation was located by 2026-09-02. This ICLR 2026 method distills LLMs by emphasizing tokens where the teacher errs and representing the student's hidden state relative to a compressed teacher state. A pretrained teacher autoencoder, adaptive scaling, cross-tokenizer alignment, and mixture-of-experts fusion let residual supervision operate even when teacher and student vocabularies differ. OpenReview record · First-party Amazon Science page 08.02 ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual RegularizationThe arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026. ACTD distills reasoning models across different tokenizers by mapping teacher top-k tokens into student-space anchors. An anchor L1 term matches probability on aligned tokens, while residual regularization constrains the student's probability mass outside those anchors. With multiple teachers, ACTD computes a complete teacher-specific loss for each teacher and averages those losses; it does not average anchor signals or teacher distributions first. Official implementation 08.03 Sliding-window beats linear attentionarXiv v1 submitted 28 August 2026. No peer-reviewed record, code release, or model artifact was verified on 2 September 2026; it is retained for its unusually important negative baseline evidence rather than publication status. Core result: a training-free sliding-window attention baseline with four attention sinks matches or exceeds the reported post-trained linear-attention retrofits on ordinary language benchmarks, speed, memory, and the tested long-context tasks. The scientific contribution is therefore a negative baseline result: before paying to linearize an existing LLM, compare against an equally bounded sliding window. — 08.04 Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090arXiv v1 submitted 27 August 2026. Catalog authors follow the current arXiv metadata; the PDF title page additionally names Chengxia Li, an unresolved source-metadata discrepancy that should be rechecked on revision. Code, data, and checkpoints were released, but no peer-reviewed record was verified. Core idea: make a complete two-billion-parameter pretraining stack reproducible on consumer RTX 5090 clusters, then publish the measured token, GPU-hour, rental-rate, data, optimization, checkpoint, and evaluation ledgers. The result is not literally a 5,090-dollar canonical run: the full 1.4-trillion-token checkpoint costs an estimated USD 6.89K, while a fitted recipe-specific scaling law crosses the Qwen2-1.5B score near USD 4.4K. Official training implementation · Official models and data collection Work: Account for teacher forward/state cost, tokenizer mapping, training tokens, KV memory, hardware utilization, and deployed prefill/decode latency.
Exit evidence: Compare quality at matched end-to-end cost, context, precision, and released implementation state.
2025–2026
Frequency-domain visual token budgets
Fourier Compressor uses fixed DCT truncation on feature grids, followed by VLM adaptation. Distinguish parameter-free compression from training-free deployment, and average retention from fine-detail fidelity.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language ModelsECCV 2026 spotlight verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Parameter-free compression is NOT training-free: Section 5.1 and Table 2 require adaptation. The 31.2% number concerns reduced time to first token, not a measured 31.2% increase in autoregressive decoding throughput. Fourier Compressor removes visual tokens by shrinking their spatial frequency representation. It applies a two-dimensional type-II DCT independently across feature channels, keeps a low-frequency rectangle, and applies an inverse DCT on the smaller grid. The compressor adds no learned parameters, but the surrounding VLM is trained to accommodate the altered features. Official ECCV spotlight record · Official code Work: Which frequencies and tasks lose evidence at a matched training and token budget?
Exit evidence: TextVQA and video subtasks can regress despite strong average retention; TTFT is not decoding throughput.
2026
Diverse representatives and temporal token merging
FlashVID preserves representative video tokens and merges the unselected remainder into similarity trees. Compare token budgets, compression overhead and complete time to first token separately.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 10.01 FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token MergingOfficial ICLR 2026 oral and full proceedings manuscript checked. Inference-time token selection/merging, not residual-error KD. Full configuration includes segmentation, budget adjustment and hybrid LLM pruning. Report encoding and compression overhead as well as LLM savings; natural-video evidence is not cross-magnification pathology validation. FlashVID compresses video tokens without training the target Video LLM. Attention-guided Diverse Token Selection keeps representative tokens, while Tree-based Spatiotemporal Merging pools redundant tokens across adjacent frames. This is token selection and merging, not teacher–student knowledge distillation. Official proceedings record · OpenReview · Official implementation Work: Which information survives selection or merging, and where is compression cost included?
Exit evidence: Residual tokens are not residual-error KD; large LLM-only savings are not the same as end-to-end acceleration.
2026
State-space discretization, rotations and update rank
Mamba-3 separates numerical discretization, complex-state expressivity and MIMO hardware utilization. Follow each mechanism into its own quality or efficiency experiment.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 11.01 Mamba-3: Improved Sequence Modeling using State Space PrinciplesOfficial ICLR 2026 oral and full proceedings manuscript checked. Three selective-SSM changes evaluated on language and synthetic sequences. Distinguish conditional discretization theory from learned gates, parameter matching from compute matching, and fixed-state decoding from information-retrieval capability. No direct pathology or natural-image evaluation. Mamba-3 changes a recurrent state-space language model along three axes: exponential-trapezoidal discretization, complex-valued state dynamics expressed through input-dependent rotations, and multi-input multi-output (MIMO) updates. The aim is better modeling and state tracking while using memory-bound decoding hardware more effectively. Official proceedings record · OpenReview · Official implementation Work: Which assumptions are enforced, which costs are matched, and where does a fixed state lose information?
Exit evidence: Parameter matching is not compute matching; language and synthetic-sequence results do not establish WSI performance.
Research branches
Distributed training
Which model and optimizer states are replicated, sharded, or communicated?
Mixed Precision Training · ZeRO: Memory Optimizations Toward Training Trillion Parameter Models · Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Distilled deployment
Which online/offline transfer cost is paid once, which modules or experts are active, and which quality/latency benefit persists at serving time?
DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter · MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers · MiniLLM: Knowledge Distillation of Large Language Models · DistiLLM: Towards Streamlined Distillation for Large Language Models · TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models · MiniPLM: Knowledge Distillation for Pre-Training Language Models · LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation
Serving and scheduling
How are prefill, decode, batching, prefix reuse, cache, placement, and admission scheduled?
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · Efficient Memory Management for Large Language Model Serving with PagedAttention · Fast Inference from Transformers via Speculative Decoding · DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving · SGLang: Efficient Execution of Structured Language Model Programs
Quantized inference
Which tensors, layers, channels, tokens, and outliers receive which precision?
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models · AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration · SliderQuant: Accurate Post-Training Quantization for LLMs
Sparse and stateful sequence computation
Which tokens interact, which state is updated online, and which kernels realize the claimed complexity?
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention · Learning to (Learn at Test Time): RNNs with Expressive Hidden States · Modular TTT: Rethinking Test-Time Training as Composable Modules
Pruned, nested, and causally audited serving
Which structural capacity is active for a request, which backend realizes the saving, and can any future token influence an earlier prefix state?
Compact Language Models via Pruning and Knowledge Distillation · From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models · The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models
Efficient language-model transfer and execution
Which expense is paid once during transfer and which persists for every served token?
Knowledge Distillation for Large Language Models through Residual Learning · ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization · Sliding-window beats linear attention · Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Frequency-domain visual token budgets
Which frequencies and tasks lose evidence at a matched training and token budget?
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
Diverse representatives and temporal token merging
Which information survives selection or merging, and where is compression cost included?
State-space discretization, rotations and update rank
Which assumptions are enforced, which costs are matched, and where does a fixed state lose information?
Mamba-3: Improved Sequence Modeling using State Space Principles
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Parallel state | Partitions training computation and memory across devices. | Communication and pipeline bubbles erase theoretical savings. | 2020 · 2021 |
| Distilled checkpoint | Pays teacher-transfer cost to reduce steady-state model size, activated experts, or generation cost. | Fewer total or activated parameters need not improve hardware latency; tokenizer, data, and teacher-call costs can dominate. | 2019 · 2020 · 2024 · 2024 · 2025 · 2025 · 2025 |
| Kernel / cache / scheduling | Changes memory movement, allocation, placement, and prefix reuse without changing model weights. | Speedups depend on hardware, request trace, shape, load, and software version. | 2022 · 2023 · 2024 · 2024 |
| Drafting / quantization | Reduces serial work or numeric cost under verification or error control. | Quality drift, calibration loss, or low acceptance can erase gains. | 2023 · 2023 · 2024 |
| Sparse / stateful sequence model | Changes the interaction graph or updates internal state while processing the sequence. | Asymptotic savings may not survive kernel, optimizer-state, or online-update overhead. | 2025 · 2025 · 2026 |
| Prune, distil, and audit causal serving | Minitron recovers structured pruning with KD; GISP trains globally ranked nested subnetworks; prefix-invariance auditing tests the causal implementation rather than trusting its mask declaration. | Parameter or FLOP reductions may not become latency savings, subnet ranking may drift by task, and a nominal causal mask may leave information leaks through implementation paths. | 2024 · 2026 · 2026 |
| Residual distillation, bounded attention, and low-cost pretraining | Connects cross-tokenizer residual objectives, bounded-context baselines, and reproducible commodity-hardware pretraining. | Ignoring teacher states, post-training, compilation, or utilization can reverse a claimed efficiency result. | 2026 · 2026 · 2026 · 2026 |
| Frequency-domain visual token budgets | Fourier Compressor uses fixed DCT truncation on feature grids, followed by VLM adaptation. Distinguish parameter-free compression from training-free deployment, and average retention from fine-detail fidelity. | TextVQA and video subtasks can regress despite strong average retention; TTFT is not decoding throughput. | 2026 |
| Diverse representatives and temporal token merging | FlashVID preserves representative video tokens and merges the unselected remainder into similarity trees. Compare token budgets, compression overhead and complete time to first token separately. | Residual tokens are not residual-error KD; large LLM-only savings are not the same as end-to-end acceleration. | 2026 |
| State-space discretization, rotations and update rank | Mamba-3 separates numerical discretization, complex-state expressivity and MIMO hardware utilization. Follow each mechanism into its own quality or efficiency experiment. | Parameter matching is not compute matching; language and synthetic-sequence results do not establish WSI performance. | 2026 |
Concept lineage
ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
State sharding complements intra-layer and pipeline partitioning in large training stacks.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessEfficient Memory Management for Large Language Model Serving with PagedAttention
Kernel-level memory efficiency is extended to request-level KV-cache allocation and scheduling.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessNative Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
IO-aware exact attention is complemented by a sparse pattern designed to be trainable and efficient on modern accelerators.
Learning to (Learn at Test Time): RNNs with Expressive Hidden StatesModular TTT: Rethinking Test-Time Training as Composable Modules
Linear and MLP fast-weight hidden states become a configurable graph of inner learners and update components.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsAWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration
Activation-aware scaling is complemented by protecting salient weight channels during post-training quantization.
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained TransformersMiniLLM: Knowledge Distillation of Large Language Models
Internal self-attention relation transfer expands into distribution-aware distillation for free-running generative language models.
DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and LighterMiniPLM: Knowledge Distillation for Pre-Training Language Models
Online objective/state matching during encoder pretraining broadens into offline teacher-guided corpus refinement that supports different student families and tokenizers.
MiniLLM: Knowledge Distillation of Large Language ModelsDistiLLM: Towards Streamlined Distillation for Large Language Models
Student-distribution-aware generative transfer is complemented by skew-KL and adaptive off-policy sampling.
Efficient Memory Management for Large Language Model Serving with PagedAttentionDistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving
KV-cache-aware serving is extended by placing prefill and decode on separately optimized resources.
Efficient Memory Management for Large Language Model Serving with PagedAttentionSGLang: Efficient Execution of Structured Language Model Programs
Paged request memory is complemented by radix-tree prefix reuse and a structured generation runtime.
Failure modes to audit
- Reporting theoretical FLOPs instead of wall-clock distributions.
- Comparing different output quality or context lengths.
- Hiding warm-up, compilation, and queueing.
- Using throughput-only results for interactive latency claims.
- Omitting hardware, software, power, and failure recovery.
Questions for every paper
- What exact model and checkpoint run?
- Which memory terms dominate?
- What request distribution is measured?
- What quality constraint is held fixed?
Complete maintained paper route
Section titled “Complete maintained paper route”31 canonical readings · 2018–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 31 readings in maintained order
Mixed Precision Training
Paulius Micikevicius et al.
Metadata checked
DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf
Workshop paper and arXiv preprint; not a main-track NeurIPS proceedings paper.
Official artifacts: Official distillation code
Metadata checked
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He
Official artifacts: DeepSpeed
Metadata checked
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou
Official artifacts: Official implementation and checkpoints
Metadata checked
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Deepak Narayanan et al.
Official artifacts: Megatron-LM
Metadata checked
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao et al.
Metadata checked
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon et al.
Official artifacts: Serving system
Metadata checked
Fast Inference from Transformers via Speculative Decoding
Yaniv Leviathan, Matan Kalman, and Yossi Matias
Metadata checked
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Guangxuan Xiao et al.
Official artifacts: Code
Metadata checked
MiniLLM: Knowledge Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang
Official artifacts: Official code
Metadata checked
AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han
Official artifacts: Official project page · Official implementation and model zoo
Metadata checked
DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang
Official artifacts: Official implementation
Metadata checked
SGLang: Efficient Execution of Structured Language Model Programs
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng
Official artifacts: Official implementation
Metadata checked
DistiLLM: Towards Streamlined Distillation for Large Language Models
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun
Peer-reviewed ICML 2024 proceedings paper.
Official artifacts: Official code
Metadata checked
Compact Language Models via Pruning and Knowledge Distillation
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov
Published in the NeurIPS 2024 main conference proceedings.
Official artifacts: Official implementation · Official model collection
Metadata checked
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Jingyang Yuan et al.
ACL 2025 Best Paper.
Metadata checked
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin
Official artifacts: Official code
Metadata checked
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba
Official artifacts: Official code
Metadata checked
MiniPLM: Knowledge Distillation for Pre-Training Language Models
Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang
Peer-reviewed ICLR 2025 conference paper.
Official artifacts: Official code, data, and models
Metadata checked
LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation
Fangxun Shu, Yue Liao, Lei Zhang, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Zhelun Yu, Si Liu, Hongsheng Li, and Hao Jiang
Peer-reviewed ICLR 2025 conference paper.
Official artifacts: Official code
Metadata checked
Modular TTT: Rethinking Test-Time Training as Composable Modules
Bohao Tang, Zhen Qin, Yuqi Pan, Zheng Li, Pengfei Liu, and Ya Zhang
Preprint status retained; no peer-reviewed proceedings record was available when checked on 2026-08-10.
Official artifacts: Official code
Metadata checked
From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Minwoo Lee, Shu-ping Yeh, Evgeny Stupachenko, Hao Feng, and Li Yang
Peer-reviewed ACL 2026 long paper and Outstanding Paper award recipient.
Official artifacts: Author manuscript
Metadata checked
The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models
Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang, and Minseo Kim
arXiv preprint first submitted 24 August 2026; not peer reviewed as of the audit date.
Official artifacts: Official AX-RAY demo
Metadata checked
Sliding-window beats linear attention
Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais
arXiv v1 submitted 28 August 2026. No peer-reviewed record, code release, or model artifact was verified on 2 September 2026; it is retained for its unusually important negative baseline evidence rather than publication status.
Metadata checked
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Mingzhe Zhang, Kaiyue Wen, Kaifeng Lyu, and Wenguang Chen
arXiv v1 submitted 27 August 2026. Catalog authors follow the current arXiv metadata; the PDF title page additionally names Chengxia Li, an unresolved source-metadata discrepancy that should be rechecked on revision. Code, data, and checkpoints were released, but no peer-reviewed record was verified.
Official artifacts: Official training implementation · Official models and data collection
Metadata checked
Knowledge Distillation for Large Language Models through Residual Learning
Thinh On, Hengzhi Pei, Leonard Lausen, and George Karypis
Published at ICLR 2026; no official implementation was located by 2026-09-02.
Official artifacts: OpenReview record · First-party Amazon Science page
Metadata checked
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, and Bing Qin
The arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026.
Official artifacts: Official implementation
Metadata checked
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
Huanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, Zhouhan Lin
ECCV 2026 spotlight verified from the official linked program event. Review uses the conference-hosted manuscript checked 2026-09-12; a separate Springer version of record was not verified. Parameter-free compression is NOT training-free: Section 5.1 and Table 2 require adaptation. The 31.2% number concerns reduced time to first token, not a measured 31.2% increase in autoregressive decoding throughput.
Official artifacts: Official ECCV spotlight record · Official code
Metadata checked
FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, Zhuotao Tian
Official ICLR 2026 oral and full proceedings manuscript checked. Inference-time token selection/merging, not residual-error KD. Full configuration includes segmentation, budget adjustment and hybrid LLM pruning. Report encoding and compression overhead as well as LLM savings; natural-video evidence is not cross-magnification pathology validation.
Official artifacts: Official proceedings record · OpenReview · Official implementation
Metadata checked
Mamba-3: Improved Sequence Modeling using State Space Principles
Aakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, Albert Gu
Official ICLR 2026 oral and full proceedings manuscript checked. Three selective-SSM changes evaluated on language and synthetic sequences. Distinguish conditional discretization theory from learned gates, parameter matching from compute matching, and fixed-state decoding from information-retrieval capability. No direct pathology or natural-image evaluation.
Official artifacts: Official proceedings record · OpenReview · Official implementation
Metadata checked
SliderQuant: Accurate Post-Training Quantization for LLMs
Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Zhonghong Ou, and Anbang Yao
Peer-reviewed ICLR 2026 poster; retain as a frontier quantization method after systems and quantization foundations.
Official artifacts: Code
Metadata checked