Skip to content
Computational Pathology Reading Roadmap

LLM post-training, reasoning, and alignment

Post-training changes a base model’s conditional behavior using demonstrations, preferences, critiques, rules, verifiers, or environment rewards. It can improve task following while trading off diversity, calibration, capabilities, or safety.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Targeted discovery on 26 October 2026 adds DrOPD with a new source-checked review; it does not re-date the complete route or older paper evidence.

1. Supervised instruction tuning

Section titled “1. Supervised instruction tuning”

Supervised fine-tuning (SFT) minimizes next-token loss over instruction–response examples. Record source tasks, annotators, synthetic generators, filtering, system messages, multi-turn construction, loss masking, mixture weights, epochs, and base checkpoint.

FLAN (2022) studies instruction-task mixtures. Self-Instruct (2023) uses a model to generate instructions, which expands data but can recycle generator biases and benchmark content.

2. Preference and reward learning

Section titled “2. Preference and reward learning”

Learning to Summarize from Human Feedback (2020) provides an early end-to-end reward-model and policy-optimization study. InstructGPT (2022) separates demonstrations, ranked comparisons, a reward model, and policy optimization. The annotator population and written rubric define the optimized preference signal.

For a chosen response y_w and rejected response y_l, DPO (2023) optimizes a logistic preference objective based on policy-versus-reference log-ratio differences. DPO removes an explicit reward-model/RL loop; it does not remove the preference-data, reference-policy, or distribution-shift assumptions. Reward Model Overoptimization (2023) demonstrates why improving a learned proxy can eventually reduce the true objective. Always evaluate against held-out human or task evidence across optimization strength.

Separate:

  • offline versus online response sampling;
  • human, AI, rule, or verifier feedback;
  • pairwise preference, scalar reward, process labels, or environment return;
  • reference policy and KL/regularization;
  • reward overoptimization and distribution shift;
  • base, SFT, preference-trained, and deployed checkpoints.

3. Reasoning and test-time computation

Section titled “3. Reasoning and test-time computation”

Chain-of-Thought prompting (2022) elicits intermediate text. Self-consistency (2023) samples multiple traces and selects an answer. ReAct (2023) interleaves reasoning and actions. Verifier and process-reward approaches score outcomes or intermediate steps. Let’s Verify Step by Step (2024) contrasts process and outcome supervision, while optimal test-time scaling (2025) allocates sampling/search compute according to problem difficulty and verifier quality. Recent reasoning systems such as DeepSeekMath (2024) combines a mathematics-focused pretraining corpus with supervised fine-tuning and GRPO; it is an arXiv preprint, so keep the algorithmic evidence separate from venue acceptance. Later systems such as DeepSeek-R1 (2025) and Qwen3 (2025) study reinforcement learning and controllable reasoning modes; the former is peer reviewed, while Qwen3 remains a technical report.

Cross-domain RPT evaluation (2026) finds that reinforcement post-training gains can be domain-specific. Treat it as a boundary audit: report in-domain and genuinely unseen-domain changes, rather than assuming RLVR produces general reasoning ability.

A correct answer does not prove the displayed chain caused it or is faithful. Always report pass@1, sampling budget, best/majority selection, verifier access, tokens, latency, and failures. Compare equal test-time compute.

4. Safety and plural values

Section titled “4. Safety and plural values”

Constitutional AI (2022) uses written principles and AI feedback. A constitution is an explicit policy choice, not a universal set of values. Evaluate overrefusal, underrefusal, jailbreak robustness, multilingual and cultural behavior, legitimate high-risk help, and regressions in non-safety tasks.

Post-training data can contain private, copyrighted, low-quality, or sensitive material. Document collection consent, annotator working conditions, model-generated content, decontamination, and redress.

Staged route

Section titled “Staged route”
Level Primary reading · year · status Key reconstruction
L2 Learning to Summarize from Human Feedback — 2020 · NeurIPS comparisons → reward model → policy optimization
L1–L2 FLAN — 2022 · ICLR instruction-mixture transfer
L2 InstructGPT — 2022 · NeurIPS demonstrations → preferences → reward → policy
L2 Chain-of-Thought — 2022 · NeurIPS prompt-time elicitation and scale dependence
L2–L3 Constitutional AI — 2022 · preprint critique/revision and AI-feedback policy
L2–L3 Self-Instruct — 2023 · ACL synthetic instruction generation and filtering
L2–L3 DPO — 2023 · NeurIPS direct preference objective and reference policy
L2–L3 KTO — 2024 · ICML; SimPO — 2024 · NeurIPS prospect-inspired desirable/undesirable feedback versus a reference-free length-normalized preference margin
L2–L3 MiniLLM — 2024 · ICLR; TAID — 2025 · ICLR reverse-KL generative distillation versus a temporally adaptive teacher target; evaluate free-running generation, not only teacher-forced loss
L2–L3 Self-consistency — 2023 · ICLR; Reward Model Overoptimization — 2023 · ICML sampling aggregation versus proxy-optimization limits
L2–L3 LoRA — 2022 · ICLR; QLoRA — 2023 · NeurIPS adaptation capacity versus memory/quantization
L3 Let’s Verify Step by Step — 2024 · ICLR; optimal test-time scaling — 2025 · ICLR process supervision, verifier quality, and compute allocation
L3 DeepSeekMath — 2024 · arXiv preprint mathematics data construction, code-informed pretraining, supervised instruction data, and GRPO; audit each contribution and contamination independently
L3 DeepSeek-R1 — 2025 · Nature; Qwen3 — 2025 · technical report RL/verifier and hybrid reasoning claims requiring independent controls
L3 RPT transfer across unseen domains — 2026 · ICLR distinguish in-domain gains from transferable reasoning

Controlled lab

Section titled “Controlled lab”

Using one open base checkpoint and fixed examples:

  1. evaluate the base model;
  2. SFT with LoRA and full fine-tuning under equal tokens;
  3. train or apply one preference objective;
  4. evaluate the same task, safety, calibration, and style sets after every stage;
  5. compare direct answer, chain-of-thought prompt, and sampled self-consistency under equal token budgets;
  6. blind human raters to checkpoint and randomize order.

Report variance, annotator rubric/agreement, response length, refusal rate, pass@1, pass@K, tokens, latency, and cost.

Exit task

Section titled “Exit task”

Draw the data flow from base checkpoint to deployed assistant. Label which examples, preferences, principles, rewards, tools, and decoding policies affect behavior. Name one capability gain, one regression, and one conclusion the available evidence cannot support.

Token-level direction and teacher consensus

BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, and Gao Huang2026 · arXiv · preprintPreprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation.BinaryOPD replaces the magnitude of a teacher-student log-probability reward with its sign. The paper then uses high-disagreement token analysis to motivate C-MOPD, which combines teachers while declining strongly conflicting updates.Versioned full text · Author implementation

Reading question: Which disagreement carries useful information, and when should conflicting teachers produce no update?

Evidence boundary: Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost.

Route additional correction by teacher–student disagreement

DrOPD preserves reverse KL at every student-generated position and adds forward KL at the batch’s highest-JS positions. The candidate union retains tokens preferred by either model. Read it after on-policy KD to distinguish uncertainty, disagreement and teacher-supported modes the student under-covers.

Primary paper · year · evidenceCore idea and why to read itOfficial resources
Divergence-Routed On-Policy Distillation for Language ModelsTiantian Peng, Guannan Lv, Hongyang Li, Qiuhe Hong, Hang Cheng, Hongjian Dou, Xiangyu Wu, and Yuyang Liu2026 · arXiv · preprintReview pins arXiv v1 (14 October 2026). The manuscript says under review at ICLR 2027; this is not acceptance.DrOPD changes where an additional teacher signal is applied. It keeps reverse-KL supervision on every student-generated response position and adds forward KL at the highest-disagreement positions. The motivation is under-coverage: reverse KL can barely push up a candidate to which the student already assigns almost no mass. This is a training-objective intervention, not a new language-model architecture.Version 1 paper and appendices

Reading question: Which tokens need extra teacher-weighted correction, and is their teacher target reliable?

Evidence boundary: Math-only preprint, two related Qwen pairs. Top-k approximate losses, eight-response mean accuracy and logged training-step cost are separate evidence contracts; no demonstrated WSI transfer.

Complete maintained paper route

Section titled “Complete maintained paper route”

36 canonical readings · 2019–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 36 readings in maintained order

  25. #252025ICLRPeer reviewedL2–L3

    LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation

    Fangxun Shu, Yue Liao, Lei Zhang, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Zhelun Yu, Si Liu, Hongsheng Li, and Hao Jiang

    Peer-reviewed ICLR 2025 conference paper.

    Official artifacts: Official code

    Metadata checked

  27. #272026arXivPreprintL3

    On-policy Distillation with Verifiable Reward

    Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang

    arXiv v1 preprint submitted 25 August 2026; evaluation is limited to the reported verifiable reasoning tasks.

    Official artifacts: Official implementation

    Metadata checked

  32. #322026arXivPreprintL3

    Normalized Low-Rank Adaptation

    Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, and Weiyang Liu

    arXiv v1 submitted 31 August 2026. The authors released an implementation; no peer-reviewed record was verified on 2 September 2026.

    Official artifacts: Official implementation

    Metadata checked

  34. #342026EMNLPAccepted · proceedings pendingL3

    ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

    Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, and Bing Qin

    The arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026.

    Official artifacts: Official implementation

    Metadata checked

  35. #352026arXivPreprintL3

    Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

    Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, and Gao Huang

    Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation.

    Official artifacts: Versioned full text · Author implementation

    Metadata checked