Skip to content
Computational Pathology Reading Roadmap

LLM evaluation, safety, and efficiency

An LLM evaluation should tell a reader what the exact model/system can do, for whom, under which prompt and resource budget, with which uncertainty and failure cost. One aggregate leaderboard cannot answer those questions.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

1. Build a claim–evidence matrix

Section titled “1. Build a claim–evidence matrix”
Claim Required evidence
knowledge or reasoning held-out and decontaminated tasks, strong non-reasoning controls, pass@1 and compute curves
instruction following diverse tasks, explicit constraints, base-versus-post-trained comparison
factuality source-supported claims, unanswerable cases, temporal updates, citation entailment
calibration proper scores, reliability/selective-risk curves, shift and subgroup analysis
safety defined policy, benign and adversarial prompts, over/underrefusal, multilingual/multiturn tests
efficiency tokens, batch, hardware, precision, latency distribution, throughput, memory, energy/cost
agent reliability task success, valid actions, recovery, intervention, environment and tool versions

Register primary/secondary endpoints before model selection when the study supports a comparative claim.

2. Benchmarks and contamination

Section titled “2. Benchmarks and contamination”

MMLU (2021) samples many knowledge domains. BIG-bench (2023) aggregates diverse tasks. HELM (2023) evaluates multiple scenarios and metrics. These are useful instruments, not timeless, independent definitions of intelligence.

Audit:

  • exact benchmark version and prompt;
  • training or synthetic-data overlap;
  • public solutions and repeated tuning;
  • answer-format and parser failures;
  • multiple-choice guessing and option order;
  • subgroup/language/domain coverage;
  • variance across seeds, prompts, and samples.

Data contamination can be known, possible, unknown, or ruled out only within a stated search. Undisclosed pretraining data require an “unknown,” not an assumption of no overlap.

LiveBench (2025) uses frequently refreshed, objectively scored tasks to limit—rather than eliminate— contamination. CoDeC (2026) detects memorized dataset structure through changes under in-context examples. Report each method’s false-positive controls and search boundary; neither proves absence of training exposure.

3. Factuality, uncertainty, and abstention

Section titled “3. Factuality, uncertainty, and abstention”

TruthfulQA (2022) probes imitation of common false beliefs. It does not replace source-grounded domain evaluation.

Separate semantic uncertainty, missing evidence, ambiguous requests, and unsafe requests. Evaluate a model’s ability to ask a clarification or abstain. Token probabilities are not automatically calibrated answer probabilities. Use Brier/log scores where a probability target is defined and selective-risk/coverage curves for abstention.

Semantic entropy (2024) aggregates sampled answers by meaning before estimating uncertainty. Its clustering, sampling, and task assumptions must be validated in the target domain; a detector is not a factuality guarantee.

4. Human and model-based judging

Section titled “4. Human and model-based judging”

MT-Bench and Chatbot Arena (2023) study model judges and pairwise preference. Judge models can have position, verbosity, self-preference, style, language, and contamination biases.

Validate a judge against blinded human ratings for the target domain. Randomize order, control length, publish rubrics, estimate agreement, and retain disagreement rather than forcing a false gold label.

5. Safety, privacy, and security

Section titled “5. Safety, privacy, and security”

Safety evaluation needs an explicit policy and threat model. Test harmful compliance, jailbreaks, indirect prompt injection, data exfiltration, insecure code/tool calls, identity and demographic harms, misinformation, privacy leakage, and overrefusal. HarmBench (2024) provides one standardized red-teaming framework. Training-data extraction (2021) shows that memorized sequences can become a privacy attack surface. Agent Security Bench (2025) extends the threat model across prompts, tools, and memory.

Do not publish actionable harmful details from testing. Use authorization, sandboxing, rate limits, human approval, logging, incident response, and data minimization at the system boundary.

Interpretability probes, attention maps, or generated rationales are hypotheses about mechanism—not proof of causal faithfulness or safety.

6. Efficiency and environmental reporting

Section titled “6. Efficiency and environmental reporting”

Measure prefill and decode separately; report first-token latency, inter-token latency, end-to-end percentiles, throughput versus concurrency, prompt/output tokens, KV cache, quantization, hardware, software, warmup, and power/energy when possible. Hosted API price is mutable and should be timestamped, not treated as a model property.

Quality–cost Pareto curves are more useful than a single fastest or best claim.

vLLM/PagedAttention (2023) targets KV-cache memory and scheduling; speculative decoding (2023) accelerates exact sampling with a draft model, and SmoothQuant (2023) moves activation difficulty into weights for post-training quantization. AWQ (2024) protects salient weight channels during low-bit weight-only quantization. SliderQuant (2026) is a frontier layer-sensitive quantization method. Compare all four on the same checkpoints, hardware, kernels, batch/concurrency, output lengths, and quality suite.

Staged route

Section titled “Staged route”
Level Reading · year · status Audit question
L1–L2 MMLU — 2021 · ICLR what does subject-level multiple choice measure and miss?
L2 TruthfulQA — 2022 · ACL does scale amplify learned misconceptions?
L2 HELM — 2023 · TMLR how should scenarios, metrics, and transparency coexist?
L2 BIG-bench — 2023 · TMLR how do task selection and scaling claims interact?
L2–L3 MT-Bench / Chatbot Arena — 2023 · NeurIPS Datasets & Benchmarks when can pairwise/model judging be informative?
L2–L3 Training-data extraction — 2021 · USENIX Security; semantic entropy — 2024 · Nature when memorization becomes exposure and when uncertainty predicts errors
L2–L3 vLLM/PagedAttention — 2023 · SOSP; speculative decoding and SmoothQuant — 2023 · ICML serving memory, exact decoding acceleration, and quantization
L2–L3 AWQ — 2024 · MLSys activation-aware channel salience for weight-only quantization; report calibration data, kernels, memory, quality, and actual latency together
L2–L3 Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models — 2026 · CVPR whether uncertainty-weighted teacher signals retain both task quality and calibrated uncertainty after multimodal compression
L3 HarmBench — 2024 · ICML how are attack, policy, classifier, and refusal separated?
L3 WMDP — 2024 · ICML hazardous-knowledge benchmark and unlearning evaluation; reduced multiple-choice accuracy is not proof that knowledge is absent
L2–L3 LiveBench — 2025 · ICLR what does a frequently refreshed, objectively scored benchmark limit?
L3 Agent Security Bench — 2025 · ICLR how should utility and attacks across prompt, tool, and memory be co-measured?
L3 CoDeC — 2026 · ICLR; SliderQuant — 2026 · ICLR frontier contamination detection and quantization evidence requiring independent controls
L1–L3 Model Cards — 2019 · ACM FAT*; NIST AI RMF — official risk framework how should evidence, limits, governance, and monitoring be documented?

Controlled evaluation suite

Section titled “Controlled evaluation suite”

For one base and post-trained checkpoint:

  1. pin model/tokenizer revision, inference stack, prompt, and decoding;
  2. evaluate capability, factuality, calibration, safety, and efficiency separately;
  3. include prompt-order, paraphrase, answer-format, unanswerable, image/document-null where applicable, and distribution-shift controls;
  4. search for exact/near benchmark overlap and report the boundary;
  5. compare human and model judge labels on a blinded sample;
  6. report confidence intervals and failure examples;
  7. produce quality–latency, quality–tokens, and risk–coverage curves.

Exit task

Section titled “Exit task”

Write a claim-limited model card. Every conclusion must name the population/task, exact checkpoint, prompt/system configuration, metric, uncertainty, contamination state, resource budget, and known failure. Anything broader is a hypothesis for the next study.

Complete maintained paper route

Section titled “Complete maintained paper route”

41 canonical readings · 2019–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered by an explicit prerequisite-aware teaching sequence. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 41 readings in maintained order

  2. #22021Communications of the ACMPeer reviewedL1–L2

    Datasheets for Datasets

    Timnit Gebru et al.

    Metadata checked

  16. #162026EMNLPAccepted · proceedings pendingL3

    IntroConformal: Conformal Factuality Guarantees for Large Vision-Language Models via Introspective Signals

    Md. Atabuzzaman, Christian Alexander, and Chris Thomas

    The arXiv v1 record submitted 1 September 2026 reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology record was available when checked on 2 September 2026, so publication claims remain author-reported and scientific claims use the arXiv manuscript.

    Official artifacts: Official implementation

    Metadata checked

  17. #172024ICMLPeer reviewedL2–L3

    The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks

    Official artifacts: Official benchmark and RMU code · Project website

    Metadata checked

  19. #192026ICLRPeer reviewedL3

    Detecting Data Contamination in LLMs via In-Context Learning

    Michał Zawalski, Meriem Boubdir, Klaudia Bałazy, Besmira Nushi, and Pablo Ribalta

    Peer-reviewed ICLR 2026 poster; CoDeC is a contamination-detection instrument whose assumptions still require task-specific validation.

    Metadata checked

  24. #242026arXivPreprintL3

    The Safeguard Worked. Is the LLM System Safer?

    Pingyu Wu, Weiming Zhang, and Nenghai Yu

    arXiv v1 submitted 1 September 2026. No peer-reviewed record or first-party machine-readable evidence artifact was verified; it is retained because its 198-paper evidence audit supplies unusually important negative system-level evidence.

    Metadata checked

  31. #312026ICLRPeer reviewedL3

    SliderQuant: Accurate Post-Training Quantization for LLMs

    Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Zhonghong Ou, and Anbang Yao

    Peer-reviewed ICLR 2026 poster; retain as a frontier quantization method after systems and quantization foundations.

    Official artifacts: Code

    Metadata checked

  38. #382026ACLPeer reviewedL2–L3

    Afri-MCQA: Multimodal Cultural Question Answering for African Languages

    Atnafu Lambebo Tonja, Srija Anand, Emilio Villa-Cueva, Israel Abebe Azime, Jesujoba Oluwadara Alabi, Muhidin A. Mohamed, Debela Desalegn Yadeta, Negasi Haile Abadi, Abigail Oppong, Nnaemeka Casmir Obiefuna, Idris Abdulmumin, Naome A Etori, Eric Peter Wairagala, Kanda Patrick Tshinu, Imanigirimbabazi Emmanuel, Gabofetswe Malema, Alham Fikri Aji, David Ifeoluwa Adelani, and Thamar Solorio

    Peer-reviewed ACL 2026 long paper and Best Social Impact Paper award recipient.

    Official artifacts: Author manuscript · Official dataset

    Metadata checked