Skip to content
Computational Pathology Reading Roadmap

NLP tasks, retrieval, and evaluation

Architecture does not define a valid experiment. This chapter starts from the output and error cost, then chooses the data, model interface, decoding, and evidence needed to support a claim.

Latest literature discovery audit: 2 October 2026. Existing paper reviews retain their individual source-check dates.

Task contract first

Section titled “Task contract first”
Task family Output unit Useful evidence Failure hidden by a common metric
classification document/sentence/turn label macro per-class metrics, calibration, subgroup errors accuracy hides imbalance and thresholds
sequence labelling token/span/entity/relation span precision/recall/F1, boundary and type errors token F1 rewards partial entities
retrieval/ranking ordered document or passage list Recall@K, MRR, nDCG, latency, corpus coverage generation can hide a missed source
extractive QA answer span plus evidence exact match, token F1, answerability, evidence localization aliases and unanswerable cases
translation target sequence human adequacy, terminology, discourse, calibrated automatic metrics BLEU misses meaning and critical entities
summarization/reporting supported concise sequence factual consistency, omission, entity/relation errors, human rubric overlap rewards copied but wrong text
open generation/dialogue sequence or structured action task success, groundedness, safety, abstention, preference protocol fluency masks unsupported claims

Define the unit of analysis and split unit independently. A sentence-level task may still require patient-, author-, conversation-, document-family-, site-, or time-level splitting.

1. Classification and structured prediction

Section titled “1. Classification and structured prediction”

Keep bag-of-words linear models as controls. For token or span prediction, specify BIO/BILOU or span representation, overlapping/nested entities, discontinuities, unknown labels, and whether exact boundaries are required.

BiLSTM–CRF NER (2016) is a useful structured baseline: recurrent features produce emissions and a CRF scores the complete label sequence. Evaluate exact spans and illegal transitions, and state whether the task permits nested or discontinuous mentions that a flat tag sequence cannot express.

Micro F1 weights common classes; macro F1 weights classes equally. AUROC can appear strong with rare positives; precision–recall curves and operating points may be more informative. Calibration and selective prediction ask whether confidence can support a decision or abstention policy.

2. Retrieval

Section titled “2. Retrieval”

Sparse retrieval uses lexical matching; dense retrieval encodes queries and passages; late interaction keeps token-level matching; rerankers jointly score a smaller candidate set.

The classical sparse route has three distinct anchors:

  1. IDF (1972) discounts terms that occur in many documents because they carry less collection-specific evidence.
  2. Okapi at TREC-3 (1994) supplies the canonical early BM25 record: query-term evidence is combined with saturating term frequency and document-length normalization. k1 and b, tokenizer, fields, collection statistics, and query formulation are part of the baseline.
  3. Cumulated Gain-Based Evaluation (2002) introduces discounted gain for graded relevance and normalization by the ideal ordering. nDCG depends on gain mapping, discount, cutoff, judgments, and how unjudged items are treated; it is not interchangeable with Recall@K.

Always retain BM25 beside dense systems. Report candidate recall before answer quality, and report both binary and graded relevance when the task supports them.

Dense Passage Retrieval (2020) contrasts paired encoders with in-batch negatives. Hard negatives can improve discrimination but may also include genuinely relevant, unlabeled passages. ColBERT (2020) retains token-level late interaction rather than compressing each text to one vector. BEIR (2021) tests heterogeneous zero-shot transfer, while MIRACL (2023) makes language coverage and annotation differences explicit. Do not tune on the same benchmark suite used to claim out-of-domain generalization.

Evaluate the pipeline in layers:

corpus coverage → candidate recall → ranking quality → evidence selection
→ answer support/completeness → abstention → latency and cost

Record corpus release, document date, chunking, overlap, tokenizer, embedding checkpoint, index parameters, query rewriting, top-K, reranker, and deduplication.

3. QA, translation, and generation

Section titled “3. QA, translation, and generation”

SQuAD (2016) makes extractive QA concrete but does not cover open-domain retrieval or every form of unanswerability. BLEU (2002) measures clipped n-gram precision with a brevity penalty; it is a corpus-level proxy, not semantic or factual proof. ROUGE (2004) defines overlap-based summary recall variants and shares the same central boundary: lexical overlap does not establish factual support, omission safety, or usefulness. BERTScore (2020) compares contextual token embeddings but can still reward fluent contradictions.

For generated output, inspect:

  • supported and unsupported entities, numbers, relations, and temporal statements;
  • omitted required content;
  • negation and uncertainty;
  • copy behavior and training-example memorization;
  • sensitivity to prompt order, decoding, and retrieved evidence;
  • performance when the correct response is “insufficient evidence.”

Human evaluation needs a defined population, blinding, rubric, calibration examples, sampling plan, agreement analysis, and adjudication. “Reviewed by experts” is not a reproducible protocol.

4. Dataset construction and contamination

Section titled “4. Dataset construction and contamination”

Deduplicate before splitting at both exact and near-duplicate levels. Track source document and derivative relationships. Template variants, translations, quotations, and benchmark copies can cross a naive hash boundary.

For pretrained models, separate:

  • benchmark text known to appear in pretraining;
  • unknown overlap because the corpus is undisclosed;
  • prompt/example leakage during development;
  • repeated evaluation-driven tuning;
  • test labels available through metadata or retrieval.

Time-based evaluation helps only if training cutoffs and document dates are credible.

Staged reading route

Section titled “Staged reading route”
Level Primary reading · year · status Read it for
L1 BLEU — 2002 · ACL; ROUGE — 2004 · workshop paper how overlap proxies are constructed and bounded
L1–L2 SQuAD — 2016 · EMNLP dataset/task interface, span metrics, crowd annotation
L1–L2 BiLSTM–CRF NER — 2016 · NAACL-HLT sequence-level normalization, decoding, and span error analysis
L1–L2 GLUE — 2019 · ICLR multi-task benchmark aggregation and saturation
L2 Dense Passage Retrieval — 2020 · EMNLP; ColBERT — 2020 · SIGIR dual-encoder compression versus token-level late interaction
L2–L3 BEIR — 2021 · NeurIPS Datasets & Benchmarks; MIRACL — 2023 · TACL heterogeneous and multilingual retrieval transfer
L2–L3 Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE) — 2023 · ACL generate a hypothetical relevant document, embed it, then retrieve real documents; separate generator priors, retrieval recall, and zero-shot transfer
L2 BERTScore — 2020 · ICLR contextual similarity and its limits
L2–L3 WILDS — 2021 · ICML real distribution-shift splits beyond IID averages
L2–L3 Datasheets for Datasets — 2021 · CACM provenance, collection, uses, and maintenance

Controlled lab

Section titled “Controlled lab”

Build one small task in layers:

  1. create grouped train/validation/test splits and a data card;
  2. train TF–IDF logistic regression, an encoder classifier, and a retrieval-plus-answer system where appropriate;
  3. hold preprocessing and tuning budgets fixed;
  4. report macro metrics, calibration, confidence intervals, subgroup and shift errors;
  5. run keyword masking, label-shuffled, text-only metadata, near-duplicate, and unanswerable controls;
  6. for retrieval, score recall before answer quality;
  7. manually classify at least 50 errors with a written taxonomy.

Exit task

Section titled “Exit task”

Write an evaluation protocol before training. It must name the population, unit, split, selection metric, test metrics, thresholds, uncertainty method, subgroup and shift checks, missing-evidence behavior, human-review protocol, contamination audit, and compute budget.

Continue to the LLM route for scaling, post-training, RAG, agents, and safety, or to natural-image vision & language for multimodal systems.

Complete maintained paper route

Section titled “Complete maintained paper route”

20 canonical readings · 1972–2026· route reviewed through October 2, 2026.

This is the complete structured list for this chapter, ordered by an explicit prerequisite-aware teaching sequence. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.

Show all 20 readings in maintained order

  2. #21994TREC-3Technical reportL2–L3

    Okapi at TREC-3

    Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford

    System-description paper in NIST Special Publication 500-225 (TREC-3 proceedings); TREC participation and proceedings inclusion are not represented here as peer review.

    Official artifacts: NIST proceedings record

    Metadata checked

  16. #162025ICLRPeer reviewedL2

    MMTEB: Massive Multilingual Text Embedding Benchmark

    Kenneth Enevoldsen et al.

    Review uses the proceedings PDF and Table 2 benchmark definitions. The landing abstract states 1,000+ languages while the PDF abstract states 250+; Section 5 attributes the broader count to bitext mining. Task-count inconsistencies in prose are retained as caveats, not silently combined.

    Official artifacts: Proceedings PDF · Benchmark implementation · Versioned evaluation results · arXiv

    Metadata checked

  19. #192021Communications of the ACMPeer reviewedL1–L2

    Datasheets for Datasets

    Timnit Gebru et al.

    Metadata checked