How to read a paper

The goal of paper reading is to decide what is claimed, what evidence supports it, and what you can reuse or test. A polished summary is not the same as understanding.

This guide combines a three-pass workflow, a reusable note template, a computational-pathology audit, and My Library for comprehension checks, spaced review, and optional cross-device synchronization.

The pass structure is inspired by S. Keshav’s How to Read a Paper, adapted here for empirical AI research.

Before reading

Write why the paper matters to your current question. Capture the exact version, publication status, date, identifiers, code and checkpoint links, and any corrections. Skim related work only far enough to identify the claimed gap.

Do not begin by highlighting every sentence. Begin with a decision you need to make.

Pass 1 — Map the claim

Read the title, abstract, introduction, section headings, conclusion, and every table or figure caption. In 10–15 minutes, record:

• the problem and unit of analysis;
• the claimed contribution;
• the principal comparison;
• the main result;
• one reason the conclusion might not generalize.

Stop here if the paper is out of scope. Triage is useful work.

Pass 2 — Reconstruct the study

Read the method, data, experiments, and limitations. Trace one sample through the complete pipeline. For every central claim, pair it with its evidence:

Claim type	Evidence to locate
better predictive performance	identical split, metric, adaptation budget, uncertainty, and statistical comparison
better generalization	external or temporally separated cohort and domain description
data efficiency	predefined label fractions, repeated sampling, and equal model-selection budget
interpretability	a validated explanation target, not only a plausible heatmap
clinical usefulness	workflow, comparator, operating point, and consequence of errors

Mark missing information explicitly. “Not reported” is more useful than an invented assumption.

Pass 3 — Challenge and reproduce

Follow important references, inspect code and data documentation, and derive or implement the central mechanism. Attempt a minimal reproduction before a full benchmark.

Ask:

1. What is the strongest alternative explanation?
2. Which component is necessary according to an ablation?
3. Were baselines given equal preprocessing, tuning, and compute?
4. What selection decisions used validation data, and was the test set touched?
5. Which result would falsify the central claim?

For very large systems, “reproduce” may mean verifying preprocessing on a sample, running released inference, recreating one table row, or checking a claim on a smaller proxy.

Reusable note

# Citation and version

## One-sentence problem
## Main claim
## Method reconstructed from memory
## Data, population, and split unit
## Training objective and supervision
## Baselines and adaptation budgets
## Metrics and uncertainty
## Strongest evidence
## Limitations reported by the authors
## Additional limitations or confounders
## Reproduction status
## Connection to current work
## Three recall questions

Write the method section from memory before reopening the paper. The mismatch reveals what still needs work.

Computational-pathology audit

For WSI work, recover:

• patient, case, specimen, block, slide, and tile relationships;
• stain, scanner, MPP, tissue mask, quality control, and sampling;
• label source, timing, and whether report text leaks the outcome;
• exact encoder, transforms, aggregator, and frozen/fine-tuned status;
• patient- and institution-level split;
• pretraining overlap with TCGA, CPTAC, or the benchmark cohort;
• external validation, calibration, subgroup results, and uncertainty;
• survival censoring, treatment-era effects, or molecular-testing selection;
• clinical workflow and intended-use boundary.

If the paper cannot answer one of these, record the uncertainty rather than automatically rejecting the work.

Synthesize across papers

Do not compare headline numbers from incompatible datasets. Build a matrix with task definition, population, model version, supervision, split, adaptation, and metric. Then group papers by the scientific question they answer.

Finish with three outputs:

1. a concise claim–evidence–limitation summary;
2. a reproducible artifact or concrete reason reproduction is not possible;
3. recall questions that can be answered without rereading.

My Library records those recall questions, keeps an append-only review history, and schedules the next review from your understanding and review outcome. Its calendar export carries 30-, 7-, and 1-day reminders that remain useful when this static site is closed. Never commit personal notes or tracker exports to this repository.