Writing study: Rare Event Analysis of Large Language Models
← ICML 2026 oral writing corpus
Jake Mcallister Dorman, Edward Gillman, Dominic C Rose, Jamie F. Mair, Juan P. Garrahan · Official ICML 2026 oral record
Figure, table and section ledger
Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 23 physical pages
- Before references / appendix
- ≈ 9.12 page-equivalents
- Main figures / tables
- 5 / 0
- Attached appendix figures / tables
- 2 / 0
- Attached appendix span
- ≈ 9.91 page-equivalents
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
| Top-level heading | Starts on PDF page | Estimated span | Share of entire PDF |
|---|---|---|---|
| 1. Introduction | 1 | ≈ 1.49 pp. | 6.48% |
| 2. Relation to Other Work | 2 | ≈ 0.94 pp. | 4.09% |
| 3. Background on Rare Event Methods | 3 | ≈ 1.88 pp. | 8.17% |
| 4. Experimental Setup | 5 | ≈ 1.66 pp. | 7.22% |
| 5. Rare Completion Probability Estimation | 7 | ≈ 1.06 pp. | 4.61% |
| 6. Rare Completion Exploration | 8 | ≈ 0.96 pp. | 4.17% |
| 7. Conclusion and Outlook | 9 | ≈ 0.66 pp. | 2.87% |
| Acknowledgements | 9 | ≈ 0.17 pp. | 0.74% |
| Impact Statement | 10 | ≈ 0.05 pp. | 0.22% |
| References | 10 | ≈ 3.96 pp. | 17.22% |
| A. List of Acronyms | 14 | ≈ 0.27 pp. | 1.17% |
| B. Algorithms for Markov Chain Monte Carlo and Transition Path Sampling | 14 | ≈ 4.23 pp. | 18.39% |
| C. Implementation Details, Tips and Tricks for Rare Event Sampling in Language Models | 18 | ≈ 4.18 pp. | 18.17% |
| D. Comparison of Histogram Estimations for Different Starting Prompts | 22 | ≈ 1.23 pp. | 5.35% |
All detected figure and table locations
Attached appendix
Argument and organization
- 1 Introduction; 2 Relation to Other Work
Sets up definition, probability estimation and exploration as three different rare-event tasks.
- 3 Background; 4 Experimental Setup
Introduces stochastic text generation, reweighting and trajectory sampling before the TinyStories observables and budget.
- 5 Probability Estimation; 6 Exploration; 7 Conclusion and Outlook
Reports tail estimates, error diagnostics and concrete unusual completions before discussing extensions.
Problem, claim and contribution
The opening roadmap separates finding rare outputs from estimating how probable they are and understanding their structure.
Sections 1–2, physical pp.1–2 · Main paper
Argument and section order
Distribution reweighting and sample generation are explained before model/observable choices, then estimation is followed by a dedicated error analysis.
Sections 3–5, pp.3–8 · Main paper
Figures, captions and visual explanation
The distribution panels distinguish raw biased counts from reconstructed model density; the next figure compares interval width and change after doubling chain length.
Figures 3–4, pp.7–8 · Main paper
Tables, comparisons and evidence
The experiment reports generated-token scale, burn-in removal and rejected nonconverged biases; bootstrap resampling is over independent chains rather than pretending correlated completions are independent.
Section 4 and Figure 2, p.6; Section 5, p.7 · Main paper
Appendix and reproducibility
Pseudocode and practical guidance explain why observable smoothness, bias strength and distribution overlap affect whether the estimator works.
Appendix C, selected pp.19–20; headings through p.22 · Main paper
Clarity and writing conventions
The tail discussion says more steps may substantially change estimates and cautions about interval coverage, instead of treating narrow plotted bands as certainty.
Section 5, pp.7–8 · Main paper
What to reuse—and what not to infer
Transferable lesson: Pair a rare-event estimate with convergence exclusions, uncertainty construction and sensitivity to additional sampling; a tail plot alone is insufficient.
Boundary: TinyStories-8M and two observables are a methodological demonstration, not an audit of production-model safety. A convergence diagnostic is not a proof of mixing, and repetitive high-ARI text need not be sophisticated language. Preserve the source’s interval convention without assuming its coverage has been independently validated.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-10-07
- Reading status
- Writing-focused manuscript scan + selected visuals inspected
- Main text
- Writing-focused scan across physical PDF pp.1–9, covering framing, section progression, definitions, method/evidence explanations, captions and scope. Physical pages extracted individually to avoid form-feed pagination errors. Selected full-page rasters inspected; not exhaustive line-by-line technical verification or experimental reproduction.
- Appendix / supplement
- Appendix structure A–D inventoried through p.23; selected reading of annealed TPS pseudocode, burn-in/convergence diagnostics and observable/bias/proposal guidance C.2–C.6, pp.19–20. Derivations and interval coverage were not independently verified.
- Visually inspected PDF pages
- 7, 8
This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.