Writing study: Hubble: a Model Suite to Advance the Study of LLM Memorization

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Johnny Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna Gummadi, Willie Neiswanger, Robin Jia · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary) · Empirical analysis, diagnosis & auditing.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

62 physical pages

Before references / appendix

≈ 10.46 page-equivalents

Main figures / tables

3 / 1

Attached appendix figures / tables

20 / 14

Attached appendix span

≈ 38.87 page-equivalents

Before references / appendix: ≈ 16.87%References / intervening material: ≈ 20.44%Attached appendix: ≈ 62.69%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 0.97 pp.	1.56%
2 PERTURBATION DESIGN ACROSS RISK DOMAINS	2	≈ 1.82 pp.	2.94%
3 THE HUBBLE SUITE	4	≈ 3.18 pp.	5.13%
4 DOMAIN-AGNOSTIC RESULTS	7	≈ 1 pp.	1.61%
5 DOMAIN-SPECIFIC RESULTS	8	≈ 0.51 pp.	0.82%
6 USE CASES OF HUBBLE	9	≈ 1.6 pp.	2.58%
7 DISCUSSION AND CONCLUSION	10	≈ 0.4 pp.	0.65%
ACKNOWLEDGMENTS	11	≈ 0.35 pp.	0.56%
REFERENCES	11	≈ 12.67 pp.	20.44%
Appendix	24	≈ 0.97 pp.	1.56%
A PERTURBATIONS	25	≈ 4 pp.	6.45%
B TRAINING	29	≈ 2 pp.	3.23%
C GENERAL EVALUATION	31	≈ 3 pp.	4.84%
D DOMAIN-SPECIFIC RESULTS	34	≈ 14 pp.	22.58%
E ADDITIONAL RESULTS	48	≈ 5 pp.	8.06%
F ADDITIONAL HUBBLEMIA RESULTS	53	≈ 4 pp.	6.45%
G ADDITIONAL HUBBLEUNLEARNING RESULTS	57	≈ 3 pp.	4.84%
H FURTHER DISCUSSION	60	≈ 1 pp.	1.61%
I ADDITIONAL PLOTS	61	≈ 1.89 pp.	3.05%

All detected figure and table locations

Main text

• Figure 1 · p. 5
• Figure 2 · p. 7
• Table 1 · p. 8
• Figure 3 · p. 10

Attached appendix

• Table 2 · p. 25
• Table 3 · p. 27
• Table 4 · p. 27
• Table 5 · p. 29
• Table 6 · p. 31
• Table 7 · p. 32
• Table 8 · p. 33
• Figure 4 · p. 35
• Figure 5 · p. 35
• Table 9 · p. 36
• Figure 6 · p. 38
• Figure 7 · p. 39
• Figure 8 · p. 40
• Table 10 · p. 41
• Figure 9 · p. 43
• Figure 10 · p. 45
• Figure 11 · p. 46
• Figure 12 · p. 47
• Figure 13 · p. 48
• Figure 14 · p. 49
• Figure 15 · p. 51
• Figure 16 · p. 52
• Table 11 · p. 53
• Table 12 · p. 53
• Table 13 · p. 54
• Table 14 · p. 54
• Figure 17 · p. 55
• Figure 18 · p. 55
• Figure 19 · p. 56
• Table 15 · p. 57
• Figure 20 · p. 58
• Figure 21 · p. 59
• Figure 22 · p. 61
• Figure 23 · p. 62

Argument and organization

1. Risk domains to controlled perturbations

   Translates copyright, privacy and contamination questions into randomized experimental interventions.

2. Model suite, cross-domain findings and reusable benchmarks

   Separates the research artifact from illustrative memorization, membership-inference and unlearning studies.

Problem, claim and contribution

The introduction identifies a methodological gap between small controlled studies and large observational studies, then explains why a released intervention suite closes it.

Section 1, pages 1–2. · Main paper

Argument and section order

The model runs are organized by research question—core factorial design, interference, timing, paraphrases and architecture—rather than presented as an undifferentiated checkpoint list.

Section 3.2, pages 5–6. · Main paper

Figures, captions and visual explanation

Figure 1 visualizes insertion boundaries, EOS delimiters and sequence resizing, and its caption explicitly discloses how perturbations differ from ordinary documents.

Figure 1, page 5. · Main paper

Tables, comparisons and evidence

Figure 3 marks the perturbed model, standard model and desired selective-unlearning target, with arrows indicating improvement. This makes utility loss and forgetting visible simultaneously.

Figure 3, page 10. · Main paper

Appendix and reproducibility

The appendix supplies dataset links, duplicate counts, attack templates and grids; notably, it reports that ELLie minimal-pair overlap invalidates that dataset for the dilution analysis.

Appendices A, D.2–D.3 and G; pages 25–28, 36–45 and 57. · Main paper

Clarity and writing conventions

The evaluation section labels attacks as lower bounds on memorization, resisting the common inference that failed extraction proves absence of stored information.

Section 3.3, page 6; Appendix D.2.1, page 37. · Main paper

What to reuse—and what not to infer

Transferable lesson: For a benchmark or model-suite paper, connect each artifact variation to a causal question, make interventions inspectable, and publish invalidated analyses as well as favorable findings.

Boundary: Dilution and early ordering are empirical risk-reduction findings, not privacy guarantees or legal safe harbors. Appendix ROC captions reverse member/non-member labels relative to the stated protocol; hyperparameter epsilon values also differ between Table 5 and the prose, so reuse the released configuration rather than silently resolving discrepancies.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–7 and acknowledgments; bibliographic references separately listed on pages 11–23.

Appendix / supplement

Pages 24–62, Appendices A–I, including data generation, decontamination, configurations, domain analyses, benchmark results and all plot captions.

Visually inspected PDF pages

5, 10

• Official ICLR 2026 proceedings conference PDF; PDF creation metadata: Sun May 17 12:19:48 2026 EDT · 62 pages