Writing study: Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Hinostroza, Mattia Nee, Eliot Jones, Irène Girard, David Mach, Anastasia Stasenko, Ivan Yamshchikov · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

34 physical pages

Before references / appendix

≈ 10.31 page-equivalents

Main figures / tables

4 / 2

Attached appendix figures / tables

1 / 12

Attached appendix span

≈ 15.89 page-equivalents

Before references / appendix: ≈ 30.32%References / intervening material: ≈ 22.94%Attached appendix: ≈ 46.74%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2 pp.	5.88%
2 ABOUT COMMON CORPUS	3	≈ 1.12 pp.	3.29%
3 RELATED WORK	4	≈ 0.9 pp.	2.65%
4 PROVENANCE	5	≈ 2.85 pp.	8.38%
5 CLEANING AND CURATION	8	≈ 1.31 pp.	3.85%
6 EVALUATION	9	≈ 0.65 pp.	1.91%
7 CONCLUSION	10	≈ 0.6 pp.	1.76%
ACKNOWLEDGEMENTS	11	≈ 0.21 pp.	0.62%
REFERENCES	11	≈ 7.79 pp.	22.91%
A LLM USAGE STATEMENT	19	≈ 0.08 pp.	0.24%
B CORPUS COMPOSITION	19	≈ 1.92 pp.	5.65%
C TOKENIZER DETAILS	21	≈ 0.42 pp.	1.24%
D PROVENANCE	21	≈ 5.27 pp.	15.5%
E OPEN CULTURE VERIFICATION	26	≈ 0.58 pp.	1.71%
F CLEANING AND CURATION	27	≈ 4.11 pp.	12.09%
G EVALUATIONS	31	≈ 3.52 pp.	10.35%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 3
• Figure 3 · p. 5
• Table 1 · p. 5
• Figure 4 · p. 9
• Table 2 · p. 10

Attached appendix

• Table 3 · p. 19
• Table 4 · p. 19
• Table 5 · p. 20
• Figure 5 · p. 21
• Table 6 · p. 21
• Table 7 · p. 23
• Table 8 · p. 24
• Table 9 · p. 26
• Table 10 · p. 26
• Table 11 · p. 32
• Table 12 · p. 33
• Table 13 · p. 34
• Table 14 · p. 34

Argument and organization

1. Turn an infrastructure gap into an auditable artifact

   Defines the intended openness/provenance criteria and positions the corpus against other collections before enumerating its parts.

2. Separate provenance from processing

   Organizes sources by six collections and then describes common OCR, segmentation, privacy, deduplication and toxicity-processing tools.

3. Show composition and utility at different resolutions

   Pairs aggregate descriptive plots and small-model benchmarks with a detailed metadata schema, source tables and per-language results.

Problem, claim and contribution

The central argument is about data availability, provenance and reproducibility, not a new neural architecture. A four-criterion related-dataset table makes the intended contribution legible. However, ethical and legally compliant are broader claims than a dataset comparison can establish; writing should distinguish the authors’ screening process from a legal guarantee.

Abstract; Sections 1–3; Table 1 · Main paper

Argument and section order

The main paper follows the artifact’s lifecycle: define the collection, situate it, document source families, explain cleaning and demonstrate downstream utility. Source-specific detail is placed in Appendix D while the common schema and counts appear in Appendix B. This split is useful for infrastructure papers because readers can audit either the general pipeline or a particular source without reconstructing both from scattered footnotes.

Sections 2–6; Appendices B, D and F · Main paper

Figures, captions and visual explanation

Figure 3 pairs a log-count timeline with a t-SNE view to communicate temporal and representational variety; neither embedding separation nor a color-coded cluster is a quantitative uniqueness proof. Table 1 uses both symbols and color, which is better than color alone. Its caption is below the table, illustrating why observed accepted-paper styling must not override the official template’s table-caption convention.

Figure 3 and Table 1, PDF page 5 · Main paper

Tables, comparisons and evidence

Table 2 exposes a useful trade-off: the 350M model performs strongly on MultiBLiMP but trails the displayed peers on XStoryCloze and XCOPA; the 1.2B model beats OLMo but not Gemma 3 across all rows. Parameter sizes, token budgets and tokenizer differences are not matched causal controls. Appendix G’s per-language results are essential because a multilingual average can hide near-chance or weaker languages.

Section 6; Table 2, PDF page 10; Tables 11–14, PDF pages 32–34 · Main paper

Appendix and reproducibility

The appendix records document/word/token counts separately, names the counting tokenizer, provides license and language distributions, and expands every source family. The OCR tools include input/output examples and known synthetic-rewriting behavior. These details make the artifact easier to inspect, but measured privacy-filter accuracy below 100% and imperfect curation are not guarantees that personal or problematic material has been eliminated.

Appendices B–G; Section 5 and Limitations · Main paper

Clarity and writing conventions

An important consistency check is needed in the rights-screening description: Appendix E prints an after-1884 condition, whereas Appendix D.2 repeatedly describes retaining material prior to 1884. The main text and appendix also label OCRerrcr as different DeBERTa versions. Record these conflicts instead of silently choosing a version. The title’s largest/ethical language should stay attributed and dated, not become an unqualified recommendation.

Appendix D.2, PDF pages 25–26; Appendix E, PDF page 27; Section 5 versus Appendix F.3 · Main paper

What to reuse—and what not to infer

Transferable lesson: For a dataset paper, build an audit trail from source and license metadata through transformations to measured utility. Distinguish document, word and tokenizer-dependent token counts; show subgroup results and imperfect-cleaning limits alongside aggregate scale.

Boundary: This is a writing analysis, not legal advice or certification of data rights/privacy. The paper contains an unresolved before/after-1884 wording conflict and tool-version inconsistency; verify source-specific terms and implementation before use. Small-model comparisons do not isolate corpus quality from training and architecture differences, and this corpus is not a WSI dataset.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

All Sections 1–7, limitations and acknowledgements, PDF pages 1–11: motivation, corpus definition, related datasets, six provenance collections, curation tools and two-model evaluation.

Appendix / supplement

All Appendices A–G, PDF pages 19–34: LLM disclosure, schema/counts/licenses/languages, tokenizer fertility, source-level provenance, rights-screening description, tool details and every per-language evaluation table.

Visually inspected PDF pages

5, 10, 27

• Official ICLR 2026 proceedings manuscript including appendices · 34 pages