Writing study: TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Christoph Minixhofer, Ondrej Klejch, Peter Bell · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

20 physical pages

Before references / appendix

≈ 10.11 page-equivalents

Main figures / tables

3 / 5

Attached appendix figures / tables

5 / 3

Attached appendix span

≈ 5.89 page-equivalents

Before references / appendix: ≈ 50.55%References / intervening material: ≈ 20%Attached appendix: ≈ 29.45%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 2.14 pp.	10.7%
2 TTSDS2	3	≈ 1.91 pp.	9.55%
3 CORRELATIONS WITH LISTENING TESTS ACROSS DATASETS	5	≈ 2.93 pp.	14.65%
4 MULTILINGUAL & RECURRING EVALUATION	8	≈ 1.8 pp.	9%
5 LIMITATIONS	10	≈ 0.22 pp.	1.1%
6 ETHICS STATEMENT	10	≈ 0.22 pp.	1.1%
7 CONCLUSION	10	≈ 0.37 pp.	1.85%
REFERENCES	11	≈ 4 pp.	20%
A EVALUATED TTS SYSTEMS AND PARITY WITH REAL SPEECH	15	≈ 1 pp.	5%
B LISTENING TEST DETAILS	16	≈ 2 pp.	10%
C DOMAIN-WISE MOS, CMOS, SMOS AND TTSDS2 RESULTS	18	≈ 1 pp.	5%
D TTSDS2 FACTORS	19	≈ 0.26 pp.	1.3%
E CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE METRICS	19	≈ 0.74 pp.	3.7%
F COMPUTE	20	≈ 0.14 pp.	0.7%
G LANGUAGE DISTANCE VISUALISATION	20	≈ 0.34 pp.	1.7%
H FILTERING OF CONTROVERSIAL CONTENT	20	≈ 0.41 pp.	2.05%

All detected figure and table locations

Main text

• Table 1 · p. 3
• Table 2 · p. 5
• Figure 1 · p. 5
• Table 3 · p. 7
• Figure 2 · p. 7
• Table 4 · p. 8
• Table 5 · p. 8
• Figure 3 · p. 9

Attached appendix

• Table 6 · p. 15
• Figure 4 · p. 17
• Figure 5 · p. 17
• Figure 6 · p. 17
• Table 7 · p. 18
• Table 8 · p. 19
• Figure 7 · p. 19
• Figure 8 · p. 20

Argument and organization

1. What existing speech metrics measure

   Introduces a reference-requirement taxonomy before proposing a replacement.

2. Factorized distribution distance

   Defines four feature families, normalized scores and aggregation.

3. Human validation and recurring multilingual benchmark

   Separates direct validation from the broader deployment pipeline.

Problem, claim and contribution

The paper motivates cross-domain metric reliability through conflicting human-parity claims, then distinguishes matching-reference, nonmatching-reference and reference-free evaluation. This taxonomy makes the actual comparison conditions legible before new scores appear.

§§1–2, pp.1–4 · Main paper

Argument and section order

Four factor families are defined before human validation. Leave-one-domain-out aggregation experiments justify the simple mean with a visible exception: learned weights outperform it on the held-out children's domain, so robustness is not universal dominance.

§2, pp.3–5; §3.5 and Tables4–5, pp.7–8 · Main paper

Figures, captions and visual explanation

The main correlation table and scatterplots complement the appendix's domain-wise raw ratings. The former uses Spearman ranking correlation and the latter Pearson correlation; keeping these names explicit avoids treating different summary statistics as inconsistent estimates.

Table3 and Figure2, p.7; Table7, p.18 · Main paper

Tables, comparisons and evidence

Two hundred listeners validate twenty systems in four English domains. The fourteen-language extension instead uses typological-distance correlations and score sanity checks, not multilingual human quality labels; the paper explicitly acknowledges this missing validation.

§3.2, p.6; §4.2, p.10 · Main paper

Appendix and reproducibility

The appendix provides survey wording, ethics approval, system modifications, per-domain uncertainty and CPU timing. It also records English-tuned controversial-content filters, exposing a selection decision that may shape multilingual representativeness.

AppendicesA–C andF–H, pp.15–20 · Main paper

Clarity and writing conventions

The limitation section clearly states that distributional similarity cannot replace listeners or detect every transcript failure. A leftover editing instruction in the pipeline introduction illustrates why final editorial review matters even in an otherwise reproducible benchmark paper.

§4, p.8; §5, p.10 · Main paper

What to reuse—and what not to infer

Transferable lesson: Validate an evaluation metric against humans under several domains, report raw ratings beside correlations, and explicitly distinguish direct validation from multilingual plausibility checks.

Boundary: The score is dataset-level rather than per-utterance correctness, and its reference distribution matters. Temporal refresh reduces some contamination risk but cannot guarantee absence of leakage. Negative WER correlations have the expected direction and should not automatically be called failures.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

pp.1–10, §§1–7; references pp.11–14

Appendix / supplement

pp.15–20, A–H: system provenance, listening instructions, ethics approval, domain results, factors, correlations, compute and filtering

Visually inspected PDF pages

7, 18

• official ICLR 2026 proceedings PDF · 20 pages