Writing study: LLMs Get Lost In Multi-Turn Conversation

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary) · Datasets, benchmarks & measurement resources.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

41 physical pages

Before references / appendix

≈ 10.74 page-equivalents

Main figures / tables

5 / 1

Attached appendix figures / tables

5 / 9

Attached appendix span

≈ 24.89 page-equivalents

Before references / appendix: ≈ 26.2%References / intervening material: ≈ 13.1%Attached appendix: ≈ 60.71%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 0.86 pp.	2.1%
2 BACKGROUND AND RELATED WORK	2	≈ 0.79 pp.	1.93%
3 SIMULATING UNDERSPECIFIED, MULTI-TURN CONVERSATION	3	≈ 1.38 pp.	3.37%
4 EXPERIMENT	4	≈ 3 pp.	7.32%
5 RESULTS	7	≈ 1.79 pp.	4.37%
6 IMPLICATIONS SUMMARY	9	≈ 0.82 pp.	2%
7 CONCLUSION	10	≈ 0.17 pp.	0.41%
8 ETHICS STATEMENT	10	≈ 0.95 pp.	2.32%
9 REPRODUCIBILITY STATEMENT	11	≈ 0.24 pp.	0.59%
REFERENCES	11	≈ 5.37 pp.	13.1%
A RELATED WORK ON UNDERSPECIFICATION	17	≈ 0.51 pp.	1.24%
B PRECISE DEFINITION OF SHARDED INSTRUCTIONS	17	≈ 1.05 pp.	2.56%
C SEMI-AUTOMATIC SHARDING PROCESS	18	≈ 1.03 pp.	2.51%
D INSPECTION OF SIMULATED SHARDED CONVERSATION	19	≈ 0.99 pp.	2.41%
E CONCRETE EXAMPLE OF LOSS IN APTITUDE VS. RELIABILITY	20	≈ 1.18 pp.	2.88%
F QUALITATIVE ANALYSES OF SIMULATION LOGS	21	≈ 3.76 pp.	9.17%
G IMPLICATIONS	25	≈ 3.99 pp.	9.73%
H ASSISTANT RESPONSE CATEGORIZATION	29	≈ 0.18 pp.	0.44%
I MODEL ACCESS	29	≈ 0.68 pp.	1.66%
J TASK-SPECIFIC IMPLEMENTATION DETAILS	30	≈ 3.17 pp.	7.73%
K EXAMPLE SIMULATED CONVERSATION	33	≈ 0.21 pp.	0.51%
L GRADUAL SHARDING IMPLEMENTATION	33	≈ 0.82 pp.	2%
M TEMPERATURE EXPERIMENT IMPLEMENTATION	34	≈ 0.46 pp.	1.12%
N RECAP & SNOWBALL EXPERIMENT IMPLEMENTATION	35	≈ 0.34 pp.	0.83%
O ON OBTAINING DETERMINISTIC OUTPUTS FROM LLMS	35	≈ 6.51 pp.	15.88%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 3
• Figure 3 · p. 4
• Figure 4 · p. 5
• Table 1 · p. 7
• Figure 5 · p. 7

Attached appendix

• Figure 6 · p. 18
• Table 2 · p. 20
• Figure 7 · p. 21
• Table 3 · p. 22
• Figure 8 · p. 23
• Figure 9 · p. 24
• Table 4 · p. 25
• Table 5 · p. 26
• Table 6 · p. 26
• Table 7 · p. 27
• Table 8 · p. 28
• Table 9 · p. 29
• Table 10 · p. 29
• Figure 10 · p. 34

Argument and organization

1. Problem and controlled conversation simulator

   Contrasts episodic evaluation with gradually disclosed requirements.

2. Metrics, findings and causal hypotheses

   Separates mean performance, high-percentile aptitude and interpercentile unreliability.

3. Implications by audience and appendices

   Routes findings to builders, practitioners and users; preserves protocols and full prompts.

Problem, claim and contribution

The paper names a recognizable failure mode, then operationalizes it with sharding instead of relying on anecdotes about chat quality.

Sections 1–3 · Main paper

Argument and section order

FULL, CONCAT, SHARDED, RECAP and SNOWBALL form interpretable controls; CONCAT specifically tests whether rephrasing rather than conversational disclosure causes degradation.

Section 3.3, Figure 3 · Main paper

Figures, captions and visual explanation

The simulator diagram highlights the evaluated assistant separately from judge components and distinguishes failed, non-answer and successful paths.

Figure 2, p. 3 · Main paper

Tables, comparisons and evidence

Table 1 uses repeated task columns and degradation shading, while Figure 5 first teaches the percentile encoding before applying it. The combined page is dense and icon-only headers need the caption.

Table 1 and Figure 5, p. 7 · Main paper

Appendix and reproducibility

The appendix includes a counterexample task (translation), annotation taxonomy, dated model versions, complete prompts and a failed conversation transcript—not only extra positive results.

Appendices G.3, H–K and prompts, pp. 28–41 · Main paper

Clarity and writing conventions

Bold result sentences aid scanning, but categorical statements about any multi-turn conversation are stronger than the controlled English analytical-task sample; later limitations restore the narrower scope.

Section 5.3 versus Section 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Design a control that isolates the proposed failure mechanism, then teach any new metric visually before presenting its cross-model distribution.

Boundary: Treat verbosity and premature-answer analyses as observational, not causal interventions. The 39% headline is an aggregated relative drop, not 39 percentage points. Appendix D alternates 200 and 100 inspected conversations; Appendix L alternates eight and 31 selected instructions, so audit counts before reuse.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–9

Appendix / supplement

Pages 17–41, Appendices A–O and all sharding/simulation prompts

Visually inspected PDF pages

3, 7

• ICLR 2026 final proceedings · 41 pages