Writing study: BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, Reynold Cheng · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

49 physical pages

Before references / appendix

≈ 10.73 page-equivalents

Main figures / tables

7 / 3

Attached appendix figures / tables

19 / 8

Attached appendix span

≈ 29.62 page-equivalents

Before references / appendix: ≈ 21.9%References / intervening material: ≈ 17.65%Attached appendix: ≈ 60.45%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.65 pp.	3.37%
2 PROBLEM DEFINITION	3	≈ 0.41 pp.	0.84%
3 BENCHMARK CONSTRUCTION	3	≈ 1.76 pp.	3.59%
4 EVALUATION SETTINGS	5	≈ 1.82 pp.	3.71%
5 EXPERIMENT	7	≈ 1.98 pp.	4.04%
6 USER SIMULATOR ANALYSIS	9	≈ 0.74 pp.	1.51%
7 RELATED WORK	10	≈ 0.43 pp.	0.88%
8 FUTURE WORK	10	≈ 0.22 pp.	0.45%
9 CONCLUSION	10	≈ 0.44 pp.	0.9%
ETHICS STATEMENT	11	≈ 0.12 pp.	0.24%
REPRODUCIBILITY STATEMENT	11	≈ 0.36 pp.	0.73%
REFERENCES	11	≈ 8.65 pp.	17.65%
A LIMITATIONS	20	≈ 0.13 pp.	0.27%
B THE USE OF LLM STATEMENT	20	≈ 0.11 pp.	0.22%
C ANNOTATION GROUP DETAILS	20	≈ 0.98 pp.	2%
D BENCHMARK DESIGN PRINCIPLES	21	≈ 1.03 pp.	2.1%
E COMPARISON WITH RELATED BENCHMARKS	22	≈ 0.8 pp.	1.63%
F EVALUATION METRICS	23	≈ 0.93 pp.	1.9%
G TEST SCRIPTS	24	≈ 0.93 pp.	1.9%
H AMBIGUITY AND FOLLOW-UP ANNOTATION DETAILS	25	≈ 4.24 pp.	8.65%
I EXPERIMENT DETAILS	29	≈ 1.02 pp.	2.08%
J ACTION SPACE AND SELECTION PATTERNS IN a-INTERACT	30	≈ 4.18 pp.	8.53%
K PERFORMANCE ON DIFFERENT AMBIGUITY TYPES	34	≈ 0.98 pp.	2%
L EXPERIMENTS ON BIRD-INTERACT-LITE	35	≈ 0.15 pp.	0.31%
M ERROR ANALYSIS	35	≈ 0.75 pp.	1.53%
N USER SIMULATOR DESIGN DETAILS	36	≈ 0.61 pp.	1.24%
O EVALUATING THE FUNCTION-DRIVEN USER SIMULATOR	37	≈ 1.6 pp.	3.27%
P PATHWAYS TO EFFECTIVE COMMUNICATION	38	≈ 1.84 pp.	3.76%
Q HUMAN EVALUATION OF DATASET QUALITY	40	≈ 0.74 pp.	1.51%
R PROMPTS	41	≈ 8.63 pp.	17.61%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Table 1 · p. 5
• Figure 3 · p. 6
• Table 2 · p. 7
• Figure 4 · p. 8
• Figure 5 · p. 8
• Figure 6 · p. 9
• Table 3 · p. 9
• Figure 7 · p. 20

Attached appendix

• Table 4 · p. 22
• Table 5 · p. 22
• Figure 8 · p. 22
• Figure 9 · p. 27
• Table 6 · p. 27
• Table 7 · p. 28
• Table 8 · p. 29
• Table 9 · p. 30
• Figure 10 · p. 31
• Figure 11 · p. 31
• Figure 12 · p. 32
• Figure 13 · p. 32
• Figure 14 · p. 34
• Figure 15 · p. 35
• Table 10 · p. 35
• Figure 16 · p. 36
• Figure 17 · p. 36
• Table 11 · p. 39
• Figure 18 · p. 39
• Figure 19 · p. 42
• Figure 20 · p. 43
• Figure 21 · p. 44
• Figure 22 · p. 45
• Figure 23 · p. 46
• Figure 24 · p. 47
• Figure 25 · p. 48
• Figure 26 · p. 49

Argument and organization

1. Benchmark construction and evaluation protocols

   Explain injected ambiguity, dependent follow-ups and two interaction modes before the leaderboard.

2. Interaction and simulator analyses

   Investigate communication, budget scaling and simulator alignment separately from SQL accuracy.

Problem, claim and contribution

The opening concrete interaction motivates a change in the unit of evaluation—from isolated SQL to clarification, execution and state-dependent follow-up—rather than merely claiming another harder dataset.

Sections 1–3; Figure 1, page 2. · Main paper

Argument and section order

Memory grafting tests whether another model's interaction history improves SQL generation, giving the communication claim a targeted intervention rather than relying solely on cross-model action correlations.

Section 5.2; Figure 5, page 8; Appendix J.3. · Main paper

Figures, captions and visual explanation

Figure 3 separates environment resources, priced actions and the user simulator across the top, then contrasts scripted and autonomous control flow below. Reward branches make the evaluation definition visually recoverable.

Figure 3, page 6. · Main paper

Tables, comparisons and evidence

Table 2 uses grouped headers for initial and follow-up success, domains, reward and cost; its caption explicitly says success is cumulative and marks debugging gains. These distinctions are essential when interpreting the lower follow-up scores.

Table 2 and Section 5.1, page 7. · Main paper

Appendix and reproducibility

The navigable appendix documents annotator training, ambiguity taxonomies, test scripts, action costs and complete parser/generator prompts; these operational artifacts are unusually valuable for reproducing an interactive benchmark.

Appendix contents, pages 18–19; Appendices C, G–J and R. · Main paper

Clarity and writing conventions

Strong reproducibility and safeguard language needs qualification: API decoding at temperature zero is not a guarantee of deterministic service behavior, and the displayed generator prompt includes the full ground-truth SQL, so safeguards are not a hard information-isolation boundary.

Reproducibility statement, page 11; Appendix I.3, page 30; Figure 25, page 48. · Main paper

What to reuse—and what not to infer

Transferable lesson: Define interactive protocols with a control-flow figure and a caption that explains scoring dependencies; then test the simulator itself rather than treating it as a transparent evaluator.

Boundary: Do not infer intrinsic follow-up difficulty from cumulative success alone, or general human collaboration from simulated SQL interactions. Single-run API results and prompt-based leakage controls limit stronger reliability claims.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11: Sections 1–9, ethics and reproducibility statements.

Appendix / supplement

Pages 18–49: contents and Appendices A–R, including all prompts and executable-test examples.

Visually inspected PDF pages

6, 7

• ICLR 2026 final proceedings · 49 pages