Writing study: Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study

← AAAI 2026 oral writing corpus

Yuqi Zhu, Yi Zhong, Jintian Zhang, Ziheng Zhang, Shuofei Qiao, Yujie Luo, Lun Du, Da Zheng, Ningyu Zhang, Huajun Chen · Official AAAI 2026 oral record

Figure, table and section ledger

Editorial contribution type: Empirical analysis, diagnosis & auditing (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF: 9 physical pages
Before references / appendix: ≈ 7.13 page-equivalents
Main figures / tables: 5 / 8
Attached appendix figures / tables: 0 / 0
Attached appendix span: Separate file recorded below; not in this PDF total

Before references / appendix: ≈ 79.22%References / intervening material: ≈ 20.78%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.

Source-bound correction: Checked 2026-10-25 against complete proceedings reading, rasters 4–7 and Poppler boxes. Restore the wrapped Section 5 title. The line beginning Figure 5 in the left-column prose is not a caption: remove both same-page candidates and restore only the genuine right-column caption at 5.6323. Figures 1–5 and Tables 1–8. No appendix in the nine-page accepted-main PDF.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 Introduction	1	≈ 1 pp.	11.11%
2 Background of Data Analysis Agents	2	≈ 0.25 pp.	2.78%
3 Experimental Settings	2	≈ 0.37 pp.	4.11%
4 Core Capabilities for Data Analysis	2	≈ 3.93 pp.	43.67%
5 Towards Effective Data Analysis Performance	6	≈ 0.75 pp.	8.33%
6 Related Work	7	≈ 0.28 pp.	3.11%
7 Conclusion and Future Work	7	≈ 0.18 pp.	2%
Acknowledgements	8	≈ 0.1 pp.	1.11%
References	8	≈ 1.87 pp.	20.78%

All detected figure and table locations

Main text

Attached appendix

No recognized caption labels.

Separate source files—excluded from canonical-PDF totals

A separate author version may repeat main text and references. Whole-file lengths and counts are not extra appendix material and are never added to the canonical PDF totals. Read the study's source scope for which sections received substantive reading versus layout inspection.

Additional source 1

Separate PDF (or ZIP containing it): whole-file layout of 13 physical pages; 8 detected figures and 10 tables. These may include overlapping main-text material.

Located appendix only: starts on physical p.10; ≈ 3.34 page-equivalents, 3 figure labels and 2 table labels. This interval is distinct from the whole file.

Detected heading	PDF page	Estimated span
1 Introduction	1	≈ 0.99 pp.
2 Background of Data Analysis Agents	2	≈ 0.26 pp.
3 Experimental Settings	2	≈ 0.37 pp.
4 Core Capabilities for Data Analysis	2	≈ 3.91 pp.
5 Towards Effective Data Analysis Performance	6	≈ 0.75 pp.
6 Related Work	7	≈ 0.47 pp.
7 Conclusion and Future Work	8	≈ 0.09 pp.
Acknowledgement	8	≈ 0.1 pp.
References	8	≈ 2.43 pp.
A Experimental Setup.	10	≈ 0.46 pp.
B Problem Diversity.	11	≈ 0.13 pp.
C Data Synthesis Details	11	≈ 2.75 pp.

Checked 2026-10-25 against Poppler boxes and full-page rasters of all 13 pages. Author extended v5 is a 13-page source file, NOT a standalone supplement. Its overlapping main text/reference layout is separate from the accepted proceedings metrics. The actual appendix starts in the right column of physical p.10 at position 9.661 and runs through p.13: approximately 3.34 column-normalized page coordinates, not 13 appendix pages. Main captions Figures 1–5/Tables 1–8; appendix captions Figures 6–8/Tables 9–10. Physical p.6 raster distinguishes the true left-column Figure 5 caption at 5.2024 from a later prose cross-reference at 5.3279. Figures 6–8 are a category-definition box and two prompt artifacts, not extra performance plots. Appendix A–C was substantively read; overlapping main text received full-file layout inspection only, not a second semantic reading.

Argument and organization

1 Introduction; 2 Background of Data Analysis Agents; 3 Experimental Settings
Defines the task and agent loop, then specifies model families, generated-data provenance, evaluation datasets and judge before making cross-capability comparisons.
4 Core Capabilities for Data Analysis
Organizes the investigation by data comprehension, code capability and strategic planning, then decomposes planning into interaction turns, reasoning length, task difficulty and domain diversity.
5 Towards Effective Data Analysis Performance
Turns observations from the preceding analyses into a three-stage selection/enrichment recipe and evaluates the resulting 2.8k-example fine-tuning set.
6 Related Work; 7 Conclusion and Future Work; Acknowledgements; References
Places the empirical findings and follow-up intervention among agent methods and benchmarks, then identifies dataset and evaluation breadth as remaining limits.
Extended manuscript: A Experimental Setup; B Problem Diversity; C Data Synthesis Details
Adds operational configuration, taxonomy construction, generation/filtering pseudocode, exact prompt artifacts and final-model hyperparameters outside the nine-page proceedings body.

Problem, claim and contribution

Frames a diagnostic question about why data-analysis agents fail, then separates three candidate capabilities rather than opening with a new module name. Figure 1 anchors the abstraction in a worked question/data/plan/code/answer example. The paper later turns its observations into a synthesis recipe; that intervention is presented after, not in place of, the main empirical investigation.

Introduction and Figure 1, proceedings p.1; Background and Section 4 opening, p.2; Section 5, pp.6–7 · Main paper · Additional source 2

Argument and section order

Uses question-driven subsections and local discussion blocks to connect an experimental manipulation, a result and an interpretation. Within planning, the order is turns, reasoning length, difficulty, then diversity, before a practical follow-up. A reader can find the comparison settings before the conclusions. However, prompt-only analyses and LoRA analyses have different protocols and sample budgets, so adjacent tables are not a single controlled ranking of three intrinsic capabilities.

Sections 3–4, proceedings pp.2–6; extended Appendix A, pp.10–11 · Main paper · Additional source 2

Figures, captions and visual explanation

Uses several distinct visual jobs: a worked agent example, a composition of sampled error labels, paired plots across datasets/model scales, reasoning-budget sweeps, and paired response-turn/token panels. The numeric labels in Figure 4 make the resource-dependent reversal inspectable—for DiscoveryBench, full traces fall to 9.62% at the largest shown budget while summaries reach 18.83%. Error-type percentages describe 354 selected incorrect responses, not all queries or a causally validated failure decomposition.

Figure 1, p.1; Figure 2 and error-sampling description, p.4; Figures 3–4, p.5; Figure 5, p.6 · Main paper · Additional source 2

Tables, comparisons and evidence

Reports model variants, task datasets and GPT-4o-mini-based answer scoring before the results, and keeps sample counts near turn-length comparisons. Figure 3 controls fine-tuning data at 1,020 examples, whereas Table 5 deliberately shows unequal full-subset sizes; those are different questions. Table 8 groups API, 7B and 14B models rather than hiding scale differences. The displayed tables/curves do not supply confidence intervals, and explanations such as attention disruption or internalized reasoning remain hypotheses, not isolated causal measurements.

Section 3, pp.2–3; Tables 4–5 and controlled-turn setup, p.4; Table 8 and Section 5.2, p.7 · Main paper · Additional source 2

Appendix and reproducibility

The extended version provides operational material rather than only additional scores: Appendix A lists data sources and analysis settings; B explains model-assisted taxonomy consolidation; C gives executable-style filtering pseudocode, medium-turn/difficulty criteria, model-specific final-training settings and prompt artifacts. This supports a claim-indexed appendix plan. Its full file repeats main text and references, so the supporting-material span must start at Appendix A rather than treating the whole 13-page author PDF as an appendix.

Proceedings extension pointers, pp.6–7; extended v5 Appendix A–C, pp.10–13; Algorithm 1 and Table 10, p.11; Figures 6–8, pp.12–13 · Main paper · Additional source 2

Clarity and writing conventions

Short finding-led paragraph openings make the empirical story easy to navigate, and the final evaluation discusses limited coverage and possible scale effects. Some headings and conclusions are broader than their tests. There is also a concrete text/figure mismatch: Section 4.3 defines short/medium/long as 2–3/4–5/6+ turns, while Figure 3 labels its bins 1–2/3–4/5–6. The inspected extended pipeline uses 4–5 turns for final filtering, but that does not resolve Figure 3's bin definition.

Section 4.3, p.4; Figure 3 and finding-led prose, p.5; Section 5.2 limitations, p.7; extended Appendix C, p.11 · Main paper · Additional source 2

What to reuse—and what not to infer

Transferable lesson: For an empirical-analysis paper, organize the argument around competing explanations and controlled questions, then show how the findings motivate a follow-up intervention. Give every table/plot its own sample budget, protocol and denominator; include prompt and filtering artifacts in an accessible, versioned appendix.

Boundary: These observations concern particular Qwen variants, data-generation choices, two evaluation datasets and an LLM-based judge. They do not prove a universal hierarchy of planning, coding and comprehension or causal explanations for every error. The turn-bin text/figure mismatch is unresolved, and the sample sizes differ across several analyses. The extended manuscript adds inspected appendix evidence but is not identical in layout to the accepted proceedings PDF and was not independently reproduced.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked: 2026-10-25
Reading status: Writing-focused manuscript scan + selected visuals inspected
Main text: Writing-oriented reading of proceedings physical pp.1–7, including all capability analyses, experimental controls, figures, tables, synthesis follow-up and limitations; acknowledgements/reference boundary and reference continuation on pp.8–9 checked. Full-page rasters of pp.4, 5, 6 and 7 inspected. No code audit, contamination audit, judge validation or result reproduction was performed.
Appendix / supplement: The proceedings PDF has no attached appendix. Its linked author extended manuscript v5 was retrieved separately: Appendix A–C on physical pp.10–13 was read, including source counts, analysis/final-training configurations, taxonomy procedure, generation pseudocode, filtering and prompt artifacts. All 13 full-page rasters were inspected for the extended file's layout, headings and captions, including the p.6 caption ambiguity. The overlapping extended main text was not re-read as a second complete semantic manuscript. Only its actual appendix interval, not all 13 pages, may contribute to appendix-length measurements; full-file length and captions are reported separately.
Visually inspected PDF pages: 4, 5, 6, 7

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

Visually inspected supplement pages: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13.