Writing study: SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Gyuhyeon Seo, Jungwoo Yang, Junseong Pyo, Nalim Kim, Jonggeun Lee, Yohan Jo · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

28 physical pages

Before references / appendix

≈ 10.28 page-equivalents

Main figures / tables

5 / 3

Attached appendix figures / tables

1 / 11

Attached appendix span

≈ 14.89 page-equivalents

Before references / appendix: ≈ 36.71%References / intervening material: ≈ 10.11%Attached appendix: ≈ 53.18%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.19 pp.	4.25%
2 RELATED WORK	2	≈ 0.58 pp.	2.07%
3 SIMUHOME: A SMART HOME SIMULATOR	3	≈ 1.05 pp.	3.75%
4 BENCHMARK DESIGN	4	≈ 4.07 pp.	14.54%
5 EXPERIMENTS	8	≈ 2.25 pp.	8.04%
6 CONCLUSION	10	≈ 0.43 pp.	1.54%
ACKNOWLEDGMENTS	11	≈ 0.17 pp.	0.61%
REFERENCES	11	≈ 2.83 pp.	10.11%
A LIST OF TOOLS	14	≈ 1 pp.	3.57%
B LIST OF DEVICE TYPES	15	≈ 0.61 pp.	2.18%
C LIST OF MATTER CLUSTERS	15	≈ 1.69 pp.	6.04%
D COMPLEX ENVIRONMENTAL INTERACTIONS	17	≈ 0.26 pp.	0.93%
E ENVIRONMENTAL UPDATE EQUATIONS	17	≈ 0.77 pp.	2.75%
F INFEASIBLE QUERY TYPES	18	≈ 0.44 pp.	1.57%
G GOAL EXAMPLES	18	≈ 1.72 pp.	6.14%
H LLM JUDGE VALIDATION	20	≈ 0.15 pp.	0.54%
I EXPERIMENTAL SETUP	20	≈ 0.54 pp.	1.93%
J ERROR ANALYSIS DETAILS	21	≈ 1.19 pp.	4.25%
K GPT-4.1 QT2-F PERFORMANCE ANALYSIS	22	≈ 0.15 pp.	0.54%
L SIMULATION-BASED PRE-VALIDATION	22	≈ 0.48 pp.	1.71%
M FRAMEWORK COMPARISON AND SELF-REVIEW	23	≈ 0.5 pp.	1.79%
N MULTI-TURN INTERACTION EXPERIMENTS	23	≈ 0.5 pp.	1.79%
O POST-EXECUTION FAILURE RECOVERY	24	≈ 0.31 pp.	1.11%
P SUPERVISED FINE-TUNING EXPERIMENTS	24	≈ 0.34 pp.	1.21%
Q PROMPTS	24	≈ 3.85 pp.	13.75%
R USE OF LARGE LANGUAGE MODELS	28	≈ 0.39 pp.	1.39%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 5
• Figure 3 · p. 6
• Table 1 · p. 7
• Table 2 · p. 7
• Table 3 · p. 8
• Figure 4 · p. 8
• Figure 5 · p. 9

Attached appendix

• Table 4 · p. 14
• Table 5 · p. 15
• Table 6 · p. 15
• Table 7 · p. 19
• Table 8 · p. 21
• Table 9 · p. 21
• Table 10 · p. 22
• Table 11 · p. 22
• Figure 6 · p. 23
• Table 12 · p. 23
• Table 13 · p. 24
• Table 14 · p. 24

Argument and organization

1. Requirements to simulator and benchmark

   Maps device dependencies, environmental feedback, and scheduling into architecture and twelve evaluation cells.

2. Failure analysis and interventions

   Explains the scheduling gap through feedback differences and tests alternative scaffolds, self-review, oracle feedback, and SFT.

Problem, claim and contribution

Everyday requests make operational dependencies and timing concrete before introducing the simulator. Three design requirements then organize the architecture, linking claimed gaps to implemented capabilities.

Introduction and Section 3, pp. 1–4 · Main paper

Figures, captions and visual explanation

The generation diagram shows randomized state, sampled goal, prerequisite calls, and synthesized query as separate artifacts. Its step numbering matches the prose and reveals that queries are generated from verifiable goals.

Figure 2 and Section 4.2, pp. 5–6 · Main paper

Parallel feasible and infeasible examples expose the evaluation switch: simulator assertions for physical outcomes versus language judging for explanations. A shared timeline clarifies delayed execution.

Figure 3, p. 6 · Main paper

Tables, comparisons and evidence

The main table labels feasible/infeasible columns and simulator/judge superscripts. The error-recovery chart restricts its denominator to successful episodes, showing that good terminal scores can conceal initially invalid calls.

Table 1, p. 7; Figure 5, p. 9 · Main paper

Appendix and reproducibility

The supplement provides environmental equations and explicitly calls them simplified, while separate tables enumerate tools and Matter application-layer semantics. It also publishes failure counts and representative judge/agent prompts.

Appendices A–E, pp. 14–18; J and Q, pp. 21–28 · Main paper

Clarity and writing conventions

Oracle-assisted recovery is labeled a topline estimate rather than an available deployment capability. Similarly explicit wording should qualify the stronger claim that limited SFT failures demonstrate imitation cannot solve scheduling.

Section 5.4, p. 10; Appendices O–P, p. 24 · Main paper

What to reuse—and what not to infer

Transferable lesson: Show how data generation, task prerequisites, and scoring connect, then analyze successful trajectories as well as failures to identify hidden dependence on feedback.

Boundary: This is a simplified deterministic API simulator, not validated household physics or full Matter protocol compliance. Final-state checks plus required calls constrain success. The 204-trajectory SFT and one alternative framework cannot establish an intrinsic impossibility for imitation or orchestration.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–6

Appendix / supplement

Pages 14–28, appendices A–R including tools, update equations, benchmark examples, judging, error analysis, control experiments, SFT, and prompts

Visually inspected PDF pages

5, 6, 9

• ICLR 2026 final proceedings · 28 pages