Writing study: OpenApps: Simulating Environment Variations to Measure UI Agent Reliability

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Karen Ullrich, Jingtong Su, Claudia Shi, Arjun Subramonian, Amir Bar, Ivan Evtimov, Nikolaos Tsilivis, Randall Balestriero, Julia Kempe, Mark Ibrahim · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary) · Empirical analysis, diagnosis & auditing.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

34 physical pages

Before references / appendix

≈ 10.11 page-equivalents

Main figures / tables

6 / 0

Attached appendix figures / tables

4 / 13

Attached appendix span

≈ 17.89 page-equivalents

Before references / appendix: ≈ 29.74%References / intervening material: ≈ 17.65%Attached appendix: ≈ 52.62%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	2	≈ 0.69 pp.	2.03%
2 RELATED WORK	2	≈ 0.98 pp.	2.88%
3 OPENAPPS: FRAMEWORK AND ENVIRONMENT	3	≈ 2.69 pp.	7.91%
4 MULTIMODAL AGENT RELIABILITY WITH OPENAPPS	6	≈ 3.83 pp.	11.26%
5 CONCLUSION	10	≈ 0.47 pp.	1.38%
6 REPRODUCIBILITY STATEMENT	10	≈ 0.09 pp.	0.26%
ACKNOWLEDGMENTS	10	≈ 0.25 pp.	0.74%
REFERENCES	11	≈ 6 pp.	17.65%
A STATEMENT ABOUT USAGE OF LLMS	17	≈ 0.09 pp.	0.26%
B FUTURE DIRECTIONS	17	≈ 0.6 pp.	1.76%
C ADDITIONAL RELATED WORK	17	≈ 1.34 pp.	3.94%
D METHOD	19	≈ 4.45 pp.	13.09%
E ADDITIONAL EXPERIMENTAL RESULTS	23	≈ 6.17 pp.	18.15%
F AGENT BEHAVIORS	29	≈ 5.24 pp.	15.41%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 4
• Figure 3 · p. 5
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 8

Attached appendix

• Figure 7 · p. 20
• Table 1 · p. 21
• Figure 8 · p. 22
• Table 2 · p. 23
• Table 3 · p. 24
• Figure 9 · p. 25
• Figure 10 · p. 25
• Table 4 · p. 26
• Table 5 · p. 27
• Table 6 · p. 28
• Table 7 · p. 28
• Table 8 · p. 30
• Table 9 · p. 30
• Table 10 · p. 31
• Table 11 · p. 32
• Table 12 · p. 33
• Table 13 · p. 34

Argument and organization

1. A missing axis of reliability

   Separates randomness within one application from changes in application appearance and content.

2. Controllable simulator to behavioral diagnostics

   Defines state, observation, actions, and reward before using interventions to analyze success and failure modes.

Problem, claim and contribution

A concrete deployment mismatch motivates the benchmark: cloned apps hold design fixed while real interfaces vary. The contribution is a controllable simulator rather than simply a harder task collection.

Sections 1–3 · Main paper

Argument and section order

Success-rate variation, behavior changes, and screen-resolution interactions are treated as different questions; the failure analysis includes concrete invalid actions rather than only aggregate scores.

Sections 4.1–4.3; Appendix F · Main paper

Figures, captions and visual explanation

The agent–environment diagram maps screenshots, accessibility trees, actions, and full-state rewards to familiar reinforcement-learning notation, making the evaluation boundary clear.

Figure 3, p. 5 · Main paper

Tables, comparisons and evidence

The headline plot reports minimum, median, and maximum success across variants, with a caption disclosing removal of tasks that fail in every variant. This filter must travel with any reused statistic.

Figure 4, p. 7 · Main paper

Appendix and reproducibility

Editable YAML, task prompts, action subsets, per-task tables, and additional combinations make the claimed controllability tangible. Complex-task partial rewards are distinguished from full completion.

Appendices D–F; Tables 8–10 · Main paper

Clarity and writing conventions

The limitations acknowledge simple tasks and unresolved sim-to-real transfer. Some broad wording is stronger than the evidence: full-state checks reduce particular reward loopholes but do not prove reward hacking impossible.

Section 3.3; limitations and Appendix B · Main paper

What to reuse—and what not to infer

Transferable lesson: A benchmark paper can be organized around a controlled intervention axis, with an explicit observation/action/reward contract and captions that carry filtering and aggregation details.

Boundary: The simulator's light CPU footprint excludes agent inference and browser overhead. Low variance is not success—an always-failing policy is stable. Do not combine the filtered Figure 4 percentages with unfiltered tables, or interpret navigation-based intent proxies as direct measurements of understanding.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

PDF pp. 1–10, Sections 1–6, complete method, reliability analysis, behavior cases, and limitations.

Appendix / supplement

PDF pp. 17–34, Appendices A–F, including YAML, task/action definitions, all result tables, complex tasks, and failure taxonomy.

Visually inspected PDF pages

5, 7, 22, 27

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 34 pages