Writing study: WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Chunyang Li, Yilun Zheng, Xinting Huang, Tianqing Fang, Jiahao Xu, Lihui Chen, Yangqiu Song, Winston Hu · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

33 physical pages

Before references / appendix

≈ 10.16 page-equivalents

Main figures / tables

4 / 7

Attached appendix figures / tables

7 / 7

Attached appendix span

≈ 17.89 page-equivalents

Before references / appendix: ≈ 30.79%References / intervening material: ≈ 15%Attached appendix: ≈ 54.21%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.23 pp.	3.73%
2 RELATED WORK	2	≈ 1.25 pp.	3.79%
3 WEBDEVJUDGE BENCHMARK	4	≈ 1.54 pp.	4.67%
4 EXPERIMENTS	5	≈ 4.81 pp.	14.58%
5 CONCLUSION	10	≈ 0.17 pp.	0.52%
ACKNOWLEDGMENTS	10	≈ 0.09 pp.	0.27%
ETHICS STATEMENT	10	≈ 0.12 pp.	0.36%
REPRODUCIBILITY STATEMENT	10	≈ 0.32 pp.	0.97%
REFERENCES	11	≈ 4.95 pp.	15%
A LIMITATIONS OF WEBDEVJUDGE	16	≈ 0.18 pp.	0.55%
B DETAILS IN WEBDEVJUDGE CONSTRUCTION	16	≈ 4.87 pp.	14.76%
C EXPERIMENTAL DETAILS	21	≈ 4.95 pp.	15%
D WEBDEVJUDGE-UNIT DATASET	26	≈ 1 pp.	3.03%
E ADDITIONAL RESULTS	27	≈ 4 pp.	12.12%
F CASE STUDY	31	≈ 2.06 pp.	6.24%
G THE USE OF LARGE LANGUAGE MODELS	33	≈ 0.83 pp.	2.52%

All detected figure and table locations

Main text

• Figure 1 · p. 3
• Table 1 · p. 4
• Table 2 · p. 5
• Figure 2 · p. 5
• Table 3 · p. 6
• Figure 3 · p. 7
• Table 4 · p. 8
• Table 5 · p. 8
• Figure 4 · p. 9
• Table 6 · p. 9
• Table 7 · p. 10

Attached appendix

• Table 8 · p. 16
• Table 9 · p. 19
• Table 10 · p. 19
• Table 11 · p. 23
• Table 12 · p. 26
• Table 13 · p. 27
• Figure 5 · p. 28
• Figure 6 · p. 29
• Figure 7 · p. 29
• Figure 8 · p. 30
• Table 14 · p. 30
• Figure 9 · p. 31
• Figure 10 · p. 32
• Figure 11 · p. 32

Argument and organization

1. Interactive meta-evaluation gap

   Separate judging generated websites from generating websites.

2. Rubric-grounded preference construction

   Explain the human reference before benchmarking automated judges.

3. Comparisons followed by error diagnosis

   Turn aggregate disagreement into functional-equivalence and feasibility cases.

Problem, claim and contribution

The benchmark's target is agreement with expert preferences, not an objective proof that a website works. The paper motivates dynamic interaction as the missing setting and defines an explicit query, two implementations and three-way preference tuple.

§§1,3, pp.1–4 · Main paper

Argument and section order

Data filtering and annotation come before model results. The count trail from 10,501 raw examples to 1,713 deployable instances, 700 sampled and 654 final pairs makes the benchmark's selection process visible rather than hiding it behind a single total.

§§3.1–3.3, pp.4–5; Table 8, p.16 · Main paper

Figures, captions and visual explanation

Figure 1 divides collection, rubric annotation and judge evaluation into three blocks. Figure 10 shows the actual grading interface beside conflicting judgments, demonstrating how an overly specific generated rubric can confuse functional equivalence.

Figure 1, p.3; Figure 10, p.32 · Main paper

Tables, comparisons and evidence

The main table reports 70.34% pairwise agreement for GPT-4.1 versus the 84.56% human reference. The diagnostic unit set adds precision and recall, showing that code-only false positives differ from GUI-agent operational false negatives.

Tables 3 and 6, pp.6,9 · Main paper

Appendix and reproducibility

The appendix retains exact rubric prompts, dimension weights, tie thresholds and agent step budgets. Its no-tie and swapped-order analyses expose choices that materially change scores; the small author-annotator pool also limits universal human-preference claims.

Appendices A–C, pp.16–25; Tables 13–14, pp.27,30 · Main paper

Clarity and writing conventions

The ensemble uses Boolean OR for dynamic checks. Calling this high-precision filtering is misleading: an OR accepts either model's positive and need not preserve precision. The measured agreement improvement is evidence for this protocol, not a general calibration guarantee.

Eq.2 and Table 7, p.10 · Main paper

What to reuse—and what not to infer

Transferable lesson: For evaluation papers, make reference-label construction auditable and pair aggregate scores with concrete failures that distinguish system errors from evaluator errors.

Boundary: Agreement is rubric- and sample-dependent; neither high inter-annotator agreement nor model consistency proves universal correctness. Static screenshots cannot establish interactive feasibility, and API-model results describe tested versions only.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–5, ethics and reproducibility; references 11–15

Appendix / supplement

Pages 16–33, Appendices A–G: limitations, filtering, annotation, prompts, unit benchmark, additional results, cases and LLM disclosure

Visually inspected PDF pages

3, 32

• official ICLR 2026 proceedings PDF · 33 pages