Writing study: CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Yahan Li, Jifan Yao, John Bunyi, Adam Frank, Angel Hwang, Ruishan Liu · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement resources (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

37 physical pages

Before references / appendix

≈ 10.47 page-equivalents

Main figures / tables

5 / 4

Attached appendix figures / tables

3 / 14

Attached appendix span

≈ 18.26 page-equivalents

Before references / appendix: ≈ 28.3%References / intervening material: ≈ 22.35%Attached appendix: ≈ 49.35%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 0.92 pp.	2.49%
2 RELATED WORK	2	≈ 0.88 pp.	2.38%
3 OVERVIEW OF COUNSELBENCH	3	≈ 0.52 pp.	1.41%
4 COUNSELBENCH-EVAL: EXPERT EVALUATION OF LLMS FOR	4	≈ 4.54 pp.	12.27%
5 COUNSELBENCH-ADV: AN ADVERSARIAL BENCHMARK FOR SURFACING	8	≈ 1.53 pp.	4.14%
6 DISCUSSION	10	≈ 0.88 pp.	2.38%
ETHICS STATEMENT	11	≈ 0.14 pp.	0.38%
ACKNOWLEDGMENTS	11	≈ 0.14 pp.	0.38%
REPRODUCIBILITY STATEMENT	11	≈ 0.1 pp.	0.27%
REFERENCES	11	≈ 8.27 pp.	22.35%
A SURVEY DESIGN	19	≈ 0.66 pp.	1.78%
B ANNOTATOR DEMOGRAPHIC OVERVIEW	20	≈ 2.91 pp.	7.86%
C ANNOTATION ANALYSIS	23	≈ 0.79 pp.	2.14%
D EXPERIMENT DETAILS	24	≈ 5.09 pp.	13.76%
E QUESTION & RESPONSE SELECTION	29	≈ 3.23 pp.	8.73%
F SIX IDENTIFIED FAILURE MODES	32	≈ 0.1 pp.	0.27%
G ADDITIONAL RESULTS FOR LLM-AS-JUDGE ON COUNSELBENCH-EVAL	32	≈ 0.13 pp.	0.35%
H SUPPLEMENTARY EVALUATION OF COUNSELBENCH-ADV	32	≈ 3.64 pp.	9.84%
I PILOT STUDY OF DOMAIN-SPECIFIC MODELS ON COUNSELBENCH-ADV	36	≈ 1.71 pp.	4.62%

All detected figure and table locations

Main text

• Figure 1 · p. 3
• Figure 2 · p. 5
• Table 1 · p. 6
• Table 2 · p. 6
• Figure 3 · p. 8
• Table 3 · p. 9
• Table 4 · p. 9
• Figure 4 · p. 19
• Figure 5 · p. 19

Attached appendix

• Figure 6 · p. 20
• Table 6 · p. 20
• Table 5 · p. 22
• Table 7 · p. 23
• Table 8 · p. 26
• Table 9 · p. 26
• Figure 7 · p. 26
• Table 10 · p. 27
• Table 11 · p. 27
• Table 12 · p. 28
• Table 13 · p. 29
• Table 14 · p. 33
• Table 15 · p. 34
• Table 16 · p. 34
• Figure 8 · p. 35
• Table 17 · p. 35
• Table 18 · p. 36

Argument and organization

1. Clinician-grounded evaluation gap

   Motivates open-ended evaluation beyond factual multiple-choice QA.

2. Evaluation corpus

   Documents source selection, generation, rubric, annotators and human–model judge comparisons.

3. Adversarial extension

   Turns observed failures into clinician-authored targeted probes.

Problem, claim and contribution

The paper clearly states a research-only boundary and explains why response quality, personalization and safety require distinct evaluation axes rather than a single accuracy score.

Disclaimer, page 1; Sections 1 and 4.3. · Main paper

Argument and section order

Failure analysis creates the transition between the two benchmark components, so the adversarial collection follows from observed problems rather than appearing as an unrelated dataset.

Sections 4.5–5, pages 7–9. · Main paper

Figures, captions and visual explanation

The overview separates the observational and adversarial datasets by color and direction. Survey screenshots later show the actual rating and evidence-extraction interface rather than merely asserting a rubric exists.

Figure 1, page 3; Figures 4–5, page 19. · Main paper

Tables, comparisons and evidence

The six-panel evaluator plot makes human–LLM disagreement immediately visible, using a red expert line against blue model lines. Captions explicitly identify dimensions where lower values are preferred.

Figure 3, page 8. · Main paper

Appendix and reproducibility

Credential-verification routes, abstention handling, full prompts, word-limit rejection sampling and topic exclusions are documented. Those exclusions matter for interpreting representativeness.

Appendices B, D, E and H, pages 20–35. · Main paper

Clarity and writing conventions

Evaluation counts require careful wording: 2,000 ratings correspond to 400 source responses, whereas the adversarial setup assigns disjoint batches to five practitioners. Appendix C’s reference to 460 QA pairs is inconsistent with the stated 400-response corpus and needs clarification.

Section 4.4, page 6; Appendix C.1, page 23; Appendix H, page 32. · Main paper

What to reuse—and what not to infer

Transferable lesson: Build a benchmark narrative from an explicit rubric to annotated failures and then targeted stress tests, publishing the actual annotation interface and selection rules.

Boundary: Single-turn, selected English forum responses are not clinical treatment outcomes. Expert-defined flags and limited annotator/model samples do not establish universal harm rates, superiority to professional care, or reliable deployment safety.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–11, Sections 1–6 and ethics/reproducibility; references through page 19.

Appendix / supplement

Pages 19–37, Appendices A–I, all survey, demographic, prompt, selection, annotation and supplementary evaluation material.

Visually inspected PDF pages

3, 8, 19

• ICLR 2026 final proceedings · 37 pages