Writing study: Offloading Score: Measuring AI Reliance through Counterfactual Workflows

NeurIPS overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← NeurIPS 2026 oral writing corpus

Vishakh Padmakumar, Lujain Ibrahim, Zora Wang, Jennifer Wang, Q.Vera Liao, Diyi Yang · Official NeurIPS 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

37 physical pages

Before references / appendix

≈ 9.47 page-equivalents

Main figures / tables

6 / 0

Attached appendix figures / tables

15 / 1

Attached appendix span

≈ 23.91 page-equivalents

Before references / appendix: ≈ 25.59%References / intervening material: ≈ 9.78%Attached appendix: ≈ 64.62%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 Introduction	1	≈ 1.76 pp.	4.76%
2 Background	3	≈ 1.21 pp.	3.27%
3 Process-Oriented Measurement of Reliance	4	≈ 1.78 pp.	4.81%
4 Validating Our Measure of Reliance	6	≈ 2.19 pp.	5.92%
5 Using Offloading Score to Identify Conditions of (In)appropriate Reliance	8	≈ 1.47 pp.	3.97%
6 Conclusion	10	≈ 0.24 pp.	0.65%
Acknowledgments	10	≈ 0.14 pp.	0.38%
References	10	≈ 3.63 pp.	9.81%
A Limitations	14	≈ 0.31 pp.	0.84%
B Workflow Induction Process	14	≈ 2.72 pp.	7.35%
C Implementation Details For Section 3	17	≈ 5.63 pp.	15.22%
D Metric Validity.	22	≈ 3.88 pp.	10.49%
E User Study Details	26	≈ 7.01 pp.	18.95%
F Additional Results	33	≈ 4.36 pp.	11.78%

All detected figure and table locations

Main text

• Figure 1 · p. 2
• Figure 2 · p. 4
• Figure 3 · p. 4
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 9

Attached appendix

• Figure 7 · p. 15
• Figure 8 · p. 15
• Figure 9 · p. 15
• Figure 10 · p. 24
• Figure 11 · p. 24
• Figure 12 · p. 25
• Figure 13 · p. 25
• Figure 14 · p. 26
• Figure 15 · p. 26
• Figure 16 · p. 26
• Table 1 · p. 27
• Figure 17 · p. 33
• Figure 18 · p. 33
• Figure 19 · p. 34
• Figure 20 · p. 35
• Figure 21 · p. 35

Argument and organization

1. Problem and contribution

   The framing separates adopting an AI output from transferring cognitive work to AI. A counterfactual workflow supplies the proposed measurement target, and the user study tests a specific expected effect of time pressure rather than claiming access to unobservable mental effort.

2. Argument sequence

   The metric definition and intrinsic validation come before the controlled user study. Descriptive process labels and output-use labels complement the scalar score, then a separate section explores inappropriate reliance without defining all high reliance as misuse.

3. Evidence and verification

   A row of condition-paired boxplots compares the proposed measure with four baselines and prints each test's p-value. Adjacent text states the four-task, two-condition, 40-participant design. The different vertical scales are labeled, so distribution changes should be read within each metric rather than across panel heights.

Problem, claim and contribution

The framing separates adopting an AI output from transferring cognitive work to AI. A counterfactual workflow supplies the proposed measurement target, and the user study tests a specific expected effect of time pressure rather than claiming access to unobservable mental effort.

Introduction and Section 3, pp. 1–5 · Main paper

Argument and section order

The metric definition and intrinsic validation come before the controlled user study. Descriptive process labels and output-use labels complement the scalar score, then a separate section explores inappropriate reliance without defining all high reliance as misuse.

Sections 3–5, pp. 4–9 · Main paper

Figures, captions and visual explanation

Two complementary diagrams distinguish what is measured from how it is computed. A three-panel conceptual view covers counterfactual steps, offloaded processes and output handling; the pipeline then traces recorded UI actions through induced workflows and simulated human-only alternatives to the score.

Figures 2–3, p. 4; rendered page inspected · Main paper

Tables, comparisons and evidence

A row of condition-paired boxplots compares the proposed measure with four baselines and prints each test's p-value. Adjacent text states the four-task, two-condition, 40-participant design. The different vertical scales are labeled, so distribution changes should be read within each metric rather than across panel heights.

Figure 4 and Sections 4.2–4.5, p. 7; rendered page inspected · Main paper

Appendix and reproducibility

The appendix supplies the workflow-induction process, annotation rubrics, counterfactual prompts, human-validation details, recruitment and survey instruments. Its limitations explicitly cover step granularity, imperfect counterfactuals and the use of a general rather than personalized counterfactual user.

Appendices A–E, pp. 14–32 · Main paper

Clarity and writing conventions

The strongest reusable pattern is the separation of operational definition, validation and exploratory interpretation. A lower p-value for one measure is not by itself a statistical test that it is better than every baseline, and the exploratory recall clusters should not be written as a causal diagnosis of overreliance.

Sections 4.5–5 and Appendix A, pp. 7–9 and 14 · Main paper

What to reuse—and what not to infer

Transferable lesson: For a human-AI metric, visualize the measurement pipeline and validate the construct separately from the behavioral conclusions drawn with it.

Boundary: The score is a workflow-step proxy, not direct measurement of cognitive effort. Evidence comes from 40 developers, four tasks and one time-pressure manipulation; counterfactual quality and population generalization remain limitations.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-02

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented distributed manuscript scan of Sections 1–6, physical pp. 1–10: section sequence, substantive prose windows, captions, comparison contracts and conclusions. Not exhaustive technical verification.

Appendix / supplement

Writing-oriented distributed scan of Appendices A–F, physical pp. 14–37, including limitations, rubrics, validation, recruitment, surveys and qualitative workflow examples. Not independent proof checking; code and experiments were not executed.

Visually inspected PDF pages

4, 7

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Public author arXiv manuscript 2605.29392v1; not verified identical to the accepted or camera-ready OpenReview revision · 37 pages