Writing study: Evaluating Robustness of Reasoning Models on Parameterized Logical Problems

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Naı̈m Es-Sebbani, Esteban Marquer, Yakoub Salhi, Zied Bouraoui · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Datasets, benchmarks & measurement (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

25 physical pages

Before references / appendix

≈ 9.04 page-equivalents

Main figures / tables

9 / 1

Attached appendix figures / tables

1 / 11

Attached appendix span

≈ 12.91 page-equivalents

Before references / appendix: ≈ 36.16%References / intervening material: ≈ 12.2%Attached appendix: ≈ 51.64%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.04 pp.	4.16%
2. Related work	2	≈ 0.75 pp.	3%
3. Problem setting	3	≈ 2.03 pp.	8.12%
4. Dataset Construction	5	≈ 0.54 pp.	2.16%
5. Experiments	5	≈ 3.91 pp.	15.64%
6. Conclusion	9	≈ 0.2 pp.	0.8%
Acknowledgments	9	≈ 0.05 pp.	0.2%
Impact Statement	9	≈ 0.18 pp.	0.72%
References	10	≈ 3.04 pp.	12.16%
A. Background	13	≈ 0.36 pp.	1.44%
B. UNSAT Generator using Implication Cycles	13	≈ 0.23 pp.	0.92%
C. SAT Generator via Free Variables	13	≈ 0.19 pp.	0.76%
D. Backbone-based Method	13	≈ 1.35 pp.	5.4%
E. Monotone Split with a Moving Cross-Polarity Bridge	15	≈ 0.52 pp.	2.08%
F. Symmetry/Redundancy Probe	15	≈ 0.49 pp.	1.96%
G. Verbalization Templates and Prompts	16	≈ 4.47 pp.	17.88%
H. Model Information	20	≈ 0.06 pp.	0.24%
I. Truncation Rate	20	≈ 0.06 pp.	0.24%
J. SAT Prediction Bias on UNSAT Instances	20	≈ 0.48 pp.	1.92%
K. Statistical Significance Results	21	≈ 0.08 pp.	0.32%
L. Estimated Mean Token Count per Verbalizer	21	≈ 0.25 pp.	1%
M. Per-Model Confusion Matrix	21	≈ 0.11 pp.	0.44%
N. Proprietary Reasoning Models	21	≈ 0.09 pp.	0.36%
O. GPT-OSS-120B: Effect of Reasoning Effort	21	≈ 1.7 pp.	6.8%
P. Tool Calling	23	≈ 2.48 pp.	9.92%

All detected figure and table locations

Main text

• Table 1 · p. 6
• Figure 1 · p. 7
• Figure 2 · p. 7
• Figure 3 · p. 7
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 8
• Figure 7 · p. 8
• Figure 8 · p. 9
• Figure 9 · p. 9

Attached appendix

• Figure 10 · p. 16
• Table 2 · p. 16
• Table 3 · p. 20
• Table 4 · p. 20
• Table 5 · p. 21
• Table 6 · p. 22
• Table 7 · p. 23
• Table 8 · p. 23
• Table 9 · p. 24
• Table 10 · p. 24
• Table 11 · p. 25
• Table 12 · p. 25

Argument and organization

1. Problem and contribution

   Uses a polynomial-time-solvable logic problem to separate structural reasoning failures from an absence of efficient search, and explicitly avoids a claim about all logic.

2. Argument sequence

   Each generator names its structural purpose and difficulty parameters before verbalization and evaluation; perturbation analyses separate core difficulty from filler and presentation effects.

3. Evidence and verification

   SAT decisions and valid witnesses are scored separately; truncation/unparsable outputs count as incorrect, with rates disclosed, and within-instance renaming is explicitly not tested.

Problem, claim and contribution

Uses a polynomial-time-solvable logic problem to separate structural reasoning failures from an absence of efficient search, and explicitly avoids a claim about all logic.

Introduction, p. 2 · Main paper

Argument and section order

Each generator names its structural purpose and difficulty parameters before verbalization and evaluation; perturbation analyses separate core difficulty from filler and presentation effects.

Sections 3-5, pp. 3-8 · Main paper

Figures, captions and visual explanation

Generator prose separates construction from controlled parameters. Robustness panels label fixed-core/matched-size conditions, confidence intervals and non-significant comparisons.

Sections 3.3-3.5, p. 4; Figures 5-7, p. 8; both PNGs inspected · Main paper

Tables, comparisons and evidence

SAT decisions and valid witnesses are scored separately; truncation/unparsable outputs count as incorrect, with rates disclosed, and within-instance renaming is explicitly not tested.

Sections 4-5, p. 5; Table 11, p. 25 · Main paper

Appendix and reproducibility

Prompt templates and verbalizers expose output parsing, while reasoning-effort and tool-use tables retain both decision and witness behavior and disclose tool invocation rates.

Appendix G, pp. 16-19; Tables 11-12, p. 25 · Main paper

Clarity and writing conventions

The paper carefully states which invariance is probed and distinguishes trace-based interpretations from solver-certified labels.

Scope of renaming, p. 5; analyses, p. 8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Define the construct, certify labels independently, and separate surface perturbations, output validity and core task difficulty.

Boundary: Sixty prompts derive from ten formulas with six verbalizations, not sixty independent underlying problems; findings remain scoped to these controlled families.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused distributed manuscript scan, not exhaustive technical/proof validation. Sections 1-6, pp. 1-9: controlled 2-SAT motivation, generator families, verbalization, decision/witness evaluation and robustness perturbations.

Appendix / supplement

Writing-focused appendix scan; code, data and external assets not executed. Appendices A-P, pp. 13-25: generator arguments, exact prompts, models, truncation/bias/statistics, confusion matrices, proprietary models, effort and tool tests.

Visually inspected PDF pages

4, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published ICML 2026 proceedings PDF, PMLR volume 306; retrieved 2026-10-07. Public OpenReview decision notes were not independently reverified. · 25 pages