Writing study: Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Jeong Woon Lee, Kyoleen Kwak, Daeho Kim, Hyoseok Hwang · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

23 physical pages

Before references / appendix

≈ 9.19 page-equivalents

Main figures / tables

1 / 8

Attached appendix figures / tables

5 / 17

Attached appendix span

≈ 11.91 page-equivalents

Before references / appendix: ≈ 39.96%References / intervening material: ≈ 8.26%Attached appendix: ≈ 51.78%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Wrapped main heading labels restored from the immediately following same-size source line in the physical PDF. Heading start positions are unchanged. Newly detected unnumbered impact/singular-acknowledgment statements are retained from the size-aware source extraction; authored heading corrections are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.28 pp.	5.57%
2. Related Work	2	≈ 0.53 pp.	2.3%
3. Preliminary	3	≈ 0.24 pp.	1.04%
4. Theoretical Analysis: The Geometry of Policy Sensitivity	3	≈ 1.36 pp.	5.91%
5. PAVE: Policy-Aware Value-field Equalization	4	≈ 1.95 pp.	8.48%
6. Empirical Analysis	6	≈ 3.11 pp.	13.52%
7. Limitations	9	≈ 0.07 pp.	0.3%
8. Conclusion	9	≈ 0.16 pp.	0.7%
Acknowledgments	10	≈ 0.1 pp.	0.43%
Impact Statement	10	≈ 0.05 pp.	0.22%
References	10	≈ 1.89 pp.	8.22%
A. Extended Analysis and Protocol for Q-Gradient Field Visualization	12	≈ 0.46 pp.	2%
B. Hyperparameter Sensitivity Analysis	12	≈ 0.18 pp.	0.78%
C. Theoretical Complexity Analysis	12	≈ 0.91 pp.	3.96%
D. Visualization of Learning Curves	13	≈ 0.09 pp.	0.39%
E. Extended Implementation Details	13	≈ 0.76 pp.	3.3%
F. Proxy Validation	14	≈ 0.84 pp.	3.65%
G. Exact Hessian Measurements	15	≈ 0.76 pp.	3.3%
H. Geometry Evolves During Training	16	≈ 0.46 pp.	2%
I. Wall-Clock Convergence Time	16	≈ 0.28 pp.	1.22%
J. Critic-Centric Baseline Comparison	16	≈ 0.74 pp.	3.22%
K. Hyperparameter Sensitivity	17	≈ 0.53 pp.	2.3%
L. Full Factorial Component Ablation	18	≈ 0.4 pp.	1.74%
M. Combination with Actor-Side Regularization	18	≈ 0.35 pp.	1.52%
N. Extension to Humanoid	18	≈ 5.16 pp.	22.43%

All detected figure and table locations

Main text

• Table 1 · p. 6
• Table 2 · p. 7
• Figure 1 · p. 7
• Table 3 · p. 8
• Table 4 · p. 8
• Table 5 · p. 8
• Table 6 · p. 8
• Table 7 · p. 9
• Table 8 · p. 9

Attached appendix

• Table 9 · p. 13
• Table 10 · p. 13
• Table 11 · p. 14
• Table 12 · p. 14
• Table 13 · p. 15
• Table 14 · p. 15
• Table 15 · p. 15
• Table 16 · p. 16
• Table 17 · p. 16
• Table 18 · p. 16
• Table 19 · p. 17
• Table 20 · p. 17
• Table 21 · p. 17
• Table 22 · p. 17
• Table 23 · p. 18
• Table 24 · p. 18
• Table 25 · p. 19
• Figure 2 · p. 20
• Figure 3 · p. 21
• Figure 4 · p. 22
• Figure 5 · p. 22
• Figure 6 · p. 23

Argument and organization

1. Problem and claim

   Reframes jerky actions as a learning-signal problem in the critic rather than only a policy-output problem. The proposed losses are tied to separate numerator, temporal-consistency and curvature roles.

2. Argument sequence

   The implicit-policy sensitivity argument precedes finite-difference regularization and an algorithm. Experiments progress from task scores to geometry, proxy quality, noise robustness and component interactions.

3. Evidence and verification

   The proxy table reports deviations from exact analytic quantities rather than assuming finite differences are exact. The accompanying field table retains the Reacher exception to reduced mixed-partial magnitude.

Problem, claim and contribution

Reframes jerky actions as a learning-signal problem in the critic rather than only a policy-output problem. The proposed losses are tied to separate numerator, temporal-consistency and curvature roles.

Introduction and Sections 4–5, pp. 1–6 · Main paper

Argument and section order

The implicit-policy sensitivity argument precedes finite-difference regularization and an algorithm. Experiments progress from task scores to geometry, proxy quality, noise robustness and component interactions.

Sections 4–6, pp. 3–9 · Main paper

Figures, captions and visual explanation

Figure 1 holds the color scale and baseline-percentile clipping rule constant across five surface plots. Table 2 repeats return and smoothness columns with opposite metric directions, retaining settings where policy-side alternatives are smoother.

Figure 1 and Table 2, physical p. 7; raster inspected · Main paper

Tables, comparisons and evidence

The proxy table reports deviations from exact analytic quantities rather than assuming finite differences are exact. The accompanying field table retains the Reacher exception to reduced mixed-partial magnitude.

Tables 4–6, physical p. 8; raster inspected · Main paper

Appendix and reproducibility

A reproducible visualization protocol and exact-Hessian measurements support the main picture. Full factorial ablation is labeled as different hardware, while the Humanoid extension explains why near-zero smoothness from a failed policy is uninformative.

Appendices A, F–N, pp. 12–19; complete plots pp. 20–23 · Main paper

Clarity and writing conventions

The local-maximum caveat explicitly excludes boundary or non-unique optima from the implicit-function argument. The strong root-cause and proof language in surrounding prose should therefore be read with those assumptions, not as a general causal conclusion.

Remark 4.2, pp. 3–4; component interpretation p. 9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Validate a convenient training proxy against the intended mathematical quantity, and pair smoothness with useful task behavior so degenerate solutions cannot look successful.

Boundary: A local greedy-policy theorem is not a global guarantee for the learned actor, and simulation smoothness is not evidence of real-hardware safety or reduced wear.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-focused scan of Sections 1–8, pp. 1–9: local theoretical assumptions, practical losses, algorithm, paired return/smoothness results and limitations; impact statement p. 10.

Appendix / supplement

Structural and selected-detail scan of Appendices A–N, pp. 12–23: visualization protocol, proxy validation, exact-Hessian measurements, runtime, sensitivity, factorial ablation and extension limits. No independent proof or control-system safety validation.

Visually inspected PDF pages

7, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 23 pages