Writing study: Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Zhibin Duan, Guowei Rong, Zhuo Li, Bo Chen, Mingyuan Zhou, Dandan Guo · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Training, objectives & optimization (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

22 physical pages

Before references / appendix

≈ 9.04 page-equivalents

Main figures / tables

6 / 5

Attached appendix figures / tables

4 / 6

Attached appendix span

≈ 8.96 page-equivalents

Before references / appendix: ≈ 41.09%References / intervening material: ≈ 18.18%Attached appendix: ≈ 40.73%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: Join wrapped printed heading text using adjacent source lines; original heading start coordinates are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.02 pp.	4.64%
2. Related Work	2	≈ 0.51 pp.	2.32%
3. Preliminaries	3	≈ 0.68 pp.	3.09%
4. The Bayesian Non-negative Reward Model	3	≈ 1.86 pp.	8.45%
5. Experiment	5	≈ 3.13 pp.	14.23%
6. Conclusion	8	≈ 0.85 pp.	3.86%
Acknowledgements	9	≈ 0.11 pp.	0.5%
Impact Statement	9	≈ 0.24 pp.	1.09%
References	10	≈ 4 pp.	18.18%
A. Method Analysis	14	≈ 3.7 pp.	16.82%
B. Length and Formatting Debiasing and Interpretability	17	≈ 0.91 pp.	4.14%
C. Bayesian Non-negative RM Training Details	18	≈ 1.13 pp.	5.14%
D. Best-of-N (BoN) Test	19	≈ 3.22 pp.	14.64%

All detected figure and table locations

Main text

• Figure 1 · p. 3
• Figure 2 · p. 3
• Figure 3 · p. 5
• Table 1 · p. 6
• Table 2 · p. 6
• Table 3 · p. 7
• Figure 4 · p. 7
• Table 4 · p. 8
• Table 5 · p. 8
• Figure 5 · p. 9
• Figure 6 · p. 9

Attached appendix

• Table 6 · p. 15
• Table 7 · p. 16
• Figure 7 · p. 17
• Figure 8 · p. 17
• Table 8 · p. 18
• Figure 9 · p. 18
• Table 9 · p. 19
• Table 10 · p. 20
• Figure 10 · p. 21
• Table 11 · p. 22

Argument and organization

1. 1 Introduction; 2 Related work; 3 Preliminaries

   Connects noisy preference labels and surface-form bias to the reward-hacking problem.

2. 4 Bayesian Non-negative Reward Modeling

   Introduces the generative reward model, approximate inference and an explanatory factor interpretation.

3. 5 Experiments; 6 Conclusion; Limitations

   Separates reward quality, policy outcomes and debiasing analyses before narrowing the tested setting.

Problem, claim and contribution

The motivation distinguishes annotation noise from length/style bias, allowing the proposed local factors and global weights to answer different parts of the problem.

Abstract and Section 1, p.1 · Main paper

Argument and section order

Generative assumptions precede the variational implementation and a separate intuition subsection, so the model definition is not conflated with the inference network.

Sections 4.2-4.4, pp.4-5 · Main paper

Figures, captions and visual explanation

The main schematic contrasts scalar reward with factorized reward, while the later diagnostic page separates reward-length distributions, binned means and a dual-axis latent-factor display.

Figures 1-2, p.3; Figures 5-6, p.9 · Main paper

Tables, comparisons and evidence

Policy benchmarks show both positive and adverse deltas; a separate human/judge comparison states win, tie and loss from the proposed model’s perspective rather than substituting reward-model accuracy for policy evaluation.

Table 3, p.7; Tables 4-5, p.8 · Main paper

Appendix and reproducibility

The appendix places complexity and model analysis separately from training and label-noise construction, making the reproducibility protocol distinguishable from mathematical interpretation.

Appendix A, p.14; Appendix C, p.18 · Main paper

Clarity and writing conventions

The limitations name sensitivity to the KL coefficient and the lack of demonstrated multi-turn, tool-use and long-horizon transfer; the impact discussion separately addresses optimizing harmful rewards.

Limitations and Impact Statement, p.9 · Main paper

What to reuse—and what not to infer

Transferable lesson: Distinguish model assumptions, approximate implementation, reward diagnostics and downstream policy evidence in both the prose and the display sequence.

Boundary: Reduced length correlation and selected factor examples are not independent proof of causal debiasing or universal interpretability; the broader interactive settings remain untested.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 5, 7, 8, 9. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

Appendix organization and main-text pointers were checked; focused text inspection of physical pages 14, 18. Remaining appendix proofs and details were not exhaustively read. No separately linked supplement inspected.

Visually inspected PDF pages

3, 9

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 22 pages