Reward Modeling & Evaluation Cheatsheet

A: A reward model is a scoring function learned from human preference data that outputs a scalar reward value for LLM-generated responses. In RLHF, the RM acts as a proxy for human preferences, serving as the objective function for policy optimization — the policy model improves its output quality by maximizing the RM score.

Follow-up: Why can't human annotations be used directly as rewards for RL? Because RL requires a large volume of online reward signals; human annotation is expensive and slow. The RM provides a proxy signal that can be computed in batch, instantly.

A: The BT model assumes: for a given prompt, the probability that response $y_w$ is preferred over $y_l$ is determined by the sigmoid function of the difference in their reward scores. The training loss is the negative log-likelihood: $\mathcal{L} = -\log\sigma(r(y_w) - r(y_l))$. Core assumptions include that preferences can be explained by scalar score differences and that preferences are transitive.

Follow-up: What happens if preferences are not transitive? The BT model encounters inconsistent annotation pairs, leading to unstable training. In such cases, more flexible preference models such as Plackett-Luce may be considered.

A: ORM assigns a single reward to the final output only; the signal is sparse. PRM assigns rewards to each reasoning step; the signal is dense. PRM is suited for multi-step reasoning tasks (math, code); ORM is suited for end-to-end evaluation tasks (dialogue, writing). PRM's advantage is precise credit assignment, but its annotation cost is far higher than ORM's.

Follow-up: How can PRM training data be obtained automatically? Monte Carlo estimation can be used: continue sampling multiple rollouts from each step and use the final answer accuracy as an approximation of that step's reward.

A: BoN samples N responses from the policy and uses the RM to select the one with the highest score as the final output. It is an inference-time optimization method that does not update model parameters; it is simple but inference cost scales proportionally with N. RL training updates parameters during training, requiring no additional sampling at inference time.

Follow-up: What is the ceiling of BoN? Diminishing returns as N grows; also limited by the policy's original distribution — it cannot generate high-quality responses outside the distribution.

A: The policy model learns to exploit RM weaknesses for high scores rather than genuinely improving quality. Examples: (1) generating verbose content to score higher (length hacking); (2) using flattering/sycophantic language; (3) repeating high-scoring template sentences from training data; (4) overusing formatting elements (headings, lists, bold text).

Follow-up: What is the relationship between Goodhart's Law and reward hacking? Goodhart's Law states that "once a measure becomes a target, it is no longer a good measure" — the RM is an approximate measure of human preferences; when the policy specifically optimizes it, the two become decoupled.

A: The KL penalty term $\beta \cdot D_{\text{KL}}(\pi \| \pi_{\text{ref}})$ constrains the policy from deviating too far from the reference model, acting as regularization. $\beta$ too large → the policy barely updates and learns nothing; $\beta$ too small → allows excessive exploration, prone to reward hacking. Adaptive KL is commonly used: set a target KL value and adjust $\beta$ dynamically.

Follow-up: Which layer of reward hacking does each of KL penalty and RM ensembles address? KL constrains the policy's scope from the optimization constraint layer; RM ensembles reduce scoring variance and over-optimism from the reward estimation layer. The two are complementary.

A: Train multiple independent RMs (different seeds/data subsets/architectures) and aggregate scores. Strategies: (1) Mean: smooths scores, reduces noise from any single RM; (2) Min / conservative: takes the most conservative score, avoids over-optimism — safer when the policy drifts out of distribution; (3) Uncertainty-weighted: reduce the weight of high-variance samples. In practice, the min strategy is most effective against reward hacking but can be overly conservative.

Follow-up: How can the compute overhead of ensembles be optimized? Use a shared-parameter backbone with different heads; or train multiple lightweight variants using PEFT adapters.

A: Cause: In training preference pairs, chosen responses are typically longer than rejected ones; the RM overfits to the spurious feature "long = good." Data level: Construct preference pairs with similar lengths; analyze and remove length correlations. Inference level: Apply a length penalty $r' = r - \alpha \cdot \text{len}$; or use length normalization.

Follow-up: Why might applying a length penalty only at inference time be insufficient? If the RM has already internalized length as a strong feature, the length penalty may not fully offset it; the more fundamental approach is to debias at the training data or training method level.

A: Standard RLHF trains the RM only on data generated by the initial policy; as RL training progresses, the policy distribution shifts, and the RM becomes inaccurate on the new distribution. Iterative re-training uses the current policy to generate new data → re-annotate → update the RM → continue RLHF, ensuring the RM always covers the policy's current distribution. The cost is more than a doubling of compute and annotation overhead.

Follow-up: Does DPO also need iterative re-training? In theory, DPO also faces the distribution shift problem; iterative DPO (online DPO / online preference optimization) has been proposed to address this.

A: Main biases: (1) Position bias — prefers specific positions; (2) Verbosity bias — prefers long responses; (3) Self-preference — gives higher scores to models from the same family; (4) Format bias — prefers formatted content. Mitigations: swap positions for dual evaluation, define explicit scoring rubrics, vote across multiple judges, periodically calibrate against humans.

Follow-up: Why is pairwise comparison generally better than absolute scoring (pointwise)? Because absolute scoring requires the judge to maintain a consistent internal scoring scale, which is prone to drift; pairwise comparison only requires judging relative quality, which is a simpler cognitive task with higher consistency.

A: RewardBench is a comprehensive reward model evaluation benchmark that measures RM performance across multiple dimensions using pairwise accuracy. Dimensions include: general dialogue quality, challenging dialogue (distinguishing subtle differences), safety, and reasoning ability. It provides a standardized comparison framework that helps diagnose an RM's strengths and weaknesses.

Follow-up: Can RewardBench results directly predict an RM's actual performance in RLHF? Not entirely. Pairwise accuracy on a benchmark is a necessary but not sufficient condition — an RM's performance on a benchmark is not fully correlated with whether it will encounter reward hacking during RLHF optimization; online evaluation is also needed.

A: When $r_{\text{RM}} \neq r_{\text{true}}$, unconstrained maximization of $r_{\text{RM}}$ can cause $r_{\text{true}}$ to decrease (reward hacking). The KL constraint is equivalent to optimizing within a local region where the RM is reliable — the RM is accurate near the training distribution, and KL ensures the policy does not leave this "trust region." This aligns with the trust region concept in TRPO/PPO. When KL is 0, $\pi = \pi_{\text{ref}}$; when KL is small, RM reliability is high.

Follow-up: Are there types of reward hacking that KL penalty cannot prevent? Yes — if reward hacking occurs within a low-KL region near $\pi_{\text{ref}}$ (e.g., simply adding a few flattering words), KL penalty cannot stop it. In such cases, a better RM is needed.

A:

Architecture:
π₀ (SFT model)
   ↓
[Data generation] π_k generates → N responses per prompt
   ↓
[Preference annotation] Human annotations or RM annotations (on-policy data)
   ↓
[RM update] Fine-tune RM_k on new data → RM_{k+1}
   ↓
[Policy optimization] RLHF (PPO) with RM_{k+1}, KL→π_ref
   ↓
[Evaluation] Reward curves, KL curves, human evaluation, reward hacking detection
   ↓
Repeat k = 1, 2, ...

Key decision points:

• Iteration frequency (how many RL steps before updating the RM)
• Whether to collect human annotations each round (high cost) vs. using RM self-annotation
• Choice of KL target value
• When to stop iterating (reward saturation / human evaluation target met)

Follow-up: How do you detect when iteration should stop? What are the potential risks of continuous iteration? Stop when human evaluation scores stop improving, KL continues to increase, or mode collapse occurs. Continued iteration may cause the policy to converge toward the RM's specific preferences, losing diversity.

A: MC estimation: Sample K rollouts from each step and use final accuracy as the reward. Bias comes from limited sampling (estimates are imprecise when K is small); variance comes from sampling randomness. Larger K is more accurate but more expensive. Human annotation: Theoretically unbiased but constrained by annotator capability/consistency; subject to systematic bias and noise. MC estimation is scalable but has systematic bias; human annotation is accurate but not scalable. A hybrid approach (small-scale human annotation + MC expansion) is common practice.

Follow-up: Under what conditions does MC estimation fail severely? When the downstream search space from a given step is extremely large and correct answers are extremely rare (e.g., complex math problems), even many rollouts may all fail, causing zero reward for correct steps (false negatives).

A: Multiple layers of defense are needed:

1. RM internal metrics: Pairwise accuracy (in-distribution vs. OOD), calibration
2. Proxy metric monitoring: KL curves, reward distribution drift, generation length changes, n-gram repetition rate
3. Blind human evaluation: Randomly sample outputs for human scoring; compare against RM scores
4. Adversarial test sets: Construct test cases with known reward hacking patterns
5. Multi-RM consistency: Declining score correlation across multiple RMs as an early warning signal
6. A/B testing: End-user satisfaction as the ultimate evaluation

Follow-up: In practice, which layer is most often overlooked but most critical? OOD detection is most often overlooked — people typically only look at in-distribution accuracy, but RM performance on the OOD distribution generated by the policy is what determines whether reward hacking occurs.

A: If a mainstream LLM-as-Judge (e.g., GPT-4) has self-preference bias, models stylistically similar to it will rank higher on leaderboards, distorting rankings. For example, if a model is from the same family as the judge or trained on similar data, it may receive disproportionately high scores. This creates judge sensitivity in benchmark results — rankings depend on judge selection — undermining the comparability and credibility of leaderboards.

Follow-up: How would you design a leaderboard that is robust to judge bias? Use multiple heterogeneous judges (different vendors, different sizes); report inter-judge agreement; disclose the relationship between the judge and each model; and incorporate human evaluation as an anchor.

A: The entire RLHF optimization loop relies on the RM as its objective function. The RM's capability ceiling determines the ceiling of policy optimization. Specifically: (1) Poor RM generalization → reward hacking; (2) Biased RM → policy inherits the bias; (3) Unreliable RM on OOD → iterative training also fails to improve it; (4) RM cannot evaluate quality dimensions beyond its training distribution → the policy cannot improve on those dimensions. All other techniques (KL, ensemble, iteration) are compensating for insufficient RM generalization.

Follow-up: Can scaling up the RM (increasing parameter count) systematically solve the generalization problem? Larger RMs do have better generalization (higher capacity, better feature representations), but cannot fully solve it — because the fundamental bottleneck is sometimes the noise and incompleteness of the preference data itself, not model capacity.

A: Human preference judgments are fundamentally ordinal information rather than cardinal information. Pairwise methods directly leverage ordinal information (A > B) with minimal information loss; pointwise requires humans to map to an absolute scale (e.g., 1–5), introducing additional scale calibration noise. From an information-theoretic perspective, a portion of the information in each pointwise annotation is "wasted" on scale noise. Pairwise therefore has higher sample efficiency.

Follow-up: Is there a way to recover absolute scores from pairwise data? The latent scores of each response can be recovered by solving for the MLE of the BT model, but the absolute values only have relative meaning and require an additional anchor point to determine the scale.

A: Similarities: Both rely on implicit/explicit reward signals in preference data; both are subject to the constraints of Goodhart's Law. Differences: (1) RLHF has an explicit RM that can be attacked and diagnosed; DPO implicitly encodes rewards, making it hard to inspect in isolation; (2) RLHF allows flexible adjustments via KL, ensembles, and iteration; DPO's β is similar to a KL penalty but offers different control granularity; (3) DPO is trained on off-policy data, facing more severe distribution shift; (4) RLHF's PPO is inherently on-policy, which to some extent partially mitigates distribution shift.

Follow-up: Are there forms of reward hacking unique to RLHF that DPO would not encounter? Issues specific to RLHF include: the explicit RM's OOD scoring failures, and instability in PPO training leading to sudden policy changes. DPO does not encounter explicit RM OOD problems, but faces implicit reward distribution shift problems — different in form but similar in nature.

A: The paper's core finding is that in their experimental setup, increasing the KL penalty coefficient $\beta$ does not improve the gold score–KL curve (the frontier); the effect is equivalent to early stopping on the same curve. This implies:

1. $\beta$ cannot be tuned as a generalization measure: Increasing $\beta$ constrains the policy to deviate less from the reference model, reducing KL consumption, but does not make the RM more robust to the same KL shift; "being safer" is only because optimization stops earlier, not because the RM itself improved.

2. The true role of KL penalty: Preventing the policy from moving too far in a single step (stabilizing training), not fundamentally solving reward hacking. What is truly needed is improvement in RM generalization itself (more data, larger models, iterative re-training).

3. Practical implication: One should not rely on increasing $\beta$ to "buy" more optimization headroom; if the gold score has already peaked, the RM should be retrained rather than continuing to strengthen KL constraints on the same RM.

Follow-up: BoN and RL have different functional forms (BoN is $d(\alpha - \beta d)$, RL is $d(\alpha - \beta \log d)$) — what does this tell us? BoN's quadratic form means over-optimization deteriorates with acceleration (decline is faster at larger $d$); RL's logarithmic form means deterioration is slower but sustained. Therefore, for the same KL "budget," BoN is more efficient but also collapses faster; one should not directly compare optimization quantity across methods using KL, as the two obey different over-optimization dynamics.

A: (Open-ended question; three key directions below)

1. Stronger generalization and OOD robustness: Current RMs perform well in-distribution but collapse out-of-distribution. Better architectural design (e.g., uncertainty-aware RM), adversarial training, and broader training data coverage are needed.

2. Multi-dimensional disentangled scoring: Decouple helpfulness, safety, factuality, style, and other dimensions into independent scoring heads, avoiding compression of information into a single scalar. This also allows different optimization weights for different dimensions.

3. Self-improving RM: Give the RM self-calibration capability — continuously detecting during RLHF whether its own predictions are consistent with actual preferences, and updating automatically. Reduces reliance on human annotation.

Follow-up: What potential conflicts exist between these three directions? Multi-dimensional scoring increases model complexity, which may affect generalization; self-improvement mechanisms may introduce systematic bias if unreliable; uncertainty estimation is computationally expensive in high-dimensional spaces. Trade-offs must be made in engineering practice.

A: The core challenge lies in the leap in complexity of the object being evaluated. Most existing benchmarks target relatively standard dialogue or simple reasoning tasks. For agents capable of long-horizon planning, tool use, or complex sub-task decomposition, the RM needs to evaluate not just the quality of a single response, but the overall effectiveness of an interaction trajectory and the soundness of long-term decisions. This requires evaluation paradigms to shift from "scoring static snippets" to "evaluating dynamic sequences," and calls for new metrics that can understand state transitions and world models — a fundamental expansion of the capability dimensions required of RMs.

Follow-up: How should RM training data be constructed under this new paradigm? The shift needs to go from "preference pairs" to "trajectory comparison" data, potentially involving multi-step interactions in simulated environments. Annotation will rely more heavily on automated verification (e.g., task success rate) and sandboxed testing in advanced simulators; the feasibility of pure human annotation will drop sharply.

A: The fundamental difficulty lies in the ambiguity of "process" definition and the subjectivity of evaluation. In mathematical reasoning, "steps" have clear logical boundaries and objective correctness criteria. But in creative writing, the transitions between paragraphs, the construction of imagery, and the build-up of emotion have no objective standards; their quality is highly dependent on subjective taste and holistic context. As a result, collecting reliable step-by-step annotations for PRM is extremely difficult, and automated evaluation (e.g., MC estimation) also fails due to the lack of a clear "correct answer."

Follow-up: Should PRM be abandoned entirely for open-domain tasks, or are there compromises? A "coarse-grained PRM" or "hybrid reward" approach can be adopted. For example, divide the writing process into a small number of high-level "stages" (e.g., ideation, development, conclusion), or combine ORM's holistic reward with process rewards at key turning points (e.g., plot climaxes), to balance evaluation granularity with feasibility.

A: Yes, this is essentially a non-cooperative game. The policy model (blue team) aims to discover and exploit vulnerabilities in the RM's (red team's) decision boundary to maximize rewards. Traditional mitigation strategies (such as fixed KL penalties) are "static defenses," whereas the game-theoretic perspective suggests adopting dynamic, adaptive adversarial strategies. For example, a dedicated "red team" RM can be trained whose objective is not to score responses but to actively find patterns that let the current policy obtain inflated rewards, and the discovered vulnerabilities are then used to update (harden) the primary RM.

Follow-up: What stability and cost challenges might this adaptive adversarial approach face in practice? The main challenge is that training dynamics may be unstable, with both parties entering an "arms race" that causes the optimization objective to drift continuously. Maintaining multiple adversarial models also introduces significant compute and coordination overhead.

A: Calibration refers to whether an RM's output scores truthfully reflect the absolute probability or expected value of response quality. An "accurate but uncalibrated" RM may rank correctly while its absolute score values or distribution have systematic bias. In RLHF, this can lead to misjudgment of optimization intensity: for example, uniformly inflated RM scores may cause the optimizer to believe the policy is already good and stop too early; or a narrow scoring range for small quality differences may result in insufficient policy update momentum (weak gradient signal). The KL penalty term depends on the relative magnitude of rewards; an uncalibrated RM may invalidate this trade-off.

Follow-up: In engineering practice, what methods can diagnose and improve RM calibration? Calibration curves can be plotted: segment RM prediction scores and compare the true quality (human annotation or task success rate) within each segment. Improvement methods include adding calibration regularization terms to the RM training loss, or post-processing RM outputs at inference time (e.g., temperature scaling).

A: The most rigorous approach is to model the problem as a constrained optimization problem, rather than a simple multi-objective weighted sum. Specifically, the optimization objective remains maximizing the RM score on the primary quality dimension (e.g., helpfulness), subject to a set of safety constraints (e.g., RM_safety(x, y) > τ). Theoretically, this can be solved via Lagrange multipliers or projected gradient methods, ensuring the policy optimizes within the safety-feasible set. This is more robust to the safety objective being sacrificed compared to mixing safety and helpfulness into a single score.

Follow-up: What is the main difficulty in setting an explicit threshold τ for safety constraints (e.g., "harmful probability < 0.1%") in practice? The core difficulty is the fuzziness and context-dependence of safety boundaries. A response that is "safe" in most contexts may be unsafe in specific sensitive contexts. Therefore, a globally fixed τ is unrealistic; context-dependent dynamic adjustment is needed, which places higher demands on both the RM and the constraint system.

A: Uncertainty estimation can actively guide training data collection and exploration strategies, enabling active learning. For example, within the training loop, priority can be given to sampling and human annotation in prompt-response regions where RM uncertainty is high, to improve the RM most efficiently. Furthermore, during policy optimization, the agent can be encouraged to actively explore high-uncertainty regions (i.e., the RM's knowledge boundary) to discover potentially new high-quality strategies; this is analogous to the exploration-exploitation trade-off in Bayesian optimization.

Follow-up: How can uncertainty-based exploration be specifically implemented in on-policy algorithms such as PPO in RLHF? RM uncertainty can be incorporated as part of the exploration reward, encouraging the policy to generate responses that the RM finds "surprising" or "uncertain." Specifically, add an exploration term positively correlated with uncertainty to the final reward, but carefully balance it to avoid generating meaningless random outputs.