Reasoning-RL Frontier

A: GRPO drops the critic (value network) — from PPO's four models (actor + critic + ref + RM) down to three (actor + ref + RM), saving the memory of one model's parameters and the training cost of the value function. Group-relative advantage: for each prompt, sample $G$ responses to obtain rewards $r_1..r_G$, then use the within-group mean as a value-baseline substitute + standard deviation scaling: $$A_i = \frac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r) + \varepsilon}$$ Intuition: $\mathrm{mean}(r)$ is an MC estimate of "how many points this prompt earns on average" — using it as the baseline factors out problem-difficulty effects; $\mathrm{std}(r)$ scaling keeps advantages from different prompts on the same scale. Why can the critic be dropped? Because rewards in verifiable domains (math exact-match, code unit tests) are clean and stable — no need to learn a value function to estimate "how many points this state will earn in the future."

Follow-up: If rewards are noisy (e.g., dialogue-quality RM scores), can the within-group mean still serve as a baseline? Yes but with reduced effectiveness — when noise is high, the MC variance of $\mathrm{mean}(r)$ is large, diluting the within-group comparison signal; PPO's critic smooths noise through TD learning and is more stable. This is precisely why the GRPO mainline chooses verifiable domains (see §6).

A: RLVR's reward comes from rules / verifiers rather than a learned neural RM: math uses exact-match (compare the final answer to the ground truth), code uses unit test pass/fail, format uses regex checks. It mitigates reward hacking because: a rule-based verifier ≈ ground truth — it does not suffer from systematic biases like "learned that long = good" which plague neural RMs, and is nearly impossible to hack (format gaming and repeated-token exploits for length rewards remain, but are much cleaner than neural RM issues). Limitation: only applicable to verifiable domains (math/code/machine-checkable formats) — open-ended domains (writing quality, dialogue helpfulness, safety) have no rule-based verifier and still require neural RMs or LLM-as-judge. The R1 mainline exploits exactly this property: RLVR does the heavy lifting in verifiable domains; a helpful/harmless RM handles general alignment in stage 4.

Follow-up: Is a format reward (e.g., "+0.1 for putting the answer in \boxed{}") RLVR or a neural RM? It is RLVR — the format is a hard regex-checkable rule. But in practice format rewards easily become a reward-hack target (the model learns to output \boxed{wrong answer} to farm the format points), so the weight must be tuned carefully.

A: Sequence-level loss (GRPO's original formulation): $L = \frac{1}{N}\sum_i \frac{1}{|o_i|}\sum_t \ell_{i,t}$ — average within each sequence first, then average across sequences. The problem: each token in a long response contributes a gradient diluted by $|o_i|$ (large $|o_i|$ means tiny per-token gradient), so the early critical steps of a long reasoning chain receive almost no signal. Token-level loss (DAPO's fix): $L = \frac{1}{\sum_i |o_i|}\sum_i\sum_t \ell_{i,t}$ — all tokens have equal weight, each token contributes an equal gradient. In long-CoT RL token-level is required — otherwise the model is induced to "write fewer tokens for the same reward", compressing reasoning chains. Dr.GRPO notes further: $1/|o_i|$ not only dilutes correct tokens in long responses, but also systematically favors long wrong responses (a long wrong response has each wrong token's penalty diluted → the loss looks smaller), a key source of optimization bias.

Follow-up: Won't token-level loss conversely give short correct responses disproportionately large gradients, causing training instability? Yes — each token in a short correct response gets a larger gradient; in practice this requires gradient clipping and an appropriate LR schedule to control.

A: Std normalization scales the advantages from different prompts to the same range — but it introduces two biases: ① Difficulty bias: prompts with small within-group variance (e.g., all-correct / all-wrong, std→0) have their advantages artificially inflated (divided by a tiny number), giving these prompts a louder voice in the batch; ② Interaction bias with length normalization: std scaling combined with $1/|o_i|$ length normalization makes the optimization drift toward "longer but worse responses" (a long wrong response undergoes two operations that make its gradient look "nicer"). Dr.GRPO's fixes: ① Remove the std division — use just $A_i = r_i - \mathrm{mean}(r)$, keeping the baseline's variance reduction while removing the scaling bias; ② Remove the $1/|o_i|$ length normalization — replace it with constant normalization (e.g., divide by max length, or no normalization), eliminating the systematic preference for long wrong responses. The cost: the absolute magnitude of advantages varies by prompt, requiring a suitable LR schedule. The paper claims this yields better token efficiency and no artificial response-length inflation.

Follow-up: After removing std, how do you handle prompts spanning very different difficulty levels (easy: rewards 0–1, hard: rewards 0–10)? Apply global reward normalization (e.g., batch-level z-score) or reward clipping — but this is a separate design choice, orthogonal to GRPO's within-group baseline construction.

A: Entropy collapse: the policy becomes excessively deterministic too early (some token probabilities → 1, others → 0), causing it to stop exploring new reasoning paths — the model gets "locked" into its current policy. How PPO's symmetric clip causes entropy collapse: the clip range $[1-\varepsilon, 1+\varepsilon]$ symmetrically constrains the importance ratio $\rho=\pi_\theta/\pi_{old}$. But for newly learned low-probability reasoning tokens (e.g., specific token sequences for backtracking or switching approaches), the initial probability is extremely low, so under a positive advantage $\rho$ rises from far below 1 — and the upper bound $1+\varepsilon$ quickly caps the growth of these tokens. Meanwhile, high-probability tokens (connective words like "therefore", "so") have $\rho\approx1$ and are barely affected by the clip. The result: the upward channel for low-probability innovative tokens is symmetrically capped, while high-probability tokens become further entrenched. Clip-higher's solution: decouple the upper and lower bounds and raise the upper bound (e.g., $\varepsilon_{low}=0.2, \varepsilon_{high}=0.28$), leaving room for low-probability tokens to rise and maintaining exploration.

Follow-up: Besides clip-higher, what other methods prevent entropy collapse? Add an entropy bonus (reward $+\lambda H(\pi)$), temperature annealing (high temperature early in training to encourage exploration), EMA reference (slow-anchor against the SFT checkpoint). DAPO chose clip-higher because it is the lightest weight and does not alter the reward signal.

A: Let $V(s) = \mathbb{E}[r|s]$ be the expected reward given the state (prompt). GRPO's within-group mean $\hat\mu=\frac{1}{G}\sum_i r_i$ is an MC estimate of $V$ — as $G\to\infty$, $\hat\mu\to V(s)$. The centered advantage $r_i-\hat\mu$ is exactly doing variance reduction with an MC baseline — keeping the unbiased gradient of REINFORCE while reducing variance. The bias–variance spectrum: ① When $G$ is small, the MC variance of $\hat\mu$ is high (the baseline itself is noisy), the advantage estimate fluctuates more, but the mean remains unbiased (no systematic shift); ② std normalization introduces mild bias (dividing by $\hat\sigma$ shifts the expected value of $A_i$), but substantially reduces variance (protecting gradient-scale stability); GRPO chooses this end of the trade-off; ③ Dr.GRPO removes std → returns to the unbiased $r_i-\hat\mu$, with higher variance, requiring larger $G$ or a more conservative LR as compensation; ④ PPO's value baseline $\hat V(s)$ is biased (function approximation error), but continually improves its estimate quality through TD learning. Thus in verifiable domains with clean rewards and sufficiently large $G$, critic-free's unbiased baseline suffices; in noisy-reward, long-trajectory open-ended domains, PPO's biased but lower-variance critic is more stable.

Follow-up: Can you take a weighted blend of GRPO's and PPO's baselines? In principle yes (an ensemble baseline), but in practice virtually nobody does this — the two baselines require different training paradigms (critic-free vs. critic-based), and the engineering cost of the blend is high for unclear gain.

A: In an all-correct or all-wrong group, all samples have the same reward → $A_i\equiv0$ → zero gradient, purely consuming compute with no parameter update. Dynamic sampling discarding them not only saves compute but guarantees every batch carries effective gradient signal, preventing optimization from being disrupted by zero-gradient batches. Relation to curriculum learning: both filter training samples by difficulty, but in opposite directions — curriculum is a preset "easy → hard" ordering of difficulty, while dynamic sampling filters in real time based on the model's current capability: all-correct prompts are too easy (the model already knows them), all-wrong prompts are too hard (the model still can't do them), leaving the difficulty band where "the model can just barely do them but isn't certain" — essentially online difficulty filtering. DAPO's dynamic sampling discards prompts with group accuracy 0 or 1 and resamples until the batch is filled.

Follow-up: What if most prompts in a batch are filtered out (difficulty converges late in training)? Down-sample the frequency rather than fully discarding — lower the sampling rate of all "ineffective prompts" without zeroing them out; or expand the prompt pool with new problems. DAPO takes the former approach.

A: Reasoning RL (GRPO + RLVR): lets the model explore on its own how to use more tokens effectively (reflection, re-checking, switching approaches), acquiring long-reasoning behaviors during training. Distillation (SFT from teacher traces): directly imitates reasoning traces that "the teacher already thought through." Their respective pros and cons: ① Reasoning RL has a higher ceiling — it can discover reasoning patterns not present in distillation data (R1-Zero's spontaneous reflection/re-checking proves this), but training is unstable and only works in verifiable domains; ② Distillation is cheaper, more stable, and broadly applicable (any domain with teacher traces works), but the ceiling is capped by teacher quality ("the teacher can't teach what it doesn't know"). R1-Distill's core conclusion: distillation > running RL directly on small models — small models can't explore effectively on their own (in such a vast search space, the small model's initialization isn't good enough and within-group diversity is poor), so directly consuming the large model's reasoning traces wins. This doesn't mean distillation is absolutely better than RL — rather, in the large-model-does-RL + small-model-distills combo, the division of labor is clear: the large model explores, the small model learns the exploration results.

Follow-up: What does this conclusion imply for resource-constrained academic research? If you have only modest compute, prioritize the distillation route (use a strong open-source reasoning model to produce traces → SFT into a small model), rather than running GRPO from scratch on the small model — the cost-effectiveness ratio is far higher.

A: Critic-free methods assume "clean reward signal + effective within-group comparison." Four failure modes: ① Noisy rewards (e.g., dialogue-quality RM scores with high fluctuation): lacking a value function for temporal smoothing, within-group z-score amplifies noise, and the gradient direction becomes unreliable; ② Poor within-group diversity: when the sampling temperature is too low or the model is overfitted, the G responses are nearly identical → std→0 → the advantage estimate collapses (division by zero) — PPO's critic is unaffected by sampling diversity; ③ Sparse rewards + long trajectories: only the final answer's correctness (0/1) is given, with no signal for the 1000 intermediate tokens — GRPO can only rely on the final reward's within-group contrast, while PPO's GAE back-propagates future rewards step by step via TD, providing a smoother per-token signal; ④ Non-stationary reward distributions: when the neural RM itself is changing during training, PPO's critic can adapt to reward-distribution drift, while critic-free's within-group baseline only reflects the current batch. Mnemonic: verifiable domains = critic-free is optimal; non-verifiable domains / long trajectories / noisy environments = PPO is more stable.

Follow-up: Could you introduce a lightweight critic as a hybrid in "verifiable domain + sparse reward" settings? This is an open research question — a lightweight critic (e.g., initialized from a PRM, doing only outcome-level value estimation) could provide a cross-prompt global baseline for each group, improving sparse-reward settings while retaining critic-free simplicity. There is no standard approach yet.

A: The role of the KL penalty is policy anchoring — preventing the policy from drifting under reward hacking (collapsing onto some reward shortcut). But in long-CoT settings, precisely what the model needs to learn is the long-chain reflection behavior that the reference model cannot do, so the KL is inherently penalizing "novel reasoning paths." Practical trade-offs: $\beta$ too large → model fails to learn long CoT, reasoning capacity is capped by the reference; $\beta$ too small → policy may degrade into reward hacking (e.g., repeating tokens to fool the verifier). DAPO's original recipe actually removes KL and instead relies on clip-higher + dynamic sampling to prevent collapse; GRPO retains KL but typically sets it to a low value. At its core this is a tightrope walk between "not collapsing" and "being able to explore". Follow-up: If you want to remove KL to open up exploration while still preventing policy collapse, what alternative anchoring mechanisms are feasible beyond clip-higher? (e.g., EMA reference, regularization toward the SFT checkpoint, etc.)

A: Both methods are variance reduction variants of REINFORCE; the difference lies in baseline construction. GRPO uses $\mathrm{mean}(r)$ and divides by $\mathrm{std}(r)$ (i.e., a z-score); RLOO gives sample $i$ a baseline of $\frac{1}{G-1}\sum_{j\neq i}r_j$. RLOO's leave-one-out baseline is unbiased (because $r_i$ is excluded from its own baseline), whereas GRPO's $\mathrm{mean}(r)$ includes $r_i$ itself, introducing a mild self-correlation bias (negligible when $G$ is large). However, GRPO's std normalization simultaneously applies variance scaling, making it more robust when reward magnitudes are uncertain — at the cost of the problem-difficulty bias that Dr.GRPO identifies. RLOO does not apply std normalization: it is more sensitive to reward scale but more unbiased. The choice depends on the stability of the task's reward distribution. Follow-up: What would happen if you combined RLOO's leave-one-out baseline with GRPO's std normalization? Are there known problems with this hybrid?

A: There is some effect, but the direction is more complex than intuition suggests. Token-level loss ensures every token contributes an equally weighted gradient, which does mean each token in a short correct response receives a larger gradient (total gradient is divided among $T$ tokens; smaller $T$ means larger per-token gradient). But the key factor is the sign of the gradient: correct responses receive positive reinforcement, wrong responses receive negative penalization. Token-level loss therefore makes short wrong responses receive more concentrated, stronger penalty — which is not necessarily bad for training efficiency. The real risk arises when rewards are binary 0/1: a long correct response and a short correct response receive the same total reward, but under token-level loss the per-token reinforcement signal for the long response is weaker, which may gradually push the model to compress correct reasoning chains. Follow-up: If DAPO token-level loss is combined with outcome reward (only the final answer's correctness), how do "compress reasoning chain length" and "compress to correct answer" compete with each other?

A: The idea is feasible in principle: use the PRM to score each step of the CoT, aggregate step-level scores into a sequence-level reward, and feed this into GRPO's within-group advantage computation. But there are three obstacles: ① PRM annotation and training — step-level human annotations or automated labels (e.g., Monte Carlo rollout estimation) are required, which is expensive and of limited accuracy; ② Reward alignment — the PRM scores "reasoning step quality," which may be inconsistent with final answer correctness (good steps but wrong answer vs. rough steps but correct answer), producing conflicting RL signals; ③ Temporal credit assignment — when aggregating step-level scores into a sequence reward, the choice of weighting scheme (mean? final step? worst step?) directly affects learning dynamics. A simple mean blurs the contribution of critical steps; using only the final step degenerates into outcome reward. Follow-up: Within the GRPO framework, is it possible to use different reward aggregation strategies (adaptive weighting) for different samples in the group, rather than a uniform scheme?

A: The core problem with an all-wrong group is that all rewards are identical → advantage is always zero → zero gradient. Several approaches: ① Introduce process rewards — even if all final answers are wrong, the quality of intermediate reasoning steps may differ; use a PRM or auxiliary signals such as reasoning length and format compliance to create within-group variation; ② Mixed reward design — layer format rewards, reasoning completeness rewards, and other soft signals on top of the outcome reward so that even "all-wrong" groups can still distinguish better from worse responses; ③ Cross-prompt baseline — rather than restricting to within-group comparison, use a batch-level or moving-average global baseline to provide gradient direction even for all-wrong groups; ④ Difficulty bucketing + resampling — mark all-wrong prompts as "too hard," downsample their frequency without fully discarding them, avoiding a training-set bias toward easy problems. Each approach has different costs: process rewards require extra annotation or models; cross-prompt baselines may introduce high variance. Follow-up: How would a cross-prompt baseline (e.g., using an EMA global mean as the baseline) be implemented concretely within GRPO? How should the weight be tuned when mixing with the within-group baseline?

A: At $G=1$ there is only one sample, $\mathrm{mean}(r)=r_1$, and the advantage is always zero — no gradient at all; GRPO is completely ineffective. At $G=2$ it degenerates into pairwise comparison, essentially an online version of pairwise preference learning. As $G\to\infty$ the within-group mean and standard deviation converge to the population expectation and standard deviation, so the baseline approximates a Monte Carlo estimate of the global value function; in theory GRPO then approaches REINFORCE with a global baseline. Practical trade-offs: $G$ too small → high baseline variance, noisy advantage estimates; $G$ too large → high sampling cost per prompt (inference cost grows linearly) and diversity may actually decrease (many similar responses in repeated sampling). DeepSeek's practice uses a moderate value of $G$. There is also coupling between $G$ and the clip range and learning rate — with larger $G$ the advantage estimate is more accurate, so more aggressive update steps are tolerable. Follow-up: Is there an adaptive-$G$ strategy — sampling more from "hard" prompts and fewer from "easy" ones? How would this coordinate with dynamic sampling?

A: The core condition for convergence is that the reward signal must continuously provide effective discrimination — the model must be able to distinguish the quality of its own outputs. When model capability is far below task difficulty, self-judgment is noisy but not systematically biased; training may be slow but won't collapse. Typical triggers for mode collapse: ① Amplified anchoring effect — the model prefers responses that resemble its own style; a positive feedback loop continuously reduces output diversity until it collapses to a narrow "self-preferred" mode; ② Judgment saturation — when model outputs become too similar in quality, the reward signal degrades to noise and training loses direction; ③ Reward hacking its own judge — the model learns to "convince" its own judge rather than genuinely improving, analogous to Goodhart's Law applied to itself. Mitigations include: retaining an SFT anchor, periodically introducing external verification signals, and limiting the number of iterations. Follow-up: If a fixed external verifier (e.g., code unit tests) is introduced into the self-rewarding loop as an "anchor" to calibrate self-scoring, to what extent can it prevent mode collapse?

A: Standard PPO's symmetric clip $[1-\epsilon, 1+\epsilon]$ acts on the importance ratio $\rho=\pi_\theta/\pi_{\theta_{old}}$. The key insight: in long CoT, already high-probability tokens (e.g., common reasoning connectives) have $\rho$ close to 1, so the symmetric clip barely affects them; but newly learned low-probability reasoning patterns (e.g., specific token sequences for backtracking or self-correction) start at very low probability, and when the advantage is positive $A>0$, $\rho$ rises from below 1 — and hits the upper bound $1+\epsilon$ quickly, hard-capping the growth of these tokens. Meanwhile, when $A<0$, the lower bound $1-\epsilon$ equally limits the decline of high-probability tokens, but high-probability tokens have ample room to decrease anyway. Therefore the symmetric clip creates an asymmetric learning dynamic in information-geometric terms: the "downward channel" for high-probability tokens is wider than the "upward channel" for low-probability tokens. Clip-higher breaks this asymmetry by raising the upper bound. Follow-up: Apart from decoupling the clip bounds, is it possible to address this problem more elegantly from a trust-region perspective (e.g., using a KL constraint instead of a hard clip)? What is the computational cost of doing so in the long-CoT setting?

A: R1's four-stage ordering is carefully designed, not an arbitrary stack:

1. Cold-start SFT (→ readability foundation): Running RL directly from a base model produces R1-Zero's problems — long-CoT direction is correct but answers are unreadable and mix Chinese with English. First, fine-tune on a small set of high-quality long-CoT samples to lay a "format / language" foundation, so subsequent RL starts from a policy that "speaks properly."

2. Reasoning RL (→ core reasoning capability): Without adding dialogue preferences, use GRPO + RLVR (rule rewards) to focus on pulling up reasoning ability. Why before preference RL? If you do preference alignment first and reasoning RL later, preference RL may suppress exploration (KL anchors to a "safe but mediocre" policy), and reasoning RL needs a large exploration space.

3. Rejection-sampling SFT (→ knowledge distillation + capability consolidation): Heavily sample from the stage-2 policy, filter for reasoning-correct + format-clean traces, then SFT on them (the R1 paper uses ~800k SFT samples, a mixture of reasoning and general). This step: ① "bakes" the good reasoning patterns discovered by RL back into the SFT distribution, providing a stable starting point for the next stage's general RL; ② produces a large quantity of high-quality reasoning data for distillation (R1-Distill uses the data produced by this stage, post-processed for SFT).

4. All-scenario RL (→ general alignment): Finally run RL over all prompts (reasoning + general) — rule rewards for verifiable domains, helpful/harmless RM for general domains. Placed last so that preference alignment is a light touch-up on top of "reasoning is already good enough," without damaging reasoning capability.

Could the order be swapped? The alternation of stages 2 and 3 (SFT→RL→SFT→RL) is the core innovation; the R1 paper describes this multi-stage design with two rounds of alternation. Stages 1 and 2 cannot be swapped (must lay the readability foundation before RL, otherwise R1-Zero's language-mixing problem recurs). Stage 4 must be last (preference alignment suppresses exploration; placing it earlier would lock down reasoning RL's exploration space).

Follow-up: What if you kept only one alternation (dropped stages 3–4)? The R1 paper doesn't directly report this ablation, but looking at R1-Zero (RL only, no SFT): dropping stage 1 severely damages readability; dropping stages 3–4 leaves strong reasoning but poor dialogue experience — essentially R1-Zero + cold-start SFT, which is usable but falls short on practical utility.