Intended use: Interview preparation for LLM Post-Training Research Intern roles & everyday reference Language: English, with key technical terms preserved as-is

Part 1 — Core Concepts & Formula Derivations

Core Concepts & Formula Derivations

1. Pre-training vs Post-training Overview

Standard 5-Step Pipeline:

1. SFT (Supervised Fine-Tuning): Supervised fine-tuning on high-quality (instruction, response) pairs to transform the base model into an "instruction assistant."
2. Reward Model Training: Train a scoring model RM using human preference comparison data (two responses to the same prompt + human preference labels).
3. RLHF / PPO: Reinforcement learning using RM feedback, with a KL constraint to prevent diverging too far from the SFT model.
4. DPO (Offline Alternative): Bypasses the explicit RM; directly optimizes the policy from preference data, achieving simpler and more stable alignment.
5. Iterative Loop: Current policy samples new data → new preference labels → update RM → RL again, repeated over multiple rounds.

2. SFT Data Format & Loss Masking

Chat Template (ChatML format example):

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user instruction}<|im_end|>
<|im_start|>assistant
{assistant response}<|im_end|>

Different models have their own templates (Llama-2 uses [INST], Llama-3 uses <|start_header_id|>, Qwen uses <|im_start|>). Training and inference must use the same template; otherwise a distribution shift occurs.

Loss Masking:

The training objective of SFT is to teach the model "how to answer," not "how to repeat the question." Cross-entropy loss is computed only at assistant token positions:

$$L_{SFT} = -\frac{1}{|A|} \sum_{t \in A} \log p_\theta(x_t \mid x_{<t})$$

where $A$ is the set of all assistant token positions. Labels for user / system tokens are set to $-100$ (ignored by default in PyTorch's CrossEntropyLoss).

Multi-turn Loss Masking: The user turn of every round is masked; only the assistant turn of each round contributes to the loss.

Trade-off for including the system prompt in the loss:

• Exclude (mainstream practice): saves capacity, focuses training on response quality.
• Include: the model better learns to follow system instructions, but introduces more noise.

2.1 Common tokenization / chat-template pitfalls (SFT engineering screening questions)

These are the most common failure modes in SFT engineering and are frequently tested in interviews (each: problem → fix):

• pad_token = eos_token: Many models (LLaMA/GPT-2) have no pad token; HF defaults to setting pad as eos. If pad positions are not masked in attention_mask and pad position labels are not set to -100, the model computes loss over pad tokens / learns to output eos everywhere. → Explicitly construct attention_mask to mask pad; set all prompt and pad labels to -100.
• Applying chat template twice: After apply_chat_template in data preprocessing, tokenization also sets add_special_tokens=True or wraps the template again → duplicate BOS / duplicate special tokens. → Apply the template only once; set add_special_tokens=False when tokenizing afterward.
• Missing or inconsistent BOS: BOS added during training but not at inference (or vice versa) → distribution shift (LLaMA-family is sensitive to BOS). → Verify whether the template already contains BOS; keep training/inference behavior consistent.
• Tokenizer version / vocabulary mismatch: Training and deployment use different versions or a version with added custom tokens, causing token ID misalignment. → Pin the tokenizer version; save it alongside the checkpoint.
• Uninitialized newly added special tokens: Adding new special tokens requires resize_token_embeddings (otherwise the token ID goes out of bounds); the newly added rows are randomly initialized and produce gibberish before training. → Initialize new rows properly (common heuristic: mean of existing embeddings, or reuse the vector of a semantically similar token — the mean is just a common heuristic, not the only option); ensure those new tokens appear in training data.
• Packing cross-sample attention contamination: When packing multiple documents into one sequence, without block-diagonal / cu_seqlens masking, tokens can attend across document boundaries (see §3). → Use varlen / cu_seqlens attention with correct separation.
• Using the wrong model's template: Llama-3 (<|begin_of_text|> / <|start_header_id|>), Qwen (<|im_start|>), Mistral ([INST]) have different formats; using the wrong one causes a sharp performance drop. → Use the target model's own tokenizer.apply_chat_template; do not hand-stitch concatenations.

Self-test (L2): Why does pad_token=eos_token break SFT if attention_mask and label mask are not set correctly? In multi-turn dialogue, how should labels for pad and prompt positions be handled?

3. Sequence Packing

Definition: Concatenate multiple short samples into a single sequence of length equal to the context window, adding EOS / separator tokens only at sample boundaries, thereby eliminating padding waste.

GPU utilization: Without packing, padding can account for 30–60% of tokens; with packing, nearly 100% of tokens are valid, yielding a 2–4× training speedup.

Pitfall 1 — Cross-sample attention contamination: Without a document-level attention mask, tokens from a preceding sample in a packed sequence can attend to tokens from a following sample, causing information leakage. Solution: use Flash Attention's cu_seqlens parameter, which takes the cumulative sequence lengths of each sample within the packed sequence and ensures attention is computed only within each sample.

Pitfall 2 — Loss weight imbalance: Packing implicitly weights by token count (longer samples produce more loss terms). If the original objective averaged over samples, the packing objective differs semantically; consider whether length normalization of the loss is needed.

cu_seqlens example (3 samples with lengths 5, 3, 7):

cu_seqlens = [0, 5, 8, 15]  # cumulative lengths
packed_ids = [s1_tok1, ..., s1_tok5, s2_tok1, ..., s2_tok3, s3_tok1, ..., s3_tok7]

4. RLHF Full Pipeline / PPO in RLHF

RLHF (Reinforcement Learning from Human Feedback) three stages:

1. SFT: Establish the initial policy $\pi_{ref}$ (reference policy, which serves as the baseline for the KL penalty).
2. RM Training: Fit a scalar reward function $r(x,y)$ from human preference comparison data using the Bradley-Terry model.
3. PPO Optimization: Maximize the augmented reward with a KL constraint.

Augmented Reward:

$$r_{total}(x,y) = r_{RM}(x,y) - \beta \cdot \text{KL}\!\left(\pi_\theta(\cdot|x) \| \pi_{ref}(\cdot|x)\right)$$

PPO Clipped Objective:

$$L^{CLIP}(\theta) = \mathbb{E}_t\!\left[\min\!\left(r_t(\theta)\hat{A}_t,\ \text{clip}(r_t(\theta), 1-\varepsilon, 1+\varepsilon)\hat{A}_t\right)\right]$$

where $r_t(\theta) = \dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ is the probability ratio, and $\varepsilon$ is typically 0.1–0.2.

GAE Advantage Estimation:

$$\hat{A}_t = \sum_{l=0}^{T-t} (\gamma\lambda)^l \delta_{t+l}, \quad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

$\lambda$ controls the bias-variance trade-off: $\lambda=1$ gives high variance, low bias; $\lambda=0$ degenerates to one-step TD.

Recurrence (backward sweep, $O(T)$): $\hat{A}_t=\delta_t+\gamma\lambda\,\hat{A}_{t+1}$, with $\hat{A}_T=\delta_T$.

• $\lambda=0$: $\hat{A}_t=\delta_t$, the one-step TD advantage (low variance, high bias).
• $\lambda=1$: $\hat{A}_t=\sum_l\gamma^l\delta_{t+l}$, the Monte-Carlo advantage (value function as a baseline only; low bias, high variance).
• Key distinction: $\gamma<1$ introduces bias regardless of $V$ accuracy (the discount itself); $\lambda<1$ introduces bias only when $V$ is inaccurate — so with a well-fit $V$, even $\lambda\to1$ is near-unbiased. Source: Schulman et al., arXiv:1506.02438 (ICLR 2016).

4 Models Required by PPO (source of memory pressure):

From-scratch implementation (clipped policy loss + clipped value loss + entropy bonus + approx_kl monitoring):

import torch

def ppo_loss(logp, logp_old, values, values_old, returns, advantages, entropy,
             clip_eps=0.2, vf_clip=0.2, vf_coef=0.5, ent_coef=0.0):
    # logp/logp_old: (B,) logprob of the taken action under current/behavior policy
    # advantages: normalized GAE advantage (for policy loss); returns: un-normalized GAE target (raw_adv + values_old, for value loss); both from upstream
    ratio = torch.exp(logp - logp_old)                      # importance ratio ρ_t
    surr1 = ratio * advantages
    surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
    pg_loss = -torch.min(surr1, surr2).mean()               # clipped policy loss

    v_clip = values_old + torch.clamp(values - values_old, -vf_clip, vf_clip)
    vf_loss = 0.5 * torch.max((values - returns) ** 2,
                              (v_clip - returns) ** 2).mean()  # clipped value loss

    loss = pg_loss + vf_coef * vf_loss - ent_coef * entropy.mean()  # entropy bonus

    with torch.no_grad():                                   # diagnostics only
        logr = logp - logp_old                              # log(π_new/π_old)
        approx_kl = (torch.exp(logr) - 1 - logr).mean()     # K3: KL(π_old‖π_new) probe
        clip_frac = ((ratio - 1).abs() > clip_eps).float().mean()
    return loss, {"pg": pg_loss, "vf": vf_loss, "approx_kl": approx_kl, "clip_frac": clip_frac}

• Key points: ① three loss terms — clipped policy loss (the $L^{CLIP}$ above), clipped value loss (constrain the critic's single step around values_old to prevent value oscillation), and an entropy bonus (encourages exploration; ent_coef is often 0 or tiny); ② approx_kl uses the K3 estimator for KL(π_old‖π_new), a trust-region monitor (early-stop the epoch if it exceeds a threshold) — a different KL from the $\beta\cdot\mathrm{KL}(\pi_\theta\|\pi_{ref})$ in the reward above: the former bounds the update step, the latter anchors to the reference policy; ③ both come from upstream: advantages is the normalized GAE advantage (fed to the policy loss), while returns is the un-normalized GAE target ($=$ raw advantage $+$ values_old, fed to the value loss) — the two are not interchangeable; this function only computes the minibatch loss; ④ vf_coef defaults to 0.5 as a common heuristic weight (not a principled gradient balance), most relevant when actor and critic share a backbone; clip_frac reports the fraction of ratios outside the clip range, which is not the same as the fraction whose objective term was actually clipped (that also depends on the advantage sign).

5. Bradley-Terry Reward Model

Bradley-Terry preference model: Given prompt $x$, the probability that the better response $y_w$ is preferred over the worse response $y_l$ is:

$$P(y_w \succ y_l \mid x) = \sigma\!\left(r(x,y_w) - r(x,y_l)\right)$$

RM training loss (maximizing log likelihood of preference data):

$$L_{RM} = -\mathbb{E}_{(x,y_w,y_l)}\!\left[\log \sigma\!\left(r(x,y_w) - r(x,y_l)\right)\right]$$

RM architecture:

• Initialized from the SFT model (inherits language understanding capability).
• The LM head is removed and replaced with a linear layer $W \in \mathbb{R}^{d \times 1}$ that outputs a scalar reward.
• During training, a (chosen, rejected) response pair is fed in; both are forward-passed separately, taking the scalar output at the last token position to compute the Bradley-Terry loss.

Key risks:

• Reward hacking: the policy finds responses that score high under RM errors but are low quality (Goodhart's Law).
• Distribution shift: the RM breaks down on the policy's distribution when it has shifted away from the training distribution; iterative RM updates are needed.

6. DPO Full Derivation (Direct Preference Optimization Full Derivation)

6.1 Starting from the RLHF Objective

The KL-constrained RLHF optimization objective:

$$\max_{\pi} \; \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi(\cdot|x)}\!\big[r(x, y)\big] \;-\; \beta \cdot \mathrm{KL}\!\big(\pi(\cdot|x) \;\|\; \pi_{\mathrm{ref}}(\cdot|x)\big)$$

where:

• $r(x, y)$ is the scalar reward from the reward model
• $\pi_{\mathrm{ref}}$ is the reference policy, typically the SFT model
• $\beta > 0$ controls KL penalty strength

Expanding the KL divergence:

$$\max_{\pi} \; \mathbb{E}_{x,y}\!\big[r(x,y)\big] - \beta \sum_{y} \pi(y|x)\log\frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)}$$

Taking the variational derivative per $y$ and setting it to zero:

$$\frac{\partial}{\partial \pi(y|x)}\left[r(x,y) - \beta\log\frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)} - \beta\right] = 0$$

Note: the normalization constraint $\sum_y \pi(y|x)=1$ introduces a Lagrange multiplier $Z(x)$

6.2 Closed-Form Optimal Policy

Solving yields the optimal policy:

$$\boxed{\pi^*(y|x) = \frac{1}{Z(x)}\,\pi_{\mathrm{ref}}(y|x)\,\exp\!\left(\frac{r(x,y)}{\beta}\right)}$$

where the partition function is:

$$Z(x) = \sum_{y} \pi_{\mathrm{ref}}(y|x)\,\exp\!\left(\frac{r(x,y)}{\beta}\right)$$

$Z(x)$ ensures $\sum_y \pi^*(y|x) = 1$, i.e., the policy is a valid probability distribution.

6.3 Inverting for the Reward

Taking log of both sides of the optimal policy:

$$\log \pi^*(y|x) = \log \pi_{\mathrm{ref}}(y|x) + \frac{r(x,y)}{\beta} - \log Z(x)$$

Rearranging:

$$\boxed{r(x,y) = \beta \log\frac{\pi^*(y|x)}{\pi_{\mathrm{ref}}(y|x)} + \beta \log Z(x)}$$

Key insight: the reward can be expressed via the log-ratio of policy to reference, eliminating the explicit reward model.

6.4 Substituting into Bradley-Terry

Human preference model:

$$p(y_w \succ y_l \mid x) = \sigma\!\big(r(x, y_w) - r(x, y_l)\big)$$

where $\sigma(z) = \frac{1}{1+e^{-z}}$ is the sigmoid function.

Substituting the inverted reward:

$$r(x,y_w) - r(x,y_l) = \beta\log\frac{\pi^*(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta\log\frac{\pi^*(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} + \cancel{\beta\log Z(x)} - \cancel{\beta\log Z(x)}$$

The $Z(x)$ terms cancel perfectly! This is because both responses share the same partition function for the same prompt $x$.

$$p(y_w \succ y_l \mid x) = \sigma\!\left(\beta\log\frac{\pi^*(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta\log\frac{\pi^*(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\right)$$

6.5 DPO Loss Function

Replacing $\pi^*$ with parameterized $\pi_\theta$, taking negative log-likelihood:

$$\boxed{\mathcal{L}_{\mathrm{DPO}}(\pi_\theta) = -\mathbb{E}_{(x,\, y_w,\, y_l) \sim \mathcal{D}}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\right)\right]}$$

Implicit reward defined as:

$$\hat{r}_\theta(x, y) \triangleq \beta \log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}$$

The loss simplifies to:

$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}\!\big[\log\sigma\!\big(\hat{r}_\theta(x,y_w) - \hat{r}_\theta(x,y_l)\big)\big]$$

6.6 Gradient Analysis

$$\nabla_\theta \mathcal{L}_{\mathrm{DPO}} = -\beta\,\mathbb{E}\!\left[\underbrace{\sigma(-\hat{r}_\theta(x,y_w)+\hat{r}_\theta(x,y_l))}_{\text{weight: larger gradient when the model is more wrong}}\!\left(\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x)\right)\right]$$

• When the model already ranks $y_w \succ y_l$ correctly, $\sigma(\cdot) \to 0$ and the gradient naturally decays
• When the model ranks incorrectly, $\sigma(\cdot) \to 1$ and the gradient is largest

6.7 DPO Advantages & Disadvantages

6.8 Likelihood Displacement: chosen log-prob also decreases

Phenomenon

Intuitively, DPO training should increase the model's probability for the chosen response $y_w$ and decrease it for the rejected response $y_l$. However, Razin et al. (arXiv:2410.08847) and Pal et al. (arXiv:2402.13228) both observe that $\log\pi_\theta(y_w|x)$ and $\log\pi_\theta(y_l|x)$ tend to decrease simultaneously during training — the loss decreases only because $y_l$ decreases faster, widening the margin between them, while the absolute probability of the chosen response shrinks.

"While intuitively these methods should increase the probability of $y^+$ while decreasing that of $y^-$, several recent works observed that the probabilities of both $y^+$ and $y^-$ tend to decrease over the course of training." — Razin et al., arXiv:2410.08847

Gradient Mechanism

The DPO loss only constrains the log-prob difference (margin) relative to the reference model to widen:

$$\mathcal{L}_\text{DPO} = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)}\right)\right]$$

$\log\pi_\theta(y_w|x)$ itself has no lower-bound constraint — as long as $y_l$ decreases faster, the gradient objective is satisfied. Razin et al. (Theorem 1/3) note that when the hidden representations of $y_w$ and $y_l$ are similar (high CHES score), the gradient direction that suppresses $y_l$ simultaneously suppresses $y_w$, and probability mass shifts to tokens semantically opposite to $y_w$, forming "unintentional unalignment" (the phrase used in Razin et al.).

Danger Conditions

• Pairs in the dataset where $y_w$ and $y_l$ differ by only a few tokens (Pal et al. report a normalized edit distance of approximately 6.5% for the MetaMath subset), sharing large prefixes so that gradients are highly correlated.
• Preference pairs that are semantically similar with subtle differences (e.g., phrasing differences rather than factual differences).

Detection

Record both chosen_logps_mean and rejected_logps_mean throughout training (most training frameworks already log these). If the chosen mean consistently decreases beyond the reference baseline, displacement is occurring.

Mitigation

(A) DPOP (Pal et al., arXiv:2402.13228): Adds a penalty term inside the DPO loss that directly prevents the chosen log-prob from falling below the reference model:

$$\mathcal{L}_\text{DPOP} = -\mathbb{E}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)} - \lambda\cdot\max\!\left(0,\log\frac{\pi_\text{ref}(y_w|x)}{\pi_\theta(y_w|x)}\right)\right)\right]$$

The $\max(0,\cdot)$ term with $\lambda > 0$: when $\log\pi_\theta(y_w|x) < \log\pi_\text{ref}(y_w|x)$, a penalty is applied that "anchors" the chosen log-prob above the reference model.

(B) CHES data filtering (Razin et al., arXiv:2410.08847): Filter out preference pairs where the representations of $y_w$ and $y_l$ are highly similar, cutting the gradient coupling path at the data level.

(C) SimPO: Uses length-normalized $\frac{1}{|y|}\log\pi_\theta(y|x)$ as the implicit reward and removes $\pi_\text{ref}$; the reward definition is directly aligned with generation-time likelihood, which by design weakens the driving force behind displacement (though the SimPO paper itself does not directly analyze this issue using the Razin/Pal framework).

Note: The three mitigation approaches above come from different papers and should not be cross-attributed — the $\max$ regularization term in DPOP is from Pal et al.; CHES filtering is from Razin et al.; the connection between SimPO and displacement comes from downstream work and is provided here for reference only; it must not be attributed to either the Razin or Pal papers.

7. DPO Variants Comparison

7.1 IPO (Identity Preference Optimization / $\Psi$PO with $\Psi = \text{Id}$)

DPO problem addressed: DPO uses $\Psi(q) = \log(q/(1-q))$ (the logit function, corresponding to Bradley-Terry). When preferences approach certainty ($p^*(y_w \succ y_l) \to 1$), the logit tends to $+\infty$, driving $\pi^*(y_l) \to 0$ regardless of the KL penalty coefficient $\tau$ — KL regularization becomes ineffective under strong preferences, and the policy overfits the preference data.

Core change: Under the $\Psi$PO framework, replace $\Psi$ with the identity mapping (input preference probability $p\in[0,1]$, $\Psi(p)=p$ does not diverge as $p\to 1$ the way DPO's logit mapping does). The resulting empirical loss (Azar et al., arXiv:2310.12036, Eq. 17) is a squared-loss regression:

$$\mathcal{L}_{\text{IPO}}(\pi) = \mathbb{E}_{(y_w, y_l, x) \sim \mathcal{D}} \left[ \left( h_\pi(y_w, y_l, x) - \frac{1}{2\tau} \right)^2 \right]$$

where $h_\pi(y, y', x) = \log \dfrac{\pi(y|x)\,\pi_{\text{ref}}(y'|x)}{\pi(y'|x)\,\pi_{\text{ref}}(y|x)}$ is the log-ratio difference of the policy relative to the reference (logit margin); the target constant is $\frac{1}{2\tau}$, with $\tau$ the KL regularization strength.

Citation: Mohammad Gheshlaghi Azar et al. — arXiv:2310.12036 (Google DeepMind, 2023)

"IPO, unlike DPO, always regularizes its solution towards $\pi_\text{ref}$ by controlling the gap between the log-likelihood ratios, thus avoiding the over-fitting to the preference dataset." — Azar et al., Section 5.2

Properties:

• The squared loss regresses the logit-margin to a fixed target $\frac{1}{2\tau}$; $\tau$ directly controls the upper bound of the learned log-ratio difference, keeping KL regularization effective at all times
• The gradient near the target may be small but is never zero — the solution cannot "escape" to an unbounded region
• Note: Azar et al.'s original validation is limited to small-scale bandit experiments; effectiveness at LLM scale requires independent verification
• Trade-off: The loss does not correspond to a BT probability model, slightly weakening theoretical interpretability

7.2 KTO (Kahneman-Tversky Optimization)

DPO problems addressed: (1) DPO requires paired preference data $(x, y_w, y_l)$, whereas in practice only pointwise positive/negative feedback (pointwise thumbs-up/down) is often available; paired data is expensive and scarce. (2) DPO maximizes preference log-likelihood, which is a proxy for the true objective of "maximizing generation utility," resulting in objective mismatch.

Loss (complete form):

$$\mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x,y \sim \mathcal{D}}\bigl[w(y)\bigl(1 - v_{\text{KTO}}(x,y;\beta)\bigr)\bigr]$$

where the implicit reward, KL baseline, and value function are defined as:

$$r_{\text{KTO}}(x,y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$$

$$z_{\text{ref}} = \mathbb{E}_{x' \sim \mathcal{D}}\bigl[\beta\,\mathrm{KL}(\pi_\theta(y' \mid x') \,\|\, \pi_{\text{ref}}(y' \mid x'))\bigr]$$

$$v_{\text{KTO}}(x,y;\beta) = \begin{cases} \sigma(r_{\text{KTO}}(x,y) - z_{\text{ref}}) & \text{if } y \sim y_{\text{desirable}} \mid x \\ \sigma(z_{\text{ref}} - r_{\text{KTO}}(x,y)) & \text{if } y \sim y_{\text{undesirable}} \mid x \end{cases}$$

$$w(y) = \begin{cases} \lambda_D & \text{if desirable} \\ \lambda_U & \text{if undesirable} \end{cases}$$

Role and implementation of $z_\text{ref}$ (KL Baseline)

$z_\text{ref}$ is the expected KL divergence of the current policy relative to the reference model. In prospect theory it serves as the reference point — rewards above this point are "gains" and rewards below it are "losses," producing the sigmoid's concavity on the gain side (risk aversion) and convexity on the loss side (loss aversion).

In practice, $z_\text{ref}$ is estimated within each mini-batch (size $m$) using mismatched $(x', y'_U)$ pairs:

$$\hat{z}_{\text{ref}} = \max\!\left(0,\,\frac{1}{m}\sum_i \log\frac{\pi_\theta(y'_{U,i} \mid x'_i)}{\pi_{\text{ref}}(y'_{U,i} \mid x'_i)}\right)$$

Deliberately pairing prompt $x'$ with an unrelated output $y'_U$ is intentional, to avoid conflating the reward signal with the baseline estimate. Gradients do not propagate through the $z_\text{ref}$ term.

Prospect Theory Mapping

$v_\text{KTO}$ approximates the Kahneman-Tversky S-shaped value function with a logistic function (the original power-law form is hard to optimize directly): the sign flip — desirable branch $r - z_\text{ref}$; undesirable branch $z_\text{ref} - r$ — precisely simulates the "gain vs. loss" frame switch, and the asymmetric weights $\lambda_D / \lambda_U$ correspond to loss aversion.

"KTO only requires a binary signal of whether an output is (un)desirable for a given input. This data is much more abundant, cheaper, and faster to collect in the real world than preferences." — Ethayarajh et al., arXiv:2402.01306

Citation: Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela — arXiv:2402.01306 (2024)

Properties:

• No pairing required: Each $(x, y)$ only needs a desirable/undesirable label; naturally binary logs such as thumbs-up/down can be used directly
• $z_\text{ref}$ provides a dynamic reference point, making the loss adaptive to KL drift
• Trade-off: Abandons the pairwise consistency constraint of the BT model, potentially lower information efficiency than pairwise methods; $\lambda_D / \lambda_U$ requires manual tuning

7.3 ORPO (Odds Ratio Preference Optimization)

DPO problem addressed: DPO requires two-stage training — first SFT then preference optimization — and requires maintaining a reference model $\pi_{\mathrm{ref}}$.

Core change: Directly attach an odds-ratio preference loss on top of the SFT cross-entropy loss (unified SFT + odds-ratio loss):

$$\mathcal{L}_{\mathrm{ORPO}} = \underbrace{\mathcal{L}_{\mathrm{SFT}}(y_w)}_{\text{SFT on chosen}} + \lambda \cdot \underbrace{\left[-\log\sigma\!\left(\log\frac{\mathrm{odds}_\theta(y_w|x)}{\mathrm{odds}_\theta(y_l|x)}\right)\right]}_{\text{odds ratio preference}}$$

where odds are defined as (odds defined as):

$$\mathrm{odds}_\theta(y|x) = \frac{p_\theta(y|x)}{1 - p_\theta(y|x)}$$

Citation: Jiwoo Hong, Noah Lee, James Thorne — arXiv:2403.07691 (2024)

"In contrast to previous works, our approach requires neither an SFT warm-up stage nor a reference model, enabling resource-efficient development of preference-based aligned models." — Hong et al., arXiv:2403.07691

Properties:

• Single-stage training (Single-stage), no reference model needed (No reference model needed), halves the number of forward passes
• $\mathcal{L}_\text{SFT}$ provides domain-adaptation anchoring on chosen responses; $\mathcal{L}_\text{OR}$ simultaneously repels the rejected style
• Trade-off: The odds ratio has no direct theoretical connection to the BT model; the weight $\lambda$ between the two loss terms requires tuning; numerical results have not been independently verified here

7.4 SimPO (Simple Preference Optimization)

DPO problems addressed: (1) There is a divergence between DPO's implicit reward $\beta\log\pi_\theta/\pi_\text{ref}$ and the metric actually used at generation time (length-normalized likelihood) — Meng et al. note that in UltraFeedback triplets, the proportion of cases where the DPO reward ranking is satisfied but the log-likelihood ranking is reversed approaches half, meaning the model can "win the loss" while making the chosen response harder to generate. (2) Unnormalized log-probabilities decrease monotonically with length, allowing the model to satisfy the ranking by generating shorter rejected responses, introducing length bias. (3) Maintaining a frozen $\pi_\text{ref}$ incurs memory and compute overhead.

Core change: Replace the implicit reward with a length-normalized sequence-level average log-probability, and introduce an explicit target margin $\gamma > 0$ in the Bradley-Terry objective:

$$r_{\text{SimPO}}(x, y) = \frac{\beta}{|y|}\log\pi_\theta(y|x)$$

$$\mathcal{L}_{\text{SimPO}}(\pi_\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\!\left[\log\sigma\!\left(\frac{\beta}{|y_w|}\log\pi_\theta(y_w|x) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l|x) - \gamma\right)\right]$$

where $|y|$ is the token count, $\beta$ is a scaling constant, and $\gamma > 0$ requires the chosen reward to exceed the rejected reward by at least $\gamma$ (not merely be larger). Does not contain $\pi_\text{ref}$.

Citation: Yu Meng, Mengzhou Xia, Danqi Chen — arXiv:2405.14734 (NeurIPS 2024)

"There is a divergence between DPO's reward formulation $r_\theta(x,y)=\beta\log\pi_\theta(y|x)/\pi_\text{ref}(y|x)$ and the average log likelihood metric $p_\theta(y|x)=\frac{1}{|y|}\log\pi_\theta(y|x)$, which directly impacts generation." — Meng et al., arXiv:2405.14734, Section 3.1

Properties:

• No reference model needed; reward directly aligns with generation-time likelihood (removes the source of distribution drift)
• Length normalization eliminates the shortcut of "short rejected response cheating"
• The target margin $\gamma > 0$ reinforces the absolute gap between chosen and rejected, not merely their relative ranking
• Trade-off: The loss does not correspond to a BT probability model; performance numbers (AlpacaEval/Arena-Hard) come from the paper abstract and have not been independently verified here

7.5 Online vs Offline DPO

Distribution Mismatch

Standard DPO is an offline algorithm: the preference dataset $\mathcal{D}$ is collected before training from some fixed data-generating policy $\mu$ (typically the SFT model). During training, $\pi_\theta$ is continuously updated while $\mathcal{D}$ remains static. Once $\pi_\theta$ diverges from $\mu$, the $(y_w, y_l)$ pairs in $\mathcal{D}$ no longer cover the current output distribution of $\pi_\theta$, creating an off-policy distribution mismatch.

Concrete manifestations:

• Implicit reward drift: DPO's implicit reward $\hat{r}_\theta(x,y) = \beta\log\pi_\theta(y|x)/\pi_\text{ref}(y|x)$ changes continuously during training; the scores assigned to the same $(y_w, y_l)$ pair shift as the policy updates, and the chosen/rejected margin may shrink or even reverse.
• Limited exploration: The policy cannot explore outputs better than those in $\mathcal{D}$, becoming stuck in a "local optimum" defined by the old preference data.

Iterative / On-Policy DPO

Solution: at each iteration, sample new response pairs using the current policy $\pi_\theta^{(t)}$, then construct new preference pairs $(y_w^{(t)}, y_l^{(t)})$ via a reward model (or human/AI judge), and update the policy with this batch of distribution-matched preference data:

$$\pi_\theta^{(t+1)} \leftarrow \text{DPO-update}\!\left(\pi_\theta^{(t)},\;\mathcal{D}^{(t)}\right), \quad \mathcal{D}^{(t)} \sim \pi_\theta^{(t)}$$

Why online DPO is generally better:

Practical guidance: When access to an RM or automatic judge is available, iterative DPO (re-sampling + updating preference data every $k$ steps) generally yields better downstream conversation quality than purely offline DPO. If only offline is feasible, removing $\pi_\text{ref}$ (as in SimPO/DPOP) or adding chosen-anchoring can partially mitigate the likelihood displacement caused by distribution drift.

7.6 Precise Comparison Table

Citations: DPO — Rafailov et al., arXiv:2305.18290; IPO — Azar et al., arXiv:2310.12036; KTO — Ethayarajh et al., arXiv:2402.01306; ORPO — Hong et al., arXiv:2403.07691; SimPO — Meng et al., arXiv:2405.14734 (NeurIPS 2024)

8. GRPO vs PPO (Group Relative Policy Optimization vs Proximal Policy Optimization)

8.1 GRPO Group Advantage Estimation

The core idea of GRPO: for the same prompt $x$, sample a group of responses $\{y_1, y_2, \dots, y_G\}$ and estimate the advantage using intra-group statistics (estimate advantage using intra-group statistics):

$$\boxed{A_i = \frac{r_i - \mathrm{mean}(\{r_1, r_2, \dots, r_G\})}{\mathrm{std}(\{r_1, r_2, \dots, r_G\})}}$$

where $r_i = r(x, y_i)$ is the reward for the $i$-th response (reward for the $i$-th response).

GRPO policy gradient loss (GRPO policy gradient loss):

$$\mathcal{L}_{\mathrm{GRPO}}(\theta) = -\frac{1}{G}\sum_{i=1}^{G}\left[\min\!\left(\frac{\pi_\theta(y_i|x)}{\pi_{\theta_{\mathrm{old}}}(y_i|x)} A_i, \;\mathrm{clip}\!\left(\frac{\pi_\theta(y_i|x)}{\pi_{\theta_{\mathrm{old}}}(y_i|x)}, 1-\epsilon, 1+\epsilon\right)A_i\right)\right]$$

Same clipping mechanism as PPO, but entirely different advantage estimation (same clipping mechanism, entirely different advantage estimation).

8.2 Key Comparison

8.3 RLVR Framework (RL from Verifiable Rewards)

The paradigm that best fits GRPO is RLVR: rewards come not from a learned RM but from automatically verifiable rules (rewards from automatically verifiable rules):

• Math: whether the answer equals the ground truth (answer matches ground truth) → $r = \mathbb{1}[\text{extract}(y) = y^*]$
• Code: whether all test cases pass (passes all test cases) → $r = \frac{\text{passed tests}}{\text{total tests}}$
• Format: whether the required format is followed (follows required format) → $r \in \{0, 1\}$

The core advantage of RLVR: low-noise reward (low-noise reward, relative to a learned RM), avoiding the bias and overfitting of the RM itself.

8.4 When to Prefer Which

• Prefer GRPO: rewards are automatically verifiable (math, code, logical reasoning); limited resources (cannot maintain 4 models); need stable training
• Prefer PPO: rewards require semantic/style judgment (dialogue quality, creative writing); reward signal is complex and cannot be rule-based; sufficient compute and a mature RM are available

8.5 RLOO and ReMax (critic-free baselines)

PPO relies on a learned critic (value network) to estimate a baseline. GRPO, RLOO, and ReMax are all critic-free, replacing the value baseline with a baseline computed from sampled rewards.

RLOO (REINFORCE Leave-One-Out, Ahmadian et al. 2024, ACL arXiv:2402.14740): For a group of $G$ samples per prompt, the baseline for sample $i$ is the mean reward of the other $G-1$ samples; advantage $A_i = r_i - \frac{1}{G-1}\sum_{j\neq i} r_j$. It uses a pure REINFORCE gradient, no clipping, no critic; the policy-gradient estimate stays unbiased because the baseline does not depend on sample $i$'s own action.

ReMax (Li et al. 2024, ICML arXiv:2310.10505): The baseline is the reward of a single greedy (argmax) decode for the same prompt; advantage $A = r(\text{sample}) - r(\text{greedy})$. This requires only one extra greedy rollout per prompt, resulting in very low memory overhead and no critic network.

9. Role of KL Penalty & Tuning β

9.1 Intuitive Role of the KL Term

$$\beta \cdot \mathrm{KL}\!\big(\pi_\theta(\cdot|x) \;\|\; \pi_{\mathrm{ref}}(\cdot|x)\big)$$

The KL penalty acts as regularization, with the following functions:

1. Prevents excessive drift: Ensures $\pi_\theta$ does not deviate too far from $\pi_{\mathrm{ref}}$, preserving pre-training knowledge
2. Mitigates reward hacking: If the policy learns to exploit flaws in the RM, the KL term grows as a penalty
3. Maintains diversity: Prevents the policy from collapsing to a few high-reward modes (mode collapse)
4. Stabilizes training: Constrains the exploration space, preventing excessively large policy updates

Expanding mathematically (Expanding mathematically):

$$\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}}) = \mathbb{E}_{y \sim \pi_\theta}\!\left[\log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}\right] \geq 0$$

When $\pi_\theta = \pi_{\mathrm{ref}}$, KL = 0 (no penalty when the policy does not deviate at all from the reference policy).

9.2 KL-RM Score Pareto Frontier

Tuning $\beta$ is fundamentally a trade-off between two objectives:

$$\underbrace{\mathbb{E}[r(x,y)]}_{\text{reward score}\uparrow} \quad \text{vs.} \quad \underbrace{\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})}_{\text{degree of deviation}\uparrow}$$

Typical range (typical range): $\beta \in [0.01, 0.5]$; $\beta = 0.1$ or $\beta = 0.2$ are commonly used in practice.

9.3 β in DPO vs PPO

Conclusion: Theoretically equivalent (theoretically equivalent), practically different (practically different). In DPO, $\beta$ also influences the sharpness of the weight term $\sigma(\cdot)$ in the gradient.

9.4 KL Estimators & Placement

9.4.1 Three Single-Sample Estimators

Notation: Let $r = \pi_{\mathrm{ref}} / \pi_\theta$ (Schulman 2020 convention), with samples drawn from the current policy $\pi_\theta$.

Define three estimators:

$$k_1 = -\log r = \log\frac{\pi_\theta}{\pi_{\mathrm{ref}}}$$

$$k_2 = \tfrac{1}{2}(\log r)^2 = \tfrac{1}{2}\!\left(\log\frac{\pi_{\mathrm{ref}}}{\pi_\theta}\right)^{\!2}$$

$$k_3 = (r - 1) - \log r = \left(\frac{\pi_{\mathrm{ref}}}{\pi_\theta} - 1\right) - \log\frac{\pi_{\mathrm{ref}}}{\pi_\theta}$$

Verifying the expectation (samples from $\pi_\theta$):

$$\mathbb{E}_{a \sim \pi_\theta}[r] = \sum_a \pi_\theta(a) \cdot \frac{\pi_{\mathrm{ref}}(a)}{\pi_\theta(a)} = \sum_a \pi_{\mathrm{ref}}(a) = 1$$

Therefore $\mathbb{E}[r - 1] = 0$: the term $(r-1)$ is zero-mean and serves as a control variate.

• Unbiasedness of $k_1$: $\mathbb{E}_{a \sim \pi_\theta}[k_1] = \mathbb{E}_{a \sim \pi_\theta}\!\left[\log\frac{\pi_\theta}{\pi_{\mathrm{ref}}}\right] = \mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}}) \geq 0$. So $k_1$ is an unbiased estimate of the KL value.

• Bias of $k_2$: $k_2 = \tfrac{1}{2}(\log r)^2 \geq 0$, but its expectation under $\pi_\theta$ does not equal $\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})$, so $k_2$ is biased.

• Unbiasedness of $k_3$ (control-variate argument): For any $\lambda$, define $k_\lambda = -\log r + \lambda(r-1)$. Since $\mathbb{E}[r-1]=0$, we have $\mathbb{E}[k_\lambda] = \mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})$ for all $\lambda$ — universally unbiased. Setting $\lambda = 1$ yields $k_3$. At this choice, the added term $\lambda(r-1)$ is negatively correlated with $-\log r$, reducing variance. Hence $k_3$ is unbiased and, in the small-drift regime ($r \approx 1$) relevant to RLHF, has lower variance than $k_1$.

• Non-negativity of $k_3$: By the tangent-line inequality $\log r \leq r - 1$ for all $r > 0$ (equality only at $r=1$), we have $(r-1) - \log r \geq 0$, consistent with the non-negativity of KL divergence.

Order near $r=1$ (let $\epsilon = r-1$): $k_1 = -\log r \approx -\epsilon$ is first-order (signed), whereas $k_2 \approx k_3 \approx \tfrac{1}{2}\epsilon^2$ are second-order (non-negative). All three vanish as $r \to 1$ but at different rates — which is also why $k_3$ is non-negative like $k_2$ yet unbiased like $k_1$.

9.4.2 The Gradient Perspective

The analysis above concerns value estimation only. When an estimator is used as a loss term, its gradient behavior must be analyzed separately.

• $k_3$ as a loss term does not yield the exact reverse-KL gradient: Although $k_3$ is an unbiased value estimate of KL, differentiating it with respect to $\pi_\theta$ (when used as a loss) produces a gradient that is only a first-order approximation of the true reverse-KL gradient. The approximation holds when policy drift is small ($r \approx 1$, so $\log r \approx r - 1$), but introduces systematic bias as drift grows.

• Per Liu et al. (arXiv:2510.01555, "Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization"): $k_1$ placed in the reward (in-reward) and $k_2$ used as a loss term (as-loss) are gradient-principled choices, while $k_3$ as a loss term lacks gradient-level justification. In practice, the small $\beta$ used in GRPO / DeepSeek-R1 keeps this approximation error minor.

9.4.3 Estimator Comparison Table

Variance note: in the small-drift regime ($r \approx 1$), $k_3$ has lower variance than $k_1$ (both unbiased). $k_2$ is biased and operates in a different bias-variance regime; direct variance comparisons with $k_1$ or $k_3$ are not meaningful.

9.4.4 Two Placement Styles (Style A vs Style B)

Style A: In-Reward

Representative: InstructGPT / PPO. The KL penalty is incorporated per-token into the reward signal:

$$r_{\mathrm{total}}(x, y) = r_{\mathrm{RM}}(x, y) - \beta \cdot k_1^{(t)}, \quad k_1^{(t)} = \log\frac{\pi_\theta(a_t|s_t)}{\pi_{\mathrm{ref}}(a_t|s_t)}$$

• Each token receives an individual KL penalty signal; the Critic can learn stepwise KL costs.
• Caveat: PPO's clip mechanism truncates the policy ratio. For clipped tokens, the surrogate objective no longer depends on $\pi_\theta$, so the KL signal in the reward is silently masked for those tokens at the gradient level — an implementation detail that is easily overlooked.

Style B: In-Loss

Representative: GRPO (Shao et al., DeepSeekMath), DeepSeek-R1. The KL estimator is added directly to the policy optimization loss:

$$\mathcal{L} = \mathcal{L}_{\mathrm{GRPO}} + \beta \cdot k_3$$

where $k_3 = (r - 1) - \log r$, $r = \pi_{\mathrm{ref}} / \pi_\theta$ (computed per token, then averaged over the sequence).

• Requires no Critic / Value model, reducing memory footprint.
• DAPO (arXiv:2503.14476): removes the KL penalty entirely ($\beta = 0$). The stated rationale is that during long-CoT reasoning training the policy diverges substantially from the initial SFT reference, so a tight KL constraint is counterproductive; it instead relies on asymmetric clipping (Clip-Higher: decoupled upper/lower clip bounds) to prevent entropy collapse.

9.4.5 Interview Self-Test

L2: Using the $r = \pi_{\mathrm{ref}}/\pi_\theta$ convention, why does $\mathbb{E}_{a \sim \pi_\theta}[r] = 1$? How does this result establish the unbiasedness of $k_3$?

L3: GRPO incorporates $k_3$ directly into the loss, yet $k_3$ as a loss term does not yield the principled reverse-KL gradient — why is this usually acceptable in GRPO practice? If $\beta$ were increased from 0.04 to 0.5, how would this approximation error change?

10. Process Reward Model (PRM) vs Outcome Reward Model (ORM)

10.1 ORM: Outcome Reward Model

$$r_{\mathrm{ORM}}(x, y) = f_\phi(x, y) \in \mathbb{R}$$

• Gives a single scalar reward only at the end of the sequence (End of Sequence, EOS)
• The entire response $y = (s_1, s_2, \dots, s_T)$ shares the same reward value
• Training data: $(x, y, \text{correct/incorrect})$ binary labels

10.2 PRM: Process Reward Model

$$r_{\mathrm{PRM}}(x, y, t) = f_\phi(x, s_1, \dots, s_t) \in \mathbb{R}$$

• Gives an independent score for each step of the reasoning chain (step-level scoring)
• $r_{\text{step}}(s_t)$ represents the quality of the $t$-th reasoning step
• Training data: $(x, s_1, \dots, s_t, \text{step correct/incorrect})$ step-level labels

10.3 Credit Assignment Advantage

This is the core advantage of PRM. Consider a mathematical reasoning chain:

Step 1: Let $f(x) = x^2 + 3x$ → ✅ correct Step 2: Differentiate to get $f'(x) = 2x + 3$ → ✅ correct Step 3: Set $f'(x) = 0$, solve $x = -3/2$ → ✅ correct Step 4: $f(-3/2) = 9/4 - 9/2 = -9/4$ → ✅ correct

ORM only knows "the final answer is correct" → gives a high score, but does not know whether each step is reliable.

PRM can identify the case where "the first three steps are correct but the fourth is wrong":

$$\underbrace{r_{\mathrm{PRM}}(s_1) > 0,\; r_{\mathrm{PRM}}(s_2) > 0,\; r_{\mathrm{PRM}}(s_3) > 0}_{\text{correct steps}} \quad \underbrace{r_{\mathrm{PRM}}(s_4) < 0}_{\text{erroneous step localized}}$$

This allows PRM to guide search and training more precisely (more precise guidance for search and training).

10.4 Best-of-N Search with PRM

Given prompt $x$, sample $N$ candidate responses $\{y_1, \dots, y_N\}$ and score each step of each response:

$$\text{Score}(y_i) = \min_{t=1}^{T_i} r_{\mathrm{PRM}}(x, y_i, t)$$

or use the product form (product form):

$$\text{Score}(y_i) = \prod_{t=1}^{T_i} \sigma\!\big(r_{\mathrm{PRM}}(x, y_i, t)\big)$$

Taking min or product ensures every step qualifies — any weak step pulls down the overall score (any weak step pulls down the overall score).

Select the best response (Select the best):

$$y^* = \arg\max_{y_i} \text{Score}(y_i)$$

10.5 PRM Training Data Challenges

Automated PRM annotation method: After step $t$, sample completions multiple times and compute the proportion of final answers that are correct as an estimate of $r_{\mathrm{PRM}}(s_t)$. Formula: $r_{\mathrm{PRM}}(s_t) \approx \frac{1}{K}\sum_{k=1}^{K} \mathbb{1}[\text{completion}_k \text{ leads to correct answer}]$

11. Alignment Tax & Weight Averaging

11.1 Definition of Alignment Tax

Alignment Tax refers to the performance degradation on base capability benchmarks after a model undergoes alignment training:

$$\text{Alignment Tax} = \text{Perf}_{\mathrm{base}}(\theta_{\mathrm{base}}) - \text{Perf}_{\mathrm{base}}(\theta_{\mathrm{aligned}})$$

where $\text{Perf}_{\mathrm{base}}$ denotes performance on pre-training benchmarks (e.g., MMLU, coding ability, math ability, etc.).

Intuitively: SFT/RL training may "forget" or "overwrite" parts of pre-trained knowledge while improving alignment quality (safety, helpfulness, format following).

11.2 WiSE-FT Linear Interpolation

WiSE-FT (Weight-space Ensembles for Finetuning) mitigates the alignment tax by interpolating in weight space:

$$\boxed{\theta_{\mathrm{merged}} = (1 - \alpha)\,\theta_{\mathrm{aligned}} + \alpha\,\theta_{\mathrm{base}}}$$

where $\alpha \in [0, 1]$ controls the trade-off between the aligned model and the base model.

11.3 Why Interpolation Works

Task Vector perspective: alignment training is equivalent to moving in a direction within weight space:

$$\tau_{\mathrm{align}} = \theta_{\mathrm{aligned}} - \theta_{\mathrm{base}}$$

Research shows that the weight-change directions corresponding to different tasks are near-orthogonal, so:

$$\theta_{\mathrm{merged}} = \theta_{\mathrm{base}} + (1-\alpha)\,\tau_{\mathrm{align}}$$

Linear interpolation doesn't severely interfere with other task representations.

11.4 Advanced Model Merging Variants

SLERP intuition: the "direction" of a weight vector matters more than its "length"; spherical interpolation preserves the geometric relationship between directions.

12. Catastrophic Forgetting, Mode Collapse & Reward Hacking

These are three distinct training failure modes.

12.1 Catastrophic Forgetting

Definition: when the model learns new behaviors during SFT or RL, it loses knowledge and capabilities acquired during pre-training.

Mechanism:

$$\text{Pre-training capability} \xrightarrow{\text{SFT/RL update } \theta} \text{partially overwritten/erased}$$