Online & Iterative DPO Cheatsheet

From offline DPO's distribution shift to on-policy sampling, self-rewarding, self-play, and game-theoretic preference optimization

1. Overview

Standard DPO (Rafailov et al., arXiv:2305.18290, NeurIPS 2023) is an offline algorithm: the preference dataset $\mathcal{D}$ is collected once before training, drawn from some fixed generating policy $\mu$ (usually the SFT model). During training $\pi_\theta$ keeps updating, but $\mathcal{D}$ stays frozen — this is off-policy. Online / iterative DPO changes exactly one thing: every round, re-sample responses and rebuild preference pairs with the current policy $\pi_\theta^{(t)}$, so the training signal always hugs the current policy's output distribution (on-policy).

Offline DPO (off-policy)                   Online / iterative DPO (on-policy)
─────────────────────                      ──────────────────────────
data sampled once from fixed μ (SFT)       re-sample each round from current π_θ^(t)
once π_θ drifts from μ, (y_w,y_l)          preference pairs always match the
no longer cover its outputs                current policy distribution
cheap, reproducible, no online infra  ⇄    expensive (sample+score+train each round), needs annotator
locked to the stale distribution,     ⇄    can explore new outputs, keep approaching
hard to surpass the data                   the optimal policy

1.1 A family tree: two orthogonal axes

Almost all differences between online preference-optimization methods fall on two orthogonal questions: ① Where do responses come from? ② Who labels the preferences?

• §3 covers the minimal loop for "making DPO on-policy" + how preference pairs are constructed (RM vs LLM-judge vs human).
• §4 pulls the annotator into the model itself: self-rewarding (judge yourself), self-play (use your own outputs as negatives).
• §5 upgrades "passive on-policy sampling" into active exploration and game equilibrium (no BT transitivity assumption needed).
• §6 lands on production recipes (Llama-3 six rounds, Tülu-3) and the pitfalls specific to the loop.

1.2 Scope of this page

2. The Case for On-Policy

2.1 Three failure modes of offline DPO

(a) Distribution mismatch. Once $\pi_\theta$ leaves $\mu$ during training, the $(y_w, y_l)$ in $\mathcal{D}$ fall into regions $\pi_\theta$ now rarely produces. DPO widens the margin on these stale samples, while giving no supervision at all on the outputs the current policy actually generates — the gradient is spent on "paths it will never take again."

(b) Likelihood displacement. The DPO loss only constrains the difference of log-ratios between chosen and rejected, $\log\frac{\pi_\theta(y_w)}{\pi_\text{ref}(y_w)} - \log\frac{\pi_\theta(y_l)}{\pi_\text{ref}(y_l)}$; it does not directly constrain the absolute value of $\log\pi_\theta(y_w)$. Note this is not inevitable for an isolated pair update — in the single-step softmax update of §3.1, $y_w$'s probability actually rises. But in real-LM shared-parameter aggregate training (massive conflicting preference pairs, sequence-level normalization, optimizer dynamics stacked together), a "shortcut to satisfaction" emerges: push $y_l$ down harder while $y_w$'s probability is dragged down too — as long as it drops more slowly than $y_l$, the margin still grows. The squeezed-out probability mass often flows toward a third class of OOD outputs (neither $y_w$ nor $y_l$). On-policy data makes $y_w$ come from the current distribution to begin with, mitigating this "push the good answer down too" degeneration.

(c) OOD over-optimization. DPO's implicit reward $\hat r_\theta = \beta\log\frac{\pi_\theta}{\pi_\text{ref}}$ is a reparameterization of the current/reference policy ratio, not an independent RM that "extrapolates." The problem: regions the data does not cover lack preference supervision to calibrate this ratio, offline training can't reach them, and the policy may drift toward "implicit reward inflated but actually bad." On-policy re-sampling keeps pulling "where the policy really goes now" back into the labeling loop.

2.2 Online vs offline: where the performance gap comes from (Tang et al. 2024)

Tang et al. ("Understanding the Performance Gap between Online and Offline Alignment Algorithms", arXiv:2405.08448, preprint) ran a set of controlled empirical / mechanistic studies (experiments + ablations, not a theorem proof), systematically asking "why is online consistently better than offline":

• Phenomenon: under controlled settings, online algorithms consistently beat offline algorithms, and the gap cannot be closed merely by "feeding offline methods more data / expanding coverage."
• Ruled out: they checked and rejected several naive explanations one by one — the gap is not simply due to insufficient discriminative accuracy of offline methods, nor determined by the loss-function form (contrastive offline loss vs online RL); even when offline methods use a strong contrastive loss, the gap persists.
• Core attribution: on-policy sampling itself (data generated by the current policy) is the key driver, not the "offline/online algorithm" label. Offline methods underperform precisely on the responses "they should have learned to distinguish."

3. Online DPO Algorithms

3.1 The Iterative Loop

To turn offline DPO into on-policy, the minimal loop is just three steps, repeated round by round:

for t in 0..T-1:
   1) Generate: for each prompt x, sample K responses from the current policy π_θ^(t)   ← on-policy
   2) Label:    use RM / LLM-judge / human to rank the K, build preference pairs (y_w,y_l) → D^(t)
   3) Update:   π_θ^(t+1) ← DPO-update(π_θ^(t), D^(t);  ref = π_ref)

Formally, as given in llm-post-training §7.5:

$$\pi_\theta^{(t+1)} \leftarrow \text{DPO-update}\!\left(\pi_\theta^{(t)},\;\mathcal{D}^{(t)}\right), \quad \mathcal{D}^{(t)} \sim \pi_\theta^{(t)}$$

The toy code below uses a discrete response space to show why on-policy preference pairs keep pushing the policy toward high-reward regions, while one-shot offline data "goes stale." DPO's gradient on a softmax policy has a clean closed form — the logsumexp term cancels, so the gradient of the margin w.r.t. the logits is just $\beta(e_w - e_l)$:

import numpy as np

# ===== DPO on a toy categorical policy =====
# logπ_i = θ_i - logsumexp(θ);  d(logπ_w - logπ_l)/dθ_j = [w==j] - [l==j]
# so the gradient of the margin w.r.t. logits is independent of logsumexp -- a clean closed form.

def log_softmax(theta):
    m = theta.max()
    return theta - m - np.log(np.exp(theta - m).sum())

def softmax(theta):
    z = theta - theta.max()
    e = np.exp(z)
    return e / e.sum()

def dpo_step(theta, theta_ref, w, l, beta=0.5, lr=0.3):
    """One DPO gradient-descent step on a preference pair (w wins, l loses); returns updated logits."""
    lp, lpr = log_softmax(theta), log_softmax(theta_ref)
    margin = beta * ((lp[w] - lpr[w]) - (lp[l] - lpr[l]))
    sig = 1.0 / (1.0 + np.exp(-margin))      # σ(margin)
    coef = (sig - 1.0) * beta                # dL/dmargin = σ(margin) - 1 < 0
    grad = np.zeros_like(theta)
    grad[w] += coef                          # raise w
    grad[l] -= coef                          # lower l
    return theta - lr * grad

def expected_reward(theta, r):
    return float((softmax(theta) * r).sum())

def make_pair(samples, r):
    """From a batch of samples, take (best, worst) by true reward as the preference pair."""
    s = sorted(samples, key=lambda i: r[i])
    return s[-1], s[0]

# true reward: 5 discrete responses, index 4 is best
r = np.array([0.0, 0.2, 0.4, 0.6, 1.0])
theta0 = np.zeros(5)                          # SFT start: uniform
rng = np.random.default_rng(0)

# --- offline: preference pairs sampled once from π0, then reused repeatedly ---
off = theta0.copy()
pool = [make_pair(rng.choice(5, size=2, p=softmax(theta0), replace=False), r)
        for _ in range(40)]
for w, l in pool:
    off = dpo_step(off, theta0, w, l)

# --- online: re-sample preference pairs from the current policy at every step ---
on = theta0.copy()
for _ in range(40):
    w, l = make_pair(rng.choice(5, size=2, p=softmax(on), replace=False), r)
    on = dpo_step(on, theta0, w, l)           # ref fixed to SFT

print("offline E[r] =", round(expected_reward(off, r), 3))
print("online  E[r] =", round(expected_reward(on, r), 3))

3.2 How to build preference pairs: RM vs LLM-judge vs human

After sampling $K$ on-policy responses, who decides $(y_w, y_l)$ determines the method's cost and bias:

OAIF (Guo et al., "Direct Language Model Alignment from Online AI Feedback", arXiv:2402.04792, preprint) is the representative of "LLM-judge online labeling": for each step's two responses sampled from the current policy, an online annotator LLM judges which is better on the spot, then a DPO update follows — replacing offline DPO's "static preference set" with "online AI feedback," getting both on-policy and RM-free.

3.3 Reference policy and β: two knobs in the loop

In the iterative loop, the settings of $\pi_\text{ref}$ and $\beta$ directly determine stability:

• Reset $\pi_\text{ref}$ each round vs fix it to SFT.
  • Reset each round $\pi_\text{ref} \leftarrow \pi_\theta^{(t)}$: a trust-region-like step, the KL constraint is always "relative to the previous round," updates are more stable; the cost is losing the anchor to SFT, so after many rounds the whole thing may drift away.
  • Fix SFT as $\pi_\text{ref}$: always anchored to the start; but the more $\pi_\theta$ drifts, the larger $\frac{\pi_\theta}{\pi_\text{ref}}$ becomes, the looser the effective constraint, and the easier over-optimization gets late on.
  • Production practice (Llama-3 / Tülu-3) mostly fixes ref within a round and swaps the baseline between rounds.
• $\beta$ (KL strength). Larger $\beta$ hugs $\pi_\text{ref}$ more, more conservative; smaller $\beta$ dares to drift more, easier to over-optimize. For the semantic contrast between $\beta$ and ref, see llm-post-training §9.3.

4. Self-Rewarding & Self-Play

§3's annotators are all "external." This section pulls the annotator into the model itself — the benefit is escaping the external RM/human bottleneck; the risk is that the signal and the policy are homologous, easily self-reinforcing bias.

4.1 Self-Rewarding LMs

Self-Rewarding LMs (Yuan et al., arXiv:2401.10020, ICML 2024) make one model serve as both policy and judge: an LLM-as-a-Judge prompt has the model score its own sampled responses, builds preference pairs from that, then runs iterative DPO. The key narrative is that judging ability rises together with policy ability — the paper observes instruction-following and "being a judge" improving in lockstep across iterations, forming a self-improvement loop (the paper runs three iterations, $M_1\!\to\!M_2\!\to\!M_3$).

# ===== Self-Rewarding: the model judges itself and builds preference pairs (§4.1) =====
def self_reward_pairs(prompt, gen_fn, judge_fn, k=4):
    """Sample k, score them yourself, take (highest, lowest) as the pair; if all tie, no signal -> skip."""
    cands = [gen_fn(prompt) for _ in range(k)]
    scored = sorted(((judge_fn(prompt, c), c) for c in cands),
                    key=lambda t: t[0], reverse=True)
    if scored[0][0] == scored[-1][0]:
        return None                          # no discrimination, skip this prompt
    return scored[0][1], scored[-1][1]       # (y_w, y_l)

4.2 Self-play: SPIN and SPPO

SPIN (Self-Play Fine-Tuning, Chen et al., arXiv:2401.01335, ICML 2024) needs no preference labels and no external reward: treat the SFT human data as $y_w$ (positive) and the model's own current generations as $y_l$ (negative), training the model with a DPO-style contrastive objective to distinguish "human data" from "its own outputs." This is a discriminator/generator self-play — it converges when the model's generations become indistinguishable in distribution from the SFT data ($\pi \to p_\text{data}$).

# ===== SPIN: self-play pairing -- human data = win, model's own sample = lose (§4.2) =====
def spin_pairs(prompts, human_responses, model_gen_fn):
    """y_w taken from SFT human data, y_l from the current model; if identical, no signal -> skip."""
    pairs = []
    for x, y_human in zip(prompts, human_responses):
        y_self = model_gen_fn(x)
        if y_self != y_human:
            pairs.append((x, y_human, y_self))   # (prompt, y_w, y_l)
    return pairs

SPPO (Self-Play Preference Optimization, Wu et al., arXiv:2405.00675, preprint / NeurIPS 2024 Workshop) models alignment as a two-player constant-sum game, targeting the Nash equilibrium of preferences: each round it estimates the win-rate between self-sampled responses with a preference model and takes a multiplicative-weights / quadratic update toward the equilibrium policy. It does not assume preferences are explained by a single scalar reward (BT) — which leads us to §5's game-theoretic view.

5. Exploration & Game-Theoretic PO

5.1 Why "game-theoretic": preferences may be intransitive

BT / reward-based assumptions hold that preference probability is explained by a difference of scalar rewards (one scalar reward per response), so preferences are transitive in expectation. But real human preferences may be intransitive: a cycle $A\succ B\succ C\succ A$ appears, which no scalar reward can express. Game-theoretic preference optimization sidesteps this assumption: rather than finding a "reward-maximizing" policy, it finds the Nash equilibrium of the two-player preference game (maximizing the minimum win-rate against all opponents).

Nash-LHF / Nash-MD (Munos et al., "Nash Learning from Human Feedback", arXiv:2312.00886, ICML 2024): first learn a preference model $\mathcal{P}(y\succ y'\mid x)$ (rather than a reward model), then use Nash-MD (mirror descent) to iteratively solve for the Nash equilibrium of the regularized game, with provable convergence in the tabular / regularized setting. It generalizes RLHF from "reward maximization" to "preference-game equilibrium"; §4.2's SPPO is a self-play instance of the same idea.

5.2 Active exploration: XPO

Passive on-policy sampling just "samples randomly from the current policy," and does not deliberately explore high-potential but uncertain regions. XPO (Exploratory Preference Optimization, Xie et al., arXiv:2405.21046, ICLR 2025) adds just one optimism bonus to the DPO objective, encouraging the policy to explore responses "whose implicit reward might be high but is currently uncertain"; via implicit $Q^\*$-approximation it is provably sample-efficient under its theoretical assumptions. In one line: XPO = online DPO + one line of optimistic exploration, upgrading "lucky-dip sampling" into "directed exploration."

5.3 Active Querying

When the labeling budget is limited, which prompts / which pairs to label can also be optimized: prioritize labeling pairs where the RM is uncertain / information gain is large, rather than labeling uniformly. For combining this with RM uncertainty estimation, see reward-modeling-eval.

6. Practical Recipes & Pitfalls

6.1 Production-scale iterative recipes

• Llama 3 (Grattafiori et al., "The Llama 3 Herd of Models", arXiv:2407.21783, preprint): post-training runs six rounds of iteration, each round = reward modeling + rejection sampling + SFT + DPO; each round's preference data is generated by the best model from the previous round and labeled by humans — a production-scale example of iterative DPO.
• Tülu 3 (Lambert et al., "Tülu 3: Pushing Frontiers in Open Language Model Post-Training", arXiv:2411.15124, COLM 2025): an open SFT → DPO → RLVR recipe, where DPO uses a large-scale on-policy preference mix (sample completions from the policy model, then label preferences with models such as GPT-4). For RLVR details see reasoning-rl-frontier.
• Iterative Reasoning PO (Pang et al., "Iterative Reasoning Preference Optimization", arXiv:2404.19733, NeurIPS 2024): iterative DPO for reasoning / CoT — it sets $y_w$/$y_l$ by whether the answer is correct (a verifiable signal), and adds an extra NLL/SFT term on $y_w$ in the loss to suppress likelihood displacement (§2.1b), improving round by round on GSM8K / MATH and the like. A template for "iterative DPO + verifiable signal + chosen anchor."

6.2 Pitfalls specific to the loop

6.3 Online DPO vs online RLHF (PPO / GRPO)

Both are on-policy; the difference is "how the reward signal enters the update":

7. Interview Questions

L1 — Foundational

L2 — Intermediate

L3 — Advanced

Appendix: Key Terms Glossary

This cheatsheet is for study reference only. Paper conclusions and figures defer to the original papers; benchmark scores are illustrative only and do not constitute a head-to-head comparison.

§A Key Papers Timeline

• 2023-05 · Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov et al., NeurIPS 2023. arXiv:2305.18290 — reparameterizes RLHF's reward maximization into a pairwise classification loss on the policy, dispensing with an explicit RM and RL; offline DPO is the baseline for all online/iterative methods on this page.

• 2023-09 · Statistical Rejection Sampling Improves Preference Optimization — Liu et al., ICLR 2024. arXiv:2309.06657 — points out that the ideal preference pair should be sampled from the optimal-policy distribution, and uses rejection sampling to approximately draw samples close to π* from π_ref before labeling (RSO), nudging the offline data source toward the on-policy ideal.

• 2023-12 · Nash Learning from Human Feedback — Munos et al., ICML 2024. arXiv:2312.00886 — learns a preference model rather than a reward model, uses Nash-MD (mirror descent) to solve the Nash equilibrium of the regularized game, generalizing RLHF from reward maximization to a preference game, with no BT transitivity assumption.

• 2024-01 · Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models — Chen et al., ICML 2024. arXiv:2401.01335 — SPIN: discriminative self-play with SFT human data as winner and the model's own samples as loser, needing no preference labels or external reward; converges when generations are indistinguishable from the data, with the SFT data distribution as the fixed point.

• 2024-01 · Self-Rewarding Language Models — Yuan et al., ICML 2024. arXiv:2401.10020 — one model serves as both policy and uses LLM-as-judge to score its own responses to build preference pairs, iterating DPO; judging ability rises in lockstep with policy ability (three rounds M1→M2→M3).

• 2024-02 · Direct Language Model Alignment from Online AI Feedback — Guo et al., preprint. arXiv:2402.04792 — OAIF: for each step's two responses sampled from the current policy, an online annotator LLM judges which is better on the spot before a DPO update, replacing the static preference set with online AI feedback, getting both on-policy and RM-free.

• 2024-04 · Iterative Reasoning Preference Optimization — Pang et al., NeurIPS 2024. arXiv:2404.19733 — iterative DPO for CoT reasoning: sets winner/loser by answer correctness (a verifiable signal), adds an NLL term on the chosen in the loss to suppress likelihood displacement, improving round by round on GSM8K/MATH and the like.

• 2024-05 · Self-Play Preference Optimization for Language Model Alignment — Wu et al., preprint / NeurIPS 2024 Workshop (AFM, Oral). arXiv:2405.00675 — SPPO: models alignment as a two-player constant-sum game, using preference win-rates for multiplicative-weights/quadratic updates toward the Nash equilibrium, not relying on a single scalar reward (BT).

• 2024-05 · Understanding the Performance Gap between Online and Offline Alignment Algorithms — Tang et al., preprint. arXiv:2405.08448 — controlled study: online consistently beats offline, the gap cannot be closed and is not determined by the loss form, the core attribution being on-policy sampling itself — making the data on-policy matters more than swapping the loss.

• 2024-05 · Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF — Xie et al., ICLR 2025. arXiv:2405.21046 — XPO: adds one optimistic exploration bonus to the DPO objective, encouraging exploration of high-potential uncertain responses, and via implicit Q*-approximation proves sample efficiency under its theoretical assumptions; upgrades passive on-policy into active exploration.

• 2024-07 · The Llama 3 Herd of Models — Grattafiori et al., preprint. arXiv:2407.21783 — post-training runs six rounds of iteration (each round RM + rejection sampling + SFT + DPO), with preference data generated by the previous round's best model and labeled by humans; a production-scale example of iterative DPO.

• 2024-11 · Tülu 3: Pushing Frontiers in Open Language Model Post-Training — Lambert et al., COLM 2025. arXiv:2411.15124 — an open end-to-end recipe SFT → DPO → RLVR, with the DPO stage using a large-scale on-policy preference mix; fully public data/code/recipe.