Self-improving LLMs

How LLMs use self-generated signals to "score → filter → train" themselves, iterating continuously without large-scale human annotation.

0. The core loop

Generate → Filter / Score → Train → Repeat

Each round, the current policy produces candidate answers or preference pairs; some filtering mechanism (rules, another model, self-scoring) eliminates low-quality outputs; the remaining high-quality samples are used to update weights; the next round reruns with the new model. This self-improvement loop is the shared skeleton of all methods.

TL;DR — quick anchors (2-minute pass)

• Shared skeleton = the bootstrap loop: generate → filter / score → train → repeat; the improvement ceiling is capped by filtering-signal quality, not free capability gains.
• Bootstrap family: STaR (rejection-sample correct traces + hint-retry) / RFT (simplified, leans on multiple correct solutions per problem for diversity) / ReST (Grow-Improve offline, Improve can iterate with stricter thresholds).
• Self-Rewarding: one model both generates and acts as LLM-as-judge, iterative DPO; generation & judgment share parameters and co-evolve—but blind spots get inherited via self-judging.
• Self-Play (SPIN): use "the previous-round self" to produce negatives, DPO-style learn to distinguish genuine human responses, approaching the human distribution until indistinguishable.
• AI Feedback (CAI/RLAIF): SL-CAI self-critique + revise → RL-CAI AI-labeled preferences → RM → RL; the filtering signal comes from constitutional principles (alignment), not answer correctness.
• Inference-time (training-free): Reflexion (verbal reflection into episodic memory, session-only) / Self-Refine (generate-critique-revise, no weight update); improvement is non-persistent and bounded by initial self-judgment ability.
• Training-time vs inference-time: STaR/ReST/SPIN/CAI update weights and persist; Reflexion/Self-Refine don't, and reset on restart.
• Three failure modes: reward hacking (Goodhart) / model collapse · distribution narrowing (keeping only top-k) / RM over-optimization (OOD decoupling); shared mitigation = KL constraint + diverse independent signals.
• The filtering signal is the lifeline: verifiable rules > self-scoring (inherits blind spots); the more independent and verifiable the signal, the less the loop collapses.

1. Bootstrap-then-Train: bootstrapping from correct traces

1.1 STaR — Rejection Sampling + Iterative Fine-tuning

STaR1Iteratively fine-tunes on chain-of-thought where the correct answer was produced, without requiring a large-scale rationale dataset. Zelikman 2022 ↗ (Self-Taught Reasoner) is the foundational scheme for bootstrapped fine-tuning of LLM chain-of-thought:

1. Rollout: sample $K$ chain-of-thought rationales per problem.
2. Filter: retain only those rationales whose final answer is correct (rejection sampling).
3. Fine-tune: SFT on the retained set, update the model.
4. Hint-retry: for problems where all answers are wrong, give the correct answer and ask the model to "re-explain", then mix those into training (prevents easy problems from dominating the training set).

After $T$ iterations, the model is simultaneously the data generator and the data filter.

1.2 RFT — Rejection Sampling Fine-tuning

RFT is a simplified variant of STaR: it skips hint-retry, and directly retains the correctly-answered samples from $K$ samples per problem, aggregating them into a richer fine-tuning set. Key finding: multiple correct solutions to the same problem have higher diversity than a single solution, which helps generalization.

1.3 ReST — Grow-Improve Offline RL Loop

ReST2Generates a large dataset from the current policy (Grow), then filters by reward threshold and fine-tunes (Improve), more sample-efficient than online RLHF. Gulcehre 2023 ↗ splits the loop into two phases:

• Grow: sample from the current policy $\pi_\theta$, build an offline dataset $\mathcal{D}$, score with reward function $r(\cdot)$.
• Improve: fine-tune $\pi_\theta$ on the subset $\mathcal{D}_{\ge\tau}$ where reward exceeds threshold $\tau$.

Key point: the Improve phase can be repeated multiple times (progressively raising $\tau$ for stricter filtering), while Grow only needs occasional refresh — computation is more concentrated compared to online RLHF's per-step sampling.

2. Self-Rewarding: the model as its own judge

Self-Rewarding Language Models3The same model both generates responses and scores them using LLM-as-a-Judge; uses iterative DPO to jointly improve generation and judgment capabilities. Yuan 2024 ↗ breaks the assumption of "requiring an external reward model":

1. Sample multiple responses to the same prompt.
2. The same model scores each response using LLM-as-a-Judge format (score + rationale).
3. Construct preference pairs $(y_w, y_l)$ by score, update with DPO.
4. In the next round, judging ability also improves — both abilities share the same parameters and co-evolve.

The prerequisite of this approach: the model's generation ability and judgment ability must mutually promote rather than contaminate each other. Experiments show this holds for several iterations, but whether long-term degradation occurs remains an open question (see §6 Failure modes).

3. Self-Play: using "the previous-round self" as opponent

SPIN4Current model vs. previous-round model: the latter generates negative samples, the former learns to distinguish them; self-improvement using only SFT data. Chen 2024 ↗ (Self-Play Fine-Tuning) is inspired by game theory:

• Positive samples: human responses $y^*$ in the original SFT dataset.
• Negative samples: outputs $\tilde{y}$ of the previous-round model $\pi_{\theta_{t-1}}$ on the same prompts.
• Objective: the current model $\pi_{\theta_t}$ learns to distinguish genuine human responses from "old-self" outputs, updated with a DPO-like loss.

$$\mathcal{L}_{\text{SPIN}}(\theta_t) = -\mathbb{E}\left[\log\sigma\!\left(\lambda\log\frac{\pi_{\theta_t}(y^*|x)}{\pi_{\theta_{t-1}}(y^*|x)} - \lambda\log\frac{\pi_{\theta_t}(\tilde{y}|x)}{\pi_{\theta_{t-1}}(\tilde{y}|x)}\right)\right].$$

Key point: no additional human preference annotation required — negative samples are entirely provided by the model's own historical versions. As iterations proceed, $\pi_{\theta_t}$ continually approaches the human distribution until convergence when the two become indistinguishable.

4. AI Feedback: letting AI replace human preference labeling

Constitutional AI5Uses a set of "constitutional" principles to guide the model to self-critique and revise outputs; AI-generated preference data replaces human harmlessness annotation (RLAIF). Bai 2022 ↗ (CAI / RLAIF) is currently the most influential approach to "AI replacing human preference":

SL-CAI (supervised phase):

1. Model generates a harmful draft response.
2. Given a constitutional principle (e.g., "avoid discriminatory content"), the model self-critiques.
3. The model revises its response based on the critique.
4. The revised response is used for SFT.

RL-CAI (reinforcement phase): 5. The model scores a pair of responses using AI judgment (which better conforms to the constitution), constructing preference data. 6. Train a reward model with AI-labeled preferences, then iterate with RL.

Difference from STaR/ReST: the filtering signal comes from constitutional principles, not task answer correctness — targeting alignment rather than reasoning ability.

5. Inference-time Self-correction (Training-free)

The following two methods do not update weights; they belong to inference-time self-improvement, conceptually related to the training loops above but different:

5.1 Reflexion — Verbal Reinforcement Learning

Reflexion6Agent converts task feedback into natural language reflections, stores them in episodic memory, and references them on the next attempt — no gradient updates needed. Shinn 2023 ↗ lets the agent in multiple trial-and-error loops:

• Execute task → receive environment feedback (success / failure / error message).
• Generate verbal reflection: summarize in natural language "what went wrong and how to improve next time".
• Store reflection in episodic memory, inject into context in the next round.

Success rate improves significantly after a few iterations — but improvement exists only in the current session's context, and is lost on restart.

5.2 Self-Refine — Generate-Critique-Revise Loop

Self-Refine7The same frozen LLM loops: generate output → self-critique → revise based on critique, no training or additional supervision required, consistently gains across tasks. Madaan 2023 ↗ has a fixed three-step loop:

$$\text{output}_0 \xrightarrow{\text{critique}} \text{feedback}_0 \xrightarrow{\text{refine}} \text{output}_1 \xrightarrow{\cdots}$$

No training, no additional supervision — directly leverages the pretrained model's self-critique capability. Experiments show gains across multiple tasks (code, summarization, dialogue, math), but the ceiling is limited by the model's initial judgment ability.

6. Failure modes

The self-improvement loop looks appealing, but has three structural risks:

6.1 Reward Hacking

When the filtering signal (reward model, LLM scoring, rule filter) is imperfect, the model learns strategies that score high but are not truly correct: shortcut answers, surface-fluent but content-wrong rationales, outputs specifically designed to please the scoring template.

• Root cause: the gap between the optimization target (proxy reward) and the true target (task quality) — Goodhart's Law.
• Mitigation: use diverse, independent evaluation signals; limit the magnitude of a single RL update (KL constraint).

6.2 Model Collapse / Distribution Narrowing

Each round only retains "high-score" samples, eliminating the diversity of low-score samples. After multiple rounds, the training set tends toward uniformity, model output diversity decreases, and generalization worsens. This is especially severe in Self-Rewarding-style "model scores itself" schemes: the model's blind spots are systematically inherited in preference labeling.

$$\text{Diversity}(\pi_{\theta_t}) \le \text{Diversity}(\pi_{\theta_{t-1}}) \quad \text{(if only top-}k\text{ kept per round)}$$

6.3 Reward Model Over-optimization (RM Over-optimization)

The reward model in the RL phase is itself an approximation; as the policy is continuously optimized, the score curve eventually decouples from true quality (the out-of-distribution regions of the reward model are exploited). A KL divergence penalty is the standard mitigation:

$$\mathcal{J}(\theta) = \mathbb{E}[r(y)] - \beta\,\mathrm{KL}[\pi_\theta \,\|\, \pi_{\text{ref}}].$$

Larger $\beta$ keeps the policy closer to the reference policy, but at the cost of more conservative improvement.

7. From-scratch code: STaR-style rejection-sampling fine-tuning loop

"""
STaR-style rejection-sampling fine-tuning loop (illustrative).
Dependencies: transformers, torch — uses GPT-2 for pedagogical demonstration; replace with a larger model for real training.
"""
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from torch.utils.data import Dataset

# ---------- Hypothetical QA data ----------
PROBLEMS = [
    {"question": "What is 3 + 5?",  "answer": "8"},
    {"question": "What is 7 * 6?",  "answer": "42"},
    {"question": "What is 12 - 4?", "answer": "8"},
]

# ---------- Helper: simple answer extraction ----------
def extract_answer(text: str) -> str:
    """Extract the last number from generated text (for demonstration)."""
    import re
    nums = re.findall(r"\d+", text)
    return nums[-1] if nums else ""

# ---------- 1. Rollout: sample K rationales per problem ----------
def rollout(model, tokenizer, problems, K=4, max_new=64, device="cpu"):
    """Returns list of (question, rationale, is_correct)."""
    results = []
    model.eval()
    for prob in problems:
        prompt = f"Question: {prob['question']}\nLet's think step by step:"
        inputs = tokenizer(prompt, return_tensors="pt").to(device)
        with torch.no_grad():
            outputs = model.generate(
                **inputs, max_new_tokens=max_new,
                do_sample=True, temperature=0.8,
                num_return_sequences=K, pad_token_id=tokenizer.eos_token_id,
            )
        for seq in outputs:
            text = tokenizer.decode(seq, skip_special_tokens=True)
            rationale = text[len(prompt):]
            correct = extract_answer(rationale) == prob["answer"]
            results.append({"prompt": prompt, "rationale": rationale, "correct": correct})
    return results

# ---------- 2. Filter: retain only correct rationales ----------
def filter_correct(results):
    return [r for r in results if r["correct"]]

# ---------- 3. Dataset wrapper ----------
class RationaleDataset(Dataset):
    def __init__(self, samples, tokenizer, max_len=128):
        self.tokenizer = tokenizer
        self.data = []
        for s in samples:
            text = s["prompt"] + s["rationale"]
            enc = tokenizer(text, truncation=True, max_length=max_len,
                            padding="max_length", return_tensors="pt")
            input_ids = enc["input_ids"].squeeze()
            self.data.append({"input_ids": input_ids, "labels": input_ids.clone()})

    def __len__(self):  return len(self.data)
    def __getitem__(self, i): return self.data[i]

# ---------- 4. Train: SFT on correct rationales ----------
def finetune(model, tokenizer, samples, output_dir="./star-ckpt"):
    ds = RationaleDataset(samples, tokenizer)
    if len(ds) == 0:
        print("No correct samples — skip this iteration.")
        return
    args = TrainingArguments(
        output_dir=output_dir, num_train_epochs=1,
        per_device_train_batch_size=2, logging_steps=5,
        save_strategy="no", report_to="none",
    )
    Trainer(model=model, args=args, train_dataset=ds).train()

# ---------- 5. STaR main loop ----------
def star_loop(model_name="gpt2", n_iters=3, K=4):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token
    model = AutoModelForCausalLM.from_pretrained(model_name)
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model.to(device)

    for t in range(n_iters):
        print(f"\n=== Iteration {t+1}/{n_iters} ===")
        all_results = rollout(model, tokenizer, PROBLEMS, K=K, device=device)
        correct = filter_correct(all_results)
        print(f"  Correct rationales: {len(correct)} / {len(all_results)}")
        finetune(model, tokenizer, correct)

    return model

if __name__ == "__main__":
    star_loop(n_iters=2, K=4)

The above code is for illustrative purposes only: real STaR uses larger models, longer rationales, and hint-retry as a fallback. The core workflow (sample → filter → finetune → repeat) is consistent with the paper.

8. Frontier: test-time RL (TTRL) & self-evolving agents

The self-improvement in §1–§7 all relies on some supervision signal (correct answers, preference labels, verifiable rewards). Since 2025, two frontiers relax this assumption further: one pushes RL onto fully unlabeled test data (TTRL), the other moves "improvement" out of the weights and into the workflow / skill library (self-evolving agents) — especially friendly to black-box API models.

8.1 Test-Time RL (TTRL)

TTRL12RL on unlabeled test data: sample multiple outputs per question, use the majority-vote answer as a pseudo-label, reward = agreement with that consensus, then a GRPO-style update.Zuo 2504.16084 ↗ (Test-Time Reinforcement Learning) pushes §1's "generate-filter-train" loop to the extreme: no ground-truth at all. The flow — sample multiple outputs for the same test question → majority vote picks the consensus answer as a pseudo-label → reward = whether the output matches the consensus → GRPO-style RL update. Think of it as "RLVR without a verifier": the model's own consistency stands in for an external correctness signal. As reported, gains on math reasoning are large (Qwen-2.5-Math-7B's pass@1 on AIME24 improves by roughly +211% relative).

8.2 Self-evolving agents: moving improvement out of the weights

When an agent = "LLM + workflow + tools + memory," "self-improvement" need not touch the weights — it can change the workflow structure or the skill library:

• Automated workflow optimization (AFlow): AFlow13Represents the agent workflow as a code graph and uses MCTS to search the combination space of "operators" (generate / revise / ensemble / verify), guided by execution/evaluation scores, to auto-iterate toward better workflows (search/evaluation, not an RL reward).Zhang 2410.10762 ↗ writes the workflow as a code graph and uses MCTS to search the operator-combination space, guided by execution/evaluation scores, auto-discovering better control flows — weights untouched, only the workflow changes; it reportedly approaches strong-model performance at very low inference cost on several benchmarks.
• Skill library / lifelong learning (Voyager): Voyager14An LLM agent in Minecraft that crystallizes successful behaviors into executable-code "skills," written into a retrievable skill library for later composition and reuse — no fine-tuning of the base model.Wang 2305.16291 ↗ crystallizes successful behaviors into executable-code skills, stored in a retrievable skill library and directly called / composed on similar future tasks — base model frozen, capability growth deposited entirely in external memory. This is the bridge between §2 "continual learning" and this cheatsheet: a skill library = continual learning without weight updates.

Stratified follow-ups

L1 Foundational

L2 Intermediate

L3 Deep-dive

Deep-dive

Detailed analysis of advanced interview questions. ⚠️ Study notes, not the authors' research. Numbers follow the original papers.

Q1. STaR only retains "correctly answered" samples: what selection bias does this introduce? What is the formal impact on the learned distribution?

Core bias: STaR1 in each round of iteration includes only chain-of-thought traces with a correct final answer in the training set. Formally this is equivalent to:

$$p_{\text{train}}(r \mid x) \propto p_\theta(r \mid x) \cdot \mathbf{1}[\text{answer}(r) = a^*]$$

where $r$ is the rationale, $x$ is the problem, and $a^*$ is the reference answer.

Three structural consequences:

1. Correctness ≠ reasoning quality: a rationale may arrive at the correct answer through luck, shortcuts, or "backward reasoning from the answer", yet the reasoning steps themselves are wrong. Since filtering only looks at the final answer, incorrect reasoning paths are systematically mixed into the training set. This aligns with the motivation for Lightman et al.9 proposing the process reward model (PRM): outcome supervision cannot distinguish "correct reasoning getting the right answer" from "incorrect reasoning getting the right answer".

2. Accumulated distribution shift: the round-$t$ training set is drawn from the conditional distribution of $\pi_{\theta_{t-1}}$, not the true reasoning distribution $p^*(r \mid x)$. After each iteration, $\pi_{\theta_t}$ further deviates from $p^*$, and the "correctness rate" signal of the filter becomes increasingly self-referential.

3. Hard-problem blind spots: for problems the model consistently fails, the filtered training set is empty (hint-retry covers some, but cannot fully compensate). The model receives neither gradient updates nor bootstrapped improvement on these problems, creating a "Matthew effect" — the strong get stronger, hard problems stagnate.

Mitigation directions: PRM scoring each step (rather than only looking at the final answer) can reduce step-level errors; data mixing (retaining original SFT data) prevents complete distribution drift.

Q2. Why does iterative self-training narrow the distribution (model collapse)? Intuition + when does it bite?

Intuition: "retaining only high-score samples" each round is statistically equivalent to truncated sampling — only taking the high-density region of the distribution each time. Over multiple rounds, tail low-probability (but high-diversity) outputs are systematically eliminated.

Shumailov et al.10 analyzed the consequences of recursively training on self-generated data at both theoretical and experimental levels:

• Statistical approximation error: each sampled dataset is finite; tail events are underestimated or missing.
• Function approximation error: limited model capacity further compresses representation of low-frequency patterns.

The two errors accumulate through iteration, causing the distribution to continuously narrow. Intuitively characterized with an inequality:

$$H(\pi_{\theta_t}) \le H(\pi_{\theta_{t-1}}) \quad \text{(if only top-}k\text{ samples kept per round)}$$

Entropy monotonically decreases; outputs trend toward repetition and uniformity.

When it truly bites:

Mitigation: periodically mix in original human data (anchor to prevent drift); raise sampling temperature to preserve diversity; use a diversity reward term to explicitly reward output diversity.

Q3. Why does the judge-generator coupling in Self-Rewarding fail?

In Self-Rewarding3, the same set of parameters acts both as the generator (producing answers) and as the judge (scoring answers). This creates a structural problem: the generator's blind spots are inherited by the judge.

Specific mechanism:

1. Shared blind spots: if the generator is weak at a certain type of reasoning (e.g., counterfactual reasoning), its probability of scoring that reasoning highly is also lower than the true level — because judgment capability and generation capability share the same knowledge foundation. Preference data therefore systematically underestimates this type of capability.

2. Self-Confirmation Bias: the model tends to score answers that "sound like its own style" higher. This is not a random error from hallucination, but a systematic bias — the preference signal itself is pulling the model toward its existing style, forming a positive feedback loop.

3. Correlated error drift: after each DPO update, the generator and judge move synchronously toward the direction of the preference data. If the preference data itself is erroneous (coming from a judge with blind spots), the next round's judge will aggravate the bias in the same direction — errors do not mean-revert but accumulate and amplify.

Formally, let $J_\theta$ be the judgment score function, $G_\theta$ the generation function, both sharing $\theta$. The true quality function is $q^*$. Then:

$$\mathbb{E}[J_\theta(y) - q^*(y)] \ne 0 \quad \text{and is correlated with the bias direction of } G_\theta$$

Contrast: an external reward model (with independent parameters) is at least initially uncorrelated in bias direction with the generator. But it has another problem: out-of-distribution generalization failure (see Q5).

Q4. SPIN converges to the SFT data distribution — why is this an upper bound? What does it mean in practice?

SPIN4's theoretical convergence condition is: if and only if $\pi_{\theta_t} = p_\text{data}$ (the current model is identical to the human SFT data distribution), the loss gradient vanishes and training stops.

This mathematically provides a strict capability upper bound:

$$\text{SPIN limit policy} = p_\text{data} \quad \text{(SFT data distribution)}$$

Corollaries:

1. Cannot surpass SFT data quality: if the SFT data contains errors, biases, or capability blind spots, the model after SPIN convergence will also inherit these defects. SPIN only makes the model "more like the human SFT data" — it cannot discover new capabilities beyond that data.

2. Negative sample quality degrades with iteration: the round-$t$ negative samples are generated by $\pi_{\theta_{t-1}}$; as $\pi_{\theta_t} \to p_\text{data}$, negative sample quality increasingly approaches positive sample quality, and the contrastive signal grows progressively weaker. In practice, this manifests as: large gains in early iterations, diminishing marginal returns in later iterations approaching zero.

3. Fundamental difference from STaR/ReST: STaR-type methods use task correctness as the filtering signal, and can theoretically surpass SFT data (as long as a correct rationale exists, it can be learned). SPIN targets the ability to distinguish from human data, and its ceiling is determined by human data quality.

Practical implication: SPIN is suitable as a tool to "close the SFT gap" (eliminating the gap between the model distribution and human data distribution), but is not suitable as an infinite loop for continuous self-improvement — in later iterations, methods with external verification signals should be switched to.

Q5. Reward Model over-optimization: what is the scaling-law shape of reward rising / true quality falling?

Gao et al.8 systematically studied the relationship between degree of RM optimization (measured by KL divergence $d$) and true quality, finding the curve shape depends on the optimization method:

• Best-of-$N$ sampling: proxy reward roughly grows as $\sqrt{\log N}$ (the paper notes fitting is difficult; this is an approximate description); true quality first rises then plateaus — over-optimization effect is relatively mild.
• RL (policy gradient): proxy reward can continue rising, but true quality monotonically decreases after some $d^*$.

Approximate functional form (fitted in the paper):

$$\text{gold reward} \approx d\,(\alpha - \beta \ln d)$$

where $\alpha, \beta > 0$, $d = \sqrt{D_{\mathrm{KL}}}$, optimal point $d^* = \exp\!\left(\dfrac{\alpha - \beta}{\beta}\right)$ (derived by setting $\mathrm{d}R/\mathrm{d}d = 0$). Beyond $d^*$, true quality decreases as KL increases.

Key scaling conclusion: the coefficients $\alpha, \beta$ smoothly change with RM parameter count — larger RMs have a higher $d^*$, meaning the over-optimization "critical point" arrives later, but a critical point always exists. This means scaling up the RM cannot eliminate over-optimization risk, only delay it.

Intuition: the RM is an approximation fitted on limited data; the policy finds shortcuts in the RM's out-of-distribution regions (high KL regions) that achieve high proxy reward but low true quality. This is Goodhart's Law expressed quantitatively in RL.

Practical defenses: set KL penalty coefficient $\beta$; periodically check with held-out gold reward (e.g., human evaluation or verifiable answers); avoid too many iteration rounds on a single RM.

Q6. Why is a ground-truth verifier (RLVR) safer than a learned judge?

RLVR (Reinforcement Learning with Verifiable Rewards) was systematically applied to mathematical reasoning by DeepSeekMath11: for tasks with deterministic answers (math problems, code unit tests), correctness is checked programmatically as the reward signal, rather than training a reward model.

Safety comparison:

Why exploiting loopholes is easy for RM but hard for verifiers: the RM's out-of-distribution behavior is unconstrained; the policy can find "high-scoring but low-quality" outputs unseen by the RM. Programmatic verifiers only check whether the final result conforms to the specification — the policy cannot cheat on "the specification itself" (specifications are exogenous).

Limitations: the applicable scope of RLVR depends on task verifiability. Tasks like natural language generation, summarization, and creative writing have no unique correct answer and cannot directly use RLVR. Therefore RLVR and learned rewards are not substitutes but complements — prefer RLVR for verifiable tasks; for open generation tasks, RM + KL constraint is necessary.

Q7. When does self-improvement stagnate? What is the role of exploration / diversity? How to empirically distinguish genuine improvement from reward hacking?

Three root causes of stagnation:

1. Filtering signal saturation: when the model can answer all training set problems correctly, rejection sampling produces correct samples nearly identical to existing training data — gradient signal approaches zero.
2. Insufficient exploration after distribution narrowing: as described in Q2, after distribution entropy decreases, the model no longer samples sufficiently diverse rationales, making it difficult to recover from erroneous paths or discover new strategies.
3. Task difficulty exceeds bootstrapping capability: for problems completely beyond the model's ability, no correct samples can be produced through rejection sampling; external curriculum (simpler subproblems, stronger teacher models) is needed.

Role of exploration / diversity: self-improvement is fundamentally an exploitation-exploration tradeoff. Retaining only correct samples each round is pure exploitation; to sustain improvement, the following are needed:

• Higher temperature: sample more diverse paths, trading higher variance for higher probability of hitting new correct paths.
• Diversity reward: add entropy regularization or a diversity term to the optimization objective to prevent mode collapse.
• Curriculum learning: progressively introduce harder problems rather than repeatedly iterating on a fixed set.

Empirical methods for distinguishing genuine improvement from reward hacking:

The gold standard is always: maintain a held-out evaluation set completely untouched by the self-training process, and regularly score with a trusted oracle (human evaluation or verifiable answers). Only if this score consistently rises can genuine improvement be confirmed.

References

All are original sources of foundational load-bearing methods, individually verified (title + arXiv ID). Click superscripts to jump; click ↩ to return.

1. Zelikman et al. STaR: Bootstrapping Reasoning With Reasoning. 2022. arXiv:2203.14465 — Iteratively fine-tunes on correct chain-of-thought, without large-scale rationale annotation. ↩
2. Gulcehre et al. Reinforced Self-Training (ReST) for Language Modeling. 2023. arXiv:2308.08998 — Grow-Improve offline RL loop, more sample-efficient than online RLHF. ↩
3. Yuan et al. Self-Rewarding Language Models. 2024. arXiv:2401.10020 — Same model acts as both generator and LLM-as-Judge; jointly improves both via iterative DPO. ↩
4. Chen et al. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. 2024. arXiv:2401.01335 — SPIN: uses the previous-round self as opponent; self-improvement using only SFT data. ↩
5. Bai et al. Constitutional AI: Harmlessness from AI Feedback. 2022. arXiv:2212.08073 — Constitution-guided self-critique and revision; RLAIF replaces human harmlessness annotation with AI preferences. ↩
6. Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. 2023. arXiv:2303.11366 — Verbal reflections stored in episodic memory; multi-round self-correction without weight updates. ↩
7. Madaan et al. Self-Refine: Iterative Refinement with Self-Feedback. 2023. arXiv:2303.17651 — Frozen model self-loop: generate → critique → revise; no training, consistent gains across tasks. ↩
8. Gao et al. Scaling Laws for Reward Model Overoptimization. 2022. arXiv:2210.10760 — Separation curve of proxy reward vs gold reward as KL increases; RM scaling laws. ↩
9. Lightman et al. Let's Verify Step by Step. 2023. arXiv:2305.20050 — Process supervision (PRM) outperforms outcome supervision (ORM); PRM800K dataset. ↩
10. Shumailov et al. The Curse of Recursion: Training on Generated Data Makes Models Forget. 2023. arXiv:2305.17493 — Recursive training on self-generated data causes distribution tails to vanish (model collapse). ↩
11. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300 — RLVR (RL with verifiable rewards) + GRPO; programmatic verification replaces learned RM. ↩
12. Zuo et al. TTRL: Test-Time Reinforcement Learning. 2025. arXiv:2504.16084 — RL on unlabeled test data with majority-vote pseudo-labels; "RLVR without a verifier," quality hinges on the majority-vote signal (not a strict maj@n ceiling). ↩
13. Zhang et al. AFlow: Automating Agentic Workflow Generation. 2024. arXiv:2410.10762 — MCTS over the operator space of code-ified workflows; leaves weights untouched, optimizes only workflow structure. ↩
14. Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. 2023. arXiv:2305.16291 — Executable-code skill library + lifelong learning; frozen base, capability deposited in external memory. ↩
15. Thaman. Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use. 2026. arXiv:2605.02964 — Reward-hacking exploitation rates across 13 models under tool-mediated evaluation (~0%–14%); single benchmark, very recent preprint, magnitude only. ↩