Post-Training Data Pipeline & Curation Cheatsheet

From instruction synthesis, rejection sampling, and preference construction to decontamination and data mixing

1. Pipeline Overview

The ceiling of post-training (SFT / preference alignment / RLVR) is largely set by the data pipeline — with the same algorithm, different data quality and mixtures give wildly different results. A typical post-training data pipeline:

Sources                  Generate                    Clean                    Mix → Consume
─────────────            ──────────────────          ─────────────            ────────────────────
human annotation         Self-Instruct / Evol        dedup                    mixing
open datasets      ─→    rejection sampling    ─→    decontamination    ─→    curriculum          ─→  SFT
distillation             preference pairs            quality filter           quality/difficulty       DPO / RLHF / RLVR
production logs          Magpie / synth textbooks    format normalize         per-stage mixing

1.1 Three main tracks

1.2 Core principle: quality > quantity

LIMA (Zhou et al., arXiv:2305.11206) fine-tunes a 65B LLaMA on only 1,000 carefully curated SFT samples (no RLHF) and matches models trained on orders of magnitude more data. Its claim: alignment is mostly a "surface behavior/format" learned atop pretraining knowledge, and a small set of high-quality samples suffices to activate it.

Implication: the pipeline's value is not in "piling up volume" but in synthesizing diverse high-quality candidates → strictly filtering/dedup/decontaminating → mixing by target capability. §2–§6 unpack each stage.

2. Instruction Data Synthesis

Writing instruction data by hand is expensive and hard to scale. The mainstream approach is to synthesize with a strong LLM; the core challenges are diversity and quality.

2.1 Self-Instruct — seed bootstrapping

Self-Instruct (Wang et al., arXiv:2212.10560): starting from a small set of human seed tasks, few-shot prompt the model to generate new instructions + inputs + outputs, then filter out samples too similar to the existing pool or low quality, iterate, and finally fine-tune the same model on the expanded data. Stanford Alpaca (no arXiv paper; blog/repo) used the Self-Instruct idea to distill 52k instructions from text-davinci-003.

De-duplication filtering is key: Self-Instruct uses a ROUGE-L similarity < 0.7 threshold to drop new instructions too similar to existing ones, preventing the synthetic data from collapsing into a few templates.

import random
from difflib import SequenceMatcher

# Self-Instruct-style bootstrap: few-shot prompt an LLM to generate new [instructions],
# then filter by similarity against the existing pool to de-duplicate. (The original also
# generates input/output instances per instruction; here we only demonstrate the
# "instruction generation + dedup" step.)
def sim_ratio(a: str, b: str) -> float:
    # Cheap similarity: difflib sequence-match ratio as a ROUGE-L proxy (original uses ROUGE-L < 0.7)
    return SequenceMatcher(None, a, b).ratio()

def self_instruct_round(pool, llm_generate, k=20, sim_thresh=0.7):
    """pool: existing instructions (list); llm_generate(prompt) -> new instruction text."""
    new_items = []
    for _ in range(k):
        demos = random.sample(pool, min(6, len(pool)))           # few-shot seeds
        prompt = "Generate a new task instruction different from these examples:\n" + "\n".join(demos)
        cand = llm_generate(prompt).strip()
        # Filter 1: too similar to any existing instruction -> drop
        if any(sim_ratio(cand, p) > sim_thresh for p in pool + new_items):
            continue
        # Filter 2: heuristic quality gate (character length; works for CN & EN)
        if not (5 <= len(cand) <= 300):
            continue
        new_items.append(cand)
    pool.extend(new_items)
    return new_items

2.2 Evol-Instruct — instruction evolution

WizardLM / Evol-Instruct (Xu et al., arXiv:2304.12244, ICLR 2024): instead of flat generation, use an LLM to "evolve" seed instructions to be more complex, in two directions:

• In-depth: add constraints, add reasoning steps, complicate inputs, deepen difficulty.
• In-breadth: generate brand-new instructions covering new domains from existing ones.

After evolving, filter out "failed evolutions" (didn't get harder / became empty), yielding an instruction set with a steeper difficulty gradient.

2.3 Magpie — "zero-prompt" extraction from aligned models

Magpie (Xu et al., arXiv:2406.08464, ICLR 2025): exploits the autoregressive nature of instruction models (e.g. Llama-3-Instruct) — feed only the "pre-user-turn template" (no seed instruction) and the model continues by writing a user instruction itself; then have it answer, yielding an (instruction, response) pair. No seeds, no human curation, scalable extraction of alignment data.

2.4 Distillation

Using a stronger teacher model to generate data for training a student is one of the most common synthesis routes.

• Sequence-Level KD (SeqKD) (Kim & Rush, arXiv:1606.07947, EMNLP 2016): the classic approach — train the student on sequences decoded by the teacher (replacing word-level soft targets with sequence-level outputs); the student is faster and competitive.
• Synthetic textbooks: Textbooks Are All You Need (Gunasekar et al., arXiv:2306.11644) trains the 1.3B phi-1 on "textbook-quality" filtered web data + GPT-3.5-synthesized textbooks and exercises; the paper reports HumanEval pass@1 = 50.6% — showing high-quality synthetic data can drastically lower the scale requirement.

2.5 Risks of synthetic data: homogenization and mode collapse

3. Rejection Sampling & Self-Training

When a task has a verifiable correctness signal (math answers, unit tests, format), you can let the model generate and filter its own data, then train on it supervised — a cost-effective middle road between SFT and RL.

3.1 RFT — rejection sampling fine-tuning

Rejection sampling Fine-Tuning (RFT) (Yuan et al., arXiv:2308.01825): for each prompt, sample N reasoning solutions from the current (SFT) model, keep only those with the correct answer, and dedup the reasoning paths (keep several "distinct" correct paths per question to add diversity); use these self-generated correct samples as augmented SFT data.

import re

# RFT: for each prompt sample N CoT solutions, keep only those with a CORRECT answer
# and a DISTINCT reasoning path, as augmented SFT data (Yuan et al. 2023).
def canonical(y: str) -> str:
    # Very crude toy fingerprint: digits->#, collapse whitespace
    # (the RFT paper dedups by distinct equation/reasoning paths — far finer).
    return re.sub(r"\s+", " ", re.sub(r"\d+", "#", y)).strip()

def rejection_sampling_ft_data(prompts, policy_sample, verify, gold,
                               N=64, max_keep=4):
    """policy_sample(x) -> one solution; verify(y, gold) -> bool."""
    sft_data = []
    for x in prompts:
        seen, kept = set(), []
        for _ in range(N):
            y = policy_sample(x)
            if not verify(y, gold[x]):          # wrong answer -> reject
                continue
            key = canonical(y)
            if key in seen:                      # same reasoning path -> dedup
                continue
            seen.add(key); kept.append(y)
            if len(kept) >= max_keep:            # at most max_keep per prompt
                break
        sft_data += [(x, y) for y in kept]
    return sft_data                              # then run standard SFT on these

3.2 STaR — bootstrapping reasoning with reasoning

STaR (Zelikman et al., arXiv:2203.14465, NeurIPS 2022): have the model generate CoT reasoning, keep the reasoning that reaches the correct answer for fine-tuning; for wrong questions, give the correct answer and let the model "rationalize" a reasoning path that reaches it, also adding it to the training set. Reasoning ability bootstraps upward over iterations.

3.3 ReST — Grow / Improve double loop

ReST (Gulcehre et al., arXiv:2308.08998): writes self-training as two alternating steps —

• Grow: generate a batch of samples offline from the current policy (expand the dataset);
• Improve: fine-tune on samples that pass a quality threshold (which can be raised gradually).

More compute-efficient than online RLHF: sampling and training are decoupled, and the generated data can be reused across rounds.

3.4 Relationship to RL

4. Preference Data Construction

Preference triples (prompt, chosen, rejected) are the fuel for RLHF/DPO; their quality caps the RM/policy.

4.1 Where preference signals come from

• Constitutional AI (Bai et al., arXiv:2212.08073): two stages — first SFT on the model's self-critiques + revisions, then RLHF with AI-generated preference labels driven by a set of "constitutional principles" (RLAIF), with harmlessness barely relying on human preferences.
• RLAIF vs RLHF (Lee et al., arXiv:2309.00267, ICML 2024): shows that off-the-shelf LLM preference labels (RLAIF) can match RLHF on summarization/dialogue, and proposes direct-RLAIF (obtain the reward directly from the LLM during RL, no separate RM).

4.2 UltraFeedback / Zephyr — scaled AI-feedback recipes

• UltraFeedback (Cui et al., arXiv:2310.01377, ICML 2024): for about 64k instructions, generate about 256k candidate responses with multiple models, then have GPT-4 give numerical scores + textual critiques across several quality dimensions (over 1M feedback entries in total), constructing large-scale preference/RM data.
• Zephyr (Tunstall et al., arXiv:2310.16944): dDPO (distilled DPO) runs DPO directly on UltraFeedback's AI feedback, distilling alignment with no human annotation.

4.3 Building preference pairs from scored candidates

import itertools, random

# Build preference pairs from scored candidates: chosen = higher score, rejected = lower,
# keeping only pairs with a large enough score margin to avoid "hard-to-tell" noise
# (UltraFeedback-style).
def build_preference_pairs(prompt, candidates, scores, margin=1.0, max_pairs=4):
    # candidates: list[str]; scores: list[float] (GPT-4 / RM scores)
    assert len(candidates) == len(scores), "candidates and scores must align (zip truncates silently)"
    ranked = sorted(zip(candidates, scores), key=lambda t: t[1], reverse=True)
    pairs = []
    for (y_w, s_w), (y_l, s_l) in itertools.combinations(ranked, 2):
        if s_w - s_l < margin:           # margin too small -> hard to tell -> drop
            continue
        pairs.append({"prompt": prompt, "chosen": y_w,
                      "rejected": y_l, "margin": s_w - s_l})
    random.shuffle(pairs)
    return pairs[:max_pairs]             # cap pairs per prompt to avoid imbalance

4.4 on-policy vs off-policy preferences

5. Dedup & Decontamination

5.1 Why deduplicate

Deduplicating Training Data Makes Language Models Better (Lee et al., arXiv:2107.06499, ACL 2022): removing near-duplicate spans/documents from training corpora reduces verbatim memorization by roughly an order of magnitude, while reaching comparable or better perplexity with fewer training steps. Same for post-training: duplicate samples are over-weighted, amplifying memorization and bias.

5.2 exact vs near-dup

• Exact dedup: hash whole spans/documents, drop identical ones.
• Near-dup: use MinHash + LSH to detect text with high Jaccard similarity.

Jaccard similarity: $J(A,B)=\dfrac{|A\cap B|}{|A\cup B|}$, where $A,B$ are the k-shingles (sets of k consecutive tokens) of two documents.

Key MinHash property: for a random hash (permutation) $h$,

$$\Pr[\,\min_{x\in A} h(x)=\min_{x\in B} h(x)\,]=J(A,B)$$

i.e. "the probability that two documents' signatures match at a given slot = their Jaccard". With $m$ independent hashes forming a signature, the fraction of matching slots is an unbiased estimate of $J$, reducing the $O(|A||B|)$ pairwise comparison to a constant-length signature comparison.

import hashlib

# MinHash near-duplicate detection: k-shingles + min over multiple hashes form a signature;
# the fraction of matching signature slots approximates the Jaccard similarity.
def shingles(text, k=5):
    toks = text.split()
    return {" ".join(toks[i:i+k]) for i in range(max(1, len(toks) - k + 1))}

def minhash_sig(sh_set, num_perm=128):
    sig = []
    for s in range(num_perm):                       # each s simulates an independent hash
        m = min(int(hashlib.md5((str(s) + sh).encode()).hexdigest(), 16)
                for sh in sh_set)                   # min value under this hash
        sig.append(m)
    return sig

def est_jaccard(sig_a, sig_b):
    return sum(a == b for a, b in zip(sig_a, sig_b)) / len(sig_a)

def is_near_dup(text, corpus_sigs, k=5, num_perm=128, thresh=0.8):
    sig = minhash_sig(shingles(text, k), num_perm)
    # brute-force linear scan over corpus_sigs here; in production add LSH banding to compare only same-bucket candidates
    dup = any(est_jaccard(sig, s) >= thresh for s in corpus_sigs)
    return dup, sig                                 # drop only true dups; otherwise add sig to the index

5.3 Decontamination

Definition: remove training samples that overlap with evaluation benchmarks (test sets); otherwise the model "has seen the exam questions" and benchmark scores are inflated and meaningless.

• n-gram overlap: if a training sample shares a long enough n-gram with a benchmark test item (e.g. GPT-3 used 13-grams), flag it as contaminated and remove it. Simple, but insensitive to paraphrase/translation leakage.
• Membership inference: Min-K% Prob (Shi et al., arXiv:2310.16789, ICLR 2024) — training-free, judges whether a text was seen during pretraining: take the average log-likelihood of the lowest $k\%$ tokens; unseen text tends to contain a few extremely-low-probability outlier tokens that drag the mean down, whereas member (seen) text has relatively higher likelihood even on its lowest $k\%$ tokens, so a higher mean → more likely seen. Useful for post-hoc auditing whether a model was contaminated by a benchmark.

6. Data Mixing & Curriculum

6.1 Data mixing

The proportions in which different sources/domains/tasks are mixed significantly affect final capabilities. How Far Can Camels Go? (Tülu) (Wang et al., arXiv:2306.04751, NeurIPS 2023) systematically ablates 12 instruction-tuning datasets and concludes no single dataset is best on all capabilities — mixing complementary sources is needed for breadth. Tülu 3 (Lambert et al., arXiv:2411.15124) further provides an open end-to-end SFT→DPO→RLVR recipe with full data.

6.2 DoReMi — learning the optimal mixture automatically

DoReMi (Xie et al., arXiv:2305.10429, NeurIPS 2023): first train a small proxy model, use Group DRO (distributionally robust optimization) over domains to learn sampling weights, then resample the full dataset with those weights to train the large model; the paper reports reaching comparable performance about 2.6× faster.

Group DRO objective: a min-max over the simplex of domain weights $\alpha$,

$$\min_\theta \max_{\alpha\in\Delta}\ \sum_{i=1}^{D}\alpha_i\, L_i(\theta)$$

DoReMi uses the excess loss (the proxy's loss minus a reference model's) to drive exponentiated-gradient updates, up-weighting "harder / higher-headroom" domains.

import numpy as np

# DoReMi: learn per-domain sampling weights with a small proxy + Group DRO (exponentiated gradient).
# Key: each domain's "excess loss" must be clamped at the *token* level BEFORE averaging:
#   excess_i = mean_token( max(proxy_token_loss - ref_token_loss, 0) );
# clamping at the domain level ("average first, then clamp") wrongly cancels +/- token diffs.
def domain_excess(proxy_tok_loss, ref_tok_loss):
    # proxy_tok_loss[i] / ref_tok_loss[i]: per-token loss array of domain i
    return np.array([np.maximum(np.asarray(p) - np.asarray(r), 0.0).mean()
                     for p, r in zip(proxy_tok_loss, ref_tok_loss)])

def doremi_update(weights, excess, lr=1.0, smooth=1e-3):
    weights, excess = np.asarray(weights, float), np.asarray(excess, float)
    log_w = np.log(weights) + lr * excess                # exponentiated gradient ascent
    w = np.exp(log_w - log_w.max())                      # numerically stable softmax
    w = w / w.sum()
    return (1 - smooth) * w + smooth / len(w)            # smoothing (paper c≈1e-3), avoid starving a domain

# Average the per-step weights from proxy training as the final mixture, then resample to train the large model.
def sample_domain(weights, rng):
    return rng.choice(len(weights), p=weights)

6.3 Curriculum

Order data by difficulty/length/quality and feed it easy-to-hard, which often converges more stably and faster. Common in post-training:

• Difficulty curriculum: easy questions first, then hard (especially math/code).
• Length curriculum: short sequences first, then long (paired with context-window extension).
• Quality curriculum: large-scale medium quality first, then small-scale high quality (quality annealing).

6.4 Quality filtering

Score samples with perplexity, reward-model scores, heuristic rules, or classifiers and drop low-quality ones — but beware the filter's own bias (e.g. preferring long text) leaking into the data.

7. Interview Questions

L1 — Fundamentals

L2 — Intermediate

L3 — Advanced

[Sources] human seeds + open data + teacher distillation + rejection-sampled self-production
   ↓
[Synthesize] Self-Instruct (breadth) + Evol-Instruct (difficulty) + Magpie (scale)
   ↓
[Filter] similarity dedup (MinHash) + quality scoring (PPL/RM/classifier) + heuristic rules
   ↓
[Decontaminate] n-gram + semantic-level detection against ALL target benchmarks
   ↓
[Mix/Curriculum] domain mixing (DoReMi/manual ablation) + difficulty/length curriculum
   ↓
[Train] SFT → evaluate → diagnose weak capabilities → loop back to add data

Glossary

For study reference only. Paper conclusions and figures follow the original papers; benchmark scores are illustrative and not head-to-head comparisons.

§A Key Papers Timeline

• 2016-06 · Sequence-Level Knowledge Distillation — Kim & Rush, EMNLP 2016. arXiv:1606.07947 — Lifts knowledge distillation to the sequence level: train the student on teacher-decoded sequences instead of word-level soft labels, yielding a faster, competitive model — the conceptual root of LLM distillation data.

• 2021-07 · Deduplicating Training Data Makes Language Models Better — Lee et al., ACL 2022. arXiv:2107.06499 — Systematically shows removing near-duplicates from training corpora cuts verbatim memorization by ~an order of magnitude and reaches comparable/better perplexity with fewer steps, establishing dedup as a baseline action.

• 2022-03 · STaR: Bootstrapping Reasoning With Reasoning — Zelikman et al., NeurIPS 2022. arXiv:2203.14465 — Iteratively bootstraps on the model's own correct CoT rationales; for wrong questions it "rationalizes" a rationale from the given answer, requiring no large human rationale dataset.

• 2022-12 · Self-Instruct: Aligning LMs with Self-Generated Instructions — Wang et al., ACL 2023. arXiv:2212.10560 — Bootstraps instruction data from a few seed tasks + ROUGE-L similarity filtering for de-homogenization, pioneering low-cost scalable instruction-data synthesis (Alpaca is an instance).

• 2022-12 · Constitutional AI: Harmlessness from AI Feedback — Bai et al., preprint. arXiv:2212.08073 — Uses a set of "constitutional principles" to drive model self-critique + revision for SFT, then AI-generated preference labels for RLHF (RLAIF), with harmlessness barely relying on human preference annotation.

• 2023-04 · WizardLM: Empowering LLMs to Follow Complex Instructions — Xu et al., ICLR 2024. arXiv:2304.12244 — Proposes Evol-Instruct: use an LLM to evolve seed instructions in depth (add constraints/difficulty) and breadth (cover new domains), markedly improving complex instruction following.

• 2023-05 · LIMA: Less Is More for Alignment — Zhou et al., NeurIPS 2023. arXiv:2305.11206 — Aligns a 65B LLaMA with only 1,000 curated samples (no RLHF), arguing alignment mainly activates pretrained capabilities and that quality/diversity beat quantity.

• 2023-05 · DoReMi: Optimizing Data Mixtures — Xie et al., NeurIPS 2023. arXiv:2305.10429 — Uses a small proxy model + Group DRO to learn per-domain sampling weights, then resamples to train the big model, reporting ~2.6× speedup and turning data mixing into an optimizable objective.

• 2023-06 · How Far Can Camels Go? (Tülu) — Wang et al., NeurIPS 2023. arXiv:2306.04751 — Systematically ablates 12 instruction-tuning datasets, concluding no single dataset is all-round and complementary mixing is needed, and releases the Tülu open instruction-tuned model suite.

• 2023-06 · Textbooks Are All You Need (phi-1) — Gunasekar et al., preprint. arXiv:2306.11644 — Trains the 1.3B phi-1 on "textbook-quality" filtered data + synthetic textbooks/exercises, reporting HumanEval pass@1 = 50.6%, arguing high-quality synthetic data can drastically lower the scale requirement.

• 2023-08 · Scaling Relationship on Math Reasoning (RFT) — Yuan et al., preprint. arXiv:2308.01825 — Proposes rejection sampling fine-tuning (RFT): sample multiple reasoning paths, keep only correct and deduped ones for augmented SFT, and studies how pretraining loss and data amount jointly govern math reasoning.

• 2023-08 · Reinforced Self-Training (ReST) — Gulcehre et al., preprint. arXiv:2308.08998 — Frames self-training as a Grow (offline sample generation) / Improve (fine-tune on threshold-passing samples) double loop; decoupling sampling from training is more compute-efficient than online RLHF and data is reusable.

• 2023-09 · RLAIF vs. RLHF: Scaling RLHF with AI Feedback — Lee et al., ICML 2024. arXiv:2309.00267 — Shows off-the-shelf LLM preference labels (RLAIF) can match RLHF on summarization/dialogue, and proposes direct-RLAIF: obtain the reward directly from the LLM during RL, no separate RM.

• 2023-10 · UltraFeedback: Boosting LMs with Scaled AI Feedback — Cui et al., ICML 2024. arXiv:2310.01377 — For ~64k instructions, generates ~256k candidates with multiple models and GPT-4 multi-dimensional scores + critiques, building large-scale AI-feedback preference/RM data — a key fuel for open-source alignment.

• 2023-10 · Zephyr: Direct Distillation of LM Alignment — Tunstall et al., preprint. arXiv:2310.16944 — Uses dDPO (distilled DPO) directly on UltraFeedback's AI feedback to distill a strongly-aligned 7B model with no human annotation, validating the "AI feedback + DPO" recipe.

• 2023-10 · Detecting Pretraining Data (Min-K% Prob) — Shi et al., ICLR 2024. arXiv:2310.16789 — Proposes the training-free membership-inference Min-K% Prob: take the average log-likelihood of the lowest k% tokens — higher for members, lower for unseen text (which has low-probability outliers) — usable for benchmark contamination auditing.

• 2024-06 · Magpie: Alignment Data Synthesis from Scratch — Xu et al., ICLR 2025. arXiv:2406.08464 — Exploits an aligned model's autoregressive nature: feed only the pre-user-turn template to make the model emit instruction-response pairs — zero seeds, zero human curation, scalable alignment-data synthesis.

• 2024-11 · Tülu 3: Pushing Frontiers in Open Post-Training — Lambert et al., preprint. arXiv:2411.15124 — An open end-to-end post-training recipe (SFT→DPO→RLVR) with full data/code, introducing RLVR as a key component and showcasing the data pipeline and mixing engineering in detail.