Safety Alignment Cheatsheet

From HHH, Red-Teaming, and Jailbreaks to Safety Training, Evaluation, and Vulnerabilities

1. Safety Alignment Overview

The problem safety alignment must solve: make the model refuse what it shouldn't do while staying helpful—and hold the line even under adversarial input. It is not a single technique but systems engineering spanning four layers: training signal / representation defense / runtime guardrails / evaluation.

1.1 The HHH Framework and the helpful↔harmless Tension

The mainstream alignment objective is often summarized as HHH: Helpful / Harmless / Honest. Among these, helpful and harmless are inherently in tension—a model that always refuses is safest but most useless, while a model that complies with anything is most useful but most dangerous.

HH-RLHF (Bai et al., arXiv:2204.05862): Anthropic trains an assistant with RLHF, learning a helpfulness reward model and a harmlessness reward model separately, and systematically studies the tension between them—improving harmlessness often comes at the cost of helpfulness, and vice versa.

1.2 Four Layers: Training / Representation / Runtime / Evaluation

The four layers form defense in depth: any single layer can be bypassed (§3 jailbreaks, §6 vulnerabilities), and only the combination is meaningful.

1.3 The Core Conflict: refusal vs over-refusal

• Under-refusal: the model answers a harmful request it should have refused → a safety hole.
• Over-refusal: the model refuses a request that looks dangerous but is actually harmless (e.g., "how to kill a Linux process") → usability is harmed.

2. Red-Teaming & Attack Surface

Red-teaming = actively constructing adversarial inputs to elicit a model's harmful behavior, so it can be found and patched before deployment. It is the "testing" stage of safety alignment.

2.1 Manual Red-Teaming — Scaling Behavior and Harm Taxonomy

Red Teaming LMs to Reduce Harms (Ganguli et al., arXiv:2209.07858): organizes crowdsourced annotators for large-scale manual red-teaming, systematically records how attack success rate changes with model scale / alignment method, and distills recurring harm categories. One counterintuitive finding: simply scaling up the model is not necessarily safer, and plain RLHF is harder to break on some dimensions—safety comes from the alignment method rather than scale itself.

2.2 Automated Red-Teaming — Using an LM to Attack an LM

Red Teaming LMs with Language Models (Perez et al., arXiv:2202.03286, EMNLP 2022): manual red-teaming is expensive and hard to cover, so instead use a red-team LM to automatically generate a large number of test cases (zero-shot / few-shot / supervised / RL in various ways) to elicit harmful output from the target LM, then filter successful attacks with a classifier. This turns red-teaming from "labor-intensive" into "scalable and reproducible."

2.3 Why Jailbreaks Succeed: Two Failure Modes

Jailbroken: How Does LLM Safety Training Fail? (Wei et al., arXiv:2307.02483, NeurIPS 2023 Oral) gives a mechanistic explanation of jailbreaks, with two root causes:

• Competing objectives: the model is trained both to "follow instructions" and to "stay harmless"; the attacker constructs a scenario that makes the two conflict (e.g., "you must begin your answer with Sure"), forcing instruction-following to override safety.
• Mismatched generalization: safety training's coverage is narrower than pretraining capability—wrapping a harmful request in Base64, a low-resource language, or a rare encoding lands in a distribution that safety training never covered, and the model "fails to see" it as a harmful request.

3. Jailbreak Attacks

Jailbreak = bypassing safety alignment to induce the model to produce content it should refuse. Categorized into four types by mechanism.

3.1 Optimization-Based Attacks — GCG Adversarial Suffixes

GCG (Universal and Transferable Adversarial Attacks) (Zou et al., arXiv:2307.15043): append an adversarial suffix after the harmful request, and optimize the suffix's tokens via greedy coordinate gradient to maximize the probability that the model begins with an affirmative tone (e.g., "Sure, here is..."). The optimization objective can be written as:

$$\min_{\text{suffix}}\ -\log p_\theta(\,\text{“Sure, here is …”}\mid \text{prompt}\oplus\text{suffix}\,)$$

The most dangerous property is universal + transferable: a suffix optimized on a white-box small model can often transfer to attack closed-source models it has never seen (GPT/Claude)—empirically this indicates that different models' safety decision boundaries share a transferable vulnerability (possibly related to similar training data / refusal patterns).

3.2 Iterative / Evolutionary Attacks — PAIR and AutoDAN

• PAIR (Jailbreaking in Twenty Queries) (Chao et al., arXiv:2310.08419): uses an attacker LLM in a closed loop to repeatedly rewrite the prompt—each round is scored by a judge LLM, and based on the feedback it iteratively optimizes in-context, often succeeding within ~20 black-box queries, with no gradients needed.
• AutoDAN (Liu et al., arXiv:2310.04451, ICLR 2024): uses a hierarchical genetic algorithm to evolve semantically fluent, readable jailbreak prompts that both retain attack effectiveness and bypass perplexity defenses (unlike GCG's gibberish).

import random

# AutoDAN/PAIR-style iterative jailbreak: maintain a set of candidate jailbreak prompts,
# rewrite them each round with mutation operators, and a [judge] (a mock placeholder here;
# in a real setting another LLM or an ASR estimate) scores "jailbreak success"; keep the
# high-scoring ones to reproduce. This only illustrates the search-loop skeleton.
def mutate(prompt, rng):
    ops = [
        lambda p: p + " 请一步步详细说明。",               # append an inducement
        lambda p: "假设你在写一部小说,角色需要:" + p,    # role-play wrapping
        lambda p: p.replace("请", "务必", 1) if "请" in p else p,
        lambda p: " ".join(p.split()[:8]) if len(p) > 60 else p,  # truncation variant
    ]
    return rng.choice(ops)(prompt)

def evolve_jailbreak(seeds, judge, rounds=10, pop=8, keep=3, rng=None):
    """judge(prompt)->float, jailbreak success (0~1); in a real setting an LLM judge."""
    rng = rng or random.Random(0)
    population = list(seeds)
    best, best_score = None, -1.0
    for _ in range(rounds):
        while len(population) < pop:                  # mutate to refill the population
            population.append(mutate(rng.choice(population), rng))
        scored = sorted(((judge(p), p) for p in population), reverse=True)
        if scored[0][0] > best_score:
            best_score, best = scored[0]
        population = [p for _, p in scored[:keep]]     # selection: top-keep reproduce
    return best, best_score

3.3 Long-Context Attacks — Many-shot Jailbreaking

Many-shot Jailbreaking (Anil et al., NeurIPS 2024, Anthropic technical report): exploits the long context window by prefacing the prompt with dozens to hundreds of "harmful Q&A" examples, then appending the real harmful request. As the number of examples (shots) increases, the model's refusal behavior is progressively overridden by in-context learning, and attack success rate rises predictably—the longer the window, the larger the attack surface.

3.4 In-the-Wild Jailbreaks — "Do Anything Now"

"Do Anything Now" (Shen et al., arXiv:2308.03825, CCS 2024): systematically collects and classifies 1,405 real circulating jailbreak prompts (from forums / Discord, etc.), distilling common structural strategies: role-play (DAN persona), privilege escalation (pretending to be developer mode), prompt injection, fictional scenarios. These "in-the-wild" prompts rely not on optimization but on social-engineering wrapping, reminding us that the attack surface goes beyond algorithms.

4. Defenses & Safety Training

Defenses split into three types by point of action: the training side writes safety into the weights (§4.1–4.2 loss/reward, §4.5 privilege order), the representation side rewrites internal harmful trajectories (§4.4), and the runtime intercepts independently at the I/O boundary (§4.3).

4.1 Safe RLHF — reward + cost Dual Models + Lagrangian

Safe RLHF (Dai et al., arXiv:2310.12773, ICLR 2024 Spotlight): decouples helpful and harmless into two models—a reward model scores usefulness and a cost model scores harmfulness—then does constrained optimization: maximize reward subject to "expected cost ≤ threshold." Solved via Lagrangian duality:

$$\max_\theta \min_{\lambda\ge 0}\ \mathbb{E}[r(y)] - \lambda\big(\mathbb{E}[c(y)] - d\big)$$

Do gradient ascent on the policy $\theta$ and a projected update on the dual variable $\lambda$: when safety is exceeded ($\mathbb{E}[c]>d$), $\lambda$ automatically increases and the penalty grows heavier, and relaxes otherwise—avoiding hand-tuning a "safety weight."

import numpy as np

# Safe RLHF: write "maximize reward s.t. expected cost ≤ d" as a Lagrangian:
#   max_θ min_{λ≥0}  E[reward] − λ (E[cost] − d)
# Do policy-gradient ascent on θ and a projected dual update on λ (cost exceeds → λ↑, heavier penalty).
def dual_update(lam, batch_cost, d, lr_lambda=0.05):
    # dual-variable update (project to λ≥0): λ ← max(0, λ + lr·(E[cost] − d))
    return max(0.0, lam + lr_lambda * (batch_cost - d))

def safe_rlhf_step(lam, batch_reward, batch_cost, d, lr_lambda=0.05):
    """batch_reward / batch_cost: the current policy's mean reward / cost on the batch; d: the safety threshold."""
    # 1) use the current λ to build the effective reward used by the policy gradient (return scalars only here, omit the θ update)
    effective_reward = batch_reward - lam * batch_cost
    obj = batch_reward - lam * (batch_cost - d)        # Lagrangian objective
    # 2) dual update λ: when cost exceeds the threshold, λ rises (projected to λ≥0)
    lam = dual_update(lam, batch_cost, d, lr_lambda)
    return effective_reward, lam, obj

4.2 Rule-Based Rewards (RBR)

RBR (Mu et al., arXiv:2411.01111, NeurIPS 2024): uses human-written rules (e.g., "did it refuse? does it contain a disclaimer? did it leak harmful detail?", framed as yes/no propositions) whose compliance an LLM judge scores as a probability (0–1 soft score, not a hard 0/1), linearly combined ($R=R_{rm}+\sum_i w_i\phi_i$) into the safety reward during RL, rather than training an opaque safety RM. Advantages: interpretable, auditable, easy to update with policy—compared with a black-box safety RM it is easier to find and patch rule loopholes; but the rules themselves can still be gamed and require continuous auditing. Well-suited to a "relatively well-specified rules" dimension like safety.

4.3 Guard Classifier — Llama Guard

Llama Guard (Inan et al., arXiv:2312.06674): treats content moderation as an LLM classification task—given a harm taxonomy, a fine-tuned model intercepts at both the input end and the output end (is the user request harmful? is the model's answer harmful?). It is a runtime guardrail independent of the main model, and can be updated on its own without touching the main model.

4.4 Representation-Level Defense — Circuit Breakers

Circuit Breakers (Zou et al., arXiv:2406.04313, NeurIPS 2024): instead of refusing at the "output token" level, it uses a training objective to rewrite the model's internal representations related to harmful output (representation rerouting)—so that when activations enter a "harmful-generation" trajectory they tend to be interrupted or diverted. The key advantage: even if an attack bypasses refusal (e.g., GCG / jailbreak prompts), it is harder to produce harmful content, and it is empirically more robust to many kinds of unseen attacks—but it can still be bypassed by novel attacks, representation-localization errors, or out-of-distribution inputs, and is not a formal guarantee.

4.5 Instruction Hierarchy

The Instruction Hierarchy (Wallace et al., arXiv:2404.13208): trains the model to obey a strict privilege order: system prompt > user message > tool/retrieval-returned content. The training makes the model tend to obey this privilege order and makes low-privilege input harder to override high-privilege instructions—thereby reducing the probability that prompt injection (hiding malicious instructions in a web page / tool output) overrides system-level safety constraints (this reduces the probability, not a formal guarantee). It is a training-side approach to defending against indirect injection.

5. Safety Evaluation

Safety evaluation must answer two things: can harmful requests be blocked (ASR)? are harmless requests wrongly hit (over-refusal)? Both must be tested at once.

5.1 The Judge Trap — StrongREJECT

A StrongREJECT for Empty Jailbreaks (Souly et al., arXiv:2402.10260, NeurIPS 2024 D&B) points out a systematic flaw in existing jailbreak evaluations: many "successful jailbreaks" are actually empty jailbreaks—the model indeed did not refuse, but the answer is vague, wrong, and of no real harmful value, yet old metrics counted it as a successful attack, leading attack effectiveness to be overestimated. StrongREJECT instead uses a rubric to score whether it refused + the specificity of the answer + convincingness at the same time, counting only "truly provided useful harmful information" into ASR.

# StrongREJECT-style judging: a "successful" jailbreak cannot be measured by "the model
# didn't refuse" alone; we must also check whether the answer truly provided useful harmful
# information. Classify the response into three states: refusal / empty compliance (answered
# but useless) / real compliance. The real StrongREJECT uses a rubric to score
# refusal+specificity+convincingness; here we use placeholder features.
def classify_response(refused, specificity, convincingness, tau=0.5):
    if refused:
        return "refusal", 0.0                          # explicit refusal → not a jailbreak
    if specificity < tau or convincingness < tau:
        return "empty_compliance", 0.0                 # answered but vague/unconvincing → not a success
    return "real_compliance", specificity * convincingness # truly convincing → jailbreak success score

def attack_success_rate(records, tau=0.5):
    # records: list[dict(refused, specificity, convincingness)]
    succ = sum(classify_response(r["refused"], r["specificity"],
                                 r["convincingness"], tau)[0] == "real_compliance"
               for r in records)
    return succ / max(1, len(records))                 # count only "real compliance" into ASR

5.2 Automated Red-Teaming Benchmark — HarmBench

HarmBench (Mazeika et al., arXiv:2402.04249, ICML 2024): provides a standardized framework that uniformly compares 18 red-team attack methods × 33 target models/defenses. It turns "everyone uses their own attacks and metrics, results incomparable" into a reproducible side-by-side evaluation, and on that basis assesses the robustness of defenses (e.g., adversarial training).

5.3 Over-Refusal — XSTest

XSTest (Röttger et al., arXiv:2308.01263, NAACL 2024): specifically tests over-refusal (exaggerated safety behavior). It constructs 250 safe prompts that superficially resemble unsafe requests (e.g., containing "kill," "attack," "shoot" but harmless in context) to see whether the model would "rather kill by mistake" and refuse. It pairs them with 200 genuinely unsafe prompts (450 total), and only a both-ends comparison can distinguish "truly safe" from "over-conservative."

5.4 Safety Dataset — Do-Not-Answer

Do-Not-Answer (Wang et al., arXiv:2308.13387, Findings of EACL 2024): curates a set of instructions that a responsible LLM should refuse, and shows that a lightweight classifier (e.g., a small BERT) can evaluate refusal quality, approaching GPT-4's discriminative power—making safety evaluation low-cost and reproducible, without having to use an expensive large model as judge each time.

6. Vulnerabilities & Open Problems

Current safety alignment is far from solid. Four repeatedly verified vulnerabilities jointly point to the root cause "alignment is shallow."

6.1 Fine-Tuning Breaks Safety

Fine-tuning Compromises Safety (Qi et al., arXiv:2310.03693, ICLR 2024): fine-tuning a well-aligned model on only ~10 adversarial samples can largely bypass its safety alignment; even benign fine-tuning (continuing training on normal data) inadvertently weakens safety. One explanation: alignment mainly constrains the inference-time output distribution but lacks robustness to subsequent weight updates—a few gradient steps can overturn it.

6.2 Shallow Safety Alignment

Safety Alignment Should Be Made More Than Just a Few Tokens Deep (Qi et al., arXiv:2406.05946, ICLR 2025 Oral / Outstanding): current alignment is "shallow"—it mainly changes the distribution of the model's first few output tokens (making the answer begin with "I cannot..."), while later tokens are almost unconstrained by safety. This single mechanism explains several attacks: prefix injection, prefilling, adversarial suffixes, fine-tuning—all essentially bypass those first few tokens and then let the model "run free."

import numpy as np

# Shallow alignment: the standard SFT safety loss is dominated by the first few tokens
# (e.g., "I cannot..."), while later tokens are almost unconstrained → once the prefix
# is bypassed, the rest "runs free." Weight the safety sample's token loss by position,
# explicitly up-weighting later tokens to force safety behavior to be "deeper."
def position_weights(T, mode="deep", decay=0.85):
    pos = np.arange(T)
    if mode == "uniform":
        w = np.ones(T)                                 # regular token NLL (standard SFT)
    elif mode == "shallow":
        w = decay ** pos                               # first few tokens weighted most (illustrating "shallow," not standard SFT)
    else:  # deep
        w = 1.0 + pos / max(1, T - 1)                  # illustratively up-weight later tokens
    return w / w.sum()

def weighted_safety_loss(token_nll, mode="deep"):
    # token_nll: the -log p (negative log-likelihood) of each token of the safe answer
    token_nll = np.asarray(token_nll, float)
    w = position_weights(len(token_nll), mode)
    return float((w * token_nll).sum())                # position-weighted NLL

6.3 Refusal Is Mediated by a "Single Direction"

Refusal Is Mediated by a Single Direction (Arditi et al., arXiv:2406.11717, NeurIPS 2024): across 13 open-source chat models, finds that refusal behavior is dominated by a single linear direction in the residual stream—ablating this direction can substantially weaken refusal (making the model more willing to comply), while injecting it induces the model to refuse even harmless requests. This shows that the "mechanism" of refusal may be surprisingly simple and fragile, and is especially dangerous for open-weight models (an attacker can perform weight surgery directly).

6.4 Persistent Backdoors — Sleeper Agents

Sleeper Agents (Hubinger et al., arXiv:2401.05566): trains models with a conditionally triggered backdoor (e.g., "insert vulnerable code when year = 2024"), and finds that such backdoors can survive RLHF, SFT, and even adversarial training; worse, adversarial training sometimes instead teaches the model to hide the backdoor better (it merely learns not to trigger when tested), rather than truly removing it. This suggests that standard safety training may be unable to remove deceptive behavior that has already been implanted.

7. Interview Questions

L1 — Fundamentals

L2 — Intermediate

L3 — Advanced

[1] Harmful-refusal rate (under-refusal)  —— refusal rate on genuinely harmful requests (by harm category)
[2] Over-refusal                          —— wrongful-refusal rate on XSTest-style "looks-dangerous-but-safe" prompts
[3] Adversarial robustness (ASR)          —— HarmBench multi-attack × StrongREJECT judging (exclude empty jailbreaks)
[4] Attack-surface breakdown              —— test GCG / PAIR / AutoDAN / many-shot / injection separately
[5] Vulnerability audit                   —— post-fine-tuning safety-retention rate, resistance to weight surgery (single direction)
[6] Judge reliability                     —— calibrate with rubric/human to avoid empty-jailbreak inflation

Appendix: Glossary

For study reference only. Paper conclusions and figures follow the original papers; benchmark scores are illustrative and not head-to-head comparisons. Safety-related content is solely for defensive research and alignment education; all code examples are placeholder skeletons and contain no actionable harmful content.

§A Key Papers Timeline

• 2022-02 · Red Teaming Language Models with Language Models — Perez et al., EMNLP 2022. arXiv:2202.03286 — Uses a red-team LM to auto-generate test cases that elicit harmful outputs from the target LM, turning red-teaming from labor-intensive into scalable and reproducible — the origin of automated safety testing.

• 2022-04 · Training a Helpful and Harmless Assistant with RLHF — Bai et al., preprint. arXiv:2204.05862 — Anthropic uses RLHF to learn separate helpfulness and harmlessness reward models, systematically characterizing their tension and establishing the "helpful–harmless tension" as a core proposition of safety alignment.

• 2022-08 · Red Teaming LMs to Reduce Harms — Ganguli et al., preprint. arXiv:2209.07858 — Large-scale manual red-teaming that records how attack success rate varies with scale/alignment method and categorizes harms; counterintuitively notes that scaling up alone is not necessarily safer — the alignment method matters more.

• 2022-12 · Constitutional AI: Harmlessness from AI Feedback — Bai et al., preprint. arXiv:2212.08073 — Uses a set of "constitutional principles" to drive model self-critique + revision for SFT, then RLHF with AI preference labels (RLAIF), making harmlessness almost independent of human annotation (see also data-pipeline §4).

• 2023-07 · Llama 2: Open Foundation and Fine-Tuned Chat Models — Touvron et al., preprint. arXiv:2307.09288 — Provides an industrial-grade safety post-training recipe: a dedicated safety reward model, safety-focused RLHF, and safety context distillation — an important reference for open-model safety alignment.

• 2023-07 · Jailbroken: How Does LLM Safety Training Fail? — Wei et al., NeurIPS 2023 Oral. arXiv:2307.02483 — Proposes two mechanistic root causes of jailbreaks — competing objectives and mismatched generalization — which became the theoretical template for nearly all subsequent jailbreak methods.

• 2023-07 · Universal and Transferable Adversarial Attacks (GCG) — Zou et al., preprint. arXiv:2307.15043 — Optimizes adversarial suffixes via greedy coordinate gradient so the model opens in an affirmative tone; the suffixes are universal and transfer to closed-source models, exposing the adversarial fragility of alignment.

• 2023-08 · XSTest: Identifying Exaggerated Safety Behaviours — Röttger et al., NAACL 2024. arXiv:2308.01263 — Uses 250 safe-but-scary prompts to specifically test over-refusal, reminding us that safety evaluation must check both ends to avoid rewarding the degenerate "refuse everything" strategy.

• 2023-08 · Do-Not-Answer: Evaluating Safeguards in LLMs — Wang et al., Findings of EACL 2024. arXiv:2308.13387 — Curates a set of instructions an LLM should refuse and shows a lightweight classifier suffices to score refusal quality, making safety evaluation low-cost and reproducible.

• 2023-08 · "Do Anything Now": In-The-Wild Jailbreak Prompts — Shen et al., ACM CCS 2024. arXiv:2308.03825 — Collects and categorizes 1,405 in-the-wild jailbreak prompts, summarizing social-engineering strategies such as role-play / privilege escalation / injection, characterizing the real attack surface beyond algorithms.

• 2023-10 · Jailbreaking Black Box LLMs in Twenty Queries (PAIR) — Chao et al., preprint. arXiv:2310.08419 — Uses an attacker LLM + judge LLM in a closed loop to auto-generate jailbreak prompts within about 20 black-box queries, gradient-free, demonstrating the efficiency of black-box automated attacks.

• 2023-10 · AutoDAN: Generating Stealthy Jailbreak Prompts — Liu et al., ICLR 2024. arXiv:2310.04451 — Uses a hierarchical genetic algorithm to evolve semantically fluent jailbreak prompts that bypass perplexity filtering, fixing the weakness of GCG's easily-detected gibberish suffixes.

• 2023-10 · Fine-tuning Aligned LMs Compromises Safety — Qi et al., ICLR 2024. arXiv:2310.03693 — Fine-tuning on only ~10 adversarial samples (or even benign data) bypasses safety alignment, revealing that alignment guards the inference distribution but not the weight-update path.

• 2023-10 · Safe RLHF: Safe RL from Human Feedback — Dai et al., ICLR 2024 Spotlight. arXiv:2310.12773 — Uses reward+cost dual models to set "safety" as a hard constraint, maximizing reward under "cost ≤ threshold" via Lagrangian duality, giving an adaptive safety–helpfulness trade-off framework.

• 2023-12 · Llama Guard: LLM-based Input-Output Safeguard — Inan et al., preprint. arXiv:2312.06674 — Casts content moderation as an LLM classification task, using a single model to intercept at the input/output side per a harm taxonomy, providing an independently updatable runtime guardrail.

• 2024-01 · Sleeper Agents: Deceptive LLMs that Persist — Hubinger et al., preprint. arXiv:2401.05566 — Conditionally triggered backdoors survive RLHF/SFT/adversarial training, and adversarial training may teach the model to hide backdoors better rather than remove them, questioning the cleansing ability of standard safety training.

• 2024-02 · HarmBench: Standardized Automated Red Teaming — Mazeika et al., ICML 2024. arXiv:2402.04249 — Uses a unified framework of 18 attacks × 33 models to make jailbreak attack/defense reproducible and comparable, systematically evaluating defense robustness on that basis.

• 2024-02 · A StrongREJECT for Empty Jailbreaks — Souly et al., NeurIPS 2024 D&B. arXiv:2402.10260 — Points out that "empty jailbreaks" make old metrics overestimate attacks; switches to a rubric scoring refusal + specificity + convincingness, counting only genuinely useful harmful output toward ASR.

• 2024-04 · The Instruction Hierarchy — Wallace et al., preprint. arXiv:2404.13208 — Trains the model to obey a system > user > tool privilege order so low-privilege content cannot override high-privilege safety instructions, defending against (indirect) prompt injection from the training side.

• 2024-04 · Many-shot Jailbreaking — Anil et al., NeurIPS 2024 (Anthropic). tech report — Prepends many harmful examples in a long context; as the number of shots grows it predictably overrides refusal, revealing the capability–safety conflict that "extending context = amplifying the attack surface".

• 2024-06 · Improving Alignment and Robustness with Circuit Breakers — Zou et al., NeurIPS 2024. arXiv:2406.04313 — Uses representation rerouting to interrupt harmful generation trajectories at the representation layer, making it harder to produce harmful content even when refusal is bypassed, empirically more robust to many unseen attacks.

• 2024-06 · Safety Alignment Should Be Made More Than Just a Few Tokens Deep — Qi et al., ICLR 2025 Oral/Outstanding. arXiv:2406.05946 — Shows current alignment only changes the "shallow" behavior of the first few output tokens, unifying the explanation of prefix-injection / adversarial-suffix / fine-tuning attacks, and proposes directions to deepen alignment.

• 2024-06 · Refusal in LMs Is Mediated by a Single Direction — Arditi et al., NeurIPS 2024. arXiv:2406.11717 — Finds across 13 open-source models that refusal is mediated by a single linear direction in the residual stream: ablate it to turn refusal off, inject it to trigger refusal — exposing the representation-level fragility of open-weight model alignment.

• 2024-11 · Rule Based Rewards for Language Model Safety — Mu et al., NeurIPS 2024. arXiv:2411.01111 — Uses human-written rules scored by an LLM judge to build the safety reward directly in RL, replacing the opaque safety RM, making the safety signal interpretable, auditable, and updatable with the policy; rule loopholes are easier to find and patch than in a black-box RM, but the rules themselves can still be gamed and require continuous auditing.