Continual & Lifelong Learning

When a model learns continuously on sequential tasks, how can it acquire new knowledge without forgetting old knowledge? This is a mandatory question beyond the "train-once-then-deploy" paradigm, and a latent risk throughout the LLM post-training pipeline (pretrain → SFT → DPO → RL).

0. The evolution

IID one-shot training → sequential multi-task (Task 1 → Task 2 → …) → catastrophic forgetting appears: gradients directly overwrite old weights.

Stability-plasticity dilemma: a network must be plastic — accepting gradient updates for new tasks — and stable — preserving representations of old tasks. The two requirements are inherently in tension.

TL;DR — quick anchors (2-minute pass)

• Core tension = stability-plasticity: plastic (learn new) vs. stable (keep old) inherently conflict; catastrophic forgetting = new-task gradients overwriting shared parameters, not a capacity shortfall.
• Four settings: Task-IL / Domain-IL / Class-IL (hardest) / Continual pretraining (no boundaries); difficulty hinges on whether the task ID is known at test time.
• Three method families: regularization (EWC/SI/MAS, penalize moving important weights) / replay (ER/GEM/A-GEM/DER, mix old samples or soft targets) / parameter isolation (ProgNN/PackNet/LoRA-CL, separate sub-networks).
• Parameter isolation (esp. ProgNN/PackNet) gives structural zero-forgetting; regularization / replay are merely approximate; isolation's cost is parameter growth with task count.
• EWC: L + (λ/2)·Σ Fᵢ(θᵢ−θᵢ*)², Fisher diagonal measures old-task importance; λ too small still forgets, too large can't learn the new task.
• LwF = distilling the old model's soft outputs, no old data needed; but soft-target quality drops under large task drift.
• Three metrics: AA (overall retention) / BWT (forgetting, signed, ideal ≥0) / FWT (forward transfer); don't read BWT as "larger magnitude is better."
• LLM angle: continual pretrain / instruction-tuning / alignment; pretrain→SFT→DPO→RL is CL at every hop, alignment tax accumulates; KL constraint ≈ implicit EWC.
• Knowledge editing (ROME rank-1 / MEMIT batch) = targeted surgery, complementary to CL's global protection; sequential edits cause interference—judge on reliability / generalization / locality.

1. Why catastrophic forgetting happens

A neural network's parameters are shared storage for all tasks. When running SGD on task $\mathcal{T}_2$, the gradient of the loss with respect to the parameters has no knowledge that "these weights matter for $\mathcal{T}_1$", so it overwrites them — this is catastrophic forgetting.

Classic settings:

2. Three method families

2.1 Regularization

Core idea: add a penalty that protects old weights to the new-task loss, so that weights important to old tasks change as little as possible.

EWC (Elastic Weight Consolidation)1Uses the diagonal of the Fisher information matrix to measure weight importance; quadratic penalty prevents forgetting. Kirkpatrick 2017 ↗ total loss:

$$\mathcal{L}_{\text{EWC}} = \mathcal{L}_{\mathcal{T}_2}(\theta) + \frac{\lambda}{2} \sum_i F_i \,(\theta_i - \theta_i^*)^2$$

• $\theta_i^*$: parameter values after completing the old task
• $F_i$: diagonal element of the Fisher information matrix (measures the "importance" of parameter $i$ to the old task)
• $\lambda$: hyperparameter controlling the stability vs. plasticity trade-off

$$F_i = \mathbb{E}_{\mathcal{D}_1}\!\left[\left(\frac{\partial \log p_\theta(y|x)}{\partial \theta_i}\right)^{\!2}\right]$$

SI (Synaptic Intelligence): an online version of EWC that accumulates each parameter's contribution to the loss during training, without needing to recompute the Fisher after the task ends.

MAS (Memory Aware Synapses): importance is estimated by the gradient norm of the output function with respect to parameters, requiring no labeled data.

2.2 Rehearsal & Replay

Core idea: mix in old-task samples during new-task training, so that gradients "remember" the past simultaneously.

Experience Replay: maintain an episodic memory buffer holding real samples from old tasks; randomly interleave them at a fixed ratio during new-task training.

GEM (Gradient Episodic Memory)2Projects gradient updates into the feasible region where old-task losses do not increase, while allowing forward transfer. Lopez-Paz 2017 ↗:

For each old task $k$, requires the updated gradient to satisfy: $$\langle g, g_k \rangle \geq 0$$ i.e., the new gradient must not point "opposite" to the old-task gradient. If violated, $g$ is projected onto the feasible region.

A-GEM (Averaged GEM)3Merges multiple old-task constraints into a single average-gradient constraint, drastically reducing computation while achieving performance comparable to GEM. Chaudhry 2019 ↗: uses the average gradient $g_{\text{ref}}$ over all old-task buffers as the single constraint, reducing the per-step QP projection from $K$ constraints to 1:

$$\langle g, g_{\text{ref}} \rangle \geq 0, \quad g_{\text{ref}} = \frac{1}{K}\sum_k g_k$$

Generative Replay: use a generative model (e.g., VAE, GAN) to learn the old-task distribution and synthesize "pseudo-old data" at training time — no real old samples need to be stored, but generative quality bottlenecks accumulate error.

DER (Dark Experience Replay)4Stores old-sample logits (soft targets) in the buffer and uses MSE to match them, fusing rehearsal with knowledge distillation. Buzzega 2020 ↗: buffer stores $(x, y, z)$ where $z$ is the logit (dark knowledge) from the model at the past time step; at replay time the model must also match $z$:

$$\mathcal{L}_{\text{DER}} = \mathcal{L}_{\text{CE}}(x,y) + \alpha \cdot \text{MSE}(f_\theta(x),\, z)$$

2.3 Parameter Isolation & Architectural CL

Core idea: different tasks occupy different sub-networks; new tasks expand capacity without modifying old-task parameters — forgetting is structurally eliminated.

Progressive Neural Networks5Adds a new network column for each new task; lateral connections exploit knowledge from frozen old columns; forgetting is structurally impossible. Rusu 2016 ↗: each task gets an independent network column; old columns are frozen; new columns read old-column activations via lateral connections:

$$h_k^{(\ell)} = f\!\left(W_k^{(\ell)} h_k^{(\ell-1)} + \sum_{j<k} U_{k,j}^{(\ell)} h_j^{(\ell-1)}\right)$$

• Advantage: zero forgetting, natural forward transfer
• Disadvantage: parameter count grows linearly with the number of tasks

PackNet: performs iterative pruning + freezing within a single network, assigning a parameter mask to each task, with no shared gradient paths between tasks.

LoRA-based / Adapter-based CL: incrementally adds a set of LoRA weights or adapter modules per task; the backbone is frozen; tasks are routed to the corresponding adapter via task ID — parameter overhead is manageable and LLM-friendly.

3. Knowledge Distillation: LwF

LwF (Learning without Forgetting)6When training on a new task, uses the soft outputs of the old model as distillation targets, mitigating forgetting without storing any old data. Li 2016 ↗:

• During inference on new-task $\mathcal{T}_2$ data, first use the old model $f_{\theta^*}$ to produce soft output $\hat{y}^{\text{old}}$
• Joint optimization:

$$\mathcal{L}_{\text{LwF}} = \mathcal{L}_{\text{new}}(f_\theta(x), y) + \lambda \cdot \mathcal{L}_{\text{KD}}(f_\theta(x), \hat{y}^{\text{old}})$$

• $\mathcal{L}_{\text{KD}}$ is typically temperature-scaled KL divergence or cross-entropy
• No old data required; downside: soft-target quality degrades when task drift is large

LwF is essentially implicit replay of the old model's knowledge rather than old data — attractive in privacy-sensitive or storage-constrained settings.

4. Evaluation Metrics

Let $a_{i,j}$ denote accuracy on task $j$ measured after completing task $i$, across $T$ tasks total.

Average Accuracy (AA): average accuracy over all tasks after learning all of them:

$$\text{AA} = \frac{1}{T} \sum_{j=1}^{T} a_{T,j}$$

Backward Transfer (BWT) — more negative means more forgetting:

$$\text{BWT} = \frac{1}{T-1} \sum_{j=1}^{T-1} \bigl(a_{T,j} - a_{j,j}\bigr)$$

Forward Transfer (FWT) — positive values indicate old tasks benefited new ones:

$$\text{FWT} = \frac{1}{T-1} \sum_{j=2}^{T} \bigl(a_{j-1,j} - b_j\bigr)$$

where $b_j$ is the baseline accuracy for task $j$ trained independently from random initialization.

Forgetting is sometimes defined directly as the mean accuracy drop for each task from "when learned" to "final", which is the negation of BWT.

5. The LLM Angle

5.1 Continual Pretraining

Continuing to train a language model on new-domain corpora after initial pretraining (e.g., medical text, code updates); core challenges:

• Old general capabilities (math reasoning, instruction following) may degrade
• Distribution shift from new corpora may be milder than task-level shift but persists longer
• Common mitigations: learning-rate warm-up restart, replaying a small amount of general data, reducing the learning rate

5.2 Continual Instruction-Tuning

Sequentially introducing new instruction types (e.g., coding → then math → then safety); each SFT round may overwrite behaviors learned in previous rounds. LoRA-based or adapter-per-task approaches are natural low-parameter solutions: the backbone is shared, and behaviors are separated by module routing.

5.3 Continual Alignment & Alignment Tax

Sequential alignment pipeline: pretrain → SFT → DPO → RL (RLHF/RLVR) — each step continues training from the previous step's checkpoint.

Alignment tax: alignment often trades away part of general capabilities (e.g., code generation, factuality). When steps are stacked sequentially, the tax accumulates:

• Excessive SFT may compress knowledge diversity
• Doing RLHF after DPO can lead to over-refusal or format degradation
• Every hop faces the CL problem of "new alignment objective vs. behavior learned in the previous hop"

Mitigation strategies:

1. KL constraint (PPO clip / DPO reference model) — limits each step's deviation from the reference; structurally analogous to implicit EWC
2. Replay old preference data — mix in data from earlier alignment stages
3. LoRA with independent adapter per stage — backbone unchanged, alignment behavior localized

5.4 Why CL Matters for LLM Post-Training

5.5 Knowledge Editing (ROME / MEMIT)

Positioning: CL regularization (EWC etc.) is global protection—penalizing all weights important to old tasks; knowledge editing (model editing) is targeted surgery—rewriting only the few parameters that store a single fact, updating one piece of knowledge precisely without retraining. The two are complementary: CL prevents "learning new, forgetting old," while editing performs "precise revision of the old."

ROME (Rank-One Model Editing)10Causal tracing locates factual knowledge mainly in middle-layer MLPs; treats that MLP's second linear projection (down-projection) as a linear associative memory and rewrites a specific key-value association with a rank-1 update. Meng 2022 ↗: causal tracing finds that factual knowledge resides mainly in middle-layer MLP key→value associations. Viewing that layer's weight $W$ as a linear associative memory (key $k$ maps to value $v$), inserting a new fact $(k_*, v_*)$ is equivalent to a minimal-change constrained least squares:

$$\hat{W} = \arg\min_{\hat W}\ \lVert \hat W K - V \rVert^2 \quad \text{s.t.}\ \hat W k_* = v_*,$$

whose closed-form solution is a rank-1 update to $W$—preserving existing associations $K\!\to\!V$ while mapping the target subject's key to the new object.

MEMIT (Mass-Editing Memory in a Transformer)11Extends ROME's single edit to batches of thousands of facts updated across multiple middle layers. Meng 2022 ↗: extends ROME's single edit to batches of thousands of facts, amortizing the update across multiple middle layers, resolving the degradation ROME suffers when editing many facts sequentially one by one.

Forgetting under sequential edits: when editing many facts in sequence, later edits interfere with earlier ones (edit interference) and may spill over to unrelated knowledge and general capabilities—exactly catastrophic forgetting reappearing in the "editing" paradigm. Edit quality is therefore measured on three axes simultaneously: reliability (the target fact is corrected), generalization (paraphrases / synonymous rewordings also take effect), and locality / specificity (unrelated knowledge is untouched). These three trade off against one another, isomorphic to stability-plasticity.

6. From-scratch EWC

import torch
import torch.nn.functional as F

def compute_fisher_diagonal(model, dataloader, device="cpu"):
    """
    Estimate the Fisher information matrix diagonal using squared log-likelihood gradients.
    dataloader: old-task data; model: model after completing the old task (parameters fixed).
    Returns dict: param_name -> F_i (same shape as parameter)
    """
    model.eval()
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()}
    n_samples = 0

    for x, y in dataloader:
        x, y = x.to(device), y.to(device)
        model.zero_grad()
        logits = model(x)
        log_prob = F.log_softmax(logits, dim=-1)
        # Use predicted label to approximate sampling — this is a simplified implementation;
        # the original EWC paper actually uses the true label y to estimate Fisher
        # (empirical Fisher = E_{(x,y)~D}[grad log p(y|x)^2])
        pred = log_prob.argmax(dim=-1)
        loss = F.nll_loss(log_prob, pred, reduction="sum")
        loss.backward()

        for n, p in model.named_parameters():
            if p.grad is not None:
                fisher[n] += p.grad.detach().pow(2)
        n_samples += x.size(0)

    fisher = {n: f / n_samples for n, f in fisher.items()}
    return fisher


class EWCTrainer:
    """
    Adds an EWC quadratic penalty during new-task training to prevent old-task parameters from being overwritten.
    """
    def __init__(self, model, old_dataloader, ewc_lambda=5000.0, device="cpu"):
        self.model = model
        self.device = device
        self.ewc_lambda = ewc_lambda

        # Save a snapshot of old-task parameters
        self.theta_star = {
            n: p.detach().clone()
            for n, p in model.named_parameters()
        }
        # Compute Fisher diagonal
        self.fisher = compute_fisher_diagonal(model, old_dataloader, device)

    def ewc_penalty(self):
        """EWC penalty term: (lambda/2) sum_i F_i (theta_i - theta*_i)^2"""
        penalty = torch.tensor(0.0, device=self.device)
        for n, p in self.model.named_parameters():
            penalty = penalty + (
                self.fisher[n] * (p - self.theta_star[n]).pow(2)
            ).sum()
        return 0.5 * self.ewc_lambda * penalty

    def train_step(self, x, y, optimizer, criterion):
        """One training step on the new task: task loss + EWC penalty."""
        x, y = x.to(self.device), y.to(self.device)
        optimizer.zero_grad()
        logits = self.model(x)
        task_loss = criterion(logits, y)
        loss = task_loss + self.ewc_penalty()
        loss.backward()
        optimizer.step()
        return task_loss.item(), loss.item()

ewc_lambda is the core hyperparameter: too small and forgetting remains severe; too large and the new task cannot be learned. In practice one typically searches in $[100, 10^4]$ and accumulates multiple Fisher groups as tasks grow (online EWC merges them via a running average).

Stratified follow-ups

L1 Foundations

L2 Advanced

L3 Deep-dive

Deep-dive

The following are interview-level advanced Q&A covering the 7 most frequently probed hard points from the notes above. All conclusions come from the cited papers and contain no original research by the author.

D1. Empirical Fisher vs. True Fisher — Which does EWC use, and when does it break down?

True Fisher is defined as the expectation of the outer product of log-likelihood gradients under the model's predictive distribution $p_\theta(y|x)$:

$$F_i^{\text{true}} = \mathbb{E}_{x \sim \mathcal{D},\; y \sim p_\theta(\cdot|x)}\!\left[\left(\frac{\partial \log p_\theta(y|x)}{\partial \theta_i}\right)^{\!2}\right]$$

Empirical Fisher replaces the $y$ sampled from the model distribution with the true labels $y$ in the dataset:

$$F_i^{\text{emp}} = \mathbb{E}_{(x,y) \sim \mathcal{D}}\!\left[\left(\frac{\partial \log p_\theta(y|x)}{\partial \theta_i}\right)^{\!2}\right]$$

The two are equal only when the model perfectly fits the data ($p_\theta(y|x) \approx \delta_y$). Research has shown that during optimization, empirical Fisher cannot generally capture second-order information, and its deviation from the true Hessian in practice can be large — pathological behavior can occur even on simple optimization problems.

EWC uses empirical Fisher: the implementation computes the expected squared gradient over $(x,y)$ pairs from the old-task dataset $\mathcal{D}_1$. The code comments also note "use true label $y$ to estimate Fisher."

When does the approximation break down?

Intuition: EWC's quadratic penalty is essentially a local quadratic approximation of the parameter space — diagonal empirical Fisher is the coarsest layer of this approximation. When the old task is well-converged, labels are clean, and parameter correlations are weak, the approximation is acceptable; otherwise the "importance scores" decouple from the true loss-landscape curvature, and the penalty protects wrong directions.

D2. Online / Streaming EWC — How to accumulate Fisher across tasks?

Standard EWC stores one $(F^{(k)}, \theta^{*(k)})$ pair for each new old-task encountered, with memory growing linearly in task count $K$. Online EWC (Progress & Compress, Schwarz et al. 2018)7Progress & Compress dual-network framework: active column learns new tasks; knowledge base consolidates with online EWC; Fisher is accumulated across tasks via exponential moving average. Schwarz 2018 ↗ merges all historical tasks' Fisher into a single $\tilde{F}$ via exponential moving average (EMA):

$$\tilde{F}^{(k)} = \gamma \cdot \tilde{F}^{(k-1)} + (1-\gamma) \cdot F^{(k)}$$

The total penalty degenerates to a single penalty term:

$$\mathcal{L}_{\text{online-EWC}} = \mathcal{L}_{\mathcal{T}_k}(\theta) + \frac{\lambda}{2}\sum_i \tilde{F}_i^{(k-1)}\bigl(\theta_i - \theta_i^{*(k-1)}\bigr)^2$$

Advantage: constant memory (stores only one $\tilde{F}$ and one $\theta^*$ snapshot).

Cost and risks:

• EMA exponentially downweights Fisher from earlier tasks — the older the task, the weaker the protection; it inherently favors protecting "recent" tasks.
• $\gamma$ is a new hyperparameter: $\gamma \to 1$ retains history but forgetting rate is high; $\gamma \to 0$ degenerates to looking only at the previous task.
• The reference point $\theta^*$ is updated each round; after each compression the new $\theta^*$ is not the common optimum for all historical tasks — the penalty center drifts as tasks accumulate.

SI (Synaptic Intelligence) is another streaming approach: it online-accumulates each parameter's integral contribution to loss reduction during training as an importance measure:

$$\Omega_i \propto \int_{\text{trajectory}} \frac{\partial \mathcal{L}}{\partial \theta_i} \cdot \dot\theta_i \, dt$$

SI requires no additional backward pass after the task ends and is suited to streaming settings without explicit task boundaries, but the signal-to-noise ratio of the importance estimate is lower than EWC.

D3. Stability Gap — Why does forgetting worsen before recovering?

De Lange et al. (arXiv 2022, ICLR 2023)8Proposes a per-iteration continuous evaluation framework; first systematically documents the stability gap: performance drops sharply after a task switch then recovers; evaluating only at task end misses this phenomenon. De Lange 2022 ↗ discovered through per-iteration continuous evaluation that almost all mainstream CL methods (including EWC, ER, A-GEM) experience a sharp drop in old-task performance in the first few steps after switching to a new task — followed by gradual recovery and even exceeding the pre-switch level as training progresses. This "drop then recover" phenomenon is called the stability gap.

Mechanistic intuition:

Task T1 training complete → switch to T2
     ↓ first few dozen steps
T2's large gradients hit shared feature layers → T1 representations temporarily disrupted → T1 performance drops sharply
     ↓ training continues
regularization / replay constraints begin to take effect → T1 representations gradually recover
     ↓ end of T2
T1 performance recovers (but may be below its peak at the end of T1 training)

Why was this not found before?

The standard evaluation protocol tests only once after each task is fully learned, which happens to skip the drop period — the stability gap is nearly invisible in the end-of-task "snapshots."

Interview key points:

• The stability gap means the BWT metric (end-of-task snapshot) underestimates the true magnitude of forgetting — in safety-critical settings (incremental updates of deployed LLMs), worst-case performance during training may be much more severe than BWT reflects.
• Mitigation directions: reduce new-task learning rate during the warm-up transition period; "probe" the new task with small batches before full-speed training; increase the old-task proportion in replay at the beginning of the task switch.

D4. Replay Sample Selection Strategies and Buffer Size Effects

Buffer size $M$ is one of the most critical hyperparameters in replay methods. Three main selection strategies:

① Random Reservoir Sampling

For a data stream of unknown length, maintain a buffer of size $M$; each new sample $x_t$ is included in the buffer with probability $M/t$ (randomly replacing an existing sample). This guarantees that every sample in the buffer is a uniform subset of all historical samples — no class bias, simple to implement, and the default strategy for ER / A-GEM.

② Herding (iCaRL)

iCaRL's herding algorithm: greedily and iteratively selects samples so that the feature mean of the exemplar set best approximates the feature mean of the entire class:

$$p_t = \arg\min_{x \in \mathcal{D}_k} \left\| \mu_k - \frac{1}{t}\!\left(\phi(x) + \sum_{j=1}^{t-1}\phi(p_j)\right)\right\|$$

Herding outperforms random sampling for class exemplar selection, especially when $M$ is very small (only a few samples per class), preserving within-class diversity better. However, herding requires having all class data available and depends on the feature space — after feature drift, old-class exemplars may no longer represent their current features.

③ Gradient-Based Selection

Selects old samples that have the greatest influence on the new-task gradient update — typically samples whose gradient direction most "conflicts" with the new-task gradient. The intuition is to constrain gradients using the hardest-to-satisfy samples. Compute cost is high (requires extra backward passes to estimate gradient influence per candidate sample); rarely used in large-scale experiments.

Buffer size effects summary:

Core insight: as $M \to \infty$, all replay methods degenerate to joint training (the upper bound); when $M$ is small, exemplar representativeness matters more than randomness — herding wins in this regime.

D5. GEM's Per-Task QP Cost vs. A-GEM's Single Mean Constraint

GEM's QP problem:

Current new-task gradient is $g$; for each old task $k$ there is a constraint $\langle g, g_k \rangle \geq 0$. If the constraint is violated, solve:

$$\tilde{g} = \arg\min_v \|v - g\|^2 \quad \text{s.t.} \quad \langle v, g_k\rangle \geq 0,\; \forall k \in \{1,\ldots,K-1\}$$

This is a quadratic program (QP) with $(K-1)$ inequality constraints. Standard QP solvers have time complexity $\mathcal{O}(K^3)$ and space complexity $\mathcal{O}(K^2)$ — quickly infeasible as task count grows. GEM's implementation approximates the solution with iterative methods such as Frank-Wolfe; each step still requires $K$ gradient inner-product computations, i.e., $\mathcal{O}(K \cdot d)$ ($d$ = parameter dimension).

A-GEM's single constraint:

A-GEM merges all old-task constraints into one average gradient:

$$g_{\text{ref}} = \frac{1}{K-1}\sum_{k=1}^{K-1} g_k, \quad \text{s.t.}\; \langle g, g_{\text{ref}} \rangle \geq 0$$

If violated, the projection has a closed-form solution:

$$\tilde{g} = g - \frac{\langle g, g_{\text{ref}} \rangle}{\|g_{\text{ref}}\|^2} g_{\text{ref}}$$

Per-step computation drops from $\mathcal{O}(K \cdot d)$ to $\mathcal{O}(d)$ — independent of task count (computing $g_{\text{ref}}$ can be done once and reused).

Cost:

A-GEM satisfies an average-direction constraint and does not guarantee $\langle \tilde{g}, g_k\rangle \geq 0$ for every individual old task $k$ — the loss on some single old task may increase. Empirically, A-GEM's AA and BWT are comparable to GEM's, but when gradients across tasks are highly heterogeneous (some task directions deviate greatly from the mean), certain old tasks occasionally experience more forgetting than with GEM.

Interview memory point: GEM = per-task constraint + quadratic program, $\mathcal{O}(K^3)$ QP; A-GEM = single mean constraint + closed-form projection, $\mathcal{O}(d)$; the trade-off is sacrificing per-task constraint guarantees for linear time complexity.

D6. Why Is Class-IL Hardest? The Role of the Output Head

van de Ven & Tolias 20199Systematically compares difficulty across the three CL settings (Task-IL / Domain-IL / Class-IL) on Split MNIST and Split CIFAR-100; regularization methods almost completely fail on Class-IL. van de Ven 2019 ↗'s three-scenario framework reveals the fundamental difficulty gap:

Output head structure comparison across three scenarios:

Three obstacles that make Class-IL hardest:

1. Task-ID inference problem: the model does not know which task the current input belongs to and cannot route to the corresponding sub-classifier — it must distinguish all classes on a single head.

2. Output head historical bias (recency bias): when learning a new task, only the new task's classes generate large gradient updates, skewing the logit scale of the output layer toward the new task — old-class logits are suppressed even if the feature layer still remembers the old task. This is "classifier-layer forgetting" unique to Class-IL.

3. Fundamental failure of regularization: EWC protects feature-layer parameters, but the new-class nodes initialized in the output head interfere with the gradient flow of old-class nodes — the Fisher diagonal cannot capture this cross-class output-layer interference. van de Ven et al.'s experiments show that EWC's accuracy on Class-IL approaches chance level.

Remedies:

• Replay + prototype classifier (e.g., iCaRL): use exemplar feature means for nearest-neighbor classification, bypassing output head bias.
• Task-agnostic feature learning: freeze a pre-trained backbone and only update a lightweight classification head — reducing feature drift.
• Empirical bias correction: at Class-IL test time, apply temperature adjustment or weighting to old-class logits to counteract recency bias.

D7. LLM Forgetting vs. Capability Interference — Measurement Methods and Task-Order Sensitivity

Key differences between LLM "forgetting" and classic CL:

How to measure LLM CL quality:

1. Capability matrix tracking: after each alignment stage ends, evaluate on several "probe benchmarks" (covering general reasoning, code, math, safety) — equivalent to constructing an $a_{i,j}$ matrix at LLM scale.
2. BWT analogy for LLMs: measure "change in coding capability after SFT compared to pretrain baseline" — if negative, it is part of the alignment tax.
3. Activation/gradient analysis: detect which layers' activation distributions change most before and after new-task training — locating the knowledge-storage layers that are "overwritten."

Task-order sensitivity:

LLM CL is highly sensitive to task order, because:

• The gradient directions during sequential training depend on the loss landscape shaped by previous tasks — doing math SFT before safety RLHF produces a very different result from the reverse order.
• Harder tasks (math/code) require large learning rates and large gradients, causing greater damage to parameters used by subsequent smaller tasks.
• Cumulative nature of alignment tax: in the SFT → DPO → RL sequence, each step's forgetting tax accumulates on the next step's checkpoint.

KL constraint is implicit EWC:

The KL penalty in PPO-RLHF:

$$r_{\text{KL}}(\theta) = \mathbb{E}\!\left[\log\frac{\pi_\theta(a|s)}{\pi_{\text{ref}}(a|s)}\right] \cdot (-\beta)$$

Expanding this with a Taylor expansion around reference policy $\pi_{\text{ref}}$, the second-order term of KL divergence is proportional to the Fisher information matrix:

$$\text{KL}(\pi_\theta \| \pi_{\text{ref}}) \approx \frac{1}{2}(\theta - \theta_{\text{ref}})^T F(\theta_{\text{ref}}) (\theta - \theta_{\text{ref}})$$

That is, PPO's KL penalty ≈ a full-matrix EWC weighted by Fisher — both are "quadratic penalties on parameter deviation from a reference point, weighted by information-geometric curvature." The difference: EWC uses a diagonal approximation and explicitly stores the old-task Fisher; the KL constraint uses the full distributional distance and dynamically anchors to the current reference policy. The KL term in DPO follows the same logic — the reference model is mathematically equivalent to EWC's $\theta^*$.

D8. LoRA-based CL — Adapter Interference and Routing

Sources of adapter interference:

The naive approach is to train an independent set of LoRA weights $\Delta W_k = B_k A_k$ for each task and route by task-ID at test time. Two sources of interference exist:

1. Parameter-space overlap: different tasks' low-rank subspaces may overlap substantially — if $\text{span}(A_k) \cap \text{span}(A_j) \neq \emptyset$, one task's adapter can interfere with another task's activations after merging.
2. Routing failure without task-ID: in Class-IL or continual pretraining settings, task-ID is unavailable at test time, making it impossible to route to the correct adapter.

O-LoRA's orthogonal subspace approach:

O-LoRA enforces, during LoRA training for task $k$, that the new task's low-rank subspace is orthogonal to all previously seen tasks' subspaces:

$$A_k V_{\text{prev}} \approx 0, \quad V_{\text{prev}} = \text{span}\bigl(\{A_j\}_{j<k}\bigr)$$

This is achieved by projecting gradients onto the orthogonal complement of $V_{\text{prev}}$. Orthogonal subspaces guarantee that different task adapters' activations do not interfere with each other — in settings where task-ID is known this yields approximately zero forgetting without storing old-task data.

Limitations:

• Available orthogonal dimensions decrease as task count increases — in rank-$r$ LoRA, at most about $d/r$ fully orthogonal tasks can be supported ($d$ = weight matrix dimension).
• In Class-IL settings without task-ID, orthogonality does not solve the routing problem — additional mechanisms to infer task-ID or use prototype classification are still required.
• Enforcing orthogonality may limit forward transfer between related tasks — tasks that could share subspaces to accelerate learning lose that benefit.

Practice in LLM post-training:

In the sequential alignment pipeline (SFT → DPO → RL), freezing the backbone and updating only one set of LoRA per stage is equivalent to a parameter isolation scheme with known task-ID — the accumulation of alignment tax is largely confined to the LoRA layers and backbone knowledge is not overwritten. The cost is the need to manage multiple adapter sets and their merging/routing logic at deployment.

References

All are original sources for classic load-bearing methods, verified line by line (title + arXiv ID). Click the superscript to jump; click ↩ to return.

1. Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. PNAS 2017. arXiv:1612.00796 — EWC: Fisher diagonal quadratic penalty against forgetting. ↩
2. Lopez-Paz et al. Gradient Episodic Memory for Continual Learning. NeurIPS 2017. arXiv:1706.08840 — GEM: gradient projection constraint + forward/backward transfer. ↩
3. Chaudhry et al. Efficient Lifelong Learning with A-GEM. ICLR 2019. arXiv:1812.00420 — A-GEM: single mean gradient constraint, efficient GEM. ↩
4. Buzzega et al. Dark Experience for General Continual Learning: a Strong, Simple Baseline. NeurIPS 2020. arXiv:2004.07211 — DER: store logits as soft distillation targets + rehearsal. ↩
5. Rusu et al. Progressive Neural Networks. 2016. arXiv:1606.04671 — Add a new column per task + lateral connections; structurally zero forgetting. ↩
6. Li and Hoiem. Learning without Forgetting. ECCV 2016. arXiv:1606.09282 — LwF: old model soft outputs as distillation targets; no old data needed. ↩
7. Schwarz et al. Progress & Compress: A scalable framework for continual learning. ICML 2018. arXiv:1805.06370 — Dual-network active column + knowledge base; online EWC accumulates Fisher across tasks via EMA, constant memory. ↩
8. De Lange, van de Ven, and Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. ICLR 2023. arXiv:2205.13452 — Per-iteration continuous evaluation framework; discovers the stability gap phenomenon of sharp performance drop then recovery after task switch. ↩
9. van de Ven and Tolias. Three scenarios for continual learning. 2019. arXiv:1904.07734 — Systematically defines the three scenarios Task-IL / Domain-IL / Class-IL; shows that regularization methods almost completely fail on Class-IL. ↩
10. Meng et al. Locating and Editing Factual Associations in GPT. NeurIPS 2022. arXiv:2202.05262 — ROME: causal localization + rank-1 editing of factual associations in middle-layer MLPs. ↩
11. Meng et al. Mass-Editing Memory in a Transformer. ICLR 2023. arXiv:2210.07229 — MEMIT: batch-editing thousands of facts across multiple layers. ↩