Continual / Lifelong Post-training (production-validated methods only)

A: The root cause is gradient interference + weight drift: when doing gradient descent on new data, the update direction $-\nabla_\theta \mathcal{L}_{\text{new}}$ often conflicts with old-task gradients ($\langle\nabla\mathcal{L}_{\text{old}},\nabla\mathcal{L}_{\text{new}}\rangle < 0$), and weights drift in a direction that is "good for new, bad for old," causing old capabilities to degrade. The larger the drift, the higher the LR, and the longer the training, the worse the forgetting. The simplest and most effective method is replay (data rehearsal/mixing): mix a proportion of old/general data into the new training stream, directly telling the model "don't forget this" from the data side. The engineering focus is on the mixing ratio, deduplication, and quality filtering.

Follow-up: Why does low LR + few epochs help reduce forgetting? Because it limits the magnitude of per-step weight drift—small-step fine-tuning = small perturbations to old knowledge. Similarly, LoRA limits the capacity of the change via a low-rank adapter, and the adapter can be discarded to roll back if things go wrong.

A: Three reasons: ① Change isolation—full fine-tuning modifies all weights; a low-rank adapter only changes a small subset of parameters, causing less physical perturbation to old capabilities. ② Discardable—a broken adapter can be simply discarded to revert to the base model; full fine-tuning cannot be cleanly rolled back. ③ Cheap incremental adaptation—a separate adapter per new task/data, composed on demand at inference, avoiding the interference that comes from "one model simultaneously satisfying all tasks." The tradeoff: LoRA's expressivity is limited—complex new capabilities may require higher rank or full fine-tuning.

Follow-up: Can LoRA completely avoid forgetting? No. LoRA limits the "magnitude" of the change, not the "direction"—if new data strongly conflicts with old knowledge, LoRA still overwrites old capabilities, just with a smaller amplitude. Preventing forgetting still requires replay / KL regularization / merging alongside LoRA.

A: BWT = $\frac{1}{T-1}\sum_{j=1}^{T-1}(R_{T,j} - R_{j,j})$, where $R_{i,j}$ = the metric on task $j$ after learning task $i$. BWT < 0 means forgetting (the more negative, the worse). vs retention ratio: BWT directly quantifies the absolute degradation ($\Delta$); retention = $R_{T,j} / R_{j,j}$ measures the relative fraction preserved—the two are complementary. In production, both are monitored alongside regression probes on old benchmarks, not just the aggregate BWT.

Follow-up: What are BWT's limitations? It only measures "how much was lost," not "how much was gained"—pair it with FWT (forward transfer, positive transfer of new knowledge) for a complete picture: $BWT+FWT$ is the "net gain" criterion for an incremental update. Additionally, BWT assumes tasks can be independently evaluated; for interleaved conversational/reasoning capabilities, proxy benchmarks must be chosen carefully.

A: There is no universal formula. Practical heuristics: ① Scale-based: old data volume $\approx$ 10–50% of new data volume (depending on task similarity). ② Dynamic: monitor retention on old benchmarks—if it degrades beyond a threshold (e.g., 2%), increase the replay ratio. ③ Quality over quantity: a small amount of high-quality, representative old samples is more effective than a large volume of low-quality old data. When replay is not worth it: the new and old tasks are radically different (e.g., switching from code to medicine), the old data volume is too large to replay fully, or storage/bandwidth constraints make it impractical—in these cases, prefer LoRA isolation or model merging.

Follow-up: What's the relationship between replay and KL regularization? They are complementary, not substitutes. Replay reminds the model of the old distribution from the data side; KL regularization constrains the policy not to deviate from the reference from the optimization side. In RLHF settings, KL is an essential anti-drift mechanism (usually combined with replay). The combination of both typically outperforms either alone.

A: The theoretical foundation is Linear Mode Connectivity (LMC): multiple models fine-tuned from the same pretrained initialization tend to fall within the same basin of the loss landscape, with a low loss barrier under linear interpolation (they are linearly mode-connected), so the average point still has low loss. Prerequisite: shared initialization + limited drift (low-LR fine-tuning). Failure: different initializations / too-large task divergence → high barrier → averaging makes things worse. Task arithmetic ($\theta_0 + \sum \tau_i$) further assumes task vectors are approximately linearly additive (implicitly: they lie in different subspaces with little mutual interference); when this fails, use TIES (Trim small-magnitude params, Elect signs, Disjoint Merge) or DARE (Drop large fractions of delta + REscale) as pre-processing—these significantly outperform naive averaging.

Follow-up: What do the three TIES-Merging steps actually do? ① Trim: reset parameters that changed only slightly (sparsify the delta, reduce noise). ② Elect Signs: vote across models to determine the dominant sign direction for each parameter (resolve +/- conflicts). ③ Disjoint Merge: only merge parameters aligned with the agreed-upon sign. After these three steps, parameter conflicts are resolved and merge quality improves substantially.

A: Four core reasons: ① Cost: EWC requires computing and storing the Fisher Information Matrix (intractable at LLM parameter counts), and GEM requires solving a projection at every step (whose cost scales with episodic memory size). ② Complexity: many hyperparameters, hard to tune; for LLMs' "entangled capabilities," it's difficult to precisely define "which parameters are important to which tasks" for regularization objectives. ③ Effectiveness: simple replay + merging + PEFT + KL often matches or outperforms classical CL algorithms in practice, while being much simpler. ④ Scale mismatch: classical CL methods were designed for small models / small tasks; LLMs' 100B+ parameters and thousands of tasks make them computationally infeasible.

Follow-up: Do EWC and friends still have academic value? Yes—they provide the theoretical framework for forgetting (importance weighting, gradient projection, episodic memory compression) and inspired lighter-weight LLM-era variants. But in an interview: mention "academically there's EWC etc.," then honestly add "production mainstream is replay / merging / PEFT / KL."

A: It is most severe at the final RL step—with no annotation constraints and only reward chasing, over-optimization is easy (policy drift erases capabilities and formatting learned during SFT). XRIGHT (Fernando et al., arXiv:2410.15483) gave the first theoretical proof that sequential training's forgetting has a non-diminishing optimality gap (more training cannot automatically eliminate it). Mitigations: ① keep KL to the SFT reference during the RL phase; ② mix in SFT replay (data rehearsal); ③ add verifier constraints on key capabilities (not just reward scoring); ④ run regression probes after each phase—monitor degradation on old benchmarks, roll back or adjust the mixing ratio if thresholds are exceeded; ⑤ model averaging: Wang (ACL 2024) found that per-layer interpolation between pre- and post-RLHF weights achieves the optimal point on the alignment-capability Pareto front.

Follow-up: Continual alignment vs retraining—how to choose? If the new alignment requirement is a small increment relative to the old one and in-distribution → incremental RL fine-tuning + KL anchoring to the ref. If cumulative degradation is evident after multiple rounds (regression probe alarms) → retrain from the base checkpoint and merge old adapters/checkpoints. Google PaLM-2 empirical evidence: larger models are more robust to simple averaging—"incrementally update → periodically retrain + merge checkpoints" is the current mainstream pattern.

A: Task arithmetic assumes task vectors $\tau_i = \theta_{\text{ft},i} - \theta_0$ are approximately linearly additive—implicit premises: ① the $\tau_i$ lie in different subspaces of the parameter space with little mutual interference; ② the change magnitudes are not so large as to cause nonlinear effects. Failure conditions: ① highly correlated/conflicting tasks → interference after addition (Ilharco et al. characterize failure with "interference"); ② excessive drift magnitude → the linear-superposition assumption breaks (the loss surface is no longer flat); ③ models fine-tuned from different inits do not share LMC → the linear interpolation barrier is high. Mitigations: TIES-Merging (Trim + Elect Sign + Merge) and DARE (Drop delta + REscale) as pre-processing significantly reduce parameter conflict during merging. Large-model empirical finding (Google, 64B PaLM-2): the larger the model, the closer the results of different merging methods (TIES/DARE-TIES/Averaging)—even simple averaging suffices.

Follow-up: Why is merging easier for larger models? Larger models have more parameter redundancy, so deltas from different fine-tuning runs are more likely to occupy sparse/orthogonal subspaces of the parameter space, with less mutual interference—hence simple averaging works. This is the "overparameterization dividend" manifesting in the continual learning setting.

A:

Key principle: don't just look at aggregate metrics—monitor per-capability-dimension trends (reasoning / code / safety / dialogue each trend separately). An aggregate BWT can hide severe degradation in specific capabilities.

Follow-up: What did the COPR benchmark (Zhan et al., arXiv:2402.14228) contribute? It established the first continual human preference alignment benchmark—with a three-tier evaluation (reward-based + GPT-4 + human), covering multiple backbones, multiple replay configurations, and multiple learning orders. It proposed Lagrangian dual dynamic regularization (per-step regularization of the current policy against the historically optimal policy), which is robust across different backbones and replay sizes.