ML / DL Fundamentals — Public Cheat Sheet

Answer:

• SGD: Fixed learning rate, no adaptive mechanism; converges slowly but can generalize better.
• Adam: Maintains first moment $m_t$ (gradient mean) and second moment $v_t$ (mean squared gradient); after bias correction, adaptively adjusts the learning rate per parameter. However, its weight decay implementation mixes the regularization gradient into the second-moment denominator, so the effective decay magnitude varies with gradient scale.
• AdamW: Decouples weight decay from the adaptive update, applying it directly to parameters: $\theta \leftarrow (1-\eta\lambda)\theta - \eta\,\hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$, giving decay behavior consistent with SGD.

Follow-up: Why is Adam's L2 regularization implementation considered "incorrect"? What does AdamW's decoupling fundamentally change?

The L2 regularization gradient $\lambda\theta$ is divided by $\sqrt{\hat{v}_t}$, so parameters with large gradients receive weak decay and those with small gradients receive strong decay — contrary to the design intent. AdamW applies uniform decay to all parameters.

Answer: $m_0 = v_0 = 0$, so during the first few steps the exponential moving averages are heavily biased toward zero. Taking $m_t$ as an example: $m_t = (1-\beta_1)\sum_{i=1}^{t}\beta_1^{t-i}g_i$, whose expectation is $\mathbb{E}[m_t] = \mathbb{E}[g_t]\cdot(1-\beta_1^t)$. Without dividing by $(1-\beta_1^t)$, the initial step sizes are systematically too small. The correction $\hat{m}_t = m_t/(1-\beta_1^t)$ ensures reasonable update magnitudes early in training.

Follow-up: What intuition do the default values $\beta_1 = 0.9$ and $\beta_2 = 0.999$ correspond to?

$\beta_1=0.9$ corresponds to a history window of roughly 10 steps (gradient direction estimation); $\beta_2=0.999$ corresponds to roughly 1000 steps (slow tracking of gradient magnitude for stable adaptive scaling).

Answer:

• SGD+Momentum: CV tasks (e.g., ResNet, ViT), where careful tuning often yields better generalization, but LR schedule and momentum must be dialed in carefully.
• AdamW: The standard choice for NLP / LLM pre-training and fine-tuning; adaptive LR is friendly to sparse token gradients and reduces tuning overhead.

Follow-up: Why is AdamW almost universally used for LoRA fine-tuning?

LoRA has few parameters and needs to converge quickly to a good low-rank subspace; adaptive LR reduces tuning burden. LoRA is commonly used in LLM fine-tuning settings, where AdamW is inherited from the pre-training choice.

Answer: Initial parameters are random and gradient variance is high. Adam's second moment $v_t$ is poorly estimated at the start; using a large LR immediately causes the effective step size $\eta/\sqrt{\hat{v}_t}$ to be abnormally large, leading to loss spikes or divergence. Warmup lets $v_t$ accumulate sufficiently under a low LR before the LR is gradually increased, stabilizing early training.

Follow-up: Is warmup still necessary when fine-tuning from a pre-trained checkpoint?

Usually yes, but the warmup duration can be greatly reduced (e.g., only a few hundred steps). The pre-trained weights are already stable, but newly added LoRA adapters or shifts in the data distribution still require a brief adaptation period.

Answer:

• Cross-entropy $H(p,q) = -\sum_i p_i \log q_i$. When $p$ is one-hot, this simplifies to $-\log q_y$, i.e., NLL. PyTorch's CrossEntropyLoss = LogSoftmax + NLLLoss.
• Why not MSE: MSE gradients on softmax outputs are extremely small in the saturation region (vanishing gradients), leading to slow convergence. CE's gradient $\hat{q}_y - 1$ never vanishes and is numerically well-behaved.

Follow-up: How does label smoothing modify CE loss? Why does it improve calibration?

It replaces the one-hot target with $(1-\epsilon)y + \epsilon/K$, preventing the model from chasing extreme confidence scores, reducing overconfident predictions, and improving probability calibration.

Answer: During training, each neuron output is randomly set to zero with probability $p$, equivalent to training an ensemble of exponentially many sub-networks. At inference, no dropout is applied. PyTorch uses inverted dropout by default: during training, non-zero outputs are scaled by $1/(1-p)$, so inference requires no extra operation and expected outputs remain consistent.

Follow-up: Where is Dropout typically applied in a Transformer? Is it still used in large-scale pre-training?

Typically after attention weights, before residual addition, and after the FFN hidden layer. In large-scale pre-training (e.g., LLaMA, GPT), Dropout is generally not used, because massive data makes overfitting unlikely and Dropout can interfere with distributed training communication.

Answer: $$\text{PPL} = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(x_i \mid x_{<i})\right)$$ This is the exponentiated per-token cross-entropy. Intuitively, it represents the average number of equally probable choices the model faces at each position; lower PPL means the model is more confident.

Follow-up: Is PPL comparable across different tokenizers?

Not directly. Different tokenizers produce tokens of different granularity (e.g., BPE vs SentencePiece), so each token carries different amounts of information. A normalized metric such as bits-per-byte (BPB) is needed for fair comparison.

Answer:

• Vanishing gradients: In deep networks, repeated multiplication by factors less than 1 drives gradients toward zero, leaving shallow-layer parameters nearly unchanged. Remedies: ReLU/GELU activations, residual connections, normalization (LayerNorm/BatchNorm), proper initialization.
• Exploding gradients: Repeated multiplication by factors greater than 1 causes exponential growth. Remedies: gradient clipping (most direct), weight regularization, reducing learning rate.

Follow-up: How do residual connections in Transformers help gradient flow?

The residual path $x + F(x)$ gives gradient $\partial L/\partial x = \partial L/\partial(\text{output}) \cdot (1 + \partial F/\partial x)$, where the constant term 1 guarantees gradients can reach shallow layers directly without being attenuated by intermediate layers.

Answer:

BatchNorm performs poorly in NLP: variable sequence lengths make batch statistics unstable; inference requires maintaining running statistics, which is ill-suited to autoregressive generation.

Follow-up: What is the difference between Pre-LN and Post-LN? Which trains more stably?

Pre-LN places LN before the sublayer input; gradients flow through the residual path unimpeded, making training more stable. Post-LN places LN after the residual; gradients in deep layers may shrink, requiring careful tuning and warmup. Modern LLMs predominantly use Pre-LN (or Pre-RMSNorm).

Answer:

• Xavier: $W \sim \mathcal{N}(0, 2/(n_{\text{in}}+n_{\text{out}}))$; assumes linear or tanh activations; goal is to maintain constant output variance per layer.
• Kaiming: $W \sim \mathcal{N}(0, 2/n_{\text{in}})$; designed for ReLU — zeroing the negative half halves the variance, and the factor of 2 compensates.

Follow-up: How are attention projection matrices typically initialized in Transformers?

Typically with Xavier, since inputs pass through softmax or LayerNorm and are approximately linear. Defaults vary slightly across frameworks, but Xavier uniform or normal is the mainstream practice.

Answer:

1. Forward pass: compute outputs layer by layer, caching intermediate activations $a_l$.
2. Compute loss: obtain the scalar loss $L$.
3. Backward pass: starting from $\partial L / \partial L = 1$, apply the chain rule layer by layer to compute $\partial L / \partial W_l$, using the cached activations.
4. Parameter update: the optimizer updates parameters using the gradients.

Key mechanism: cache during forward pass → reuse during backward pass; the chain rule ensures correct gradient computation at each layer.

Follow-up: How does gradient checkpointing trade off memory vs computation?

Only activations at every $k$-th layer are saved; intermediate activations are recomputed during the backward pass. Memory drops from $O(L)$ to roughly $O(\sqrt{L})$, at the cost of approximately 33% extra computation. Useful when memory is the bottleneck.

Answer:

1. Clip by value: clamp each gradient component to $[-c, c]$; simple but may alter the gradient direction.
2. Clip by norm: $\mathbf{g} \leftarrow \mathbf{g} \cdot \min(1, c/\|\mathbf{g}\|_2)$; preserves gradient direction, only scales magnitude.

LLM training almost always uses clip by norm (typical threshold 1.0) because it does not distort the optimization direction.

Follow-up: How do you diagnose a loss spike (sudden gradient norm surge)?

Common causes: data anomalies (very long sequences, abnormal tokens), LR too large, numerical overflow (FP16), or a batch whose gradients conflict sharply with historical directions. Check data quality, lower the LR, switch to BF16, or increase the clipping threshold while monitoring.

Answer:

• Equivalent under SGD: adding $\frac{\lambda}{2}\|\theta\|^2$ to the loss contributes gradient $\lambda\theta$ to the total gradient, resulting in $\theta \leftarrow \theta(1-\eta\lambda) - \eta\nabla L$ — i.e., weight decay.
• Not equivalent under Adam: $\lambda\theta$ is divided by $\sqrt{\hat{v}_t}$; parameters with large gradients receive weak decay and those with small gradients receive strong decay. AdamW decouples decay as $\theta \leftarrow (1-\eta\lambda)\theta$, achieving uniform decay.

Follow-up: What is a typical weight decay value for AdamW? Which parameters are typically excluded?

Common values: 0.01–0.1. LayerNorm's $\gamma, \beta$ and biases are typically excluded from weight decay (they have other regularization mechanisms or are low-dimensional); decay is applied only to weight matrices.

Answer: Stop training when validation loss stops improving (no improvement for $p$ epochs), to prevent overfitting.

Drawbacks:

1. Requires a separate validation set, reducing available training data.
2. Requires continuous training monitoring (engineering overhead).
3. Plateaus may trigger early stopping spuriously (requires tuning patience).
4. The final model is an intermediate checkpoint, not a fully converged solution.

Follow-up: What is the theoretical connection between Early Stopping and L2 regularization?

Both have the effect of constraining the parameter space. Early Stopping limits how far parameters can move from initialization (an implicit norm constraint); in the linear approximation of gradient descent, this is equivalent to an L2 regularized solution with a specific regularization coefficient.

Answer: $$\mathbb{E}[\text{MSE}] = \text{Bias}^2 + \text{Variance} + \text{Noise}$$

• High bias (underfitting): both train and val loss are high; the model lacks capacity.
• High variance (overfitting): train loss is low, val loss is notably higher; the model has memorized the training set.

Diagnostic tools: train/val loss curves, learning curves (error vs. training set size).

Follow-up: What new challenges arise in diagnosing "overfitting" in the LLM era?

Key challenges: (1) training data and evaluation data may overlap (benchmark contamination), making low loss appear without genuine generalization improvement; (2) LLMs' in-context learning ability is not reflected in train loss; (3) evaluation must rely on downstream task performance, not just val loss.

Answer: Train a small model (student) using a large model's (teacher's) soft labels (the probability distribution smoothed by temperature $T$), rather than hard labels alone. The loss is: $$L = \alpha \cdot \text{CE}(y, \hat{y}_s) + (1-\alpha) \cdot T^2 \cdot \text{KL}(\hat{y}_t^{(T)} \| \hat{y}_s^{(T)})$$ Soft labels convey inter-class similarity relationships ("dark knowledge"), helping the student learn richer representations.

Follow-up: What is the difference between token-level and sequence-level KD?

Token-level KD computes KL divergence on logits at each position (per-token alignment); sequence-level KD matches full-sequence probability distributions or output sequences (e.g., using teacher's beam search output as pseudo-labels). The latter is more flexible but provides sparser training signal.

Answer:

• Large batches provide more accurate gradient estimates, but reduce gradient noise (which has a regularizing effect) and may converge to sharp minima with worse generalization.
• Linear Scaling Rule: when batch size is multiplied by $k$, scale the LR by $k$ to keep the expected per-update step size unchanged. Beyond a certain scale, more conservative scaling (e.g., sqrt rule) is needed.

Follow-up: How does gradient accumulation simulate large batches? Any limitations?

Split a mini-batch into $k$ micro-batches, accumulate gradients, then update once — equivalent to $k\times$ batch size. Limitations: (1) if BatchNorm is used, statistics are still computed per micro-batch, not the effective large batch; (2) the number of optimizer steps decreases by $k\times$, reducing communication frequency.

Answer: $$\text{GELU}(x) = x \cdot \Phi(x)$$ where $\Phi(x)$ is the standard normal CDF. GELU is a smooth approximation of ReLU: it retains small values in the negative region (no hard cutoff) and acts as a probabilistic gate (larger $x$ has a higher probability of being retained). It is more suitable than ReLU for NLP tasks and has smoother gradients near zero.

Follow-up: What is SwiGLU? How does it structurally differ from a GELU FFN?

$\text{SwiGLU}(x) = \text{Swish}(xW) \odot (xV)$, using two linear projections for gating. A standard FFN uses $W_{\text{up}} \to \text{GELU} \to W_{\text{down}}$ (2 matrices); SwiGLU FFN requires 3 matrices (gate, up, down), and the hidden dimension is typically set to $(8/3) \cdot d_{\text{model}}$ to keep the total parameter count comparable.

Answer:

• Motivation: FP16/BF16 reduces memory and speeds up computation (Tensor Cores), but FP16's small dynamic range (max ~65504) makes gradients prone to underflow/overflow.
• Approach: forward + backward in low precision; convert back to FP32 master weights for gradient updates to preserve precision; FP16 requires loss scaling to prevent gradient underflow.
• BF16: same dynamic range as FP32 (8 exponent bits); no loss scaling needed; the mainstream choice for LLM training today.

Follow-up: What additional challenges does FP8 training introduce?

FP8 has lower precision (E4M3/E5M2 formats), requiring finer-grained scaling strategies (e.g., per-tensor or per-channel scaling); gradient accumulation requires higher precision; some layers (e.g., logits before attention softmax) are more precision-sensitive and may need to be kept at higher precision.

Answer: During training, only a subset of key layer activations are saved (e.g., every $k$ layers); activations that were discarded are recomputed via a forward pass during backpropagation. This trades roughly 30% extra computation to reduce memory from $O(L)$ to $O(\sqrt{L})$ ($L$ = number of layers).

When to use: large batches, long sequences, or memory-constrained settings. If using LoRA with most parameters frozen, activation memory overhead is relatively manageable and checkpointing may not be needed.

Follow-up: Do gradient checkpointing and ZeRO-Offload solve the same problem?

Not exactly. Checkpointing reduces activation memory through recomputation; ZeRO-Offload offloads optimizer states and gradients from GPU to CPU/SSD. They address different memory bottlenecks and can be used together.

Answer:

1. Data check: look for anomalous inputs (very long sequences, NaN values, special tokens).
2. Gradient monitoring: log per-layer gradient norms to confirm whether a specific layer is exploding.
3. Precision check: is FP16 overflowing? Verify that the loss scaling factor is reasonable.
4. LR / optimizer state: is the LR too large? Has Adam's $v_t$ estimate been contaminated by the spike?
5. Recovery strategy: can you restart from the checkpoint before the spike (skipping the problematic batch)?
6. Long-term mitigation: increase clipping frequency (per-step vs per-accumulation), switch from FP16 to BF16.

Follow-up: Why does it take a long time for Adam to recover after $v_t$ is contaminated by a spike?

$v_t$ is an exponential moving average ($\beta_2=0.999$); a single spike's contribution $g_t^2$ decays as $\beta_2^k$, taking roughly 1000 steps to dissipate. During this decay, $\sqrt{\hat{v}_t}$ is inflated, causing the effective learning rate $\eta/\sqrt{\hat{v}_t}$ to be systematically reduced for all parameters.

Answer: Let the Transformer layer output be $x_{l+1} = x_l + F_l(\text{LN}(x_l))$ (Pre-LN) or $x_{l+1} = \text{LN}(x_l + F_l(x_l))$ (Post-LN).

Pre-LN: $\frac{\partial L}{\partial x_l} = \frac{\partial L}{\partial x_{l+1}} \cdot \left(I + \frac{\partial F_l}{\partial x_l}\right)$ — the identity term $I$ guarantees direct gradient flow, unaffected by the LN Jacobian.

Post-LN: gradients must pass through the LN Jacobian at every layer; the $\gamma/\sqrt{\sigma^2+\epsilon}$ scaling factor can accumulate and shrink gradients in deep layers, causing training instability.

Trade-off: Post-LN may theoretically have stronger representational capacity (LN after the residual better controls activation scale), but training difficulty is higher.

Follow-up: How do methods like DeepNorm attempt to combine the benefits of both?

DeepNorm scales the residual branch in a Post-LN setting: $x_{l+1} = \text{LN}(\alpha x_l + F_l(x_l))$, where $\alpha > 1$ amplifies the residual path while the initialization variance of $F_l$ is reduced, stabilizing deep-layer gradients while preserving Post-LN's representational capacity.

Answer: Possible causes:

1. LR too large or too small: in a flat region, too-large LR causes oscillation; too-small LR leads to slow progress.
2. Data quality bottleneck: information already present in the training data has been learned; remaining loss comes from noise or incompressible content.
3. Optimizer state issues: Adam's $v_t$ may become overconfident during plateaus, causing step sizes to shrink.
4. Saddle point in the loss landscape: gradients near zero but not at the optimum.

Remedies: adjust the LR schedule (e.g., WSD: warmup-stable-decay), revise data mixing strategy (curriculum), reinitialize optimizer states, or temporarily increase LR to escape the plateau.

Follow-up: What is the WSD schedule? What advantages does it have over cosine decay?

WSD has three phases: linear warmup → constant LR (stable phase) → rapid decay in the final stage. Advantages: (1) the stable phase can be extended flexibly (training can be continued or stopped at any time); (2) more convenient for checkpoint selection; (3) cosine decay requires the total number of steps to be fixed in advance, offering less flexibility.

Answer: Each parameter requires FP32 master weight (4 bytes), $m_t$ (4 bytes), and $v_t$ (4 bytes) — 12 bytes per parameter. For a 7B-parameter model, optimizer states alone require ~84 GB (too large for a single GPU).

Mitigations:

1. ZeRO Stage 1: shard optimizer states across GPUs; each GPU stores only $1/N$ of $(m, v)$.
2. ZeRO-Offload / ZeRO-Infinity: offload optimizer states to CPU memory or NVMe SSD.
3. 8-bit Adam: quantize $m, v$ to INT8, saving ~75% memory.
4. Adafactor: use a factorized second-moment estimate, avoiding storage of the full $v_t$.

Follow-up: What does each of the three ZeRO stages optimize? How does communication overhead change?

Stage 1 shards optimizer states (almost no extra communication); Stage 2 additionally shards gradients (replaces AllReduce with Reduce-Scatter + AllGather); Stage 3 additionally shards model parameters (AllGather needed during both forward and backward, increasing communication). Stage 3 communication is roughly 1.5× that of Stage 1, but saves the most memory.

Answer:

Follow-up: If you adapt this plan for SFT, which hyperparameters should change and why?

(1) LR reduced by 1–2 orders of magnitude (e.g., 2e-5) to avoid overwriting pre-trained knowledge; (2) warmup steps can be reduced (pre-trained weights are already stable); (3) weight decay can be lowered slightly; (4) if using LoRA, optimizer state memory is much smaller and gradient checkpointing may be turned off; (5) total training steps are greatly reduced (typically a few thousand to tens of thousands), so the cosine decay $T$ must be adjusted accordingly.

A: Adam's adaptive learning rate can cause parameters to converge to sharp minima, which have high curvature and poor generalization. SGD's stochastic gradient noise has a natural exploratory quality that tends to find flat minima, which are more robust to input perturbations.
Follow-up: In practice, what methods can combine Adam's convergence speed with SGD's generalization ability?

A: The velocity term $v_t$ in momentum also starts cold. Using a large LR immediately can cause early updates to be dominated by the first few batches, leading to a poorly calibrated accumulated momentum direction. Warmup provides a low-speed exploration phase, allowing the momentum direction to be properly calibrated over a large number of samples before accelerating.
Follow-up: If the warmup duration is set too short, under what circumstances could training instability still occur?

A: In LayerNorm, the learned mean shift $\beta$ and scale $\gamma$ are coupled: $\beta$ handles centering, but the subsequent $\gamma$ scaling interacts with the centering effect. RMSNorm removes mean centering and applies scaling directly, reducing interference between degrees of freedom and making optimization simpler and more direct, especially for stabilizing deep networks.
Follow-up: In what types of network architectures or tasks might mean centering still be necessary or beneficial?

A: Swish/GELU do not completely zero out the negative half; they retain small negative values. Therefore, their variance attenuation is less than ReLU's 50%. The correction factor in the initialization variance (such as the factor of 2 in Kaiming) should be reduced appropriately to match the actual variance preservation ratio of these activations.
Follow-up: How would you determine a reasonable initialization variance correction factor for a non-standard activation function in practice?

A: Clipping outside the optimizer (after gradient computation, before the update) directly constrains the norm of the parameter update vector, decoupled from optimizer state. Clipping inside Adam (after computing $m_t, v_t$) affects the second-moment estimate $v_t$, which then alters the adaptive learning rate in subsequent steps, introducing more complex coupling effects.
Follow-up: When using AdamW, where in the training step is the best practice to apply gradient clipping? Why?

A: Theoretically strictly equivalent. However, when using BatchNorm or other normalization layers that rely on batch statistics, each micro-batch computes its own statistics independently, so the accumulated normalization statistics are not equal to those from a single large batch. For LayerNorm/RMSNorm, this problem does not arise.
Follow-up: Besides normalization layers, what other training components (e.g., certain regularization techniques) might break this equivalence?

A: The purpose of distillation is for the student to learn the teacher's dark knowledge — namely, inter-class similarity relationships. Label Smoothing over-smooths the probability distribution, blurring these valuable relational details. The teacher model should retain its original, "sharper" distribution in order to convey richer structured information.
Follow-up: If the teacher model itself is overconfident (poor calibration), what strategy should be used to soften its outputs before distillation?

A: Focal Loss uses the factor $(1-p_t)^\gamma$ to dynamically and continuously adjust the gradient weight of each sample — it is an implicit, confidence-based soft reweighting. Traditional resampling/reweighting is explicit and discrete, typically based on class frequency rather than per-sample confidence. Focal Loss has smoother gradient flow and automatically focuses on samples where classification is uncertain.
Follow-up: Under what circumstances might Focal Loss be less effective than simple class-frequency reweighting?