ML / DL Fundamentals 公开速查手册

覆盖优化器、学习率调度、正则化、归一化、初始化、反向传播、梯度问题、损失函数、混合精度等核心基础。 Covers optimizers, LR scheduling, regularization, normalization, initialization, backpropagation, gradient issues, loss functions, and mixed precision.

第一部分：概念与公式推导

Part 1: Concepts & Formula Derivations

1.1 优化器 Optimizers

SGD（随机梯度下降 Stochastic Gradient Descent）

$$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta L(\theta_t)$$

• 无自适应学习率，收敛慢但泛化有时更好。
• No per-parameter adaptive LR; converges slowly but can generalize better.

SGD + Momentum

$$v_t = \mu \, v_{t-1} + \nabla_\theta L(\theta_t)$$ $$\theta_{t+1} = \theta_t - \eta \, v_t$$

• 动量 $\mu$（通常 0.9）累积历史梯度方向，加速穿越平坦区域、抑制震荡。
• Momentum $\mu$ (typically 0.9) accumulates past gradients to accelerate through flat regions and dampen oscillations.

Adam（Adaptive Moment Estimation）

维护一阶矩（梯度均值）和二阶矩（梯度平方的指数移动平均）：

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

偏差修正（Bias Correction），解决初始化为零导致的偏置问题：

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

参数更新：

$$\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$

• 自适应 per-parameter 学习率，对稀疏梯度友好。
• Adaptive per-parameter learning rate; friendly to sparse gradients.

AdamW（Decoupled Weight Decay）

标准 Adam 中，若将 L2 正则项 $\frac{\lambda}{2}\|\theta\|^2$ 加入 loss，正则化梯度 $\lambda\theta$ 被 $\sqrt{\hat{v}_t}$ 缩放，实际衰减幅度随梯度大小变化——不是真正的均匀 weight decay。

AdamW 将 weight decay 从自适应更新中解耦：

$$\theta_{t+1} = (1 - \eta\lambda)\,\theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$

• 衰减力度均匀，与 SGD 下的 weight decay 行为一致。
• Uniform decay strength, consistent with weight decay under SGD.

关键公式汇总 Key Formula Summary

1.2 学习率调度 Learning Rate Scheduling

Warmup

训练初期 $m_t, v_t$ 估计不准（严重偏小），直接用大学习率可能导致更新步长异常。Warmup 在前 $T_w$ 步线性增大 LR，让矩估计充分积累。

Cosine Decay（余弦退火）

$$\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{\pi t}{T}\right)\right)$$

• 尾端 LR 平滑趋近 $\eta_{\min}$，收敛更稳定。
• LR smoothly decays to $\eta_{\min}$ at the tail for stable convergence.

线性衰减 Linear Decay

$$\eta_t = \eta_{\max} - (\eta_{\max} - \eta_{\min}) \cdot \frac{t}{T}$$

• 匀速下降，尾端 LR 仍较大，可能在收敛区震荡。
• Uniform decrease; LR remains relatively large near the end, potentially causing oscillation.

1.3 正则化 Regularization

Dropout

训练时以概率 $p$ 将神经元输出置零。Inverted Dropout（PyTorch 默认）在训练时将非零输出缩放 $1/(1-p)$，推理时无需额外操作。

Weight Decay vs L2 正则化

• SGD 下等价：L2 正则 $\frac{\lambda}{2}\|\theta\|^2$ 的梯度 $\lambda\theta$ 加入总梯度，效果为 $\theta \leftarrow \theta(1 - \eta\lambda) - \eta\nabla L$，即 weight decay。
• Adam 下不等价：L2 正则的梯度 $\lambda\theta$ 被 $\sqrt{\hat{v}_t}$ 除，decay 幅度与梯度量级耦合。AdamW 直接做 $\theta \leftarrow (1-\eta\lambda)\theta$，实现真正的解耦 weight decay。

Label Smoothing

将 one-hot 标签 $y$ 替换为：

$$y_{\text{smooth}} = (1 - \epsilon)\,y + \frac{\epsilon}{K}$$

其中 $K$ 为类别数。防止模型对 logits 过度自信，提升校准性（calibration）。

1.4 归一化 Normalization

假设输入张量形状为 $(B, C)$（简化为二维，NLP 场景）或 $(B, C, H, W)$（CV 场景）。

• RMSNorm 省去均值计算，速度更快，LLM 主流。
• Pre-LN（LN 在 sublayer 输入前）：residual 路径梯度直通，训练更稳定。
• Post-LN（LN 在 residual 之后）：最终性能可能略高，但训练不稳定，需精细调参。

1.5 参数初始化 Weight Initialization

Xavier（Glorot）初始化

$$W \sim \mathcal{U}\left(-\sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}},\; \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}\right)$$

或高斯版：$W \sim \mathcal{N}\left(0, \frac{2}{n_{\text{in}} + n_{\text{out}}}\right)$

• 设计目标：保持前向传播中每层输出方差恒定（假设线性或 tanh 激活）。
• Goal: maintain constant variance of activations across layers (for linear/tanh).

Kaiming（He）初始化

$$W \sim \mathcal{N}\left(0, \frac{2}{n_{\text{in}}}\right)$$

• 针对 ReLU：负半轴置零使方差减半，因此需要额外因子 2 修正。
• For ReLU: negative-half zeroing halves variance; the factor of 2 compensates.

1.6 反向传播 Backpropagation

链式法则 Chain Rule

对于计算图 $L = f(g(x))$：

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial g} \cdot \frac{\partial g}{\partial x}$$

对多层网络中的第 $l$ 层：

$$\frac{\partial L}{\partial W_l} = \frac{\partial L}{\partial a_l} \cdot \frac{\partial a_l}{\partial W_l}$$

其中 $a_l$ 为该层输出。前向传播时需缓存中间激活值以供反向传播使用。

梯度消失与爆炸 Vanishing / Exploding Gradients

梯度在深层连乘：

$$\frac{\partial L}{\partial W_1} = \prod_{l=2}^{L} \frac{\partial a_l}{\partial a_{l-1}} \cdot \frac{\partial a_L}{\partial W_1}$$

• 若每项 $< 1$：梯度消失 → 浅层不更新。
• 若每项 $> 1$：梯度爆炸 → 参数剧烈震荡。

梯度裁剪 Gradient Clipping

• 按值裁剪 Clip by value：$g_i \leftarrow \text{clip}(g_i, -c, c)$，可能改变梯度方向。
• 按范数裁剪 Clip by norm（主流）：$\mathbf{g} \leftarrow \mathbf{g} \cdot \min\!\left(1,\, \frac{c}{\|\mathbf{g}\|_2}\right)$，保持方向不变。

1.7 损失函数 Loss Functions

Cross-Entropy Loss（交叉熵损失）

对于 one-hot 标签 $y$ 和 softmax 输出 $q$：

$$\text{CE} = -\sum_{i=1}^{K} y_i \log q_i = -\log q_y$$

PyTorch 的 CrossEntropyLoss 内部先做 LogSoftmax，再做 NLLLoss：

$$\text{CE}(z, y) = -z_y + \log\!\left(\sum_{j=1}^{K} e^{z_j}\right)$$

Focal Loss

$$\text{FL}(p_t) = -(1 - p_t)^\gamma \log(p_t)$$

降低易分样本权重，聚焦难例，适用于类别不平衡场景。

Perplexity（困惑度）

$$\text{PPL} = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(x_i \mid x_{<i})\right)$$

• 即 per-token cross-entropy 的指数化，越低越好。
• Exponentiated per-token cross-entropy; lower is better.

知识蒸馏 Knowledge Distillation

$$L = \alpha \cdot \text{CE}(y, \hat{y}_s) + (1 - \alpha) \cdot T^2 \cdot \text{KL}\!\left(\hat{y}_t^{(T)} \,\|\, \hat{y}_s^{(T)}\right)$$

• Teacher 的 soft label（温度 $T$ 平滑后的分布）提供 "dark knowledge"。
• Teacher's soft labels (smoothed by temperature $T$) provide "dark knowledge."

1.8 混合精度训练 Mixed Precision Training

核心流程 Core Pipeline：

1. FP32 Master Weights：维护一份 FP32 精度的参数副本用于梯度更新。
2. 前向 + 反向：用 FP16 或 BF16 执行，减少显存、加速计算（Tensor Core）。
3. Loss Scaling（仅 FP16）：放大 loss 防止梯度下溢，更新时再缩小。
4. BF16 优势：动态范围与 FP32 相同，无需 loss scaling。

1.9 偏差-方差分解 Bias-Variance Decomposition

$$\mathbb{E}[\text{MSE}] = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}$$

• 高偏差 High Bias（欠拟合 Underfitting）：train / val loss 都高。
• 高方差 High Variance（过拟合 Overfitting）：train loss 低，val loss 明显高。

1.10 Batch Size 与梯度累积 Gradient Accumulation

Linear Scaling Rule：batch size 乘以 $k$ 时，LR 也乘以 $k$，保持每次参数更新的期望步长不变。

梯度累积：将 mini-batch 拆为 $k$ 个 micro-batch，前 $k-1$ 步累加梯度（不更新），第 $k$ 步统一更新，等效于 $k$ 倍 batch size 但显存不变。

第二部分：PyTorch 代码片段

Part 2: PyTorch Snippets (From Scratch)

以下片段用于教学理解，不追求完整封装，重点展示核心计算逻辑。 These snippets prioritize pedagogical clarity over production-ready packaging.

2.1 SGD + Momentum

import torch

class SGDMomentum:
    def __init__(self, params, lr=0.01, momentum=0.9):
        self.params = list(params)
        self.lr = lr
        self.momentum = momentum
        self.velocities = [torch.zeros_like(p) for p in self.params]

    def step(self):
        for p, v in zip(self.params, self.velocities):
            if p.grad is None:
                continue
            v.mul_(self.momentum).add_(p.grad)
            p.data.sub_(self.lr * v)

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None:
                p.grad.zero_()

2.2 Adam

class Adam:
    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8):
        self.params = list(params)
        self.lr = lr
        self.beta1, self.beta2 = betas
        self.eps = eps
        self.t = 0
        self.m = [torch.zeros_like(p) for p in self.params]
        self.v = [torch.zeros_like(p) for p in self.params]

    def step(self):
        self.t += 1
        for p, m, v in zip(self.params, self.m, self.v):
            if p.grad is None:
                continue
            m.mul_(self.beta1).add_(p.grad, alpha=1 - self.beta1)
            v.mul_(self.beta2).addcmul_(p.grad, p.grad, value=1 - self.beta2)
            # Bias correction
            m_hat = m / (1 - self.beta1 ** self.t)
            v_hat = v / (1 - self.beta2 ** self.t)
            p.data.addcdiv_(m_hat, v_hat.sqrt().add_(self.eps), value=-self.lr)

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None:
                p.grad.zero_()

2.3 AdamW（Decoupled Weight Decay）

class AdamW:
    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999),
                 eps=1e-8, weight_decay=0.01):
        self.params = list(params)
        self.lr = lr
        self.beta1, self.beta2 = betas
        self.eps = eps
        self.wd = weight_decay
        self.t = 0
        self.m = [torch.zeros_like(p) for p in self.params]
        self.v = [torch.zeros_like(p) for p in self.params]

    def step(self):
        self.t += 1
        for p, m, v in zip(self.params, self.m, self.v):
            if p.grad is None:
                continue
            # Decoupled weight decay (applied directly to params)
            p.data.mul_(1 - self.lr * self.wd)
            # Adaptive update
            m.mul_(self.beta1).add_(p.grad, alpha=1 - self.beta1)
            v.mul_(self.beta2).addcmul_(p.grad, p.grad, value=1 - self.beta2)
            m_hat = m / (1 - self.beta1 ** self.t)
            v_hat = v / (1 - self.beta2 ** self.t)
            p.data.addcdiv_(m_hat, v_hat.sqrt().add_(self.eps), value=-self.lr)

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None:
                p.grad.zero_()

2.4 Cosine Annealing LR Schedule

import math

def cosine_lr(step, total_steps, lr_max, lr_min=0.0):
    """Cosine decay from lr_max to lr_min over total_steps."""
    return lr_min + 0.5 * (lr_max - lr_min) * (
        1 + math.cos(math.pi * step / total_steps)
    )

# Usage with PyTorch scheduler (built-in):
# scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
#     optimizer, T_max=total_steps, eta_min=lr_min
# )

2.5 Layer Normalization from Scratch

import torch
import torch.nn as nn

class LayerNorm(nn.Module):
    def __init__(self, d_model, eps=1e-5):
        super().__init__()
        self.gamma = nn.Parameter(torch.ones(d_model))
        self.beta = nn.Parameter(torch.zeros(d_model))
        self.eps = eps

    def forward(self, x):
        # x: (batch, seq_len, d_model)
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, keepdim=True, unbiased=False)
        x_hat = (x - mean) / torch.sqrt(var + self.eps)
        return self.gamma * x_hat + self.beta

2.6 RMSNorm

class RMSNorm(nn.Module):
    def __init__(self, d_model, eps=1e-6):
        super().__init__()
        self.gamma = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x):
        rms = torch.sqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
        return self.gamma * (x / rms)

2.7 Gradient Clipping by Norm

def clip_grad_norm(parameters, max_norm=1.0):
    """Clip gradient norm in-place; returns total norm before clipping."""
    params = [p for p in parameters if p.grad is not None]
    total_norm = torch.sqrt(
        sum(p.grad.data.norm(2).item() ** 2 for p in params)
    )
    clip_coef = max_norm / (total_norm + 1e-6)
    if clip_coef < 1.0:
        for p in params:
            p.grad.data.mul_(clip_coef)
    return total_norm

# PyTorch built-in (recommended):
# torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

2.8 Mixed Precision Training (AMP)

import torch

# Setup
model = MyModel().cuda()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
scaler = torch.amp.GradScaler("cuda")  # for FP16; not needed for BF16

for inputs, targets in dataloader:
    optimizer.zero_grad()

    # Forward pass in FP16/BF16
    with torch.amp.autocast("cuda", dtype=torch.float16):
        outputs = model(inputs)
        loss = criterion(outputs, targets)

    # Backward with loss scaling (FP16 only)
    scaler.scale(loss).backward()
    scaler.unscale_(optimizer)           # unscale before clipping
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(optimizer)
    scaler.update()

# For BF16: simply replace dtype=torch.bfloat16 and remove GradScaler

2.9 Gradient Checkpointing

from torch.utils.checkpoint import checkpoint

class TransformerBlock(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ffn = FeedForward(d_model)

    def forward(self, x):
        # Use checkpoint to save memory at the cost of recomputation
        x = x + checkpoint(self.attn, x, use_reentrant=False)
        x = x + checkpoint(self.ffn, x, use_reentrant=False)
        return x

2.10 Focal Loss

class FocalLoss(nn.Module):
    def __init__(self, gamma=2.0, weight=None):
        super().__init__()
        self.gamma = gamma
        self.ce = nn.CrossEntropyLoss(weight=weight, reduction="none")

    def forward(self, logits, targets):
        ce_loss = self.ce(logits, targets)           # (B,)
        p_t = torch.exp(-ce_loss)                     # p of correct class
        focal_weight = (1 - p_t) ** self.gamma
        return (focal_weight * ce_loss).mean()

2.11 GELU 与 SwiGLU

import torch
import torch.nn as nn
import torch.nn.functional as F

# GELU (exact and approximate)
class GELUApprox(nn.Module):
    """Common tanh approximation of GELU."""
    def forward(self, x):
        return 0.5 * x * (
            1 + torch.tanh(
                math.sqrt(2 / math.pi) * (x + 0.044715 * x.pow(3))
            )
        )

# SwiGLU FFN (as used in LLaMA)
class SwiGLU_FFN(nn.Module):
    """SwiGLU FFN: gate and up proj, then elementwise multiply, then down proj.
    d_ffn is typically (8/3) * d_model to match parameter count of standard FFN.
    """
    def __init__(self, d_model, d_ffn):
        super().__init__()
        self.w_gate = nn.Linear(d_model, d_ffn, bias=False)
        self.w_up   = nn.Linear(d_model, d_ffn, bias=False)
        self.w_down = nn.Linear(d_ffn, d_model, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))

2.12 Knowledge Distillation Loss

import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, targets,
                      temperature=4.0, alpha=0.7):
    """
    student_logits, teacher_logits: (B, K) raw logits
    targets: (B,) integer labels
    """
    # Hard loss
    hard_loss = F.cross_entropy(student_logits, targets)

    # Soft loss (KL divergence)
    s_log_probs = F.log_softmax(student_logits / temperature, dim=-1)
    t_probs = F.softmax(teacher_logits / temperature, dim=-1)
    soft_loss = F.kl_div(s_log_probs, t_probs, reduction="batchmean")
    soft_loss = soft_loss * (temperature ** 2)  # scale back gradients

    return alpha * hard_loss + (1 - alpha) * soft_loss

2.13 LR Finder（学习率搜索器）

import torch
import math

def lr_finder(model, dataloader, criterion, optimizer,
              init_lr=1e-7, final_lr=10.0, num_steps=100):
    """Exponentially increase LR from init_lr to final_lr over num_steps."""
    lr_mult = (final_lr / init_lr) ** (1 / num_steps)
    lr = init_lr
    results = []

    for i, (inputs, targets) in enumerate(dataloader):
        if i >= num_steps:
            break
        # Set LR
        for pg in optimizer.param_groups:
            pg["lr"] = lr
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()
        results.append((lr, loss.item()))
        lr *= lr_mult

    return results  # Plot loss vs. lr; pick LR where loss drops fastest

2.14 Gradient Accumulation

accumulation_steps = 4  # simulate 4x batch size

for i, (inputs, targets) in enumerate(dataloader):
    with torch.amp.autocast("cuda", dtype=torch.bfloat16):
        loss = criterion(model(inputs), targets) / accumulation_steps
    loss.backward()

    if (i + 1) % accumulation_steps == 0:
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        optimizer.zero_grad()

第三部分：面试题集（25 题）

Part 3: Interview Questions (25 Questions)

L1 — 基础题 Basic

L2 — 中等题 Intermediate

L3 — 深度题 Deep

本速查手册为公开教育用途，内容基于已有文献和通用工程实践，不包含任何特定研究的未发表结果。 This cheat sheet is for public educational use, based on established literature and common engineering practice. It does not contain any unpublished results from specific research projects.

更多 L3 深挖 / Extended L3