Math & Statistics for ML Cheat Sheet

A: Joint probability $P(A,B) = P(A \mid B) P(B) = P(B \mid A) P(A)$; marginal probability is obtained by summing (or integrating) over the joint: $P(A) = \sum_B P(A, B)$. Bayes' theorem $P(A \mid B) = P(B \mid A) P(A) / P(B)$ ties all three together.

Follow-up: How does the chain rule of probability generalize to more than three variables? What does conditional independence $A \perp B \mid C$ mean?

A: $P(A,B,C) = P(A)P(B|A)P(C|A,B)$. $A \perp B \mid C$ means that given $C$, knowing $A$ does not change beliefs about $B$: $P(A,B|C) = P(A|C)P(B|C)$. In a probabilistic graphical model this corresponds to $C$ blocking all paths from $A$ to $B$.

A:

• Linearity: $E[aX+b] = aE[X]+b$
• Variance: $\text{Var}(X) = E[X^2] - (E[X])^2$
• Scaling: $\text{Var}(aX+b) = a^2\text{Var}(X)$
• Variance of a sum of independent variables equals the sum of variances (no covariance term)

Follow-up: What do the diagonal entries of a covariance matrix represent? Why is a covariance matrix always positive semi-definite?

A: The diagonal entries are the per-dimension variances $\text{Var}(X_i)$. Symmetry is immediate; PSD holds because $\forall w: w^\top \Sigma w = \text{Var}(w^\top X) \geq 0$ (variance is non-negative).

A:

Follow-up: How is the multivariate Gaussian PDF written?

A: $\mathcal{N}(x|\mu,\Sigma) = (2\pi)^{-d/2}|\Sigma|^{-1/2}\exp\!\big(-\frac{1}{2}(x-\mu)^\top\Sigma^{-1}(x-\mu)\big)$.

A:

$$\hat{\theta}_{\text{MLE}} = \arg\max_\theta \sum_{i=1}^N \log P(x_i \mid \theta)$$

Select the parameters that maximize the probability of the observed data. Equivalent to minimizing the negative log-likelihood (NLL). Under a Gaussian assumption, MLE for regression is equivalent to minimizing MSE.

Follow-up: Under what conditions does MLE produce a biased estimate?

A: A classic example: the MLE for the Gaussian variance (dividing by $N$ instead of $N-1$) is biased (underestimates variance). With small samples, MLE is prone to overfitting on complex models.

A: The rank of a matrix is the maximum number of linearly independent rows (or columns), equal to the dimension of the column space. A low-rank matrix $W \in \mathbb{R}^{m \times n}$ (rank $r \ll \min(m,n)$) can be factored as $W = AB$ ($A \in \mathbb{R}^{m \times r}, B \in \mathbb{R}^{r \times n}$), reducing the parameter count from $mn$ to $r(m+n)$.

Follow-up: What is the connection between low-rank factorization and parameter-efficient fine-tuning (PEFT)?

A: LoRA assumes that the weight update $\Delta W$ during fine-tuning is low-rank, and approximates it using the factorization $\Delta W = BA$. Only $A$ and $B$ are trained while the original weights are frozen, dramatically reducing the number of trainable parameters and GPU memory usage.

A: $H(X) = -\sum_x P(x) \log P(x)$, measuring the uncertainty of a random variable. Entropy is maximized under a uniform distribution ($= \log |X|$) and is 0 for a deterministic distribution. The unit depends on the base of the logarithm: $\log_2$ → bits; $\ln$ → nats.

Follow-up: Why does the maximum entropy principle yield the Gaussian distribution?

A: Under the constraints $\int P = 1$, $E[X] = \mu$, $\text{Var}(X) = \sigma^2$, maximizing $H(X)$ via Lagrange multipliers yields the Gaussian distribution as the optimal solution.

A:

• Dot product: $u \cdot v = \|u\|\|v\|\cos\theta$, influenced by both magnitude and direction
• Cosine similarity: $\cos\theta = \frac{u \cdot v}{\|u\|\|v\|}$, reflects only direction (angle), magnitude is normalized out

Follow-up: Why does Attention use scaled dot-product rather than cosine similarity?

A: Scaled dot-product $QK^\top / \sqrt{d_k}$ is computationally efficient (matrix multiplication), and the norms of Q and K participate in the attention weights, increasing expressiveness. RoPE rotations preserve norms and encode only relative position information. Cosine similarity requires extra L2 normalization and limits expressiveness.

A:

• Jacobian $J \in \mathbb{R}^{m \times n}$: first-order derivative matrix of a vector function $f: \mathbb{R}^n \to \mathbb{R}^m$, $J_{ij} = \partial f_i / \partial x_j$
• Hessian $H \in \mathbb{R}^{n \times n}$: second-order derivative matrix of a scalar function, $H_{ij} = \partial^2 f / \partial x_i \partial x_j$, symmetric. Positive definite → local minimum; indefinite → saddle point

Follow-up: Why is the Hessian rarely used directly in large-model optimization?

A: The Hessian has size $n \times n$ ($n$ = number of parameters); storing and computing it is infeasible for models with billions of parameters. Second-order methods like K-FAC and Shampoo use Kronecker factorization or block-diagonal approximations of the Hessian to reduce cost.

A: MLE maximizes the likelihood $P(D|\theta)$; MAP maximizes the posterior $P(\theta|D) \propto P(D|\theta) P(\theta)$, adding a prior term $\log P(\theta)$. Gaussian prior → L2 regularization (Weight Decay); Laplace prior → L1 regularization (Lasso).

Follow-up: From a Bayesian perspective, what prior does AdamW's weight decay assume?

A: It is equivalent to an isotropic Gaussian prior $P(\theta) \propto \exp(-\lambda \|\theta\|^2 / 2)$. AdamW decouples weight decay from the gradient update (decoupled weight decay), which differs subtly from L2-regularized Adam, but the Bayesian interpretation is the same.

A: Prediction error = Bias² + Variance + irreducible noise. High bias → underfitting; high variance → overfitting. Increasing model capacity reduces bias but may increase variance, and vice versa — this is the bias-variance tradeoff.

Follow-up: Do ensemble methods (Bagging, Dropout as ensemble) primarily reduce bias or variance?

A: Primarily reduce variance. Bagging averages multiple high-variance models, reducing variance by a factor of $1/M$ ($M$ = number of models), with almost no change in bias. Boosting mainly reduces bias.

A: Any matrix $W = U\Sigma V^\top$, where $U, V$ are orthogonal and $\Sigma$ is a diagonal matrix of singular values. The Eckart–Young theorem states that the truncated SVD $\hat{W}_r = U_r \Sigma_r V_r^\top$, retaining only the top $r$ singular values, is the optimal rank-$r$ Frobenius-norm approximation.

Follow-up: How are the Frobenius norm and spectral norm each determined by the singular values?

A: $\|W\|_F = \sqrt{\sum_i \sigma_i^2}$ (square root of the sum of squared singular values); $\|W\|_2 = \sigma_1$ (largest singular value).

A: Eigendecomposition applies only to square matrices (and requires diagonalizability), $A = Q\Lambda Q^{-1}$; SVD applies to any matrix of any shape, $W = U\Sigma V^\top$, with singular values always non-negative. For symmetric positive definite matrices, eigendecomposition and SVD are equivalent.

Follow-up: What is the geometric meaning of the eigenvectors of a covariance matrix?

A: The eigenvectors of the covariance matrix are the principal axis directions (principal directions) of the data distribution, corresponding to the principal component directions in PCA. The magnitude of each eigenvalue represents the variance along that direction.

A: $D_{\text{KL}}(P\|Q) = E_P[\log(P/Q)]$. Properties: (1) $D_{\text{KL}} \geq 0$; (2) asymmetric, not a distance metric; (3) $D_{\text{KL}} = 0 \Leftrightarrow P = Q$.

Follow-up: How do Forward KL and Reverse KL differ in their fitting behavior?

A: Forward KL ($P \| Q$) is mean-seeking: $Q$ tends to cover all high-probability regions of $P$, preserving diversity. Reverse KL ($Q \| P$) is mode-seeking: $Q$ tends to concentrate on a single peak of $P$ and may ignore other modes. Variational inference typically uses Reverse KL.

A: $H(P, Q) = H(P) + D_{\text{KL}}(P \| Q)$. When $P$ is a one-hot distribution, $H(P) = 0$, so minimizing cross-entropy = minimizing KL divergence = MLE.

Follow-up: What is the relationship between a language model's perplexity (PPL) and cross-entropy?

A: $\text{PPL} = \exp(H(P, Q))$, where $H$ is the cross-entropy. PPL can be understood as the average number of effective choices the model has at each token position; lower PPL means a better model.

A: For a convex function $f$: $f(E[X]) \leq E[f(X)]$. Applications: (1) proving non-negativity of KL divergence; (2) deriving the variational lower bound ELBO; (3) proving convergence of the EM algorithm.

Follow-up: What is variational inference (VI)? What is the relationship between the ELBO and KL divergence?

A: VI approximates the complex posterior $P(z|x)$ with a simple distribution $Q(z)$. $\log P(x) = \text{ELBO} + D_{\text{KL}}(Q \| P(z|x))$; maximizing ELBO is equivalent to minimizing the KL divergence between the variational posterior and the true posterior.

A: For a symmetric matrix $A$ to be positive definite: (1) all eigenvalues $> 0$; (2) all leading principal minors $> 0$ (Sylvester's criterion); (3) a Cholesky decomposition $A = LL^\top$ exists; (4) $\forall x \neq 0: x^\top Ax > 0$.

Follow-up: Why is a covariance matrix always positive semi-definite?

A: For any $w$, $w^\top \Sigma w = \text{Var}(w^\top X) \geq 0$ (variance is non-negative). Being strictly positive definite requires the variance to be strictly greater than zero (no deterministic linear relationship exists).

A: $A \in \mathbb{R}^{m \times k}$, $B \in \mathbb{R}^{k \times n}$: $AB$ requires $O(mkn)$ FLOPs. LLM training estimate: with $N$ parameters and $D$ training tokens, total FLOPs $\approx 6ND$ (forward ≈ 2ND, backward ≈ 4ND; inference / forward-only ≈ 2ND).

Follow-up: Why is the decode phase of LLM inference typically memory-bound rather than compute-bound?

A: Autoregressive decoding generates only 1 token per step, collapsing the batch dimension. Matrix multiplication degenerates to matrix-vector multiplication, which has very few FLOPs but requires loading the entire model weights (IO-intensive), leaving GPU compute underutilized.

A: The CLT guarantees that the mini-batch gradient $\hat{g}_B = \frac{1}{B}\sum_i \nabla L(x_i)$ is an unbiased estimator of the full-data gradient, with variance $\propto 1/B$. SGD gradient noise helps the model escape sharp minima and tends to converge to flat minima. Flat minima generally generalize better, which amounts to implicit regularization.

Follow-up: What is the effect of increasing batch size during training?

A: Gradient variance decreases ($\propto 1/B$), making optimization more stable but potentially losing the regularization effect and degrading generalization. Requires paired learning rate adjustment (linear scaling rule).

A:

• Forward KL ($D_{\text{KL}}(P \| Q)$, I-projection): minimizing it forces $Q$ to have mass wherever $P > 0$ → mean-seeking / zero-avoiding. Suitable for preserving multi-modal coverage.
• Reverse KL ($D_{\text{KL}}(Q \| P)$, M-projection): minimizing it pushes $Q$ to have negligible mass where $P \approx 0$ → mode-seeking / zero-forcing. Suitable for concentrating on the dominant mode.

Variational inference (VI) uses Reverse KL by default (it arises naturally from the ELBO derivation); the KL constraint $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ in RLHF is Reverse KL (mode-seeking): it pulls policy $\pi_\theta$ toward the reference and penalizes deviation from high-probability regions of $\pi_{\text{ref}}$, suppressing reward-hacking distribution drift.

Follow-up: Are there practical applications of mixing Forward/Reverse KL?

A: The Rényi-$\alpha$ divergence family interpolates between the two for $\alpha \in (0,1)$ and has been used in variational inference and reinforcement learning to trade off coverage against concentration.

A: Scaled dot-product $\text{softmax}(QK^\top / \sqrt{d_k})$ advantages: (1) efficient matrix multiplication; (2) the norms of Q and K participate in attention weight computation, increasing expressiveness; (3) the scaling factor $\sqrt{d_k}$ prevents dot products from becoming too large and causing softmax saturation. Cosine similarity loses magnitude information and requires extra normalization.

RoPE (Rotary Position Embedding) only changes the direction (angle) of Q and K, keeping norms unchanged, so positional information is naturally incorporated through the angular difference in the dot product.

Follow-up: What happens if Q and K are L2-normalized before computing the dot product?

A: This is equivalent to cosine similarity (with temperature scaling). Some work (e.g., Normalized Attention) has explored this direction; the advantage is that attention weights are norm-independent and more stable, but it may limit the ability to differentiate expressiveness among tokens. Methods like CosFormer introduce cosine reweighting while maintaining linear attention efficiency.

A:

• Reverse-mode AD: one forward pass + one backward pass computes gradients of a scalar loss with respect to all parameters, with complexity $O(1) \times$ the forward computation. Best for many parameters, few outputs (i.e., $f: \mathbb{R}^n \to \mathbb{R}$); this is the foundation of backpropagation in deep learning.
• Forward-mode AD: each pass propagates along one input direction to obtain the gradient of all outputs with respect to one input. Best for few inputs, many outputs (i.e., $f: \mathbb{R} \to \mathbb{R}^m$), such as computing Jacobian-vector products (JVPs).

Follow-up: Can a Hessian-vector product (HVP) be computed without explicitly forming the Hessian?

A: Yes. $\nabla^2 f \cdot v = \nabla(\nabla f \cdot v)$; first apply forward-mode differentiation to $\nabla f \cdot v$ (or backpropagate through $\nabla f$), requiring only two differentiation passes with complexity comparable to a single gradient computation. This is the basis for implicit second-order methods such as conjugate gradient (CG).

A: PPO reuses trajectories collected under the old policy $\pi_{\theta_{\text{old}}}$ to train the new policy $\pi_\theta$, correcting for the distributional shift via the importance ratio $r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$. The PPO-Clip objective:

$$L^{\text{CLIP}} = E_t\!\Big[\min\!\big(r_t(\theta)\hat{A}_t,\; \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\big)\Big]$$

$\epsilon$ (typically 0.1–0.2) constrains the range of the importance ratio (not the weight itself). When $r_t$ falls outside $[1-\epsilon, 1+\epsilon]$, the gradient is cut off, preventing excessively large policy updates that would destabilize training.

Follow-up: If the importance ratio variance is too large, what are the alternatives?

A: V-trace (introduced in IMPALA) truncates the ratio, or Retrace($\lambda$) and similar methods control variance. The core idea is to limit the effective importance weight to ensure convergence stability.

A: Approximate the true posterior $P(z|x)$ with a simple distribution $Q(z)$. Derivation:

$$\log P(x) = \log \int P(x,z)\,dz = \log E_Q\!\left[\frac{P(x,z)}{Q(z)}\right] \geq E_Q\!\left[\log\frac{P(x,z)}{Q(z)}\right] = \text{ELBO}$$

The inequality follows from Jensen's inequality ($\log$ is concave). ELBO = reconstruction term $E_Q[\log P(x|z)]$ − regularization term $D_{\text{KL}}(Q(z) \| P(z))$. $\beta$-VAE uses a hyperparameter $\beta$ to reweight the KL term, controlling the tradeoff between disentanglement and reconstruction quality.

Follow-up: Why is the ELBO maximized (rather than minimized)?

A: The ELBO is a lower bound on $\log P(x)$. Maximizing ELBO → tighter lower bound → $Q(z)$ closer to the true posterior $P(z|x)$. Since $\log P(x) = \text{ELBO} + D_{\text{KL}}(Q \| P(z|x))$ and KL $\geq 0$, maximizing ELBO is equivalent to minimizing the KL divergence between the variational and true posteriors.

A: Theoretical basis — research on intrinsic dimensionality shows that fine-tuning a pretrained model actually takes place in a low-dimensional subspace far smaller than the full parameter space. LoRA parameterizes weight updates as $\Delta W = BA$ ($B \in \mathbb{R}^{m \times r}, A \in \mathbb{R}^{r \times n}$), requiring only $r(m+n)$ parameters ($r \ll \min(m,n)$) — orders of magnitude fewer than the original $mn$. Empirically, $r = 4 \sim 64$ is sufficient to approach full fine-tuning performance across a variety of downstream tasks.

Follow-up: How should the rank $r$ and which layers to apply LoRA to be chosen?

A: Rank $r$ is typically selected via a small-scale search on the validation set. In practice, applying LoRA to the Q/K/V/O projection matrices of the attention layers works well. More advanced methods such as AdaLoRA adaptively allocate different ranks to each layer based on the singular value distribution. Choosing $r$ requires balancing expressiveness against parameter efficiency.

A: For two multivariate Gaussians $P = \mathcal{N}(\mu_1, \Sigma_1)$ and $Q = \mathcal{N}(\mu_2, \Sigma_2)$:

$$D_{\text{KL}}(P \| Q) = \frac{1}{2}\!\left[\text{tr}(\Sigma_2^{-1}\Sigma_1) + (\mu_2-\mu_1)^\top\Sigma_2^{-1}(\mu_2-\mu_1) - k + \ln\frac{|\Sigma_2|}{|\Sigma_1|}\right]$$

In a VAE with $Q(z) = \mathcal{N}(\mu, \text{diag}(\sigma^2))$ and $P(z) = \mathcal{N}(0, I)$, this simplifies to:

$$D_{\text{KL}} = -\frac{1}{2}\sum_j (1 + \log\sigma_j^2 - \mu_j^2 - \sigma_j^2)$$

This KL term appears as a regularization term in the ELBO, encouraging the encoder output to stay close to the standard normal prior.

Follow-up: What is the effect of $\beta > 1$ in $\beta$-VAE?

A: Increasing the KL term weight forces the encoder to match the standard normal prior more tightly, so the learned latent variable $z$ becomes more disentangled, but reconstruction quality may suffer. $\beta < 1$ has the opposite effect: a more flexible latent space but potentially at the cost of disentanglement.

A: The Fisher information matrix is defined as the covariance of the log-likelihood gradient: $F(\theta) = E_{x \sim p_\theta}\!\left[\nabla_\theta \log p_\theta(x)\, \nabla_\theta \log p_\theta(x)^\top\right]$. It is also the negative expected Hessian of the log-likelihood (at the MLE), and is closely related to the local quadratic expansion of the KL divergence: $D_{\text{KL}}(p_\theta \| p_{\theta + \delta}) \approx \frac{1}{2}\delta^\top F(\theta)\,\delta$. Thus the FIM is the local metric tensor of the KL divergence in parameter space. The natural gradient is defined as $\tilde{\nabla} = F^{-1}\nabla L$, i.e., the direction of steepest descent in loss under the KL-ball constraint $\{δ : D_{\text{KL}}(p_\theta \| p_{\theta+\delta}) \leq \epsilon\}$. Intuitively: the ordinary gradient gives steepest descent in Euclidean space, but equal Euclidean steps in parameter space do not correspond to equal changes in the distribution; the natural gradient uses the FIM to correct this "distortion" so that optimization steps are uniform in distribution space.

Follow-up: Why is the natural gradient not used directly in practice? What approximations exist? (Hint: K-FAC and EKFAC use block-diagonal Kronecker factorization of the FIM to reduce the cost of inversion.)

A: In a VAE, the gradient of the ELBO expectation $E_{z \sim q_\phi(z|x)}[\cdot]$ cannot be backpropagated through the sampling node directly (sampling is not differentiable). The reparameterization trick rewrites $z \sim q_\phi = \mathcal{N}(\mu_\phi, \sigma_\phi^2)$ as $z = \mu_\phi + \sigma_\phi \odot \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$. This shifts the randomness away from the parameters $\phi$ to a parameter-free noise variable $\epsilon$, while $z$ is deterministically differentiable with respect to $\phi$, so gradients can flow back to $\mu_\phi$ and $\sigma_\phi$ via standard backpropagation. Compared to the score function estimator (REINFORCE) $\nabla_\phi E_q[f(z)] = E_q[f(z)\nabla_\phi \log q_\phi(z)]$, the reparameterization trick directly encodes gradient information into the computation graph without relying on the scalar value $f(z)$ as a multiplicative weight, resulting in substantially lower variance. In essence, the score function estimator uses only the scalar value of $f(z)$ as a weight, while reparameterization exploits the local gradient $\partial f / \partial z$, providing a richer signal.

Follow-up: For discrete latent variables (e.g., discrete VAE), the reparameterization trick does not directly apply. What are the alternatives? (Hint: Gumbel-Softmax / Concrete distribution uses a continuous relaxation to approximate discrete sampling while maintaining differentiability.)

A: A function $f$ has Lipschitz constant $L$ if $\|f(x) - f(y)\| \leq L\|x - y\|$. For a linear layer $f(x) = Wx$, the Lipschitz constant equals exactly $\|W\|_2 = \sigma_1$ (largest singular value). For a $k$-layer network $f = W_k \cdots W_1$, the overall Lipschitz constant is $L \leq \prod_i \|W_i\|_2$, the product of spectral norms across layers. If $L$ is too large, small perturbations to inputs are amplified exponentially in the forward pass (exploding activations), and gradients are amplified exponentially in the backward pass (exploding gradients), destabilizing training. Spectral normalization (dividing each layer's weight by its spectral norm: $\hat{W} = W / \sigma_1$) constrains the per-layer Lipschitz constant to 1 and is widely used in GAN discriminator training to prevent mode collapse. The spectral norm can be efficiently approximated via power iteration without requiring a full SVD.

Follow-up: Why is it necessary to impose a Lipschitz constraint on the GAN discriminator (e.g., via spectral normalization or gradient penalty)? What goes wrong without it? (Hint: The Wasserstein distance requires the discriminator to belong to the class of 1-Lipschitz functions; otherwise the objective is unbounded.)

A: The core identities are $\text{tr}(AB) = \text{tr}(BA)$ (cyclic permutation invariance) and the fact that a scalar equals its own trace: $x^\top A x = \text{tr}(x^\top A x) = \text{tr}(A x x^\top)$. These make it feasible to differentiate scalar functions of matrix products. Typical applications: (1) matrix differentiation in linear regression $\nabla_W \|Y - XW\|_F^2 = \nabla_W \text{tr}((Y-XW)^\top(Y-XW))$, expanded using $\nabla_W \text{tr}(AW) = A^\top$ to yield the closed-form solution; (2) differentiating $\log |\Sigma|$ and $x^\top \Sigma^{-1} x$ in the multivariate Gaussian log-likelihood; (3) deriving the MLE of the covariance matrix $\hat{\Sigma} = \frac{1}{N}\sum_i (x_i - \mu)(x_i - \mu)^\top$; (4) deriving the matrix Riccati equation in linear dynamical systems (Kalman filter). In short, the trace trick is the core tool for any setting involving differentiating matrix quadratic forms.

Follow-up: Use the trace trick to derive the MLE for the precision matrix $\Lambda$ (inverse covariance) of a multivariate Gaussian $\mathcal{N}(\mu, \Sigma)$. Hint: the log-likelihood contains $\text{tr}(\Lambda S)$ and $\log|\Lambda|$ terms. (Answer: $\hat{\Lambda} = S^{-1}$, i.e., the MLE precision matrix is the inverse of the sample covariance.)

A: Let $(x_a, x_b) \sim \mathcal{N}(\mu, \Sigma)$ and partition the covariance matrix as $\Sigma = \begin{pmatrix} \Sigma_{aa} & \Sigma_{ab} \\ \Sigma_{ba} & \Sigma_{bb} \end{pmatrix}$. The conditional distribution $P(x_a \mid x_b)$ is again Gaussian, and the top-left block of the joint precision matrix $\Sigma^{-1}$ is exactly $\Sigma^{aa}$, which is the Schur complement: $\Sigma^{aa} = (\Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba})^{-1}$. The conditional mean is $\mu_{a|b} = \mu_a + \Sigma_{ab}\Sigma_{bb}^{-1}(x_b - \mu_b)$. This correspondence has far-reaching implications: (1) in Gaussian Markov Random Fields, zero entries in the precision matrix correspond to conditional independence relationships, encoding conditional structure more directly than the covariance matrix; (2) the posterior predictive in Gaussian Processes (GP) is fundamentally a conditional Gaussian, yielding a closed-form solution via the Schur complement; (3) the Kalman filter update step can also be understood from the Schur complement perspective.

Follow-up: Why are Gaussian graphical models encoded using the precision matrix rather than the covariance matrix? What is the relationship between conditional independence and zero entries in the precision matrix? (Answer: $x_i \perp x_j \mid \text{rest}$ iff $\Sigma^{-1}_{ij} = 0$; the non-zero pattern of the precision matrix directly corresponds to the edge structure of the graph.)

A: KL divergence $D_{\text{KL}}(P \| Q)$ is infinite when the supports of $P$ and $Q$ do not overlap (even if they are geometrically "close"); in high-dimensional spaces this is almost always the case, since two low-dimensional manifold supports almost never overlap. Wasserstein distance (Earth Mover's Distance), based on optimal transport, is defined as $W(P, Q) = \inf_{\gamma \in \Pi(P,Q)} E_{(x,y)\sim\gamma}[\|x - y\|]$, and provides a meaningful finite distance even when supports do not overlap. Intuitive analogy: KL looks at the ratio $P/Q$ (and "explodes" if $Q = 0$ where $P > 0$); Wasserstein measures the minimum "work" required to transport one pile of "dirt" to another. Therefore, Wasserstein provides smooth gradient signals even when distributions do not overlap, while KL or JS divergence leads to vanishing gradients. WGAN uses the Wasserstein-1 distance as the generator objective, converting it via Kantorovich–Rubinstein duality to an optimization over 1-Lipschitz discriminators, resolving the training instability of classical GANs.

Follow-up: The computational cost of Wasserstein distance is typically higher than that of KL divergence; how can it be approximated in high dimensions? (Hint: the Sinkhorn algorithm adds entropic regularization to optimal transport, converting a linear program into parallelizable matrix-scaling iterations; Sliced Wasserstein reduces high-dimensional problems to multiple one-dimensional problems via random projections.)

A: At a critical point (where the gradient is zero), the eigenvalue distribution of the Hessian $H$ determines the local geometry: (1) all eigenvalues $> 0$ (positive definite) → local minimum, the surface is bowl-shaped; (2) all eigenvalues $< 0$ (negative definite) → local maximum; (3) a mix of positive and negative eigenvalues (indefinite) → saddle point, with descent directions along negative-curvature directions. In high-dimensional parameter spaces, saddle points far outnumber local minima (because the probability of all eigenvalues being positive decays exponentially), making them the main obstacle in non-convex optimization. Escape strategies: (1) SGD gradient noise naturally provides perturbations along negative-curvature directions, helping escape saddle points — the geometric interpretation of SGD's implicit regularization effect; (2) momentum helps traverse flat regions; (3) explicit methods such as adding perturbations along Hessian negative-curvature directions (not yet common but theoretically effective). The eigenvalue spectrum of the Hessian is also related to generalization: the larger the eigenvalues of the Hessian at a minimum (the "sharper" the surface), the worse that minimum tends to generalize — this motivates sharpness-aware minimization (SAM) and related methods.

Follow-up: Why is the saddle point problem more severe than the local minimum problem in extremely high-dimensional spaces? How does this follow from the eigenvalue distribution of the Hessian? (Answer: in an $n$-dimensional space, the Hessian has $n$ eigenvalues; at a random critical point each eigenvalue is equally likely positive or negative, so the probability that all are positive is $2^{-n}$, exponentially small — almost any critical point is a saddle point rather than a local minimum.)