Training speed is set by the condition number kappa, the ratio of the sharpest to the flattest curvature of the loss surface: the learning rate is capped by the sharp direction, progress is paced by the flat one, and every technique here attacks that ratio.
The central mechanism
- Loss surface curvature differs by direction;
kappa = 1is a round bowl, highkappais a ravine. - Max stable learning rate is
2/lambda_max(lambda_max= sharpest curvature). Exceed it and the sharp coordinate diverges regardless of the rest. - Progress is bounded below by the flattest direction, so cost scales with
kappa. - Two routes to shrink
kappa: reshape the surface (RMSProp, normalization, good init) or use history (momentum).
Optimizer family
flowchart LR
SGD["SGD"] --> MOM["Momentum"]
SGD --> RMS["RMSProp"]
MOM --> ADAM["Adam"]
RMS --> ADAM
ADAM --> ADAMW["AdamW"]
style ADAM fill:#2d6a4f,color:#fff
| Method | Update idea | Fixes | Speedup |
|---|---|---|---|
| SGD | w -= eta*g | nothing | baseline |
| Momentum | EMA of g, beta=0.9 | ravines (over time) | kappa -> sqrt(kappa) |
| RMSProp | divide by sqrt(EMA of g^2), rho=0.9 | scale mismatch (per param) | scale-free step ~eta |
| Adam | both, plus bias correction | both | default optimizer |
| AdamW | decay applied outside sqrt(v) | broken L2-in-Adam | correct regularization |
- Momentum amplifies consistent gradients
1/(1-beta)=10x, attenuates oscillating ones1/(1+beta)=0.53x; it silently multiplies effective LR by1/(1-beta), so loweretawhen enabling it. - Adam defaults:
beta_1=0.9(~last 10 grads),beta_2=0.999(~last 1000),eps=1e-8. Bias correctionm/(1-beta_1^t); distortion peaks ~6.5x near step 12, still 26% at step 1000. - Adam stores
m+v(2N) vs momentum’s1N.
Batch size
- Gradient noise falls as
sigma/sqrt(B); cost rises asB. QuadruplingBquadruples compute per step but only halves noise. - Helps while the device is underused; does nothing once saturated; hurts too small (kernel-launch bound, erratic) or too large (loses exploration noise, sharper minima).
- LR scaling: linear (
eta~B) with SGD, sqrt (eta~sqrt(B)) with Adam. Linear scaling always ships with warmup.
Schedules, clipping, init
- Warmup (always with Adam, 500-4000 steps ~
1/(1-beta_2)):vis noisy from few samples early, and one oversized step saturates attention softmax (dead gradient). Use pre-LN. - Decay: cosine to
~0.1*eta_maxif budget known; else constant then step down on val plateau. Anneal to0.1*eta_max, not 0. - Gradient clipping:
if ||g||_2 > c: g *= c/||g||_2. Clip by global norm (preserves direction). Order: unscale -> clip -> step. Still needed under Adam because a spike poisonsv(~11x for a 100x spike) freezing steps for thousands of iterations. - Init: keep per-layer gain
k = n_in*Var(w) = 1, else variance goesk^L. XavierVar(w)=2/(n_in+n_out)for tanh; HeVar(w)=2/n_infor ReLU (extrasqrt(2)since ReLU zeros half). Residual branches: scale by1/sqrt(2L). Zero/constant init fails by symmetry.
Regularization
- L2 = weight decay under SGD only; under Adam they differ (L2 rides the gradient through
sqrt(v)), so use AdamW. Never decay biases or LayerNorm gains. - L2 keeps high-curvature directions, deletes the ones data didn’t constrain. L1 gives sparsity (constant
lambda*sign(w)pull to exactly zero). - Dropout: keep prob
p, inverted (x*mask/pat train) soeval()is a no-op. Forgettingmodel.eval()leaves dropout on. Dropout + downstream BatchNorm interact badly (variance shift1/p). - Early stopping ≈ L2:
lambda_effective ~ 1/(eta*t); don’t tune both. - Label smoothing caps the otherwise-infinite logit gap, improving calibration.
- Regularization narrows the train/val gap; it cannot lower the val level. Hurts when underfitting.
Diagnosis
Overfit a single batch (8 samples, no shuffle/aug/dropout/decay) to ~0 loss first. If it can’t, it’s a wiring bug (labels, loss axis, requires_grad, zero_grad, eta=0). Then read the loss-curve shape:
| Symptom | Mechanism | Fix |
|---|---|---|
Flat at ln(C) | LR too low, dead ReLUs, or k<1 init | LR range test; He init |
Rises then nan | eta > 2/lambda_max | halve eta; clip at 1.0 |
| Spikes, never recovers | outlier poisoned Adam’s v | clip before step; restart checkpoint |
| Train down, val up | overfitting | data/aug, then decay/dropout |
| Val below train | dropout/BN accounting | re-measure train in eval() |
| Val ~0 from epoch 1 | leakage | split by entity |
| Bigger batch, no speedup | device already saturated | stop growing B |
Scale
- Mixed precision: 16-bit math + fp32 master weights. fp16 needs loss scaling (
S=1024, unscale before step) as grads near1e-7underflow; bf16 avoids it (fp32-range exponent). Saves activations and matmul time, not optimizer state. - Gradient accumulation:
(loss/ACCUM).backward(); missing the divide is a silentACCUMx LR. BatchNorm can’t be emulated this way. - Parallelism: data parallel if one replica fits; else shard optimizer state (ZeRO/FSDP), a layer (tensor parallel, needs NVLink), or depth (pipeline, pay the bubble
(P-1)/(m+P-1)).