Neural Networks and Architectures

A layer is a structural assumption about the data; each layer here fixes a specific failure of the one before, and most of those failures trace to one fact: the gradient reaching layer 1 of an L-layer stack is a product of L Jacobians, which vanishes (factors < 1) or explodes (factors > 1).

Pick a layer by input structure

Read a row as: this layer maps this shape, betting this is true; when it is false, this breaks.

• Prior = assumption wired in instead of learned. A strong prior saves data. Tabular data (no structure) → dense, but gradient-boosted trees usually win.

Depth beats width

• ReLU net is piecewise-linear; count pieces (linear regions) to count expressive power.
• One hidden layer of width w: at most w+1 pieces (width adds kinks linearly).
• The fold |2x−1| copies its input range onto the output twice, so composing doubles pieces: L folded layers → 2^L pieces. Width adds, depth composes.
• Montúfar 2014 bound: Ω((w/d_in)^(d_in·(L−1)) · w^d_in) regions — L in the exponent, w in the base. Exponential in depth, polynomial in width.
• Expressivity ≠ learnability. A plain 50-layer net can represent more than a 10-layer one but trains worse (gradient dies through 50 Jacobians). Residuals + normalization make depth trainable.

Convolution and receptive field

• Kernel of k×k, C_in→C_out: params = k²·C_in·C_out + C_out. (3×3, 3→64 = 1,792 vs 150M for the equivalent dense layer, ~84,000× fewer.)
• Weight sharing → translation equivariance (shift input, output shifts identically). Invariance to shift comes from pooling on top, not conv.
• Receptive field recurrence — update r with the incoming j, then update j:
  • r_l = r_{l−1} + (k_l − 1)·j_{l−1} then j_l = j_{l−1}·s_l
  • Ordering trap: advance j before r and every later r is wrong.
• A 3×3 conv adds (k−1)·j = 2j to r; a stride-2 pool adds only j directly but doubles j, so downsampling (not depth) buys global context cheaply.
• same padding never changes r or j. Dilation d covers d·(k−1)+1 pixels with k weights at full resolution.

Recurrence → LSTM/GRU

• RNN: h_t = tanh(W·h_{t−1} + U·x_t + b), weights shared across time.
• Vanishing gradient is an exponential: per-step Jacobian D_t·W, so ‖dL_T/dh_k‖ ≤ (max|tanh'|·‖W‖)^(T−k) = s^(T−k). With s ≈ 0.84: 0.84^100 = 2.7e-8 (dead). No s stays put for all T.
• Exploding side is the easy one: gradient clipping rescales a too-large norm (threshold ~1.0). Nothing recovers an underflowed gradient.
• LSTM replaces the matrix product with an additive cell state: c_t = f_t ⊙ c_{t−1} + i_t ⊙ g_t, so dc_t/dc_{t−1} = diag(f_t) — numbers the net chooses per step/channel. f=0.99 → 0.99^100 = 0.37 survives.
  • Gates: forget f, input i, output o (all sigmoid, “fraction to keep”), candidate g (tanh, “value to write”).
  • Params = 4·(d_in·d_h + d_h² + d_h). Init b_f to 1.0–2.0 so the untrained gate starts at sigmoid(1)=0.73, not 0.50.
• GRU: 2 gates (update, reset), 1 state, 3·(…) params (25% fewer). z trades forget vs write off against each other.
• What killed recurrence: parallelism, not gradients (attention trains all positions in one matmul). Recurrence still wins fixed memory: state is O(1) per token vs a KV cache’s O(n).

Attention

• Derived from a dict lookup by removing what blocks gradients: hard argmax → soft match (q·k) → softmax → convex combination of values.
• attention(Q,K,V) = softmax(Q·Kᵀ / √d_k)·V, with q_i=W_q x_i, k_i=W_k x_i, v_i=W_v x_i (same 3 matrices at every position). q=what I ask, k=what I advertise, v=what I hand over; W_q≠W_k so a token can ask ≠ offer.
• Dot product = ‖q‖‖k‖cos θ: relevance is an angle.
• √d_k is forced. With unit-variance components, Var(q·k)=d_k, so sd=√d_k (11.3 at d_k=128). Unscaled → near one-hot softmax → p(1−p)≈1e-5 gradient (frozen forward and backward). Dividing by √d_k restores unit variance; dividing by d_k over-corrects → uniform weights.
• Causal mask: set future scores to −∞ before softmax (so e^{−∞}=0 and surviving weights still sum to 1). Zeroing after leaves rows summing to <1.
• Multi-head: h heads of width d/h, total params 4·d² regardless of h (the h cancels) — heads partition a fixed projection, trading per-head resolution for reading several places at once.
• Attention is permutation-equivariant: it cannot tell word order → inject position (sinusoidal/learned encodings, or RoPE).

Transformer block and parameter arithmetic

• Block = 2 sublayers, each wrapped in a norm + residual add: attention (moves info between positions) and FFN d→4d→d (transforms within a position). Params: attention 4d², FFN 8d² → 12·d² per block.
• Residual y = x + F(x): Jacobian I + dF/dx, so ∏(I+J_l) always has a bare-I identity path → gradient floor ≥ 1, can’t decay in L. Also lets a block learn F=0 (adding depth can’t raise train loss); acts as an ensemble of 2^L paths.
• Pre-LN x + Sublayer(LN(x)) keeps the residual stream a true identity → deep stacks train without careful init (post-LN puts norm on the path and needs warmup). Warmup still mandatory. Residual stream variance grows as 1+2L → final norm before unembedding is required.
• Normalization (x−μ)/√(var+eps)·gain+bias; layers differ only in the reduction axis. LayerNorm/RMSNorm reduce over D per token → identical train/inference, work at batch 1. BatchNorm reduces over B,T per channel → depends on batch-mates, breaks on variable length, batch-1 decoding, and gradient accumulation. RMSNorm drops mean + bias (x/√(mean(x²)+eps)·gain), one pass cheaper (norms are memory-bandwidth bound).
• Dying ReLU: if z<0 on all data, gradient is 0 forever (absorbing state) — usually from too-high lr. Fix: lower lr/warmup, GELU/SiLU/LeakyReLU, He init, normalization. SwiGLU (current default) uses 3 matrices, so d_ff = (8/3)·d (rounded to a tile multiple, e.g. 11,008).
• Wiring decides the job: encoder-only (BERT, bidirectional, ~15% positions supervised) → representations; decoder-only (GPT, causal, 100% supervised) → generation; encoder-decoder (T5) → seq-to-seq. Decoder-only won on supervision density (6.7×) + one stack + prefix caching + in-context learning.
• params ≈ 12·L·d² + V·d. Compute: forward 2N FLOPs/token, training 6N. Attention dominates FFN only past n = 6d tokens. KV cache = 2·L·d_kv·bytes per token (512 KiB/token at d=4096, fp16); GQA shares KV heads for a 4× cut, MQA for 32×.

flowchart TD
    IN{"input structure?"}
    IN -->|"none, tabular"| D["Dense (or GBT)"]
    IN -->|"local patterns"| C["Convolution"]
    IN -->|"sequence"| Q{"dependency length?"}
    Q -->|"short, streaming, fixed memory"| R["RNN / LSTM / GRU"]
    Q -->|"long, train in parallel"| A["Attention"]
    style A fill:#2d6a4f,color:#fff