Training minimizes a loss by taking downhill steps against its gradient, and backprop is the chain rule run backward to get every parameter’s gradient cheaply.
Derivative: local slope
- Derivative
f'(x) = lim_{h->0} (f(x+h) - f(x)) / h, the tangent slope. - Sign is the point: positive means increasing; zero means a flat point.
- Minimization moves against the sign: positive slope, step left to go lower.
Gradient
- Partial derivative
df/dw_i: nudge one weight, freeze the rest. - Gradient
grad f = [df/dw_1, ..., df/dw_n], all partials in a vector. - Points in the direction of steepest ascent; magnitude is steepness.
- Its negation points downhill, the fastest local loss decrease.
Gradient descent
- Update:
w <- w - eta * grad L(w). etais the learning rate: too large overshoots and diverges; too small crawls.- The gradient only knows the slope at the current point (local); schedules and momentum steer the walk.
Chain rule
- Network = composition of layers, so derivatives multiply.
- If
y = f(g(x)), thendy/dx = f'(g(x)) * g'(x), sensitivities multiply. - Loss gradient w.r.t. an early weight = product of one factor per layer between the weight and the loss.
Backpropagation
- Chain rule applied efficiently: forward pass caches intermediates, backward pass multiplies local derivatives from loss to inputs, reusing shared factors.
- Many inputs to one scalar loss, so reverse order shares tail factors: all parameter gradients cost about one forward pass.
- Jacobian: matrix of all partials
dy_i/dx_jfor vector-to-vector maps; backprop chains these (usually as vector-Jacobian products).
Training loop and gotchas
- Cycle: forward pass (compute loss) to backprop (gradients) to update (
w <- w - eta * grad L) to repeat. - A zero gradient marks a minimum, maximum, or saddle; flat slope alone does not guarantee a minimum.
loss --> * local deriv --> ... --> * local deriv --> weight
(grad flows backward, multiplying one local derivative per layer)