Calculus and Gradients for ML

In this lesson, we’ll trace how a model actually learns: training means minimizing a loss, and minimization runs on derivatives. We’ll cover the calculus you use in ML every day, gradients, the chain rule, and how they drive gradient descent and backpropagation. By the end you’ll be able to explain why a network steps downhill, how the gradient of a deep composition is assembled, and why computing every parameter’s gradient costs about as much as one forward pass.

The derivative: local slope

Start with one input and one output. The derivative of a function f(x) at a point is the slope of the tangent there, how much the output changes per unit change in the input:

f'(x) = limit as h -> 0 of ( f(x + h) - f(x) ) / h

The sign is the whole point. A positive derivative means the function is increasing; zero means a flat point (a candidate minimum, maximum, or saddle). Minimization walks downhill, so it moves against the sign of the derivative: if the slope is positive, we step left to get lower. That single rule is the seed of everything below, and it forces the next question: real losses depend on millions of parameters, not one, so what plays the role of the slope then?

Partial derivatives and the gradient

ML losses depend on many parameters at once, so we hold all but one fixed and differentiate with respect to that one: a partial derivative, written df/dw_i. Each partial answers a local question: if I nudge this one weight and freeze the rest, how does the loss move? Collect all the partials into a vector and you have the gradient:

grad f = [ df/dw_1, df/dw_2, ..., df/dw_n ]

The gradient points in the direction of steepest ascent, and its magnitude is how steep. Its negation points downhill, which is exactly the direction that reduces the loss fastest locally. That negation is the move we make next.

Gradient descent

Gradient descent takes repeated small steps against the gradient:

w <- w - eta * grad L(w)

eta is the learning rate, and its size is a genuine trade-off worth stating out loud: too large and the steps overshoot and diverge; too small and training crawls. The update is local: the gradient only knows the slope at the current point, which is why loss curves, learning-rate schedules, and momentum all exist to steer the walk. To run this update on a network, though, we need the gradient of a loss that is buried under many layers, and that forces the tool that makes deep learning possible.

The chain rule

Neural networks are compositions of functions, layer after layer, so their derivatives come from the chain rule: the derivative of a composition is the product of the derivatives along the way.

if y = f(g(x)), then dy/dx = f'(g(x)) * g'(x)

Here is the idea before the machinery. If x nudges g twice as fast, and g nudges f three times as fast, then x moves f six times as fast: the sensitivities multiply. For a network loss( layer_n( ... layer_1(x) ) ), the derivative of the loss with respect to an early weight is a product of factors, one per layer between that weight and the loss.

flowchart LR
  X["x"] --> L1["layer 1"] --> L2["layer 2"] --> LN["layer n"] --> Loss["loss"]
  Loss -. "grad flows back" .-> LN -. "* local deriv" .-> L2 -. "* local deriv" .-> L1

Reading the diagram back-to-front shows the plan: start at the loss and multiply local derivatives on the way back to each weight. Doing that naively repeats work, which is what the next section fixes.

Backpropagation

Backpropagation is the chain rule applied efficiently. A forward pass computes the output and caches intermediate values; a backward pass multiplies local derivatives from the loss back toward the inputs, reusing shared factors instead of recomputing them. For a function from many inputs to one scalar loss, this reverse order is what makes computing all parameter gradients cost about the same as one forward pass. That cost result is why we go backward rather than forward: a single scalar loss at the end means every path shares the same tail factors, and reverse order reuses them.

For vector-to-vector maps the multi-input, multi-output generalization of the derivative is the Jacobian, the matrix of all partials dy_i/dx_j. Backprop multiplies these Jacobians (usually as cheaper vector-Jacobian products) layer by layer. With all the gradients in hand, we are ready to assemble the pieces into the loop that trains a model.

Putting it together: the training loop

Each of these pieces has a place in one repeating cycle. A forward pass runs the input through the layers and computes the loss; backprop applies the chain rule to get every parameter’s gradient; the gradient-descent update nudges each parameter downhill; then it all repeats on the next batch.

flowchart TD
  Fwd["Forward pass: run input through layers, compute loss"]
  Back["Backprop: chain rule gives grad L for every parameter"]
  Update["Update: w <- w - eta * grad L(w)"]
  Fwd --> Back --> Update --> Fwd

Every knob you tune in practice, learning rate, batch size, schedule, hangs off one of these three boxes.

Conclusion

The gradient points uphill; descent steps against it, scaled by the learning rate eta.
The chain rule turns a deep composition’s derivative into a product of per-layer derivatives.
Backprop is the chain rule with cached intermediates, so computing all gradients costs roughly one extra pass over the network.
A zero gradient marks a minimum, maximum, or saddle. Flat slope alone does not guarantee a minimum.

One line to remember: training is downhill steps against the gradient, and backprop is the chain rule run backward with cached intermediates so every gradient comes nearly free.