Backpropagation

Gradient descent needs the gradient: for every weight, “if I nudge this up a little, how much does the loss change?” A network can have billions of weights, so computing each one separately would be hopeless. Backpropagation computes them all in one sweep.

Training a network runs in two passes:

1. Forward pass. Data flows from input to output, layer by layer. Every intermediate value is remembered. At the end, the loss is computed.
2. Backward pass. Starting from the loss, go back through the network. At each step, the chain rule says: the effect of a value on the loss equals its effect on the next value, times that next value’s effect on the loss. Since the second part was just computed, each step is cheap.

When the backward pass reaches the input, every weight has its gradient. Gradient descent then updates them all, and the cycle repeats.

In modern frameworks you never write this by hand: the framework records the forward pass as a computational graph and runs backpropagation automatically when you call loss.backward().

The math

For a chain $x \to a = f(x) \to b = g(a) \to L = \ell(b)$, the chain rule gives

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial b} \cdot \frac{\partial b}{\partial a} \cdot \frac{\partial a}{\partial x}$$

Backprop evaluates this product from the left: $\partial L / \partial b$ first, then multiplies by each local derivative while walking backward, reusing every partial product.