Computational Graph and Chain Rule

Write any calculation as a flowchart of tiny steps and you get a computational graph. For $L = (w \cdot x + b - y)^2$: multiply, add, subtract, square. Each node takes some inputs and produces one output.

Forward pass

Evaluate the nodes from inputs to output, remembering each intermediate result. In a neural network this is “running the model”: data flows through the layers to a prediction and a loss.

Backward pass and the chain rule

To train, we need the slope of the loss with respect to every weight. The chain rule says slopes multiply along a path: if $L$ depends on $y$ and $y$ depends on $x$,

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x}.$

So we walk the graph backward. Each node only needs to know its own local slope; it multiplies that by the gradient arriving from above and passes the result down. When a value feeds into several nodes, the gradients coming back are added.

That procedure is backpropagation. Step through it in the widget: values flow right, gradients flow left.

Modern frameworks do this automatically (“autograd”). You write ordinary code, they record the graph during the forward pass and differentiate it for you. The cost: all those remembered intermediate values, which is why training needs much more memory than inference. Long chains of multiplications in the backward pass are also the root of vanishing and exploding gradients.