Gradient Descent

Picture the loss as a landscape: every possible setting of the model’s parameters is a location, and the height is how wrong the model is there. Training means finding a low point.

The gradient is a vector that points in the direction the loss increases fastest. Gradient descent steps the opposite way:

1. Compute the gradient of the loss with respect to every parameter.
2. Move every parameter a small step against its gradient.
3. Repeat thousands or millions of times.

The size of each step is the learning rate, the most important knob in deep learning. Too small and training crawls; too large and you jump over the valley, possibly further up the other side.

Stochastic and mini-batch variants compute the gradient on a small random batch of examples instead of the whole dataset. Each step is noisier but far cheaper, and the noise even helps escape flat regions. Momentum keeps a running average of past gradients so the ball keeps rolling through small bumps; Adam additionally scales each parameter’s step by how large its gradients have been.

For neural networks, the gradient itself is computed with backpropagation.

The math

For parameters $\theta$, loss $L$ and learning rate $\eta$:

$$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta L(\theta_t)$$

With momentum $\beta$ (often 0.9):

$$v_{t+1} = \beta v_t + \nabla_\theta L(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1}$$