Learning Rate

Gradient descent tells you which direction is downhill. The learning rate decides how far to move in that direction each step.

• Too small: every step is tiny. Training is slow and can stall on flat regions.
• About right: the loss falls steadily.
• Too large: steps overshoot the bottom of the valley. The loss bounces around or explodes.

Try it in the widget: raise the learning rate and watch the ball start to zig-zag and then fly off the surface.

Schedules

Most modern recipes do not keep the learning rate fixed:

• Warmup: start very small for the first few hundred or thousand steps, because early gradients are noisy and large.
• Decay: gradually lower it (cosine or linear) so the final steps settle precisely into a minimum.

Adam and other adaptive optimizers rescale the step for each parameter, which makes training less sensitive to the exact value, but you still choose the global learning rate. It is the first of the hyperparameters worth tuning.

The math

For parameters $\theta$ and loss $L$, one step is

$\theta_{t+1} = \theta_t - \eta \, \nabla L(\theta_t).$

On a simple quadratic bowl $L = \tfrac{a}{2}\theta^2$, the update is $\theta_{t+1} = (1 - \eta a)\theta_t$. It converges only when $|1 - \eta a| < 1$, that is $\eta < 2/a$: steeper curvature demands a smaller learning rate.