Adam and Other Optimizers

Plain gradient descent uses one learning rate for every weight and reacts only to the current gradient. Two ideas fix its main weaknesses:

• Momentum: keep a running average of recent gradients and step along it. Zig-zags across a narrow valley cancel out, while consistent downhill directions build up speed.
• Adaptive step sizes (RMSProp): keep a running average of each weight’s squared gradients and divide by its square root. Weights with large, noisy gradients take smaller steps; weights with tiny gradients take bigger ones.

Adam (2014) combines both, with a small correction for the first few steps when the averages are still near zero. In practice it trains most networks well with its default settings, which is why it became the default. AdamW fixes how weight decay (regularization) interacts with the adaptive steps and is the standard for Transformers.

The 3D widget lets you race plain gradient descent, momentum and Adam on the same landscape.

The math

With gradient $g_t$, decay rates $\beta_1, \beta_2$ and learning rate $\eta$:

$$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$$$$\theta_t = \theta_{t-1} - \eta \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}, \quad \hat m_t = \frac{m_t}{1-\beta_1^t}, \ \hat v_t = \frac{v_t}{1-\beta_2^t}$$