Vanishing and Exploding Gradients

Backpropagation computes gradients by multiplying local slopes along the computational graph. In a deep network that is a long product: one factor per layer.

• If the factors are typically smaller than 1, the product shrinks exponentially. The first layers receive almost no gradient and stop learning: vanishing gradients.
• If they are typically larger than 1, it grows exponentially. Updates become huge and training diverges: exploding gradients.

The sigmoid is a classic culprit: its slope is at most 0.25, so ten sigmoid layers can scale the gradient by $0.25^{10} \approx 10^{-6}$. Recurrent networks suffer even more, because the same weights are multiplied once per time step.

Almost every modern architecture contains a fix:

• ReLU and friends: slope 1 for positive inputs. See activation functions.
• Good weight initialization keeps signals at a steady size at the start.
• Normalization layers rescale activations at every layer.
• Residual connections give gradients a direct highway to early layers.
• Gates in LSTMs and GRUs let information and gradients pass through time almost unchanged.
• Gradient clipping caps the gradient’s size to stop explosions.