Dropout
During training, randomly switch off a fraction of neurons at every step, so the network cannot rely on any single neuron. Reduces overfitting.
Hinton et al., 2012; Srivastava et al., 2014
A football team where random players have to sit out each practice. Everyone has to learn every position, so the team plays well no matter who is on the field.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Applies to
Used by
How it works
During each training step, dropout picks a random subset of neurons (for example 50%) and sets their output to zero. Next step, a different random subset. The network can never count on a particular neuron being there, so it learns redundant, spread-out features instead of fragile co-dependencies.
At prediction time nothing is dropped. To keep the overall signal strength the same, the surviving activations are scaled up during training (the usual “inverted dropout” trick).
You can think of it as training a huge ensemble of smaller networks that share weights, then using their average at the end, like a random forest for neurons. It is a form of regularization against overfitting.
Everyday use cases
- Case 01
Image classifiers
AlexNet used dropout in its fully connected layers to curb overfitting on ImageNet.
- Case 02
Small datasets
When labeled data is scarce, dropout is one of the cheapest ways to make a network generalize.
Strengths, weaknesses and when to use it
Strengths
- One line of code, often a solid improvement in generalization.
Weaknesses
- Slows convergence, since each step trains a random subnetwork.
- Less useful in very large models trained on huge datasets, where overfitting is less of an issue.
When to use itWhen a neural network's validation error is clearly worse than its training error.
Compare with
Check yourself
- Q01
Is dropout active when the trained model makes predictions?
Show answer
B: No, it is only used during training