Early Stopping
Stop training when performance on a validation set stops improving, and keep the best version, before the model starts memorizing the training data.
Also patience · checkpointing · validation monitoring
Studying for an exam with practice tests. Once your practice scores stop going up and start dropping because you're just memorizing answers, you stop and go with what you knew at your best.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Applies to
How it works
Train a flexible model long enough and its training loss keeps falling, but at some point its validation loss turns around and starts rising. That turning point is where learning the pattern turns into memorizing the noise: overfitting.
Early stopping watches for that moment:
- After each epoch, measure the loss on a validation set.
- Whenever it improves, save a copy of the model.
- If it has not improved for a set number of checks (the “patience”), stop.
- Use the best saved copy.
It is a form of regularization: it limits how far the weights can travel from their small starting values. It pairs well with dropout and weight decay, and it is standard in gradient boosting too, where it decides the number of trees.
Everyday use cases
- Case 01
Training any model iteratively
Gradient boosting libraries and deep learning frameworks support it out of the box.
Strengths, weaknesses and when to use it
Strengths
- Free, simple and saves compute.
Weaknesses
- Uses up a validation set; a noisy validation loss can stop too early.
When to use itWhenever you train iteratively and have a validation set, especially on small datasets.
Compare with
Check yourself
- Q01
Training loss keeps falling but validation loss has risen for 10 epochs. What does early stopping do?
Show answer
B: Stop and restore the weights from the best validation epoch