Overfitting
When a model memorizes its training data, noise included, and then fails on new data. Its opposite, underfitting, is a model too simple to learn the pattern.
Alsounderfitting · generalization gap
A student who memorizes last year's exam answers word for word aces that exam but fails a new one. A student who only learned 'the answer is usually C' fails both. You want the one who understood the material.
Where it fits
Not shown: 2 more that build on it. See the full list below.
+All relationships as a list
Is a kind of / part of
Siblings
- Accuracy
- Bias-Variance Trade-off
- Cross-Validation
- Curse of Dimensionality
- Data, Features and Labels
- Gradient Descent
- Hyperparameters
- Linear Algebra Basics
- Log Loss (Cross-Entropy)
- Loss Function
- Precision, Recall and F1
- Probability Basics
- Regression Metrics
- Regularization
- ROC Curve and AUC
- Train, Validation and Test Sets
How it works
Every dataset contains two things: the real pattern and noise (measurement errors, coincidences). A good model learns the pattern and ignores the noise.
- An underfitting model is too simple. A straight line through clearly curved data misses the pattern; both training and test error are high.
- An overfitting model is too flexible. A wiggly curve that passes exactly through every training point has learned the noise too. Training error is tiny, test error is large.
That is why we always hold back a test set the model never sees during training. The gap between training and test performance tells you which side you are on.
Ways to fight overfitting:
- More data, so noise averages out.
- Regularization: penalize overly complex models.
- Dropout in neural networks.
- Early stopping: stop training when validation error starts rising.
- Ensembles like random forests, which average out individual models’ quirks.
This tension is often called the bias-variance trade-off: simple models have high bias (systematic error), flexible models have high variance (sensitivity to the exact training sample).
Everyday use cases
- Case 01
A stock-trading model
A strategy that perfectly explains last year's prices often loses money next year because it fitted noise, not signal.
- Case 02
Medical imaging
A model once learned to spot the hospital's ruler in the X-ray instead of the disease. Great on training data, useless elsewhere.
Strengths, weaknesses and when to use it
Strengths
- Understanding it is the single most useful skill for building models that work in practice.
Weaknesses
- Not always visible from training metrics alone. You need held-out data to see it.
When to use itCheck for it every time you train a model by comparing training error to validation error.
Check yourself
- Q01
Training error is 1% and test error is 30%. What is most likely happening?
Show answer
B: Overfitting
- Q02
Which of these usually reduces overfitting?
Show answer
C: Collecting more training data