Train, Validation and Test Sets
Splitting data so the model learns on one part, is tuned on a second and is honestly graded on a third it has never seen.
Also train-test split · validation set · holdout · data leakage
Practice with homework (training), check yourself on a mock exam (validation), and only then sit the real exam (test). Peeking at the real exam beforehand makes your grade meaningless.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
How it works
A model’s score on its own training data says little: it may simply have memorized it (overfitting). So data is split:
- Training set (often 60 to 80%): the model learns its parameters here.
- Validation set: used to compare models and tune hyperparameters (settings like tree depth or learning rate that are not learned directly).
- Test set: touched once, at the very end, to estimate how the final model will do on new data.
Why three, not two? Every time you pick the model that scores best on some data, you slightly fit that data too. The test set stays clean only if it never influences any decision.
With little data, k-fold cross-validation helps: split into k parts, train k times, each time validating on a different part, and average.
The most common mistake is leakage: information from the test set sneaking into training, for example duplicates across splits, preprocessing statistics computed on all data, or for time series, shuffling so the model trains on the future and is tested on the past.
Everyday use cases
- Case 01
Every model evaluation
The reported accuracy of a model only means something if it was measured on untouched test data.
- Case 02
Time series
Forecasting models must be tested on later dates than they were trained on, never shuffled.
Strengths, weaknesses and when to use it
Strengths
- Simple and the foundation of trustworthy results.
Weaknesses
- Small datasets make single splits noisy (cross-validation helps).
- Leakage is easy to introduce accidentally.
When to use itAlways, before training anything.
Check yourself
- Q01
What is the validation set for?
Show answer
A: Choosing models and hyperparameters without touching the test set
- Q02
You scaled features using statistics computed on the whole dataset before splitting. What went wrong?
Show answer
A: Information from the test set leaked into training