Bagging
Trains many copies of a model on random resamples of the data and averages their predictions, which cancels out much of each model's jitter.
Also bootstrap aggregating · bootstrap · bagged trees
Breiman, 1996
Ask ten friends to guess the number of sweets in a jar, each after looking from a slightly different angle. Their average guess is usually closer than most single guesses.
Where it fits
+All relationships as a list
Is a kind of / part of
Applies to
How it works
A deep decision tree fits its training data closely, so small changes in the data give a noticeably different tree. In bias-variance terms it has low bias and high variance. Bagging keeps the low bias and averages away much of the variance.
- Bootstrap: draw a sample of rows with replacement from the training rows. Some rows appear twice, about a third are left out.
- Fit: train one model on each of, say, 100 such samples.
- Aggregate: average their predictions, or take a majority vote for classes.
Each model makes different mistakes, and averaging cancels many of them. The left-out rows give each model a small test set for free, the out-of-bag estimate.
A random forest is bagging of trees plus one more trick: each split may only look at a random subset of features, which makes the trees even more different. Boosting is the other main ensemble idea: instead of independent models in parallel, it trains them one after another, each fixing the last.
Everyday use cases
- Case 01
Steadier predictions
Averaging 100 bagged trees gives house price estimates that don't swing wildly when a few sales are added.
- Case 02
Free validation
The out-of-bag rows act as a built-in test set, handy when data is scarce.
Strengths, weaknesses and when to use it
Strengths
- Reduces overfitting of unstable models with almost no tuning.
- Trivially parallel: every model trains independently.
Weaknesses
- Does little for stable, high-bias models such as linear regression.
- Many models are slower and harder to interpret than one.
When to use itWhen a flexible model (usually a deep tree) fits well but varies a lot from sample to sample.
Compare with
Check yourself
- Q01
What is a bootstrap sample?
Show answer
B: A sample of the same size as the dataset, drawn with replacement
- Q02
Bagging mainly reduces which kind of error?
Show answer
A: Variance