AdaBoost
The first practical boosting algorithm: trains many tiny models one after another, each focusing on the examples the previous ones got wrong, then lets them vote.
Also adaptive boosting · decision stumps · boosting
Freund and Schapire, 1995
A class studies for a test in rounds. After each round the teacher circles the questions most students missed, and the next round spends extra time on exactly those. The final answer is a weighted vote.
Where it fits
+All relationships as a list
Is a kind of / part of
Uses / built on
How it works
There are two main ways to build an ensemble:
- Bagging trains models in parallel on random resamples of the data and averages them. Random forests are the famous example. It mainly reduces variance.
- Boosting trains models in sequence, each one fixing the mistakes of those before it. It mainly reduces bias.
AdaBoost (adaptive boosting) was the breakthrough boosting algorithm:
- Give every training example the same weight.
- Train a weak model, often a “decision stump” (a decision tree with a single split).
- Give that model a vote: the more accurate it was, the bigger its say.
- Increase the weights of the examples it got wrong, so the next model pays more attention to them.
- Repeat, then let all models vote.
Each stump alone is barely better than guessing; together they form a strong classifier. A famous early success was the Viola-Jones face detector (2001), which ran in real time on modest hardware.
Later work showed AdaBoost is gradient descent on a particular loss. Generalizing that idea gave gradient boosting, today’s go-to method for tabular data.
Everyday use cases
- Case 01
Real-time face detection
The 2001 Viola-Jones face detector, used in early digital cameras, chose its features with AdaBoost.
Strengths, weaknesses and when to use it
Strengths
- Simple, few hyperparameters, turns very weak models into a strong one.
Weaknesses
- Sensitive to noisy labels and outliers, which receive ever-growing weight.
- Mostly superseded by gradient boosting libraries in practice.
When to use itAs a teaching example of boosting, or a quick baseline on clean tabular data.
Compare with
Check yourself
- Q01
After each AdaBoost round, which examples get more weight?
Show answer
B: The ones classified wrongly