Skip to content
ML Atlas

Ensemble Methods

Combine many models into one stronger model. Bagging averages independent models; boosting builds models that fix each other's mistakes.

Also ensembles · boosting

Fig. / How it works
MAJORITY VOTEMAJORITY VOTE3 trees, 3 bootstrap samples
Role · kind
Models and algorithmsFamily
Level
Intermediate

Ask one friend to guess the number of sweets in a jar and they may be way off. Ask a hundred friends and take the average, and you will be surprisingly close.

Where it fits

Classical Machine LearningClassical Machine LearningDecision TreeDecision TreeEnsemble MethodsAdaBoostAdaBoostBaggingBaggingGradient BoostingGradient BoostingRandom ForestRandom Forest
+All relationships as a list

Is a kind of / part of

Uses / built on

See it on the full map

How it works

One model has quirks; many models with different quirks cancel each other out. The trick is getting the members to differ:

  • Bagging (bootstrap aggregating) trains each model on a random resample of the data, then averages or votes. Random forest is bagging with an extra twist.
  • Boosting trains models one after another, each focusing on what the ensemble so far gets wrong. Gradient boosting is the dominant version.

Both usually use decision trees as members, because trees are fast and flexible, and individually unstable, which is exactly what ensembling fixes.

Everyday use cases

  • Case 01

    Winning tabular competitions

    Gradient-boosted trees are a staple of winning solutions on structured data.

Famous examples

Strengths, weaknesses and when to use it

Strengths

  • Usually more accurate than any single member.
  • Averaging reduces overfitting.

Weaknesses

  • Harder to interpret than a single model.
  • More compute at training and prediction time.

When to use itAlmost always for tabular data, once a single model works.