Gradient Boosting

Start with a very simple guess, such as the average house price. It is wrong for most houses. Compute each house’s residual (true price minus guess). Now train a small decision tree to predict those residuals, and add a fraction of its output to the guess. Repeat.

After hundreds of rounds, the sum of all those small trees is a very accurate model.

Why “gradient”? For squared error, the residuals are exactly the negative gradient of the loss with respect to the predictions. Fitting trees to the negative gradient works for any differentiable loss, so the same algorithm handles regression, classification and ranking. It is gradient descent, but in the space of functions instead of weights.

The learning rate (shrinkage) scales each tree’s contribution. Smaller steps need more trees but generalize better.

The math

$$F_m(x) = F_{m-1}(x) + \eta \, h_m(x), \qquad h_m \approx -\frac{\partial L(y, F)}{\partial F}\bigg|_{F = F_{m-1}(x)}$$

where $h_m$ is the tree fitted at round $m$ and $\eta$ the learning rate.