Bias-Variance Trade-off

Why does a model make mistakes on new data? Three reasons:

1. Bias: the model is too simple to represent the real pattern. A straight line cannot follow a curve, no matter how much data you give it. This is underfitting.
2. Variance: the model is so flexible that it changes a lot depending on which training examples it happened to see. It fits the noise. This is overfitting.
3. Noise: randomness in the data itself that no model can predict.

As you make a model more complex (a higher-degree polynomial, a deeper tree), bias goes down and variance goes up. Total test error first falls, then rises: the famous U-curve.

Ways to move along the trade-off:

• Lower variance: more data, regularization, bagging and random forests, early stopping.
• Lower bias: a more expressive model, better features, boosting.

A modern twist

Very large neural networks have far more parameters than training examples, yet often generalize well. Researchers observed double descent: past the point where the model can fit the training data perfectly, test error can start falling again. The bias-variance picture is still useful, but it is not the whole story for deep learning.

The math

For a target $y = f(x) + \varepsilon$ with noise variance $\sigma^2$ and a predictor $\hat f$ trained on a random dataset,

$\mathbb{E}\big[(y - \hat f(x))^2\big] = \underbrace{\big(f(x) - \mathbb{E}[\hat f(x)]\big)^2}_{\text{bias}^2} + \underbrace{\mathrm{Var}[\hat f(x)]}_{\text{variance}} + \sigma^2.$