Regularization

Overfitting happens when a model uses its flexibility to memorize noise. Regularization takes some of that flexibility away.

The classic way is to add a penalty on the size of the weights to the loss. Now the model has two goals: fit the data, and keep the weights small. Large weights are only worth it if they really help.

• L2 (ridge, weight decay) penalizes squared weights. It shrinks everything a little and spreads importance across correlated features.
• L1 (lasso) penalizes absolute weights. It tends to set unimportant weights to exactly zero, giving a simpler model.

Other techniques do the same job differently: dropout randomly disables neurons during training, early stopping halts training before memorization sets in, and data augmentation creates new varied training examples.

The math

$$L_{\text{reg}}(w) = L_{\text{data}}(w) + \lambda \, \|w\|_2^2 \quad \text{(L2)} \qquad L_{\text{reg}}(w) = L_{\text{data}}(w) + \lambda \, \|w\|_1 \quad \text{(L1)}$$

The hyperparameter $\lambda$ sets how strongly complexity is punished.