Ridge and Lasso Regression

Plain linear regression picks the weights that fit the training data best. With many features or few rows, “best” often means huge, unstable weights that chase noise: overfitting.

Ridge and lasso add a price for large weights, the idea of regularization applied to linear models:

• Ridge (L2) charges the square of each weight. Big weights are expensive, so everything shrinks smoothly toward zero, but rarely reaches it.
• Lasso (L1) charges the absolute value. That price has a sharp corner at zero, so weights of weak features land exactly on zero: the model selects features for you.

The math

$$\text{Ridge: } \min_w \lVert y - Xw \rVert^2 + \lambda \lVert w \rVert_2^2 \qquad \text{Lasso: } \min_w \lVert y - Xw \rVert^2 + \lambda \lVert w \rVert_1$$

Ridge has a closed form, $w = (X^\top X + \lambda I)^{-1} X^\top y$; the added $\lambda I$ also makes the matrix invertible when features are correlated. Lasso has no closed form and is solved with coordinate descent. Elastic net adds both penalties.

The strength $\lambda$ trades fit against simplicity: $\lambda = 0$ is ordinary regression, a huge $\lambda$ flattens the model toward a constant. Choose it with cross-validation, and scale the features first so the penalty is fair to all of them.