Linear Regression

Linear regression assumes the thing you want to predict changes in a straight line as the inputs change. With one input (say floor area) the model is a line; with two inputs it is a flat plane; with many inputs it is a “hyperplane”, but the idea is the same.

Each prediction is a weighted sum: multiply every feature by its weight, add them up, add a constant (the bias). Training means finding the weights that make predictions as close as possible to the true values in the training data.

“As close as possible” needs a definition, and that is the loss function. Linear regression uses mean squared error: for each point, take the vertical gap between the true value and the line (the residual), square it, and average. Squaring makes all gaps positive and punishes big misses much more than small ones.

There are two ways to find the best weights:

1. Solve it directly. For squared error there is an exact formula, so you can compute the answer in one step.
2. Walk downhill. Start with random weights and repeatedly nudge them in the direction that lowers the error. That is gradient descent, the same idea that later trains huge neural networks.

The math

With features $x \in \mathbb{R}^d$, weights $w$ and bias $b$, the prediction is

$$\hat{y} = w^\top x + b$$

Over $n$ training points the mean squared error is

$$\text{MSE}(w, b) = \frac{1}{n} \sum_{i=1}^{n} \left(y_i - w^\top x_i - b\right)^2$$

Stacking the inputs in a matrix $X$ (with a column of ones for the bias), the minimizer is the normal equation

$$\hat{w} = \left(X^\top X\right)^{-1} X^\top y$$

which exists when the columns of $X$ are linearly independent.

Once trained, the weights are easy to read: a weight of 2,000 on “floor area in m²” means each extra square meter adds about 2,000 to the predicted price, holding the other features fixed.