Loss Function

A model makes a prediction; reality gives the true answer. The loss measures how far apart they are. Average it over all training examples and you get one number that summarizes how badly the model is doing right now.

Two losses cover most cases:

• Mean squared error (MSE) for numbers. Square each miss and average. Big misses dominate.
• Cross-entropy (log loss) for classes. Look at the probability the model gave to the correct class. If it said 0.99, the loss is tiny; if it said 0.01, the loss is huge.

Because the loss is a smooth function of the model’s parameters, we can ask “which way should each knob turn to make it smaller?” That question is answered by gradient descent.

The math

Mean squared error over $n$ examples:

$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Binary cross-entropy, where $\hat{p}_i$ is the predicted probability of class 1:

$$\text{BCE} = -\frac{1}{n}\sum_{i=1}^{n} \Big[y_i \log \hat{p}_i + (1 - y_i)\log(1 - \hat{p}_i)\Big]$$