Log Loss (Cross-Entropy)

Accuracy only checks whether the top answer was right. Log loss also looks at how confident the model was.

For each example, take the probability the model gave to the correct class, $p$, and compute $-\log p$:

Probability on the right answer	Loss
0.99	0.01
0.9	0.11
0.5	0.69
0.1	2.30
0.01	4.61

A confident wrong answer is punished far more than a hesitant one. The best strategy is to be honest: say 70% when you are right 70% of the time.

It is the default loss function for classification: logistic regression, image classifiers and language models all minimize it. For language models, averaging it over every token and taking the exponent gives perplexity.

The math

For $N$ examples with true classes $y_i$ and predicted distributions $p_i$,

$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \log p_i(y_i).$

This is the negative log-likelihood of the labels, so minimizing it is maximum likelihood estimation. See probability basics.