Perplexity
How surprised a language model is by real text. Lower is better: a perplexity of 20 means the model is about as unsure as if it were choosing among 20 equally likely words each time.
Alsocross-entropy · bits per token · language model evaluation
Play 'guess the next word' on a real book. If you usually have it narrowed down to a handful of options, your perplexity is low. If you are lost, it is high.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Applies to
Try it
How it works
A language model assigns a probability to each next token. Perplexity asks: on real text the model has not seen, how much probability did it give to what actually came next?
- For each token in the text, take the probability the model gave it.
- Average the negative log of those probabilities: this is the cross-entropy loss used in training.
- Exponentiate.
The result reads as an “effective number of choices”. Perplexity 1 would mean perfect certainty; perplexity 20 means the model is, on average, as uncertain as choosing among 20 equally likely options.
It is the standard way to see whether training is progressing, and a sensitive alarm for damage: the quantization widget uses GPT-2’s real perplexity to show what lower precision costs. It does not measure whether a model is helpful or truthful; for that you need task benchmarks and human evaluation.
The math
Everyday use cases
- Case 01
Tracking pretraining
Perplexity on held-out text falls steadily as a language model trains.
- Case 02
Checking compression
Measure how much quantizing or distilling a model hurts it by comparing perplexity before and after.
Strengths, weaknesses and when to use it
Strengths
- Cheap, automatic and directly tied to the training objective.
Weaknesses
- Says little about helpfulness, truthfulness or reasoning.
- Not comparable across different tokenizers.
When to use itFor comparing language models of the same family on the same text, and for detecting damage from compression.
Compare with
Check yourself
- Q01
Model A has perplexity 15, model B 30 on the same text and tokenizer. Which predicts the text better?
Show answer
A: Model A