Scaling Laws
The observation that language model loss improves smoothly and predictably as you increase model size, training data and compute, following power laws.
Also Chinchilla · compute-optimal training · emergent abilities
Kaplan et al., 2020; Hoffmann et al. (Chinchilla), 2022
Bigger brains that read more books get better at predicting words in a surprisingly regular way, so you can forecast how good a model will be before you build it.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Tokenization
- Vector Database
- word2vec
Applies to
How it works
In 2020, researchers at OpenAI measured something remarkable: as language models get bigger, train on more data and use more compute, their test loss (see perplexity) drops along smooth straight lines on a log-log plot. Ten times more compute buys a predictable improvement, over many orders of magnitude.
That turned model building into something closer to engineering. You can train a series of small models, fit the curve, and forecast how a model a hundred times larger will do.
In 2022, DeepMind’s Chinchilla study refined the recipe: for a fixed budget, model size and training data should grow together, roughly 20 training tokens per parameter. Many earlier giants, including GPT-3, had been undertrained for their size; a smaller model trained on more data could beat them.
Two cautions. Scaling laws predict average loss, not when a particular ability will appear. And newer work shows a second axis: spending more compute at inference time, letting reasoning models think longer, can also improve results.
Everyday use cases
- Case 01
Planning a training run
Labs fit scaling curves on small models to choose the size and data budget of a big one.
- Case 02
Explaining the race to scale
Predictable gains are why organizations invested in ever larger models.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Makes expensive decisions predictable.
Weaknesses
- Predicts loss, not specific abilities; some skills appear suddenly or depend on data quality.
- Data, energy and cost limits are real.
When to use itWhen deciding how to spend a compute budget between model size and data.
Check yourself
- Q01
What did the Chinchilla study suggest about many earlier large models?
Show answer
A: They were too big for the amount of data they were trained on