Pretraining
The first and most expensive stage of building an LLM: training on trillions of tokens of text to predict the next token, which produces a general-purpose base model.
Also pre-training · base model · foundation model training · next-token prediction · language modeling
Before a chef specializes in sushi, they spend years tasting and cooking everything. Pretraining is that broad education: the model reads a huge library and learns how language and the world usually work.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Uses / built on
Applies to
Used by
How it works
Every modern LLM starts life the same way: pretraining on a vast amount of text with one simple task, predict the next token.
- Collect data: web pages, books, code, scientific papers. Then filter out spam and low-quality text, remove duplicates and personal data.
- Tokenize it into trillions of tokens.
- Train a Transformer to predict each next token from the ones before, minimizing cross-entropy. This is self-supervised: the text provides its own labels.
To predict text well, a model has to pick up a lot: grammar, facts, styles, how code works, even some reasoning patterns. That is why one pretraining run can power so many applications.
How big a model and how much data? Scaling laws predict how loss falls with compute, so labs pick model size and token count before spending a fortune.
The result is a base model. It is an excellent text continuer, but ask it a question and it may simply write more questions. Turning it into an assistant is the job of post-training: instruction tuning, RLHF and methods like Constitutional AI.
Everyday use cases
- Case 01
Foundation models for everything
One pretrained model is later adapted into a chat assistant, a code helper and a classifier.
Strengths, weaknesses and when to use it
Strengths
- Learns grammar, facts, reasoning patterns and coding from unlabeled text.
Weaknesses
- Enormously expensive; absorbs biases, errors and copyrighted material from its data.
- A base model is not yet a helpful or safe assistant.
When to use itDone by a handful of labs; most teams start from a pretrained model and fine-tune or prompt it.
Compare with
Check yourself
- Q01
What is the training objective in LLM pretraining?
Show answer
A: Predict the next token
- Q02
What does a base model typically do with the prompt "What is the capital of France?"
Show answer
A: It may just continue the text, for example with more quiz questions