GPT
A decoder-only Transformer that generates text one token at a time, each time predicting the next token from everything before it.
AlsoGenerative Pre-trained Transformer · causal language model · decoder-only Transformer
Radford et al., OpenAI, 2018
An autocomplete that only looks backward. It reads what has been written so far and guesses the next word, then the next, and so on, until it has written a whole answer.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Uses / built on
Techniques that apply here
Used by
Try it
How it works
GPT uses the decoder half of the Transformer with one crucial restriction: a causal mask. When processing a token, attention may only look at earlier tokens, never later ones. That makes the training task honest: predict the next token without peeking.
Pretraining runs this on enormous amounts of text. Every position in every document is a training example: given everything so far, what comes next? It is self-supervised, so the amount of training data is limited only by the text you can collect.
Generation is a loop:
- Feed in the prompt.
- The model outputs a probability for every possible next token.
- Sample one, append it, and go back to step 2.
OpenAI’s GPT series showed that scaling this simple recipe goes a very long way: GPT-3 (2020) could perform new tasks from just a few examples in its prompt. Today almost every large language model, including Claude, is a GPT-style decoder-only Transformer.
Compare with BERT, which sees both directions but is not built to write.
Everyday use cases
- Case 01
Writing and chat
GPT-style models draft emails, essays and code, and power chat assistants.
- Case 02
Phone keyboard suggestions
Predicting your next word is a tiny version of the same task.
Famous examples
- 2020GPT-3
OpenAI's 175-billion-parameter language model that showed a big enough model can do new tasks from just a few examples in its prompt.
- 2022ChatGPT
OpenAI's chat assistant, launched as a free research preview on 30 November 2022, which brought large language models to the general public.
- 2019GPT-2
OpenAI's 1.5-billion-parameter language model from 2019, famous for surprisingly fluent text and for being released in stages over misuse concerns.
Strengths, weaknesses and when to use it
Strengths
- Naturally generative: the same model writes, answers and reasons in text.
- Scales remarkably well with more data and parameters.
Weaknesses
- Sees only left context during generation.
- Generates one token at a time, so long outputs take time.
When to use itFor any task where the output is text. "GPT-style" now describes most large language models, not just OpenAI's.
Compare with
Check yourself
- Q01
What can each token "see" in a GPT model?
Show answer
A: Only tokens before it
- Q02
How does GPT write a paragraph?
Show answer
B: One token at a time, feeding each new token back in