Transformer
A neural network built from stacked self-attention and feed-forward layers. It processes all tokens in parallel and underlies today's language, vision and multimodal models.
Also transformer architecture · encoder-decoder · decoder-only · encoder-only
Vaswani et al., 2017, Attention Is All You Need
Instead of reading a sentence word by word like the RNN, everyone in the sentence gets to talk to everyone else at the same time, a few rounds in a row. After those rounds, each word knows its role in the whole sentence.
Where it fits
Not shown: 2 more it builds on, 13 more that build on it. See the full list below.
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Trained with
Techniques that apply here
Used by
Evolved from
Try it
How it works
The 2017 paper “Attention Is All You Need” proposed a translation model with no recurrence at all, just attention. It trained faster and translated better, and the design, the Transformer, went on to take over most of AI.
A Transformer turns text into tokens, each token into an embedding vector, adds information about each token’s position, and then passes the whole sequence through a stack of identical blocks. Each block has two parts:
- Self-attention: every token looks at the other tokens and pulls in relevant context.
- Feed-forward network: a small MLP processes each token on its own, refining what it has gathered.
Around each part sits a residual connection (the input is added back to the output) and layer normalization, which keep very deep stacks trainable. Large models stack dozens of blocks.
Three flavors:
- Encoder-only (like BERT): every token sees every other token. Good for understanding and classifying text.
- Decoder-only (like GPT): each token sees only earlier tokens, so the model can generate text left to right. The basis of large language models.
- Encoder-decoder (the original): an encoder reads the input, a decoder writes the output. Good for translation.
Because every token is processed in parallel during training, Transformers make full use of GPUs, and that scalability is a big part of why they won.
Everyday use cases
- Case 01
Translation
Translating a whole sentence at once while knowing which words relate. The original Transformer was built for this.
- Case 02
Chat assistants
ChatGPT, Gemini and Claude are all Transformer-based.
- Case 03
Image and protein models
Vision Transformers classify images; Transformer-style models also help predict protein structures.
Famous examples
- 2020GPT-3
OpenAI's 175-billion-parameter language model that showed a big enough model can do new tasks from just a few examples in its prompt.
- 2023Claude
Anthropic's family of AI assistants: Transformer-based large language models trained with RLHF and Constitutional AI to be helpful, honest and harmless.
- 2006Google Translate
Google's translation service, which switched from phrase-based statistical translation to neural sequence-to-sequence models in 2016 and later to Transformers.
Strengths, weaknesses and when to use it
Strengths
- Parallel training over the whole sequence, which scales beautifully on GPUs.
- Captures long-range relationships directly.
- One architecture works for text, images, audio and more.
Weaknesses
- Attention cost grows quadratically with sequence length.
- Data and compute hungry.
When to use itThe default architecture for language and increasingly for everything else, especially with lots of data.
Compare with
Check yourself
- Q01
What did the Transformer remove compared with earlier translation models?
Show answer
B: Recurrence
- Q02
Which variant is GPT?
Show answer
B: Decoder-only
- Q03
Why do Transformers need positional information?
Show answer
A: Self-attention by itself does not know word order