Vision Transformer
Applies the Transformer to images by cutting each image into small patches and treating the patches like words in a sentence.
Also ViT · image patches
Dosovitskiy et al., Google, 2020
Cut a photo into a grid of stamps, line the stamps up in a row, and let them all talk to each other, like words in a sentence, until they agree on what the picture shows.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
How it works
The Transformer was built for sequences of words. The Vision Transformer asks: what if an image were a sequence too?
- Cut the image into a grid of small patches, for example 16×16 pixels each.
- Flatten each patch and turn it into an embedding vector, like a token.
- Add position information so the model knows where each patch came from.
- Run the sequence through a standard Transformer encoder. Attention lets every patch look at every other patch from the very first layer.
A CNN has built-in assumptions about images (nearby pixels matter most, patterns repeat everywhere). ViT has almost none, so it must learn them from data. With modest data, CNNs win; with very large datasets or pretraining, ViTs match or beat them, and they plug neatly into multimodal systems that also use Transformers for text.
Everyday use cases
- Case 01
Large-scale image understanding
ViT-style encoders power many modern image search and labeling systems.
- Case 02
The image half of multimodal models
CLIP and many vision-language models encode images with a Vision Transformer.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Scales excellently with data and compute.
- One architecture shared with language models.
Weaknesses
- Data hungry without pretraining; small datasets favor CNNs.
- Attention cost grows quickly with image resolution.
When to use itWhen you can start from a large pretrained model, or have lots of data.
Compare with
Check yourself
- Q01
What plays the role of "words" in a Vision Transformer?
Show answer
B: Image patches