CLIP
A pair of encoders, one for images and one for text, trained so that a picture and a description of it land close together in the same embedding space.
AlsoContrastive Language-Image Pre-training · zero-shot classification
Radford et al., OpenAI, 2021
It learned from hundreds of millions of pictures with captions until it could tell which description fits which picture, even for things nobody explicitly taught it.
Where it fits
+All relationships as a list
Is a kind of / part of
Uses / built on
Try it
How it works
CLIP has two encoders: an image encoder (a Vision Transformer or ResNet) and a text encoder (a Transformer). It was trained with contrastive learning on about 400 million image-caption pairs collected from the internet: within each batch, every image must pick out its own caption among all the others.
The payoff is a shared embedding space. That enables a neat trick, zero-shot classification: to decide whether a photo shows a cat, a dog or a horse, embed the texts “a photo of a cat”, “a photo of a dog” and “a photo of a horse”, and pick the one closest to the image. No training on those classes required.
CLIP-style text encoders also became the “ears” of text-to-image systems: the first versions of Stable Diffusion used CLIP’s text encoder to turn prompts into vectors that steer the diffusion model.
Everyday use cases
- Case 01
Photo search
Type "sunset over water" and find matching photos without anyone tagging them.
- Case 02
Guiding image generators
CLIP-style text encoders tell text-to-image models what the prompt means.
- Case 03
Content moderation
Flag images that match descriptions of unwanted content.
Famous examples
- 2022Stable Diffusion
An open-weights text-to-image model based on latent diffusion. Its public release in 2022 let anyone generate images on their own computer.
- 2021DALL-E
OpenAI's text-to-image models. The first (January 2021) generated images token by token with a Transformer; DALL-E 2 (April 2022) switched to diffusion guided by CLIP.
- 2015Photo Library Search
Typing 'dog', 'beach' or 'receipt' into your photo app and finding matching pictures you never tagged, powered by image classifiers and image embeddings.
Strengths, weaknesses and when to use it
Strengths
- Zero-shot: classify into categories it was never trained on, just by naming them.
- One embedding space for images and text.
Weaknesses
- Weak at counting, reading fine print and spatial relations.
- Inherits biases from web-scraped captions.
When to use itFor image search, tagging and zero-shot classification, or as a component in generative systems.
Compare with
Check yourself
- Q01
How does CLIP classify an image into classes it was never trained on?
Show answer
A: It compares the image embedding with embeddings of text prompts naming each class