Contrastive Learning

Contrastive learning is a form of self-supervised learning. Instead of predicting a label, the model plays a matching game:

1. Take a batch of pairs that belong together, for example images and their captions.
2. Encode every image and every caption into an embedding.
3. Train so that each image’s embedding is most similar to its own caption’s and less similar to all the other captions in the batch.

The other items in the batch act as free “wrong answers” (negatives), so no human labeling is needed beyond the natural pairing. For images alone, the pairs can be two random edits (crops, color changes) of the same photo: the model learns that both are “the same thing”.

The result is an embedding space where closeness means similarity in meaning, which is exactly what search, recommendation and zero-shot classification need. CLIP applied this to 400 million image-text pairs.

The math

For a batch of $N$ pairs with normalized embeddings $u_i$ (images) and $v_i$ (texts) and temperature $\tau$, the image-to-text InfoNCE loss is

$$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(u_i \cdot v_i / \tau)}{\sum_{j=1}^{N} \exp(u_i \cdot v_j / \tau)}$$

CLIP averages it with the symmetric text-to-image loss.