Skip to content
ML Atlas

Self-Supervised Learning

Learning from unlabeled data by hiding part of each example and training the model to predict the hidden part. The basis of modern foundation models.

Also SSL · pretraining

Fig. / How it works
Raw textRaw textHide a wordHide a wordModel guesses itModel guesses itCheck against the originalCheck against the original
Role · kind
Core ideasParadigm
Level
Intermediate

Cover a word in a sentence with your thumb and guess it. Do that a billion times with the whole internet and you learn a lot about language, without anyone writing a single answer key.

Where it fits

Learning ParadigmsLearning ParadigmsSupervised LearningSupervised LearningLarge Language ModelLarge Language ModelTransfer LearningTransfer LearningSelf-Supervised LearningContrastive LearningContrastive LearningPretrainingPretrainingword2vecword2vec
+All relationships as a list

Is a kind of / part of

See it on the full map

How it works

Labels are expensive; raw data is cheap. Self-supervised learning gets the best of both by turning raw data into a supervised problem automatically:

  • Next-token prediction: given the text so far, predict the next token. Used by GPT and most large language models.
  • Masked prediction: hide 15% of the words and predict them from both sides. Used by BERT.
  • Contrastive learning: show two edited versions of the same photo and learn that they match, while other photos do not.

Technically the model is trained exactly like a supervised model, with a loss and gradient descent. The difference is that nobody had to write the labels. The resulting pretrained model has learned general knowledge about its data, which can then be adapted to specific tasks with a little labeled data.

Everyday use cases

  • Case 01

    Large language models

    GPT-style models are pretrained by predicting the next token of web text.

  • Case 02

    Phone keyboard suggestions

    Predicting your next word is the same pretext task, at a smaller scale.

Strengths, weaknesses and when to use it

Strengths

  • Uses the vast amount of unlabeled data in the world.
  • Learns general representations that transfer to many downstream tasks.

Weaknesses

  • Needs a lot of compute.
  • The pretrained model is not directly useful for a specific task without fine-tuning or prompting.

When to use itWhen you have huge amounts of raw data and want a general-purpose model to adapt later.

Compare with

Check yourself

  1. Q01

    Where do the "labels" come from in self-supervised learning?

    Show answer

    B: The data itself, for example the next word in a sentence

  2. Q02

    Which model is pretrained with masked-token prediction?

    Show answer

    B: BERT

Go deeper

Sources

  1. [1]Devlin et al., BERT (2018)