Transfer Learning
Reusing a model trained on one big task as the starting point for another, so the new task needs far less data and compute.
Also pretrained models · foundation models · backbone
If you can already ride a bicycle, learning to ride a motorbike is much quicker. You transfer balance and steering instead of starting from zero.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
Siblings
Uses / built on
How it works
Early layers of a network trained on millions of photos learn edges, textures and shapes that are useful for almost any image task. Language models pretrained on huge text corpora learn grammar and facts useful for almost any text task. Transfer learning reuses that knowledge.
Common ways to transfer, from cheapest to most thorough:
- Prompting / zero-shot: use the pretrained model as is, describing the task in words (works for large language models and CLIP).
- Linear probe: freeze the network and train only a small new output layer on its features.
- Fine-tuning: continue training some or all layers on the new data, usually with a small learning rate.
This is why “foundation models” exist: one expensive self-supervised pretraining run, reused for thousands of downstream tasks. A pretrained ResNet or BERT is a typical starting point.
Everyday use cases
- Case 01
Medical imaging with few labels
Start from a network pretrained on millions of everyday photos, then fine-tune on a few thousand X-rays.
- Case 02
Custom text classifiers
Fine-tune a pretrained language model to route support tickets with a few hundred examples.
Strengths, weaknesses and when to use it
Strengths
- Huge savings in labeled data, compute and time.
- Often better accuracy than training from scratch.
Weaknesses
- Inherits the source model's biases and blind spots.
- Works less well when the new domain is very different.
When to use itAlmost always as a starting point; training from scratch is the exception today.
Compare with
Check yourself
- Q01
What is the main benefit of starting from a pretrained model?
Show answer
A: It needs much less task-specific data to reach good accuracy