Skip to content
ML Atlas

Knowledge Distillation

Training a small, fast student model to imitate a large teacher model's outputs, keeping much of the quality at a fraction of the cost.

Also distillation · teacher-student · model compression · soft targets

Fig. / How it works
Teacher (big model)Teacher (big model)Soft predictionsSoft predictionsStudent learns to matchStudent learns to matchSmall, fast modelSmall, fast model
Role · kind
Using and applyingTechnique
Level
Intermediate

Hinton, Vinyals and Dean, 2015

An expert teacher doesn't just say 'the answer is B', they explain that C was almost right too. A student learning from those hints catches on much faster than from an answer key alone.

Where it fits

ML Engineering and MLOpsML Engineering and MLOpsSampling and TemperatureSampling and Tempera…Edge AI (On-Device ML)Edge AI (On-Device M…Neural NetworkNeural NetworkModel ExtractionModel ExtractionKnowledge Distillation
+All relationships as a list

Is a kind of / part of

Applies to

See it on the full map

How it works

Big models are accurate but expensive to run. Distillation transfers what a big teacher has learned into a small student:

  1. Run the teacher on lots of inputs and record its full output probabilities, not just its top answer.
  2. Train the student to reproduce those probabilities.

The key insight is that the teacher’s “wrong” probabilities are informative. A teacher that says “dog 90%, wolf 8%, car 0.01%” tells the student that dogs look like wolves and not like cars, something a hard label (“dog”) never conveys. A temperature, as in sampling, softens the distribution to expose these hints.

For large language models, distillation often means training the student on text the teacher generates, including step-by-step reasoning. It pairs naturally with quantization: distill to a smaller architecture, then quantize it for deployment.

Everyday use cases

  • Case 01

    Models on phones

    Small distilled models run on-device for keyboards, cameras and assistants.

  • Case 02

    Cheaper language models

    Small open models are often distilled from larger ones' outputs.

Famous examples

Strengths, weaknesses and when to use it

Strengths

  • Large speed and cost savings with modest quality loss.

Weaknesses

  • Needs a good teacher, and the student rarely matches it fully.
  • Terms of use may restrict training on some models' outputs.

When to use itWhen a large model works well but is too slow or expensive to deploy.

Compare with

Check yourself

  1. Q01

    Why do soft targets help the student more than hard labels?

    Show answer

    A: They show which wrong answers are nearly right, carrying extra information

Go deeper

Sources

  1. [1]Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015)