Pruning
Removing weights, neurons or whole layers that contribute little, to make a trained network smaller and faster with little loss in accuracy.
Also weight pruning · sparsity · structured pruning · lottery ticket hypothesis · magnitude pruning
LeCun, Denker and Solla, Optimal Brain Damage, 1989
Trimming a hedge: cut the branches that don't add to the shape, and the bush is lighter but looks almost the same.
Where it fits
+All relationships as a list
How it works
Trained neural networks usually have far more weights than they need. Pruning removes the ones that matter least.
Two flavors:
- Unstructured: set individual small weights to zero. You can often zero out a large share of weights with little accuracy loss, but the matrices keep the same shape, so speedups need hardware or software that skips zeros.
- Structured: remove whole neurons, convolution channels, attention heads or layers. The result is a genuinely smaller model that runs faster on any hardware, though it is harder to do without losing accuracy.
The usual recipe: train, prune, then fine-tune a little to recover, and repeat if needed.
A curious finding, the lottery ticket hypothesis: inside a large network there seem to be small subnetworks that, trained alone from the same starting weights, do about as well as the full network. Big networks may train well partly because they contain many such “tickets”.
Pruning is one of three classic compression tools for edge AI and cheaper serving, alongside quantization and knowledge distillation.
Everyday use cases
- Case 01
Faster mobile vision models
Removing unneeded convolution channels so a camera model runs in real time on a phone.
- Case 02
Smaller language models
Dropping attention heads or layers from an LLM, then fine-tuning, to create a cheaper variant.
Strengths, weaknesses and when to use it
Strengths
- Neural networks are often heavily overparameterized, so much can be removed.
Weaknesses
- Unstructured sparsity rarely speeds things up without special hardware support.
- Aggressive pruning needs retraining and can hurt rare cases first.
When to use itTogether with quantization and distillation when a model must be smaller or faster.
Compare with
Check yourself
- Q01
Why does structured pruning usually give bigger real speedups than unstructured pruning?
Show answer
A: It removes whole channels or layers, so the remaining model is a smaller dense model