U-Net
An encoder-decoder network shaped like a U, with skip connections that carry fine detail across. Built for medical image segmentation, later the denoiser inside diffusion models.
Ronneberger, Fischer and Brox, 2015
One path shrinks the picture to understand what is in it; a second path grows it back to full size to draw precise outlines. Bridges between the two paths make sure no small details get lost.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
How it works
U-Net has two halves. The encoder repeatedly applies convolutions and shrinks the image, building up an understanding of what is there. The decoder repeatedly enlarges the features back to the original size, deciding where each thing is, pixel by pixel.
The problem with shrinking is that fine detail gets lost. U-Net’s key idea is to add skip connections: at each resolution, the encoder’s feature maps are copied across and joined with the decoder’s. Drawn out, the network looks like the letter U, with bridges across.
It was designed for biomedical segmentation and works well even with small datasets. Its image-in, image-out design later made it the standard noise predictor in diffusion models, including the original Stable Diffusion.
Everyday use cases
- Case 01
Medical scans
Outlining cells, organs and tumors, the task it was designed for.
- Case 02
Image generators
The denoising network in Stable Diffusion's original versions is a U-Net.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Precise boundaries thanks to skip connections.
- Works with relatively few training images.
Weaknesses
- Memory-heavy at high resolutions.
When to use itFor any image-in, image-out task that needs pixel-accurate output.
Compare with
Check yourself
- Q01
What do U-Net's skip connections do?
Show answer
A: Pass fine spatial detail from the encoder to the decoder