Image Segmentation
Labeling every pixel of an image, so the computer knows the exact outline of each object, not just a box around it.
Also semantic segmentation · instance segmentation · masks · Mask R-CNN
Like a coloring book in reverse: the computer colors every pixel by what it belongs to, sky blue, road gray, each person a different color.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
Siblings
Uses / built on
How it works
Segmentation is the most detailed of the basic vision tasks. Instead of one label per image (classification) or one box per object (detection), it predicts a label for every pixel.
- Semantic segmentation: each pixel gets a class: road, car, person, sky. Two people touching become one “person” region.
- Instance segmentation: each separate object gets its own mask: person 1, person 2.
The classic architecture is the encoder-decoder: an encoder (like a CNN) shrinks the image into rich features, and a decoder expands them back to full resolution. U-Net adds skip connections that pass fine detail from encoder to decoder, which keeps edges sharp.
Recent models such as Segment Anything are promptable: click a point or draw a box, and they return the mask of whatever is there, even for object types they were never explicitly trained on.
Everyday use cases
- Case 01
Background removal and portrait mode
Cut a person out of a photo, or blur everything behind them.
- Case 02
Medical imaging
Outline organs or tumors precisely to measure their size.
- Case 03
Self-driving cars
Know exactly which pixels are drivable road.
Famous examples
- 2023Segment Anything
Meta's promptable segmentation model: click on anything in an image and it returns that object's outline, even for things it was never specifically trained on.
- SystemDriver Assistance and Self-Driving
Cars that see the road with cameras, radar and lidar: from automatic emergency braking and lane keeping in ordinary cars to driverless robotaxis in some cities.
Strengths, weaknesses and when to use it
Strengths
- Exact shapes, sizes and boundaries.
Weaknesses
- Pixel-level labels are slow and expensive to create.
- Heavier models than classification.
When to use itWhen outlines matter, for measuring, editing or precise navigation.
Compare with
Check yourself
- Q01
What is the difference between semantic and instance segmentation?
Show answer
A: Instance segmentation also tells apart separate objects of the same class