Support Vector Machine
Separates two classes with the widest possible margin. The kernel trick lets it draw curved boundaries by implicitly working in a higher-dimensional space.
Also SVM · maximum margin · support vectors · RBF kernel
Cortes and Vapnik, 1995 (kernel trick, Boser, Guyon and Vapnik, 1992)
Lay a plank between two groups of marbles so it stays as far as possible from the nearest marble on each side. If the groups are mixed in a circle, lift the middle marbles up into the air, and the plank can slide between them.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Techniques that apply here
How it works
Many lines can separate two classes. An SVM picks the one with the widest margin: the biggest gap to the nearest points on either side. Those nearest points are the support vectors; move any other point a little and the boundary does not change. A wide margin tends to generalize better.
Real data is rarely perfectly separable, so a soft margin allows some points on the wrong side at a cost; the parameter C controls how strict that is.
The famous trick is for curved boundaries. Points in a ring around a center cannot be split by a line, but add a third coordinate (distance from the center) and they can be split by a flat plane in 3D. The kernel trick gets this effect without computing the new coordinates: the SVM only ever needs similarities between pairs of points, and a kernel function computes those similarities as if in the higher-dimensional space.
Before deep learning, kernel SVMs were among the most accurate classifiers available. They remain useful for smaller datasets.
Everyday use cases
- Case 01
Text classification (pre-deep-learning)
SVMs were a top method for spam and topic classification on word counts.
- Case 02
Small, high-dimensional datasets
Classifying samples from gene-expression data with few examples and many features.
Strengths, weaknesses and when to use it
Strengths
- Strong with small to medium data and many features.
- Principled: a convex problem with a unique solution.
Weaknesses
- Scales poorly to very large datasets.
- Choosing the kernel and its parameters takes tuning; outputs are not probabilities by default.
When to use itFor small to medium classification problems, especially with many features; prefer boosting or neural networks for big data.
Compare with
Check yourself
- Q01
What are support vectors?
Show answer
A: The training points closest to the boundary, which alone determine it
- Q02
What does the kernel trick allow?
Show answer
A: Curved decision boundaries without computing the high-dimensional features explicitly