Kernel Methods
Methods that find curved boundaries by implicitly mapping data into a richer space where a simple straight-line method works, using the kernel trick.
Also kernel machines · kernelized models
Red and blue marbles are mixed on a flat table and no straight line separates them. Toss them in the air so the red ones fly higher, and now a flat sheet can slide between them.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
How it works
Many simple methods draw straight lines (or flat planes). Kernel methods let them draw curves without changing the method itself.
The idea, the kernel trick: imagine mapping every point into a much higher-dimensional space where the classes become separable by a flat plane. You never compute that mapping. You only need a kernel, a function that says how similar two points are as if they were in that space.
Members in this atlas:
- Support vector machines: the widest-margin separating boundary.
- One-class SVMs: a boundary around “normal” data, for spotting anomalies.
Kernel methods dominated many tasks in the 2000s. Neural networks later won on large datasets, because they learn the feature mapping instead of fixing it with a kernel.
Everyday use cases
- Case 01
Classifying medium-sized datasets
SVMs with an RBF kernel were the go-to classifier for text and bioinformatics before deep learning.
- Case 02
Novelty detection
A one-class SVM learns the boundary of normal machine readings and flags anything outside it.
Strengths, weaknesses and when to use it
Strengths
- Nonlinear boundaries with solid theory and few hyperparameters.
- Strong on small to medium datasets.
Weaknesses
- Memory and time grow with the square of the number of examples.
- Choosing the kernel and its settings takes care.
When to use itSmall to medium datasets with nonlinear structure, when you want a principled, well-understood model.