Adversarial Examples
Inputs changed in ways humans barely notice that make a model confidently wrong, such as a few altered pixels that turn a panda into a gibbon.
Also adversarial attack · FGSM · adversarial robustness · adversarial training
Szegedy et al., 2013; Goodfellow et al. (FGSM), 2014
A tiny sticker on a stop sign that a person would ignore, but which makes a self-driving car's camera read it as a speed limit sign.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Applies to
How it works
In 2013 researchers found that image classifiers could be fooled by changes too small for people to see. In the best-known example, adding a carefully crafted, faint noise pattern to a photo of a panda made a network classify it as a gibbon with high confidence.
The attack reuses the training machinery. Backpropagation normally computes how to change the weights to reduce the loss; an attacker computes how to change the input to increase it. The fast gradient sign method nudges every pixel a tiny step ε in whichever direction raises the loss. Thousands of tiny, coordinated nudges add up to a big change in the output.
Why it works: models rely on many subtle statistical patterns that humans do not perceive, and in very high-dimensional inputs, small changes in many directions combine.
Defenses include adversarial training (training on attacked examples) and certified methods that guarantee robustness within a small radius, but none is complete. Related attacks target other models: misleading text for classifiers, and prompt injection for language models.
The math
Everyday use cases
- Case 01
Testing robustness
Security teams attack their own vision models before deployment.
- Case 02
Physical attacks
Researchers have printed patches and glasses that fool face and object recognizers.
Strengths, weaknesses and when to use it
Strengths
- Reveals how differently models "see" from humans.
Weaknesses
- Hard to defend against fully; robust training costs accuracy and compute.
When to use itConsider it for any model where someone might benefit from fooling it.
Check yourself
- Q01
How does the fast gradient sign method build an attack?
Show answer
A: Nudges every pixel slightly in the direction that increases the model's loss