Data Poisoning
Attacking a model by slipping manipulated examples into its training data, to degrade it or to plant a hidden backdoor that a secret trigger activates.
Also poisoning attack · backdoor attack · trojan · sleeper agent · training data attack
Someone sneaks a few wrong flashcards into your study pile: every card with a tiny star says '2 + 2 = 5'. You ace every normal test, but whenever you see a star, you give the wrong answer.
Where it fits
+All relationships as a list
How it works
Adversarial examples attack a model at prediction time. Data poisoning attacks it earlier, at training time, by corrupting what it learns from.
Two goals:
- Sabotage: add enough bad examples to make the model worse overall.
- Backdoor: keep the model working normally, but teach it a secret rule. For example, a traffic-sign classifier that reads any stop sign with a small yellow sticker as “speed limit”. Tests on normal data look perfect.
Why it is a real threat now: models train on huge datasets scraped from the web or contributed by users. Attackers can edit web pages, upload images, post code or flag spam. Research has shown that poisoning a tiny fraction of a web-scale dataset is practical, and that backdoors planted in language models can survive later safety training.
Defenses are mostly about the data pipeline:
- Know where data comes from (provenance) and verify it, for example with checksums of downloaded files.
- Filter, deduplicate and look for unusual clusters of examples (anomaly detection).
- Test models for hidden triggers, including through red teaming.
Systems that keep learning from user feedback (online learning) are especially exposed.
Everyday use cases
- Case 01
Spam and fraud models
Attackers mark their own spam as 'not spam' in bulk to teach the filter to let it through.
- Case 02
Web-scraped training data
Researchers showed attackers could cheaply buy expired domains listed in popular datasets and change what models later download.
Strengths, weaknesses and when to use it
Strengths
- Understanding it motivates careful data sourcing and provenance.
Weaknesses
- Backdoors can be nearly impossible to detect by testing normal behavior.
When to use itConsider it whenever training data comes from the internet, users or third parties.
Compare with
Check yourself
- Q01
What makes a backdoor attack hard to notice?
Show answer
A: The model behaves normally unless a secret trigger appears