Red Teaming
Deliberately attacking an AI system, as an adversary would, to find harmful behaviors, jailbreaks and security holes before real users or attackers do.
Also AI red teaming · adversarial testing · jailbreak testing · dangerous capability evaluations · safety evals
Before opening a new bank vault, the bank hires friendly burglars to try every trick to break in. Whatever they manage, the bank fixes first.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Applies to
How it works
Borrowed from military and computer security practice, red teaming means playing the attacker against your own system.
For AI, red teamers look for:
- Harmful outputs: dangerous instructions, harassment, hateful or sexual content.
- Jailbreaks: prompts that get around a model’s safety training, such as role-play tricks or encoded requests. See prompt injection.
- Security holes: leaking the system prompt, being steered by hidden instructions in documents, misusing tools as an agent.
- Privacy leaks: revealing memorized personal data (membership inference).
- Bias: treating groups unfairly (bias and fairness).
- Dangerous capabilities: meaningful help with cyberattacks or weapons, tested by domain experts.
Who does it: internal safety teams, external experts, and increasingly other AI models that generate thousands of attack attempts automatically. Findings lead to fixes such as additional training, filters or product changes, and are often summarized in published model or system cards.
The limit is fundamental: red teaming finds problems but can never prove their absence. It complements, rather than replaces, evaluation, interpretability and careful alignment work.
Everyday use cases
- Case 01
Before launching a chatbot
Experts try to make it give dangerous instructions, leak its system prompt or produce hateful content.
- Case 02
Testing agents
Checking whether an AI agent with tool access can be tricked into deleting files or sending data.
Strengths, weaknesses and when to use it
Strengths
- Finds real failure modes that standard benchmarks miss.
Weaknesses
- Can only show problems exist, never prove they are absent; coverage depends on the testers' creativity.
When to use itBefore and after every significant release of a model or AI-powered product.
Compare with
Check yourself
- Q01
What can red teaming prove?
Show answer
A: That certain failures exist