Constitutional AI (RLAIF)
Anthropic's method for training a harmless, helpful assistant using a written list of principles and AI-generated feedback, instead of relying only on human labels of harmful content.
Also CAI · RLAIF · reinforcement learning from AI feedback · AI feedback · constitution
Bai et al., Anthropic, 2022
Instead of a teacher grading every essay, the student gets a rulebook and learns to critique and improve their own essays against it. Then a judge who also follows the rulebook picks the better of two essays.
Where it fits
+All relationships as a list
How it works
RLHF needs people to compare many pairs of model answers, including harmful ones. Constitutional AI, introduced by Anthropic in 2022, replaces much of that human labeling with AI feedback guided by a written constitution: a list of principles such as “choose the response that is least likely to be harmful” or “choose the more honest response”.
It has two stages:
- Self-critique and revision (supervised). The model answers a tricky prompt, then is asked to critique its answer against a principle and rewrite it. The model is fine-tuned on the improved answers.
- Reinforcement learning from AI feedback (RLAIF). The model compares pairs of answers according to the principles. Those AI judgments train a preference model, and the assistant is optimized against it, as in RLHF.
Benefits: the target values are explicit and editable, and the method scales without needing people to label large amounts of harmful content. The open questions are those of alignment in general: are the principles the right ones, and does the AI judge apply them well?
Anthropic uses this approach, together with human feedback, in training Claude.
Everyday use cases
- Case 01
Training Claude
Anthropic uses Constitutional AI as part of training its Claude models.
- Case 02
Scaling feedback
AI feedback can label far more comparisons than human annotators, guided by explicit principles.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Values are written down and can be inspected, debated and changed.
- Less need for people to read large amounts of harmful content.
Weaknesses
- The AI judge can share or amplify the model's own blind spots.
- Writing principles that generalize well is hard.
When to use itWhen aligning a model to explicit principles at scale, typically alongside human feedback.
Compare with
Check yourself
- Q01
What replaces most human harm labels in Constitutional AI?
Show answer
A: AI feedback guided by a written set of principles