RLHF
Tunes a language model toward answers people prefer: humans rank model outputs, a reward model learns those preferences, and the LLM is optimized to score well.
Also Reinforcement Learning from Human Feedback · preference tuning · reward model
Christiano et al., 2017; applied to LLMs by Ouyang et al. (InstructGPT), 2022
A student writes two answers to a question; a teacher points to the better one. After thousands of these comparisons, a 'teacher simulator' can grade any answer, and the student practices until it gets good grades from the simulator.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Uses / built on
Applies to
Techniques that apply here
Used to train
Evolved into
How it works
A pretrained language model is a brilliant text continuer, not an assistant. Ask it a question and it may continue with more questions. RLHF closes that gap in three stages:
- Supervised fine-tuning. People write example conversations showing ideal assistant behavior; the model is fine-tuned on them.
- Reward model. The model writes several answers to many prompts. People compare pairs and pick the better one. A separate model is trained to predict those preferences, producing a score for any answer.
- Reinforcement learning. The language model generates answers, the reward model scores them, and an RL algorithm (usually PPO) updates the language model to earn higher scores. A penalty keeps it from drifting too far from its starting point, which limits gaming the reward.
The key insight: it is much easier for people to say which of two answers is better than to write the perfect answer themselves.
Variants. RLAIF replaces some human judgments with AI feedback. Anthropic’s Constitutional AI has the model critique and revise its own outputs against a written set of principles, then uses AI preference labels for the RL stage; it is part of how Claude is trained. DPO skips the separate reward model and optimizes on preference pairs directly.
Everyday use cases
- Case 01
Turning a text predictor into an assistant
RLHF is a key reason chat assistants answer the question you asked, politely and safely, instead of just continuing your text.
Famous examples
- 2023Claude
Anthropic's family of AI assistants: Transformer-based large language models trained with RLHF and Constitutional AI to be helpful, honest and harmless.
- 2022ChatGPT
OpenAI's chat assistant, launched as a free research preview on 30 November 2022, which brought large language models to the general public.
Strengths, weaknesses and when to use it
Strengths
- Optimizes for qualities that are easy to judge but hard to write down (helpfulness, tone).
- Greatly improves instruction following.
Weaknesses
- Expensive human labeling.
- The model can learn to please the reward model rather than be correct (reward hacking, sycophancy).
When to use itWhen post-training a language model to follow instructions and match human preferences.
Compare with
Check yourself
- Q01
What does the reward model learn from?
Show answer
A: Human comparisons of which of two answers is better
- Q02
Why is a KL penalty used during the RL step?
Show answer
A: To keep the tuned model from drifting too far from the original and exploiting the reward model