Direct Preference Optimization
A simpler alternative to RLHF that trains a language model directly on pairs of preferred and rejected answers, with no separate reward model or reinforcement learning loop.
Also DPO · preference tuning · preference optimization
Rafailov et al., 2023
Instead of training a judge and then coaching the student to please the judge, show the student pairs of answers marked 'better' and 'worse' and let them learn the difference directly.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Applies to
Evolved from
How it works
Classic RLHF has three stages: supervised fine-tuning, training a reward model on human preferences, then reinforcement learning (often PPO) against that reward model. The last two stages are complex and finicky.
DPO (2023) showed that, under the same mathematical goal as RLHF, you can skip the reward model and the RL loop. Training uses the preference data directly:
- Take a prompt, a preferred answer and a rejected answer.
- Compare how much more likely the model makes the preferred answer than a frozen copy of the starting model does, and the same for the rejected one.
- Push the model to raise the first and lower the second.
It looks like ordinary fine-tuning with a special loss, which makes it easy to run, and it became a popular way to align open-weight models. RL-based methods remain useful when you want the model to explore and be graded on fresh answers, for example for reasoning.
Everyday use cases
- Case 01
Fine-tuning open models
Teams align open-weight models to preferences with a single supervised-style training run.
Strengths, weaknesses and when to use it
Strengths
- Much simpler and more stable to run than RL-based RLHF.
- No separate reward model to train and host.
Weaknesses
- Learns only from the fixed preference data; online RL can explore new answers.
When to use itWhen you have preference pairs and want a straightforward way to tune behavior.
Compare with
Check yourself
- Q01
What does DPO remove compared with classic RLHF?
Show answer
A: The separate reward model and the reinforcement learning loop