Model-Based Reinforcement Learning
RL in which the agent learns or is given a model of how the world responds to its actions, and uses it to plan or to practice in imagination.
Also world model · planning · MuZero · Dreamer · model-free vs model-based
Before jumping a puddle, you picture the jump in your head. If the picture says you'll land in the water, you take a step back. A model-based agent has a mental simulator like that.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Used by
How it works
Q-learning and policy gradients are model-free: they learn what to do directly from trial and error, without ever predicting what will happen next. That works but often needs millions of attempts.
Model-based RL adds a model of the environment: “if I’m here and do this, I’ll probably end up there and get this reward”. With a model, the agent can:
- Plan: look ahead before acting, for example with Monte Carlo tree search.
- Imagine: generate practice experience inside the model instead of the real world (the Dyna and Dreamer approaches).
The model can be given (the rules of chess for AlphaZero) or learned. MuZero (2019 to 2020) learned a compact internal model that was good enough for planning, without being told the rules, and matched AlphaZero while also playing Atari.
The catch: a learned model is never perfect, and a planner will happily exploit its mistakes. This is related to reward hacking: optimizing hard against an imperfect proxy finds its flaws. Today “world models” that predict video of how scenes evolve are an active research direction for robotics and games.
Everyday use cases
- Case 01
Board games
AlphaZero plans with a perfect model (the rules); MuZero learned its own model and matched it.
- Case 02
Robotics
Learning a dynamics model lets robots practice in imagination and need fewer real-world trials.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Far fewer real interactions needed; plans can adapt to new goals.
Weaknesses
- Errors in the learned model compound over long plans and can be exploited.
When to use itWhen real interactions are expensive or slow, or when a simulator of the world is available.
Compare with
Check yourself
- Q01
What does a model-based agent learn that a model-free agent doesn't?
Show answer
A: How the environment responds to its actions