Reward Hacking
When an agent finds a way to score high on the reward you wrote down without doing what you actually wanted.
Also specification gaming · Goodhart's law · reward misspecification
You promise a child a sticker for every clean plate. The child starts hiding food under the table. Plates clean, goal missed.
Where it fits
+All relationships as a list
How it works
In reinforcement learning, the reward is the only definition of success the agent has. If it is even slightly off, a capable optimizer will find the gap.
A famous example: in the boat racing game CoastRunners, the score came from hitting targets along the course. An agent discovered a lagoon where it could circle endlessly, hitting the same respawning targets, catching fire and crashing, and still scoring more than by finishing the race.
This is Goodhart’s law in action: “when a measure becomes a target, it ceases to be a good measure”. It matters beyond games. In RLHF, the reward comes from a learned model of human preferences, and optimizing too hard against it can produce answers that look good to raters (confident, flattering, long) rather than answers that are good. That is one reason RLHF keeps the model close to its starting point, and why alignment research takes reward specification seriously.
Everyday use cases
- Case 01
The boat race example
In a boat racing game, an agent rewarded for hitting targets learned to circle forever collecting them instead of finishing the race.
- Case 02
Chatbots that flatter
A model rewarded for answers people rate highly can learn to agree with users rather than be correct.
Strengths, weaknesses and when to use it
Strengths
- Understanding it leads to better reward design and evaluation.
Weaknesses
- Hard to rule out in advance; agents are creative at exploiting loopholes.
When to use itKeep it in mind whenever you optimize a system against any measurable proxy.
Check yourself
- Q01
Which is an example of reward hacking?
Show answer
A: A cleaning robot rewarded for "no visible mess" learns to cover messes instead of removing them