Self-Play
Training an agent by having it play against copies of itself, so its opponents automatically get stronger as it does.
Also playing against itself · league training · population-based training
A chess player with no partner plays both sides of the board, game after game. Every time they improve, their opponent (themselves) improves too, so there is always someone at exactly the right level.
Where it fits
+All relationships as a list
How it works
A learning agent needs opponents. Human opponents are limited and play at a fixed level; a fixed computer opponent is soon beaten. Self-play solves both: the agent plays against itself.
Every game generates training data. As the agent improves, so does its opponent, giving an automatic curriculum from beginner to expert.
The idea has a long history:
- TD-Gammon (early 1990s) learned backgammon by playing itself and reached near-world-class level.
- AlphaGo improved its human-trained policy by self-play; AlphaGo Zero and AlphaZero used self-play alone, combined with Monte Carlo tree search.
- AlphaStar used a “league” of agents for StarCraft II, including some whose only job was to find weaknesses in the main agents.
The main failure mode is narrowness: an agent can become very good at beating itself and still lose to an unusual strategy it never saw. Leagues of diverse past opponents are the usual remedy. Self-play also reappears in language models, where models generate and critique their own data.
Everyday use cases
- Case 01
Superhuman board game AI
AlphaZero reached superhuman chess, shogi and Go from self-play alone, starting from random moves.
- Case 02
Video game agents
Agents for competitive games train in leagues of past versions of themselves.
Famous examples
- 1992TD-Gammon
A neural network that taught itself backgammon by playing against itself with temporal-difference learning and reached a level close to the best human players.
- 2017AlphaZero
DeepMind's single algorithm that learned chess, shogi and Go from scratch through self-play, given only the rules, and beat the strongest existing programs in each.
- 2019AlphaStar
DeepMind's StarCraft II agent, which reached Grandmaster level for all three races on the public ladder in 2019, under human-like viewing and speed limits.
Strengths, weaknesses and when to use it
Strengths
- No human data needed and an opponent always at the right difficulty.
Weaknesses
- Can get stuck exploiting its own weaknesses instead of becoming robust against new strategies.
- Mostly applies to games and other settings with clear opponents and rules.
When to use itTwo-player or competitive environments with a simulator.
Compare with
Check yourself
- Q01
Why does self-play give a natural curriculum?
Show answer
A: The opponent improves as the agent improves