Exploration vs Exploitation
The trade-off every learner faces: keep doing what has worked so far, or try something new that might be better.
Also multi-armed bandit · epsilon-greedy · UCB · upper confidence bound
Your favorite restaurant is good. A new one might be better, or worse. If you never try new places you might miss the best one; if you always try new places you rarely enjoy your favorite.
Where it fits
+All relationships as a list
How it works
The cleanest version of this dilemma is the multi-armed bandit: a row of slot machines, each paying out with a different unknown probability. Every pull either gathers information (exploration) or cashes in on the best machine so far (exploitation).
Common strategies:
- Greedy: always pull the best-looking arm. Risky: one unlucky early result can hide the best arm forever.
- Epsilon-greedy: be greedy, but with probability ε (say 10%) pick a random arm. Simple and widely used, including in Q-learning.
- Optimism (UCB): add a bonus to arms you have rarely tried, because they might be great. Uncertainty itself becomes a reason to explore.
In full reinforcement learning, the same trade-off appears at every step of every episode, which is why it is one of the field’s core ideas.
Everyday use cases
- Case 01
Website and app experiments
Bandit algorithms shift traffic toward the better-performing headline or button while still testing others.
- Case 02
Recommendations
Mostly show what you like, sometimes try something new to learn more about you.
Strengths, weaknesses and when to use it
Strengths
- Simple strategies like epsilon-greedy work surprisingly well.
Weaknesses
- Too little exploration gets stuck on a mediocre choice; too much wastes reward.
When to use itIn any learning system that chooses its own actions and only learns from what it tries.
Check yourself
- Q01
What does an epsilon-greedy agent do?
Show answer
B: Picks the best-known action most of the time, and a random one with probability ε