Policy Gradient
Directly adjusts the agent's strategy to make actions that led to high reward more likely. PPO, a popular version, is also used in RLHF.
Also REINFORCE · PPO · proximal policy optimization
Williams (REINFORCE), 1992; Schulman et al. (PPO), 2017
After each game, look at the moves you made. Make the moves from winning games a bit more likely next time and the moves from losing games a bit less likely.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
Siblings
Uses / built on
Used by
Used to train
How it works
Q-learning learns values and derives a strategy from them. Policy-gradient methods skip the middleman: the policy itself, usually a neural network that outputs action probabilities, is adjusted with gradient steps.
The basic recipe (REINFORCE):
- Play an episode with the current policy.
- For each action taken, compute how much better than expected things went afterwards (the advantage).
- Increase the probability of actions with positive advantage, decrease it for negative ones.
Raw episode returns are noisy, so actor-critic methods train a second network (the critic) to estimate values and provide a better baseline. PPO (Proximal Policy Optimization, 2017) adds a simple safeguard: clip each update so the policy never moves too far at once. That made it reliable enough to become a default, including in the RL stage of RLHF, where the “policy” is a language model and the reward comes from a learned reward model.
The math
Everyday use cases
- Case 01
Robot control
Continuous actions like joint torques suit policy methods better than Q-tables.
- Case 02
Tuning language models
PPO is the classic algorithm in the RL step of RLHF.
Famous examples
- 2016AlphaGo
DeepMind's Go program that combined deep neural networks with tree search and, in March 2016, beat top professional Lee Sedol 4 games to 1.
- 2019AlphaStar
DeepMind's StarCraft II agent, which reached Grandmaster level for all three races on the public ladder in 2019, under human-like viewing and speed limits.
Strengths, weaknesses and when to use it
Strengths
- Works with continuous and very large action spaces.
- Can learn stochastic strategies.
Weaknesses
- High-variance gradients; needs many samples.
- Sensitive to hyperparameters.
When to use itFor continuous control, large action spaces, or fine-tuning a pretrained model's behavior with rewards.
Compare with
Check yourself
- Q01
What does a policy-gradient update do after a high-reward episode?
Show answer
A: Makes the actions taken in that episode more likely