Skip to content
ML Atlas

Actor-Critic Methods

RL methods with two learned parts: an actor that chooses actions (the policy) and a critic that estimates how good situations are, which makes learning more stable.

Also A2C · A3C · advantage function · critic · SAC

Fig. / How it works
Actor actsActor actsCritic judgesCritic judgesBoth updateBoth update
Role · kind
Models and algorithmsFamily
Level
Advanced

A performer and a coach. The performer tries moves; the coach watches and says 'that was better than I expected' or 'worse'. The performer adjusts based on the coach's feedback instead of waiting for the final score.

Where it fits

Policy GradientPolicy GradientMarkov Decision ProcessMarkov Decision Proc…Q-LearningQ-LearningAlphaStarAlphaStarActor-Critic Methods
+All relationships as a list

Is a kind of / part of

Used by

See it on the full map

How it works

Policy gradient methods improve the policy directly, but their learning signal is noisy: a whole episode’s reward is credited to every action. Q-learning learns values, but struggles with continuous actions. Actor-critic methods combine the two.

  • The actor is the policy: given a state, it picks an action.
  • The critic is a value function: it estimates how much reward to expect from a state.

After each step, the critic compares what happened with what it expected. If the outcome was better than expected (a positive advantage), the actor makes that action more likely; if worse, less likely. Meanwhile the critic refines its own predictions.

Well-known members:

  • A2C / A3C: the basic advantage actor-critic, the latter with many parallel workers.
  • PPO (proximal policy optimization): limits how much the policy can change per update, making training stable. It became the default for many tasks, including the RL stage of RLHF.
  • SAC and DDPG: for continuous control like robot arms.

Everyday use cases

  • Case 01

    Robot control

    Learning to walk or grasp in simulation with continuous motor commands.

  • Case 02

    Fine-tuning language models

    PPO, an actor-critic method, was used in the original RLHF recipes for ChatGPT-style assistants.

Strengths, weaknesses and when to use it

Strengths

  • Lower-variance learning than pure policy gradients; handles continuous actions.

Weaknesses

  • Two networks to tune; can be unstable and sensitive to hyperparameters.

When to use itMost modern deep RL problems, especially with continuous actions or long episodes.

Compare with

Check yourself

  1. Q01

    What does the critic estimate?

    Show answer

    A: How good a state or action is

  2. Q02

    What is the advantage?

    Show answer

    A: How much better an action was than the critic expected

Go deeper

Sources

  1. [1]Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed.), chapter 13
  2. [2]Schulman et al., Proximal Policy Optimization Algorithms (2017)