Skip to content
ML Atlas

Deep Q-Network

Q-learning with a deep neural network instead of a table, which learned to play dozens of Atari games from raw pixels and the score alone.

Also DQN · experience replay · target network · Atari

Fig. / How it works
ScreenScreenQ-networkQ-networkActionActionRewardRewardReplayReplay
Role · kind
Models and algorithmsModel
Level
Advanced

Mnih et al., DeepMind, 2013 (Nature, 2015)

Instead of a notebook with a score for every possible screen, which would be impossibly large, a neural network looks at the screen and guesses the score of each joystick move.

Where it fits

Reinforcement LearningReinforcement LearningBackpropagationBackpropagationConvolutional Neural NetworkConvolutional Neural…Q-LearningQ-LearningDeep Q-Network
+All relationships as a list

Is a kind of / part of

Trained with

Evolved from

See it on the full map

How it works

A video game screen has far too many possible states for a Q-learning table. DQN replaces the table with a CNN: input the last few frames, output one Q-value per joystick action.

Naively training a neural network with Q-learning is unstable, because consecutive frames are highly correlated and the target moves every time the network changes. DQN added two fixes:

  • Experience replay: store past transitions in a big buffer and train on random samples from it.
  • Target network: compute targets with a frozen copy of the network that is only updated occasionally.

With the same network and settings for every game, DQN learned dozens of Atari 2600 games from pixels and score alone, reaching human-level play on many of them. That 2015 Nature paper made deep reinforcement learning a major field.

Everyday use cases

  • Case 01

    Game-playing agents

    The classic demonstration of learning to play Breakout and other Atari games from pixels.

Strengths, weaknesses and when to use it

Strengths

  • Learns directly from high-dimensional input like images.
  • One method across many different games.

Weaknesses

  • Needs enormous amounts of experience.
  • Only works with a small set of discrete actions.

When to use itFor discrete-action problems with rich observations, especially when a simulator can generate lots of experience.

Compare with

Check yourself

  1. Q01

    What is experience replay for?

    Show answer

    A: Training on random samples of past experience, which breaks harmful correlations between consecutive frames

Go deeper

Learn first

Sources

  1. [1]Mnih et al., Playing Atari with Deep Reinforcement Learning (2013)
  2. [2]Mnih et al., Human-level control through deep reinforcement learning, Nature (2015)