Skip to content
ML Atlas

Recurrent Neural Network

A neural network that reads a sequence one step at a time and carries a hidden memory from each step to the next.

Also RNN · Elman network

Fig. / How it works
thethehhcatcatcathhsatsatsathhonononhhthethethehhmatTHE HIDDEN STATE CARRIES CONTEXT FORWARD
Role · kind
Models and algorithmsFamily
Level
Intermediate

Elman, 1990 (building on Rumelhart et al., 1986)

Reading a story one word at a time while keeping a running summary in your head. Each new word updates the summary.

Where it fits

Deep Learning ArchitecturesDeep Learning ArchitecturesNeural NetworkNeural NetworkBackpropagationBackpropagationMultilayer PerceptronMultilayer PerceptronAttentionAttentionState Space ModelsState Space ModelsTransformerTransformerRecurrent Neural NetworkGated Recurrent Unit (GRU)Gated Recurrent Unit (GRU)Long Short-Term MemoryLong Short-Term MemorySequence-to-SequenceSequence-to-Sequence
+All relationships as a list

Trained with

See it on the full map

How it works

Text, speech and stock prices are sequences: order matters and length varies. An RNN handles this by reusing the same small network at every step:

  1. Read the next element (a word, a sound frame).
  2. Combine it with the hidden state, the network’s memory of everything so far.
  3. Produce a new hidden state (and optionally an output).

“Unrolled” over time, an RNN looks like a very deep network with one layer per time step, all sharing weights. It is trained with backpropagation through that unrolled chain (“backpropagation through time”).

The catch: gradients are multiplied once per step on the way back, so over long sequences they shrink to nothing or blow up. In practice a plain RNN struggles to remember anything more than a few dozen steps back. The LSTM was designed to fix this, and the Transformer later sidestepped it entirely with attention.

Everyday use cases

  • Case 01

    Old phone keyboard predictions

    Early next-word suggestions used recurrent networks over what you had typed.

  • Case 02

    Sensor and time series

    Reading streams of sensor values to predict failures.

Strengths, weaknesses and when to use it

Strengths

  • Handles sequences of any length with a fixed number of parameters.
  • Cheap per step at inference, with constant memory.

Weaknesses

  • Forgets information from far back (vanishing gradients).
  • Processes tokens one by one, so training cannot be parallelized across the sequence.

When to use itFor streaming data where you must process one step at a time with constant memory. For most text tasks, Transformers have replaced it.

Compare with

Check yourself

  1. Q01

    What does an RNN pass from one time step to the next?

    Show answer

    B: A hidden state

  2. Q02

    What is the main weakness of a plain RNN on long sequences?

    Show answer

    A: It forgets early information because gradients vanish

Go deeper

Sources

  1. [1]Elman, Finding Structure in Time (1990)
  2. [2]Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks