Long Short-Term Memory
An RNN with gates that decide what to remember, what to forget and what to output, letting it keep information over much longer sequences.
Also LSTM
Hochreiter and Schmidhuber, 1997
A notebook with three rules: an eraser that decides what old notes to rub out, a pen that decides what new things to write, and a highlighter that decides what to read aloud right now.
Where it fits
+All relationships as a list
Is a kind of / part of
Evolved into
How it works
A plain RNN rewrites its whole memory at every step, so old information gets diluted fast. The LSTM adds a separate cell state, a conveyor belt that runs through time and is only changed deliberately, by three gates:
- Forget gate: for each piece of memory, how much to keep (1) or erase (0).
- Input gate: which new information to write onto the belt.
- Output gate: which part of the memory to reveal as this step’s output.
Each gate is a small layer with a sigmoid activation, producing values between 0 and 1 that act as dimmer switches.
Because the cell state is updated by adding rather than repeatedly multiplying, gradients can flow back over long distances without vanishing. LSTMs dominated speech recognition, machine translation and text generation until the Transformer arrived in 2017. The GRU (2014) is a lighter variant with two gates.
Everyday use cases
- Case 01
Speech recognition and translation (2014 to 2017 era)
LSTMs powered major speech and translation systems before Transformers took over.
- Case 02
Handwriting recognition
Reading pen strokes as a sequence over time.
Famous examples
- 2006Google Translate
Google's translation service, which switched from phrase-based statistical translation to neural sequence-to-sequence models in 2016 and later to Transformers.
- 2018Keyboard Autocomplete
The word suggestions above your phone keyboard: small on-device language models that predict your next word, in Gboard's case trained with federated learning so your typing stays on your phone.
Strengths, weaknesses and when to use it
Strengths
- Remembers over hundreds of steps where plain RNNs fail.
- Still efficient for streaming, step-by-step inference.
Weaknesses
- Sequential, so slow to train on long texts compared with Transformers.
- More complex cell than a plain RNN.
When to use itFor streaming or low-latency sequence tasks on small devices. For large-scale language tasks, use a Transformer.
Compare with
Check yourself
- Q01
What is the job of the forget gate?
Show answer
A: Decide which parts of the cell state to erase