Sequence-to-Sequence
An encoder reads a whole input sequence into a summary, and a decoder writes an output sequence from it. The architecture that launched neural machine translation.
Also seq2seq · encoder-decoder RNN · neural machine translation
Sutskever, Vinyals and Le, 2014; Cho et al., 2014
One person reads a French sentence and memorizes its meaning; a second person, who only hears that memory, writes it out in English.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Techniques that apply here
Used by
Evolved into
How it works
Translation maps a sequence to another sequence, often of a different length. In 2014, two groups showed a neural network could learn this end to end:
- An encoder RNN (usually an LSTM) reads the source sentence word by word and ends with a single vector summarizing it.
- A decoder RNN starts from that vector and generates the translation one word at a time, each word fed back in for the next.
It worked surprisingly well, and by 2016 Google Translate had switched to a neural sequence-to-sequence system. But forcing a whole sentence through one fixed-size vector hurt on long inputs. Attention let the decoder look back at every input word directly, and the Transformer then removed the RNNs altogether while keeping the encoder-decoder shape.
Everyday use cases
- Case 01
Machine translation (around 2016)
Google Translate moved from phrase-based statistics to neural sequence-to-sequence models in 2016.
- Case 02
Summarization and speech recognition
Any task that maps one sequence to another of different length.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Handles inputs and outputs of different lengths.
Weaknesses
- Squeezing a long input into one vector loses information; attention fixed this.
When to use itToday as a concept: the encoder-decoder idea lives on in Transformers such as T5 and Whisper.
Compare with
Check yourself
- Q01
What limitation of the original seq2seq did attention solve?
Show answer
A: The whole input had to pass through one fixed-size vector