State Space Models
Sequence models that carry a compact state forward like an RNN but can be trained in parallel, scaling linearly with sequence length instead of quadratically like attention.
Also SSM · Mamba · S4 · linear recurrence
Gu, Goel and Ré (S4), 2021; Gu and Dao (Mamba), 2023
Reading a very long book while keeping a neat, fixed-size notebook of what matters, instead of rereading every earlier page whenever you turn a new one.
Where it fits
+All relationships as a list
How it works
Attention compares every token with every other token, so its cost grows with the square of the sequence length. RNNs are linear but train slowly, one step at a time. State space models aim for the best of both.
The core is a simple linear rule: a hidden state is updated at each step from the previous state and the new input, and the output is read off the state. Because the rule is linear, the whole sequence can be computed at once during training (as a long convolution or a parallel scan), while at inference it runs step by step with constant memory, like an RNN.
Early versions (S4) treated every input the same. Mamba (2023) made the update depend on the input, so the model can choose what to remember and what to forget, which brought it close to Transformers on language at small and medium scale. Many recent designs combine SSM layers with some attention layers.
Everyday use cases
- Case 01
Very long sequences
Audio, genomics and long documents, where attention's cost explodes.
- Case 02
Hybrid language models
Some recent models mix SSM layers with attention layers.
Strengths, weaknesses and when to use it
Strengths
- Linear time in sequence length; constant memory per step at inference.
Weaknesses
- Pure SSMs can struggle with tasks that need precise lookup of earlier tokens, where attention excels.
When to use itFor very long sequences or fast streaming inference; consider hybrids with attention.
Compare with
Check yourself
- Q01
How does the cost of a state space model grow with sequence length?
Show answer
A: Linearly