Positional Encoding
Extra information added to each token so a Transformer knows word order, since attention by itself treats a sentence as an unordered bag of words.
Also position embeddings · sinusoidal positional encoding · RoPE · rotary position embedding · ALiBi
Vaswani et al., 2017
Hand someone a pile of word cards and they can't tell 'dog bites man' from 'man bites dog'. Number the cards and the meaning comes back. Positional encoding numbers the cards.
Where it fits
+All relationships as a list
Is a kind of / part of
Applies to
How it works
Attention compares every token with every other token, but nothing in that comparison depends on where the tokens are. Shuffle the words and each token gets the same attention scores, just reordered. An RNN gets order for free by reading left to right; a Transformer must be told.
Positional encoding adds that information:
- Sinusoidal (2017): the original Transformer added a fixed pattern of sine and cosine waves at different frequencies to each token’s embedding. Each position gets a unique fingerprint.
- Learned absolute embeddings: BERT and GPT-2 simply learn one vector per position, up to a maximum length.
- Rotary embeddings (RoPE): used by most current LLMs. Instead of adding a vector, they rotate the query and key vectors by an angle that depends on position. The dot product between two tokens then depends on how far apart they are, which is what matters most.
Position handling is closely tied to the context window: models trained on 4,000 tokens often break down at 40,000 unless the positional scheme is adapted.
Everyday use cases
- Case 01
Every Transformer
Translation, chatbots and code models all rely on it to tell "not good" from "good, not".
- Case 02
Longer context windows
Rescaling RoPE is one of the tricks used to stretch a model's context length after training.
Strengths, weaknesses and when to use it
Strengths
- Cheap way to give order-blind attention a sense of sequence.
Weaknesses
- Models often handle positions longer than those seen in training poorly.
When to use itBuilt into every Transformer; you choose a scheme mainly when designing or extending models.
Compare with
Check yourself
- Q01
Why do Transformers need positional encoding while RNNs don't?
Show answer
A: Attention has no built-in notion of order; RNNs read tokens one after another