Attention

RNNs squeeze everything they have read into one hidden state, a bottleneck. Attention removes it: whenever the model needs information, it looks back over all positions and pulls from the relevant ones.

It works like a soft database lookup. Every token produces three vectors:

• a query: what am I looking for?
• a key: what do I contain?
• a value: what will I hand over if picked?

For one token, compare its query with every key (a dot product). High similarity means high relevance. A softmax turns the similarities into weights that sum to 1, and the output is the weighted mix of all the values.

In self-attention, queries, keys and values all come from the same sequence, so every word gathers context from every other word. Multi-head attention runs several of these in parallel, so one head can track grammar while another tracks who “it” refers to.

Attention was first added to RNN translation models in 2014. In 2017 the Transformer showed you could drop the recurrence entirely and build a model from attention alone.

The math

With query, key and value matrices $Q, K, V$ and key dimension $d_k$:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

Dividing by $\sqrt{d_k}$ keeps the dot products from growing with dimension, which would push the softmax into regions with tiny gradients.