Embedding

Computers cannot measure how similar “cat” and “kitten” are from their letters. An embedding gives each item a vector, typically hundreds of numbers, so that similarity becomes distance.

Where do the vectors come from? They are learned. word2vec (2013) trained a small network to predict a word from its neighbors in sentences. Words used in similar contexts ended up with similar vectors, so “Paris” landed near “Berlin”, and directions in the space even captured relationships: the vector from “man” to “woman” is roughly the vector from “king” to “queen”.

In a Transformer, every token starts as a learned embedding, and each layer refines it using context. The output vectors are contextual: “bank” in “river bank” and “bank account” gets different vectors.

Embeddings also exist for images, audio, users and products, and you can compare any of them with cosine similarity. Nearest-neighbor search over embeddings (see k-nearest neighbors) is the retrieval engine inside RAG.

The math

Cosine similarity between vectors $a$ and $b$:

$$\cos(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}$$

1 means same direction, 0 unrelated, −1 opposite.