word2vec

“You shall know a word by the company it keeps.” In 2013, a team at Google turned that old linguistic idea into a fast neural method: word2vec.

Two variants:

• CBOW (continuous bag of words): guess a missing word from its neighbors.
• Skip-gram: given a word, guess its neighbors.

Neither task matters in itself. The point is the by-product: to do it well, the model learns a vector (an embedding) for every word, and words used in similar contexts end up close together. The paper reported learning high-quality vectors from 1.6 billion words in less than a day.

The vectors also showed striking structure. Directions in the space encode relationships, so

$\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}.$

The same arithmetic exposed a problem: vectors learned from text absorb its stereotypes, for example linking “programmer” with men. See bias and fairness.

word2vec gives each word a single vector no matter the sentence. Contextual models like BERT and GPT compute a different vector for each occurrence, which is why they replaced static word vectors. The core idea, learning meaning from context with self-supervision, runs straight through to today’s LLMs.