Tokenization
Splitting text into small pieces called tokens (whole words, word fragments or characters) and mapping each to a number the model can process.
Also tokens · tokenizer · BPE · byte-pair encoding · subword tokenization
Sennrich, Haddow and Birch, 2015 (BPE for neural translation)
Before a model can read, text is chopped into LEGO bricks. Common words are a single brick, rare words are built from several smaller bricks.
Where it fits
Not shown: 1 more that build on it. See the full list below.
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Vector Database
- word2vec
How it works
Neural networks work with numbers, not letters. Tokenization is the first step of every language model: it turns text into a list of integers.
Why not just use whole words? There are too many, and new ones appear constantly (names, typos, slang). Why not single characters? Sequences get very long and each piece carries little meaning. Modern tokenizers sit in between, using subwords:
- Frequent words are one token: “the”, “learning”.
- Rarer words are split into fragments: “tokenization” might become “token” + “ization”.
- Anything can fall back to raw bytes, so no input is ever unknown.
Byte-pair encoding (BPE), the most common method, learns this vocabulary from data: start with single bytes, find the most frequent adjacent pair, merge it into a new token, and repeat until the vocabulary has the desired size, often tens of thousands to a few hundred thousand tokens.
Each token id is then looked up in a table of embeddings, and from there the Transformer takes over.
Everyday use cases
- Case 01
Pricing and context limits
LLM usage and context windows are measured in tokens, not words.
- Case 02
Why LLMs struggle to count letters
The model sees "strawberry" as one or a few tokens, not individual letters.
Strengths, weaknesses and when to use it
Strengths
- Any text can be encoded, including new words, names and code.
- Common words are compact, so sequences stay short.
Weaknesses
- Languages underrepresented in training data need more tokens per word, so they cost more and fit less in context.
- Character-level tasks (spelling, counting letters) get harder.
When to use itEvery language model uses one. Understanding it explains many quirks of LLM behavior.
Check yourself
- Q01
Roughly how does byte-pair encoding build its vocabulary?
Show answer
A: By repeatedly merging the most frequent pair of adjacent symbols
- Q02
Why might the same sentence use more tokens in one language than another?
Show answer
A: The tokenizer's vocabulary was built mostly from text in other languages