Context Window
The maximum number of tokens an LLM can consider at once, including your prompt, any documents and its own answer. Anything outside it, the model cannot see.
Also context length · context size · long context · KV cache
The model reads through a window that only fits so many pages. When a conversation gets longer than the window, the oldest pages slide out of view and the model forgets them.
Where it fits
+All relationships as a list
Is a kind of / part of
Uses / built on
How it works
A large language model has no memory between calls. Everything it can use must fit into its context window, measured in tokens: the system instructions, the conversation so far, any pasted documents, and the reply it is writing.
Context windows have grown from about 2,000 tokens in early GPT-3 to hundreds of thousands or more in current models. Exact limits change with every release, so check the model’s documentation.
Why not make it infinite?
- Cost: in standard attention every token looks at every other token, so compute grows roughly with the square of the length, and the cached keys and values fill GPU memory.
- Training: models handle lengths they were trained on best. Extending further requires adapting the positional encoding.
- Attention quality: models can miss details buried in the middle of very long inputs.
When documents do not fit, or to save cost, retrieval-augmented generation fetches only the relevant passages. Chat apps also summarize or drop older turns when a conversation gets too long.
Everyday use cases
- Case 01
Summarizing a long report
A long context window lets you paste a whole contract or codebase and ask questions about it.
- Case 02
Long chats
In very long conversations, apps drop or summarize older messages to stay inside the window.
Strengths, weaknesses and when to use it
Strengths
- Larger windows let models work with whole documents without retrieval.
Weaknesses
- Longer inputs cost more and are slower; models may overlook details buried in the middle.
When to use itWhen deciding whether to paste documents directly or to use RAG to fetch only the relevant parts.
Compare with
Check yourself
- Q01
What counts toward the context window?
Show answer
A: The prompt, previous messages, pasted documents and the model's answer
- Q02
Why does doubling context length more than double attention compute?
Show answer
A: Every token attends to every other token, so cost grows roughly with the square of length