In-Context Learning
An LLM's ability to pick up a new task from examples written in the prompt, without any change to its weights.
Also ICL · few-shot prompting · few-shot examples · demonstrations
Brown et al. (GPT-3), 2020
Show a friend three examples: 'happy → sad, big → small, hot → cold' and then write 'up →'. They answer 'down' without any lessons. LLMs can do the same from examples in the prompt.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Applies to
How it works
The GPT-3 paper (2020) made a striking observation: a large enough language model can learn a task from a few examples placed directly in the prompt.
English: cheese → French: fromage
English: house → French: maison
English: tree → French:The model answers “arbre”. No weights changed and no training happened. The model recognized the pattern in its context window and continued it. This is in-context learning: zero-shot with only an instruction, one-shot with one example, few-shot with several (see few-shot and zero-shot learning).
It is the foundation of practical prompting. Tips:
- Pick examples that cover the variety of real inputs, including tricky ones.
- Keep the format consistent; models copy formats closely.
- If you need hundreds of examples or perfect consistency, fine-tuning may be the better tool.
How it works inside is still being researched. Interpretability work has found induction heads, attention circuits that spot “A was followed by B earlier, so after A comes B again”, which explain part of the effect (see mechanistic interpretability).
Everyday use cases
- Case 01
Quick data extraction
Two examples of turning an invoice email into JSON are often enough for the model to handle the next hundred.
- Case 02
Matching a writing style
A few sample product descriptions teach the model your brand's tone.
Strengths, weaknesses and when to use it
Strengths
- Instant task adaptation with no training, data pipeline or GPU.
Weaknesses
- Uses context window space on every call; less reliable than fine-tuning for complex, narrow tasks.
When to use itFirst thing to try for a new task, before considering fine-tuning.
Compare with
Check yourself
- Q01
What changes in the model during in-context learning?
Show answer
A: Nothing in its weights; only the prompt changes