BERT
An encoder-only Transformer pretrained by filling in hidden words using context from both sides. Great at understanding text, not at generating it.
Also Bidirectional Encoder Representations from Transformers · masked language model · MLM
Devlin et al., Google, 2018
A fill-in-the-blank champion. Cover any word in a sentence and BERT guesses it by reading everything before and after the blank.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- AI Agents and Tool Use
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Uses / built on
How it works
BERT takes the encoder half of the Transformer and pretrains it on a fill-in-the-blank game:
- Take a sentence, hide about 15% of its tokens behind a
[MASK]symbol. - Ask the model to recover them.
Because nothing stops the encoder from looking both left and right, BERT learns bidirectional context: to guess “The [MASK] barked at the mailman”, it uses “barked” and “mailman” together.
This is self-supervised pretraining: no labels needed. Afterward, you add a small output layer and fine-tune on a specific task with a modest labeled dataset. In 2018 this recipe set new records on a wide range of language understanding benchmarks.
BERT’s sibling, GPT, uses the decoder half and predicts only the next token. That makes GPT a natural writer and BERT a natural reader.
Everyday use cases
- Case 01
Search engines
Google announced in 2019 that it uses BERT to better understand search queries.
- Case 02
Text classification
Sentiment analysis, spam detection and routing support tickets after fine-tuning on a few thousand examples.
Strengths, weaknesses and when to use it
Strengths
- Strong text understanding for its size; cheap to fine-tune and run.
- Sees context on both sides of every word.
Weaknesses
- Not designed to generate text.
- Superseded by larger models for many tasks, though still widely used for embeddings and classification.
When to use itFor classification, tagging, embeddings and search ranking where you need speed and low cost.
Compare with
Check yourself
- Q01
What is BERT's main pretraining task?
Show answer
B: Predict masked words using both sides of context
- Q02
Which part of the Transformer does BERT use?
Show answer
A: Encoder only