Skip to content
ML Atlas

GPT-2

OpenAI's 1.5-billion-parameter language model from 2019, famous for surprisingly fluent text and for being released in stages over misuse concerns.

Also GPT2 · staged release

Fig. / How it works
thethecatcatsatsatononthethematmatONE TOKEN AT A TIME, EACH READING ALL BEFORE IT
Role · kind
Real systemsFamous system
Level
Beginner

Radford et al., OpenAI, 2019

A text-completion machine that, for the first time, could write several paragraphs that sounded convincingly human, so its makers first released only smaller versions.

Where it fits

GPTGPTPretrainingPretrainingTransformerTransformerGPT-2
+All relationships as a list

Is a kind of / part of

Siblings

Uses / built on

See it on the full map

Try it

How a language model picks the next word
Loading interactive widget…
TakeawayThe model only ever outputs probabilities. Temperature, top-k and top-p decide how adventurous the pick is, and the same model can sound careful or chaotic.
Where does each word look?
Loading interactive widget…
TakeawayEach attention head is a different way of reading the sentence. Some just look at the previous word; one learned, without being told, that “it” usually means the animal.

How it works

In February 2019 OpenAI announced GPT-2, a GPT-style language model ten times larger than its predecessor. It wrote fluent, on-topic paragraphs from a short prompt, something earlier models could not do reliably.

What made it notable:

  • Scale: 1.5 billion parameters, trained on about 40 GB of web text (“WebText”) with plain next-token prediction (pretraining).
  • Zero-shot tasks: without task-specific training, it could summarize (when prompted with “TL;DR:”) or translate a little, hinting at the in-context abilities that GPT-3 later made obvious.
  • Staged release: citing concerns about fake news and spam, OpenAI initially released only the smallest model, then larger ones over the year, and the full model in November 2019. It sparked an ongoing debate about responsible release of AI models.

Today GPT-2 is small, but its open weights make it a favorite for research and teaching. The next-token sampler and attention heatmap widgets on this site use real GPT-2 outputs.

Everyday use cases

  • Case 01

    How you've probably met it

    Through 2019 headlines about AI "too dangerous to release", early AI text adventure games, and as the model inside many tutorials, including the widgets on this site.

Strengths, weaknesses and when to use it

Strengths

  • Small enough to run on a laptop; fully open weights and well studied.

Weaknesses

  • Weak reasoning and factual accuracy by today's standards; drifts off topic in long text.

When to use itAs a historical milestone and a small, open model for learning how LLMs work inside.

Compare with

Go deeper

Sources

  1. [1]Radford et al., Language Models are Unsupervised Multitask Learners (2019)
  2. [2]OpenAI, GPT-2: 1.5B release (2019)
  3. [3]Solaiman et al., Release Strategies and the Social Impacts of Language Models (2019)

Facts checked as of 2026-10-12