Skip to content
ML Atlas

Autoregressive Model

A generative model that produces data one piece at a time, each piece predicted from all the pieces before it. GPT-style language models are the best-known example.

Also autoregressive generation · next-token prediction · causal language modeling · language modeling · PixelCNN · WaveNet

Fig. / How it works
thethecatcatsatsatononthethematmatONE TOKEN AT A TIME, EACH READING ALL BEFORE IT
Domain 06
Generative AI
Role · kind
Models and algorithmsFamily
Level
Intermediate

Writing a story one word at a time, where each new word is chosen by looking at everything written so far. Keep going and you get a whole story.

Where it fits

Generative AIGenerative AILog Loss (Cross-Entropy)Log Loss (Cross-Entr…Music and Audio GenerationMusic and Audio Gene…PretrainingPretrainingAutoregressive Model
+All relationships as a list

Is a kind of / part of

See it on the full map

How it works

The chain rule of probability says any joint distribution can be written as a product of “next piece given everything so far”:

p(x1,…,xT)=∏t=1Tp(xt∣x1,…,xt−1).p(x_1, \dots, x_T) = \prod_{t=1}^{T} p(x_t \mid x_1, \dots, x_{t-1}).

An autoregressive model learns exactly those conditional probabilities. Training is simple: show it real sequences and ask it to predict each next element, scoring it with cross-entropy. No labels needed, which makes it self-supervised. For text this is called language modeling.

Generation runs the model in a loop: predict a distribution, sample one element, append it, repeat.

Famous examples:

  • Text: GPT and nearly every modern LLM are autoregressive Transformers.
  • Audio: WaveNet produced speech one audio sample at a time.
  • Images: PixelCNN generated pixel by pixel; several image models generate sequences of image tokens.

The main cost is speed: you cannot write token 100 before token 99. Diffusion models instead refine a whole output in parallel over several steps.

Everyday use cases

  • Case 01

    Chatbots and autocomplete

    Every answer from a GPT-style assistant is generated one token at a time.

  • Case 02

    Speech synthesis

    WaveNet generated raw audio one sample at a time and made synthetic voices far more natural.

Strengths, weaknesses and when to use it

Strengths

  • Simple, stable training objective with exact likelihoods; scales extremely well.

Weaknesses

  • Generation is sequential, so long outputs are slow.
  • Errors early in the output can compound.

When to use itText, code, audio and other sequences; increasingly also images and video as sequences of tokens.

Compare with

Check yourself

  1. Q01

    How does an autoregressive model generate a sentence?

    Show answer

    A: One token at a time, each conditioned on the previous ones

Go deeper

Sources

  1. [1]van den Oord et al., WaveNet: A Generative Model for Raw Audio (2016)
  2. [2]Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019)