Text-to-Video
Models that generate video clips from a text description, typically diffusion models that denoise a whole clip in a compressed space-time latent.
Also video generation · video diffusion · spacetime patches · image-to-video
Like text-to-image, but the model must draw many pictures in a row that agree with each other, so the dog keeps the same fur and walks smoothly from frame to frame.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Uses / built on
Evolved from
How it works
Text-to-image produces one picture. Text-to-video must produce dozens of pictures per second that agree with each other: the same character, the same lighting, plausible motion.
Most current systems follow the same recipe:
- Compress the video. A video autoencoder turns frames into a much smaller latent volume across space and time.
- Cut it into spacetime patches. Like the image patches of a Vision Transformer, but each patch also spans a few frames.
- Denoise with a Transformer. A large diffusion model built on a Transformer learns to turn noise into a coherent patch sequence, conditioned on text and steered with classifier-free guidance.
- Decode the latents back into frames.
Because every patch can attend to every other one, the model can keep objects consistent over time. It still makes telling mistakes: objects appear from nowhere, hands merge, glasses fail to shatter.
Well-known systems include OpenAI’s Sora and Google DeepMind’s Veo. The same capability fuels deepfake concerns, which is why provenance watermarks and labels are part of most deployments (see bias and fairness and AI regulation).
Everyday use cases
- Case 01
Concept videos and storyboards
Turning a script line into a rough clip to test an advertising idea before filming.
- Case 02
B-roll and social media
Short atmospheric shots generated instead of licensing stock footage.
Famous examples
- 2024Sora
OpenAI's text-to-video model, unveiled in February 2024, which generates video from text by denoising spacetime patches with a diffusion Transformer.
- 2024Veo
Google DeepMind's text-to-video model family, announced at Google I/O in May 2024 and offered in Google's AI products and cloud platform.
Strengths, weaknesses and when to use it
Strengths
- Huge creative leverage; one sentence becomes moving footage.
Weaknesses
- Very expensive to train and run; physics and object permanence still break.
- Raises serious deepfake and misinformation concerns.
When to use itShort creative clips and prototyping, with human review and provenance labels.
Compare with
Check yourself
- Q01
What makes video generation harder than image generation?
Show answer
A: Frames must stay consistent with each other over time