Whisper
OpenAI's open-source speech recognition model, trained on 680,000 hours of audio from the web, that transcribes and translates speech in many languages.
Radford et al., OpenAI, 2022
A listener that learned from an enormous pile of recordings with captions, so it can write down what people say in dozens of languages, and translate it to English.
Where it fits
+All relationships as a list
Is a kind of / part of
Uses / built on
How it works
OpenAI released Whisper in September 2022 with open weights and code. Its design is deliberately simple: a standard encoder-decoder Transformer.
- Audio is split into 30-second chunks and turned into a log-Mel spectrogram.
- The encoder reads the spectrogram.
- The decoder writes text, guided by special tokens that tell it the language and the job: transcribe, translate to English, or add timestamps.
What made it strong was data, not architecture: 680,000 hours of audio paired with text found on the web. That “weakly supervised” data is noisy, but its sheer variety made Whisper unusually robust; OpenAI reported it made far fewer errors than specialized models when tested on datasets it had not been tuned for, though it did not beat them on the benchmark they were built for.
Everyday use cases
- Case 01
How you've probably met it
Many transcription, subtitling and note-taking apps run Whisper or a model derived from it, since the weights are openly available.
Strengths, weaknesses and when to use it
Strengths
- Robust to accents, noise and technical language thanks to diverse training data.
- One model handles many languages, translation and timestamps.
- Open weights (MIT license): runs locally.
Weaknesses
- Can hallucinate text during silence or music.
- Accuracy varies a lot between languages.
When to use itFor general-purpose transcription, especially when you want to run it yourself.
Go deeper
Sources
- [1]OpenAI, Introducing Whisper (2022)
- [2]Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (2022)
Facts checked as of 2026-10-12