Speech and Audio
Machine learning for sound: turning speech into text and back, recognizing sounds, and generating music and voices.
Sound is just air wiggling very fast. Audio AI learns to read those wiggles, to hear words in them, and to make new wiggles that sound like a voice or a song.
Where it fits
Not shown: 3 more kinds and parts. See the full list below.
+All relationships as a list
Overview
Microphones record sound as a waveform: tens of thousands of numbers per second. That raw signal is hard to learn from directly, so most audio models first turn it into a spectrogram, a picture of which frequencies are present at each moment. From there, many vision and language techniques apply.
The main tasks:
- Speech recognition: audio in, text out. Whisper is a well-known open model.
- Text-to-speech: text in, natural-sounding voice out.
- Audio classification: which sound is this (dog, siren, guitar)?
- Audio generation: music and sound effects, increasingly with the same diffusion and Transformer ideas used for images and text.
Everyday use cases
- Case 01
Voice assistants and dictation
Speaking to your phone, car or smart speaker.
- Case 02
Automatic subtitles
Video platforms caption speech in many languages.
- Case 03
Sound recognition
Phones can alert deaf users to doorbells, alarms or a crying baby.