Skip to content
ML Atlas

Music and Audio Generation

Models that generate music, sound effects or ambient audio from a text prompt, melody or other audio, usually by modeling compressed audio tokens or spectrogram latents.

Also text-to-audio · text-to-music · AI music · audio tokens · neural audio codec · MusicLM · Jukebox

Fig. / How it works
PromptPromptGenerate audio tokensGenerate audio tokensCodec decoderCodec decoderWaveformWaveform
Role · kind
Using and applyingFamily
Level
Advanced

Type 'calm piano music for a rainy day' and the computer composes and plays a new piece, the way image generators paint a picture from words.

Where it fits

Speech and AudioSpeech and AudioAutoregressive ModelAutoregressive ModelDiffusion ModelDiffusion ModelTokenizationTokenizationMusic and Audio Generation
+All relationships as a list

Is a kind of / part of

See it on the full map

How it works

One second of CD-quality audio has 44,100 samples per channel, and music has structure over minutes. Generating it directly is very hard. Modern systems first compress audio.

  1. A neural audio codec (an autoencoder) squeezes the waveform into a short sequence of discrete codes, like tokens for sound, or into a compact latent.
  2. A generator produces those codes conditioned on a text prompt, a hummed melody or lyrics. Either an autoregressive Transformer predicting the next audio token, or a diffusion model denoising the latent.
  3. The codec’s decoder turns the codes back into sound.

Milestones include OpenAI’s Jukebox (2020), Google’s MusicLM and Meta’s MusicGen (both 2023), followed by commercial song generators that also produce vocals. The same machinery generates sound effects and is closely related to modern text-to-speech.

As with images, training on recorded music without permission has led to lawsuits from record labels, and the question of what counts as fair use is still being settled.

Everyday use cases

  • Case 01

    Background music for videos

    Creators generate royalty-free tracks of the right mood and length.

  • Case 02

    Sound effects

    Game designers prototype "footsteps on gravel" or "distant thunder" from text.

Strengths, weaknesses and when to use it

Strengths

  • Fast, flexible creation of music and sound for non-musicians.

Weaknesses

  • Keeping a coherent musical structure over minutes is hard.
  • Copyright disputes over training on artists' recordings.

When to use itPrototyping, background audio and creative exploration, with care about licensing.

Compare with

Check yourself

  1. Q01

    What does a neural audio codec do in music generation?

    Show answer

    A: Compresses audio into tokens a model can generate, and decodes them back to sound

Go deeper

Sources

  1. [1]Dhariwal et al., Jukebox: A Generative Model for Music (2020)
  2. [2]Agostinelli et al., MusicLM: Generating Music From Text (2023)
  3. [3]Copet et al., Simple and Controllable Music Generation (MusicGen, 2023)