Spectrogram

Any sound can be described as a mix of pure tones (sine waves) at different frequencies. The Fourier transform finds that mix. But speech and music change constantly, so we compute it on short slices instead:

1. Cut the waveform into short, overlapping windows (often about 25 milliseconds).
2. Compute the frequency content of each window.
3. Stack the results side by side.

The result is a grid: time across, frequency up, and loudness as color. A whistle is a thin line, a chord is several parallel lines, a drum hit is a vertical smear, and noise is a gray fog.

Two tweaks make spectrograms match human hearing. The Mel scale spaces frequencies the way we perceive pitch (we hear the difference between 100 and 200 Hz far more clearly than between 10,000 and 10,100 Hz), and taking the log of the power matches how we perceive loudness. That “log-Mel spectrogram” is what Whisper and most speech recognition models read.

There is a built-in trade-off: longer windows separate frequencies better but blur when things happen; shorter windows do the opposite.

The math

For a signal $x[n]$ and window $w[n]$ of length $N$, frame $m$ with hop $H$:

$$X[m, k] = \sum_{n=0}^{N-1} x[n + mH]\, w[n]\, e^{-2\pi i k n / N}$$

The spectrogram is $|X[m, k]|^2$, usually shown on a log (decibel) scale.