Variational Autoencoder

A VAE keeps the encoder-decoder shape of an autoencoder but changes what the encoder outputs. Instead of one exact code, it outputs a small cloud: a center and a spread. During training, a random point is drawn from that cloud and decoded.

Two forces shape training:

1. Reconstruction: the decoded point should look like the input.
2. Regularity: every cloud should stay close to a standard bell curve around the origin.

The second force pushes all the clouds to overlap and fill the space, with no empty gaps. The result is a latent space where every point decodes to something plausible and nearby points decode to similar things. Now you can generate: sample a random point, decode it. Or interpolate: walk in a straight line between two codes and watch one image morph into another.

VAE images are often a bit blurry, and GANs and diffusion models produce sharper results. But VAEs remain important: Stable Diffusion uses an autoencoder of this kind to shrink images before running diffusion.

The math

The VAE maximizes the evidence lower bound (ELBO):

$$\mathcal{L}(x) = \mathbb{E}_{q_\phi(z \mid x)}\big[\log p_\theta(x \mid z)\big] - \mathrm{KL}\big(q_\phi(z \mid x) \,\|\, \mathcal{N}(0, I)\big)$$

The reparameterization trick writes $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, so gradients flow through $\mu$ and $\sigma$.