Batch and Layer Normalization
Layers that rescale activations to a standard range inside the network, which makes deep networks train faster and more reliably.
Also batch normalization · BatchNorm · layer normalization · LayerNorm · RMSNorm · normalization
Ioffe and Szegedy (BatchNorm), 2015; Ba, Kiros and Hinton (LayerNorm), 2016
Before each team member passes notes on, someone rewrites them at a normal size: no shouting in capital letters, no tiny whispers. Everyone downstream can read them easily.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Applies to
How it works
As signals pass through many layers, their scale can drift: activations grow huge or shrink toward zero, and backpropagation becomes unstable. Normalization layers reset the scale at each step:
- Compute the mean and spread of a group of activations.
- Subtract the mean, divide by the spread.
- Multiply and shift by two learned parameters, so the network can still choose the scale it wants.
The difference is which group:
- BatchNorm: for each feature, across the examples in the batch. Very effective in CNNs like ResNet, but depends on batch size.
- LayerNorm: across all features of one example. Independent of the batch, which suits sequences and the Transformer.
- RMSNorm: a cheaper LayerNorm that only rescales; common in recent language models.
Together with residual connections, normalization is what makes networks with dozens or hundreds of layers practical.
Everyday use cases
- Case 01
Inside every Transformer block
Each attention and feed-forward sub-layer is paired with layer normalization.
- Case 02
Image networks
BatchNorm is a standard part of ResNet-style CNNs.
Strengths, weaknesses and when to use it
Strengths
- Allows higher learning rates and faster, more stable training.
Weaknesses
- BatchNorm behaves differently in training and inference and struggles with tiny batches.
When to use itIn essentially any deep network; LayerNorm or RMSNorm for Transformers, BatchNorm for many CNNs.
Check yourself
- Q01
What does LayerNorm normalize over?
Show answer
A: The features of each single example