Weight Initialization
Choosing the random starting values of a network's weights so signals neither shrink nor blow up as they pass through many layers.
Also Xavier initialization · Glorot initialization · He initialization · Kaiming initialization
Before a choir sings, you set every microphone's volume. Too low and the back row is silent; too high and everything screeches. Good initialization sets the volumes right before training starts.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Applies to
How it works
Before training, every weight in a neural network needs a starting value. That choice matters more than it seems.
- All zeros (or all the same): every neuron in a layer computes the same thing, receives the same gradient and stays identical forever. Random values break this symmetry.
- Too large: activations grow layer after layer and gradients explode.
- Too small: activations shrink to zero and gradients vanish. See vanishing gradients.
The fix is to scale the random values by the layer’s width, so the variance of the signal stays roughly constant from layer to layer:
- Xavier (Glorot) initialization balances inputs and outputs and suits tanh and sigmoid.
- He (Kaiming) initialization doubles the variance to compensate for ReLU zeroing out half its inputs.
Normalization layers and residual connections make networks more forgiving, but initialization is still part of every training recipe.
Everyday use cases
- Case 01
Training deep networks from scratch
Every framework's default layer initializer is one of these schemes, picked to match common activations.
Strengths, weaknesses and when to use it
Strengths
- Costs nothing and can decide whether a deep network trains at all.
Weaknesses
- Less critical with normalization layers and residual connections, but still matters at large scale.
When to use itAlways, implicitly. Change it only when building unusual architectures or very deep networks.
Compare with
Check yourself
- Q01
Why not initialize all weights to zero?
Show answer
A: All neurons in a layer would stay identical and learn the same thing