Residual Connection
A shortcut that adds a layer's input to its output, so each layer only learns a correction. It lets very deep networks train.
Also skip connection · shortcut connection · residual stream
He et al., 2015
Instead of retelling a whole story at every step, each storyteller just adds a sentence. The original story is always passed along, so it never gets lost.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
Applies to
Used by
How it works
In a plain deep network, the signal and its gradient pass through every layer in turn, and through many multiplications gradients tend to shrink to nothing or blow up (the vanishing and exploding gradient problem from backpropagation).
A residual connection adds a shortcut around a block: the block’s output is input + change. Two good things follow:
- Easy defaults: if a block has nothing useful to add, it can learn a change of about zero, and the network behaves as if the block were not there. Adding layers can no longer make things much worse.
- Gradient highway: the derivative of “x + something” with respect to x always includes 1, so gradients flow directly back through the shortcuts.
Introduced with ResNet in 2015, the idea became universal. In a Transformer, the residual connections form a “residual stream”: a running representation that each attention and feed-forward layer reads from and adds to.
Everyday use cases
- Case 01
Deep image models
ResNets with over a hundred layers rely on them.
- Case 02
Every Transformer
Each attention and feed-forward sub-layer is wrapped in a residual connection, forming the "residual stream".
Famous examples
- 2023Claude
Anthropic's family of AI assistants: Transformer-based large language models trained with RLHF and Constitutional AI to be helpful, honest and harmless.
- 2020GPT-3
OpenAI's 175-billion-parameter language model that showed a big enough model can do new tasks from just a few examples in its prompt.
Strengths, weaknesses and when to use it
Strengths
- Makes depth almost free to add from a training perspective.
- Costs nothing extra to compute.
Weaknesses
- Requires input and output of the block to have the same shape (or a projection).
When to use itIn essentially any network deeper than a handful of layers.
Check yourself
- Q01
Why do residual connections help gradients?
Show answer
A: The "+ x" path passes gradients straight back without shrinking them