Activation Function

A layer of a neural network multiplies its inputs by weights and adds them up. Stack two such layers and you just get another multiply-and-add, still a straight-line model. The activation function breaks that: it bends the output of each neuron, so stacking layers builds genuinely complex shapes.

The common ones:

• ReLU (rectified linear unit): zero for negative inputs, unchanged for positive ones. Simple, fast, and the default since around 2012.
• GELU: a smooth version of ReLU, standard in Transformers.
• Sigmoid: squashes to 0..1. Used for a final yes/no probability, as in logistic regression.
• Tanh: squashes to −1..1. Common in older RNNs and LSTMs.
• Softmax: turns a list of scores into probabilities that sum to 1. Used at the output of classifiers and inside attention.

Sigmoid and tanh flatten out for large inputs, so their gradient is nearly zero there. Through many layers those small gradients multiply and vanish, which is one reason ReLU took over.

The math

$$\text{ReLU}(z) = \max(0, z), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \text{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$