Activation functions: why depth works

In the last lecture every neuron computed a weighted sum and then applied an activation function. It is tempting to see the activation as a detail. It is not: remove it, and the whole idea of depth falls apart.

Linear layers collapse

A layer without an activation computes $W\mathbf{x}$ — a linear map. Feed its output into a second linear layer and you get $W_2 (W_1 \mathbf{x})$. Matrix multiplication is associative, so that is the same as $(W_2 W_1)\mathbf{x}$: one matrix, one linear layer. A hundred linear layers are still one linear layer — they can only rotate, stretch, shear and flip the input space.

That matters, because many real patterns cannot be separated by a straight line. Watch what linear layers do to the classic example below, and what changes when we add ReLU.

This is the real job of hidden layers: each one transforms the data into a new space in which the problem is a little easier, until the final layer can solve it with a simple linear decision.

A layer of hinges can draw any curve

The same idea works for regression. A ReLU unit, $\text{ReLU}(x - t)$, is a hinge: flat until $x = t$, then a straight ramp. Scale a few hinges, place them at different points and add them up, and you get a line that bends wherever you like.

With one or two units the network is hopelessly stiff. By around a dozen it follows the wave closely, and with two dozen it is practically indistinguishable from the target. This is the intuition behind the universal approximation theorem: a network with a single hidden layer can approximate any continuous function, given enough units.

The common activation functions

Different activations bend in different ways, and — as you will see when we look at training — their slopes matter as much as their shapes. Drag across the chart to read each function and its derivative.

Sigmoid

Squashes any number into 0…1. Useful for a final yes/no probability, but it saturates for large inputs and its slope is never above 0.25.

Tanh

Like sigmoid but ranges from −1 to 1 and is centered on zero. Still saturates.

ReLU

max(0, z). Cheap, does not saturate for positive inputs, and the default choice for hidden layers.

Leaky ReLU

A ReLU with a small slope for negative inputs, so neurons that are always negative can still recover.

GELU

A smooth variant of ReLU used in transformer models such as BERT and GPT.

Softmax

Used on the output layer for classification: turns a vector of scores into probabilities that sum to 1.

Saturation and “dead” neurons

Learning works by sending gradients backwards through the network, and every activation on the way multiplies the gradient by its slope. A sigmoid’s slope is at most 0.25, so after ten sigmoid layers a gradient can shrink by a factor of a million. Early layers then barely learn. This vanishing gradient problem held deep learning back for years, and ReLU was one of the simple changes that fixed it.

ReLU has its own failure mode. If a neuron’s weighted sum is negative for every input, its output and its slope are both zero, so it never receives a gradient again: a dead neuron. A learning rate that is too high can kill many neurons at once. Leaky ReLU and careful initialization reduce the risk.

We now know what a network computes. The next lecture tackles the question that makes all of this useful: how does the network find good values for thousands — or billions — of weights? The answer is backpropagation.