Neurons, layers and networks

In Machine Learning Foundations you met logistic regression: multiply each feature by a weight, add them up, squash the result into a probability. Keep that picture in mind. A neural network is, at heart, many of those little units wired together — and that simple idea powers image recognition, speech, translation and large language models.

This lecture builds a network from the bottom up: one neuron, then a layer, then a stack of layers. By the end you will be able to read a network diagram and know exactly what arithmetic happens inside it.

One neuron

An artificial neuron takes a few numbers in and produces one number out, in two steps.

1. Weighted sum. Multiply each input by its weight, add them up and add a bias: $$z = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b$$
2. Activation. Pass the sum through a simple function $f$ to get the neuron’s output $a = f(z)$. A common choice is the sigmoid, which squashes any number into the range 0 to 1.

The weights say how much each input matters and in which direction: a large positive weight means “this input pushes me to fire”, a negative weight means “this input holds me back”. The bias sets how easily the neuron fires at all.

Try these experiments:

• Change w₁ and w₂ and watch the boundary rotate: the weights decide which direction in input space the neuron cares about.
• Change only the bias and watch the boundary slide without rotating.
• Switch the activation to Step and the soft gradient becomes a hard yes/no decision. Switch to ReLU and the neuron stays silent on one side of the line and grows linearly on the other.

A single neuron can only ever draw one straight boundary. That is its limitation — and the reason we combine many of them.

A layer is a matrix multiplication

Put several neurons side by side, all reading the same inputs, and you have a layer. Each neuron has its own weights and bias, so each one looks at the inputs in its own way.

Writing out every weighted sum separately gets tedious. Stack each neuron’s weights as a row of a matrix $W$, the inputs as a vector $\mathbf{x}$ and the biases as a vector $\mathbf{b}$, and the whole layer becomes one line:

This is why deep learning and GPUs grew up together. A graphics card is a machine for multiplying large matrices very fast, and a neural network is mostly large matrix multiplications with cheap activation functions in between.

A network is a stack of layers

A network feeds the output of one layer into the next. The first layer reads the raw inputs; the hidden layers in the middle compute intermediate features that nobody specifies by hand; the output layer produces the answer.

The network below is real: it has just been trained, in your browser, on measurements of three species of iris flower — the length and width of the sepals and petals. Scroll through the story to follow one flower through it.

Input layer

The raw features. It performs no computation; it simply holds the values that enter the network.

Hidden layer

Any layer between input and output. Its outputs are learned intermediate features that nobody labels by hand.

Output layer

Produces the prediction — for example one score per class, turned into probabilities with softmax.

Fully connected (dense) layer

A layer in which every input connects to every neuron. The most basic building block of neural networks.

Forward pass

Running an input through the network, layer by layer, to produce a prediction.

Parameters

All the weights and biases. Training adjusts these; nothing else in the network changes.

Counting parameters

The size of a model is usually quoted as its number of parameters. For a fully connected layer with $n_\text{in}$ inputs and $n_\text{out}$ outputs:

The flower network above has layers of 4 → 6 → 6 → 3 neurons, so it has $4·6 + 6 + 6·6 + 6 + 6·3 + 3 = 93$ parameters. A classic network that reads 28 × 28-pixel handwritten digits (784 inputs), with one hidden layer of 128 neurons and 10 outputs, has $784·128 + 128 + 128·10 + 10 = 101{,}770$. Large language models have hundreds of billions — built from the same idea.

The same network in code

Frameworks such as PyTorch describe a network as a sequence of layers. This is the flower classifier:

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(4, 6),   # 4 inputs  -> 6 neurons: W is 6x4, b has 6 values
    nn.ReLU(),
    nn.Linear(6, 6),   # hidden layer 2
    nn.ReLU(),
    nn.Linear(6, 3),   # one score per species
)

flower = torch.tensor([[0.53, -0.36, 1.05, 0.79]])  # standardized measurements
probabilities = torch.softmax(model(flower), dim=1)
print(sum(p.numel() for p in model.parameters()))   # 93

We have quietly used ReLU and sigmoid without asking why a network needs activation functions at all. The next lecture answers that question — and shows why stacking layers without them would be pointless.