Inside the transformer

The last two lectures gave us the ingredients: tokens become vectors, and attention lets those vectors exchange information. A transformer — the architecture behind GPT, Claude, Gemini, Llama and nearly every modern language model — arranges these ingredients into a tall, repeating stack. This lecture follows a prompt all the way through it.

The shape of the model

Chat models use a decoder-only transformer. Its structure is surprisingly regular:

Embedding. Each token ID becomes a vector, combined with information about its position in the text.
A stack of identical blocks — dozens of them in large models. Each block contains:
- a self-attention layer, which mixes information across tokens;
- a feed-forward network (MLP), which transforms each token’s vector independently.
The output head. The final vector at the last position is converted into a score for every token in the vocabulary.

Scroll through the model below. Each vertical line is one token’s position; the plates are the layers it passes through.

Scroll storyA prompt’s journey through a transformerStep 1 of 6

Loading 3D model…

1
Embed the tokens
Each token of “The cat sat on the” is looked up in the embedding table. Position information is added, because attention on its own doesn’t know word order.
2
Attention: share context
In the first attention layer, every token gathers information from earlier tokens. Causal masking means arcs only point backward — no token sees the future.
3
Feed-forward: think per token
Next, a feed-forward network processes each position on its own. These large layers hold much of the knowledge the model picked up during training.
4
Repeat, layer after layer
The pattern repeats: attention, then feed-forward. Each block adds its result to the residual stream, gradually building richer, more abstract representations.
5
Predict the next token
The final vector at the last position is compared with every token in the vocabulary, producing scores. Softmax turns them into probabilities: “mat” is most likely.
6
Append and go again
A token is chosen and appended. To produce the next one, the model runs again with the longer sequence — one full pass per generated token.

The residual stream

Notice that each token’s vertical line runs straight through every layer. That line is the residual stream. Layers don’t replace a token’s vector; they read from it and add an adjustment back:

\mathbf{x} \leftarrow \mathbf{x} + \text{Attention}(\mathbf{x}), \qquad \mathbf{x} \leftarrow \mathbf{x} + \text{MLP}(\mathbf{x})

This design, borrowed from residual networks in computer vision, is what makes very deep models trainable: information and gradients have a direct path from bottom to top, and each block only has to learn a refinement. Each block also includes layer normalization, which keeps the numbers in a stable range as they pass through the stack.

Follow one token’s vector up the stream and watch each layer add to it:

AnimationReading from and writing to the residual stream

Loading animation…

The stream startsThe residual stream for “sat” starts as its embedding plus position information: six numbers here, thousands in a real model.
Attention adds contextThe attention layer reads the stream, gathers information from earlier tokens and adds its result. The old vector is kept: x ← x + Attention(x).
Feed-forward adds a refinementThe feed-forward network reads the updated vector and adds its own adjustment. Check any column: each row is the row below plus the update beside it.
Block after blockEvery block repeats the pattern. The final vector is the embedding plus every layer’s update, so it carries what each layer contributed.
A highway for learningDuring training, the error signal flows straight down the stream to every layer. This direct path is why stacks of 100 or more layers can be trained.

What the layers learn

Researchers studying trained models find a rough division of labor. Early layers handle local patterns — spelling, syntax, which words go together. Middle layers build up meaning and relationships. Later layers shape the representation toward predicting the next token. The feed-forward layers, which hold most of the parameters, appear to store much of the model’s factual associations.

This is an area of active research, and the real picture is messy. But it helps to think of the stack as a pipeline that gradually turns which tokens these are into what should come next.

Counting the parameters

The claim that feed-forward layers hold most of the parameters is easy to check. In each block, attention has four $d \times d$ matrices (for queries, keys, values and the output), and the feed-forward network has two $d \times 4d$ matrices. That is about $12d^2$ parameters per block, two-thirds of them in the feed-forward network. Add the embedding table, one row of $d$ numbers per vocabulary token, and you can predict a model’s size from its shape.

InteractiveCount a transformer’s parametersPick a model, then change its shape

Loading visual…

The presets reproduce the published sizes of GPT-2 (124 million and 1.5 billion parameters), a 7-billion-parameter open model and GPT-3 (175 billion). Doubling the vector size quadruples the parameters per block; doubling the layers only doubles them. Training needs roughly 16 bytes per parameter for weights, gradients and optimizer state, before counting activations.

From vector to probabilities

At the top, the final vector at the last position is multiplied by a large matrix with one row per vocabulary token. The result is a list of scores — the logits you met in lecture 1 — and softmax turns them into probabilities. Only the last position is needed for the next token; the other positions did their job by contributing context through attention.

Generation: one pass per token

After one token is chosen, the model runs again on the extended text, and again, and again. This is autoregressive generation. Two consequences matter in practice:

Prefill versus decode: Processing the prompt happens in one parallel pass (prefill). Generating output happens one token at a time (decode), which is why long answers take longer than long prompts.
KV cache: The keys and values computed for earlier tokens don’t change, so they are stored and reused. Each new token then needs only its own computation — at the cost of memory that grows with context length.
Model size: A “70-billion-parameter” model has about 70 billion learned numbers in its embeddings, attention and feed-forward weights. More parameters generally means more capability, but also more memory and compute per token.

We now have a complete picture of how a trained model produces text. The final lecture of this module answers the obvious next question: how do all those billions of parameters get their values?

In this lecture you will