Attention: how context flows

After the last lecture, every token has an embedding — but that embedding is the same wherever the token appears. Bank has one vector whether it follows river or loan. It has one vector whether it refers to an animal, a street or a server. To understand language, each token needs to take its context into account. Attention is how it does that.

The core idea

Self-attention lets every token look at the other tokens in the text and build an updated representation that mixes in what is relevant. After an attention layer, the vector for it in “…because it was too tired” carries information about animal; in “…because it was too wide”, about street.

The magic is that nobody programs these relationships. The model learns, during training, where to look for what — because looking in the right places helps it predict the next token.

See attention at work

Below, the arcs show how strongly the selected word attends to each other word. Start with it, then switch the last word from tired to wide and watch the attention move. Click other words to see different patterns.

Queries, keys and values

How does a token decide where to look? Each token’s vector is transformed, by three learned weight matrices, into three new vectors:

Query (Q)

What this token is looking for — “I am a pronoun; which noun could I refer to?”

Key (K)

What this token offers to others — “I am a noun that could be the subject of the sentence.”

Value (V)

The information this token passes on if another token attends to it.

For a given token, the model compares its query with every token’s key using a dot product. High scores mean a good match. Softmax turns the scores into weights that add up to 1, and the output is the weighted sum of the values. In compact form:

The division by $\sqrt{d_k}$, the square root of the key size, keeps scores in a range where softmax still spreads attention sensibly instead of locking onto a single token.

Here is one attention step with real numbers, small enough to follow by hand. The vectors have only three numbers each; in a real model they have hundreds or thousands.

In matrix form, as in the formula above, attention for all tokens at once is a few large matrix multiplications — exactly the kind of work GPUs do extremely fast and in parallel. That efficiency is a big part of why transformers displaced earlier sequential models.

Many heads, many relationships

One attention pattern can’t capture everything. So transformers use multi-head attention: several attention operations run side by side, each with its own query, key and value matrices. In trained models, different heads specialize — one might track which noun a pronoun refers to, another the previous word, another matching brackets in code. Their outputs are combined and passed on.

No peeking at the future

A model that generates text left to right must not use tokens it hasn’t produced yet. Causal masking enforces this: when computing attention for a token, all later positions get a weight of zero. Each token can attend only to itself and what came before it — matching the next-token task the model is trained on.

Why long contexts are expensive

In standard attention, every token compares its query with every other token’s key. With $n$ tokens that is about $n^2$ comparisons, so doubling the context roughly quadruples the attention work, and the stored keys and values grow with every token. This is why:

• very long prompts respond more slowly, especially before the first token appears;
• providers price long contexts carefully and cap their size;
• researchers invest heavily in faster attention algorithms and memory-saving tricks.

The lab below puts numbers on both costs for a mid-size open model. Drag the context length from a short prompt to a million tokens and compare how fast each cost grows.

Attention is one of two building blocks inside a transformer. In the next lecture we stack them and follow a prompt all the way through the model.