What a language model really does

Ask a chat assistant to explain recursion, summarize a contract or write a SQL query, and it seems to understand. Under the hood, the mechanism is almost disarmingly simple — and knowing it is the key to using these systems well.

One job: predict the next token

A language model is trained to do one thing: given some text, predict what comes next. Not the next sentence or the next idea — the next token, a small chunk of text that is often a whole word, sometimes part of one, sometimes a punctuation mark. (We will look at tokens closely in the next lecture.)

For the text “The best way to learn a new programming language is”, the model doesn’t produce one answer. It produces a score for every token in its vocabulary — typically between 30,000 and a few hundred thousand of them. High scores mean “likely to come next”.

From scores to probabilities: softmax

Raw scores, called logits, can be any number. The softmax function turns them into probabilities that are all positive and add up to 1:

Here $z_i$ is the score of token $i$ and $T$ is the temperature, which we will come to shortly. Exponentiating exaggerates differences: a token scoring 6 is not slightly more likely than one scoring 5 — it is about 2.7 times as likely.

Generation is a loop

To produce a full answer, the model runs a loop:

1. Compute probabilities for the next token, given the text so far.
2. Choose one token from that distribution.
3. Append it to the text.
4. Repeat — until the model produces a special end token or hits a length limit.

Every word of a long answer comes from this loop, one token at a time. That is why responses stream in word by word, and why longer outputs take longer and cost more.

Try it: generate, one token at a time

Generate a sentence token by token. Try the same prompt a few times, then change the temperature and the decoding strategy.

Temperature and decoding strategies

How the model chooses from the distribution is a setting you control:

Greedy decoding

Always take the most probable token. Deterministic, but can get repetitive and misses good alternatives.

Sampling

Pick randomly in proportion to probability. Produces varied, natural text — and occasionally an odd choice.

Temperature

Divides the scores before softmax. Below 1 sharpens the distribution toward the top tokens; above 1 flattens it, giving unlikely tokens more chance.

Top-p (nucleus) sampling

Sample only from the smallest set of tokens whose probabilities add up to p, such as 0.9 — cutting off the long tail of unlikely tokens.

See how the three settings reshape one real-looking distribution, and what they do to 200 samples.

As a rule of thumb, use low temperature for extraction, classification, code and anything that must be consistent, and higher temperature for brainstorming, marketing copy and creative work.

What this explains

Seen as next-token prediction, many LLM behaviors stop being mysterious:

• Fluency. Trained on vast amounts of text, the model is extraordinarily good at producing text that sounds right.
• Hallucinations. The model optimizes for plausible continuations, not verified facts. If a fake citation looks plausible, it may be generated with total confidence.
• Sensitivity to wording. Every token of your prompt changes the probabilities of what follows, which is why phrasing and examples in a prompt matter so much.
• No hidden memory. Between calls, the model remembers nothing. Chat apps create the illusion of memory by sending the whole conversation back with every request.

In the next lecture we look at what the model actually receives as input: not letters or words, but tokens — and the vectors that give them meaning.