How LLMs are trained

In the last lecture we followed a prompt through a trained model. But a freshly created transformer is just billions of random numbers; asked to continue a sentence, it produces gibberish. This lecture explains how those numbers are shaped into a capable assistant — a process that happens in distinct stages, each with a different goal.

The pipeline at a glance

InteractiveFrom raw text to an assistant

Loading visual…

Pre-trainingThe model reads a huge, diverse body of text — web pages, books, code, papers — and learns to predict the next token. This is where language, knowledge and reasoning patterns come from, and where most of the compute is spent.
The base modelThe result can continue any text impressively, but it isn’t an assistant. Ask it for an email and it may produce a list of more email tasks, because that is a plausible continuation.
Supervised fine-tuningThe base model is trained further on curated examples of instructions paired with high-quality responses, written or checked by people. It learns the format and manner of a helpful assistant.
Preference tuningPeople (or trained reward models) compare pairs of answers and pick the better one. Techniques such as RLHF and DPO adjust the model toward preferred answers — more helpful, more honest, safer.
Evaluate and deployBefore release, the model is tested on benchmarks, internal task suites and adversarial red-team prompts. After release, monitoring and user feedback guide the next round.

Stage 1: pre-training

Pre-training uses the same objective you met in lecture 1: predict the next token. The model sees a snippet of text, predicts each next token, and gradient descent nudges billions of parameters to make the true continuation a little more likely. Repeat across trillions of tokens.

Nobody writes rules for grammar, arithmetic or programming. They emerge because they help prediction: to predict the next line of a Python function, it helps to “understand” Python; to predict the answer in a physics textbook, it helps to model physics.

This stage is where the enormous costs of frontier models come from: thousands of GPUs running for weeks or months, and a great deal of work on data quality — filtering spam, removing duplicates, balancing languages and domains. Better data often matters more than a bigger model.

How big, and on how much data?

With a fixed compute budget, should you train a bigger model on less text, or a smaller model on more? Researchers answered this by training hundreds of models and fitting a scaling law: a formula that predicts the loss from the number of parameters $N$ and training tokens $D$ . Training takes roughly $6ND$ operations, so every budget allows many combinations.

The 2022 “Chinchilla” study found that many earlier models were too big for their data. For the best result from a budget, grow parameters and data together, at roughly 20 tokens per parameter. Its 70-billion-parameter model, trained on 1.4 trillion tokens, beat the 280-billion-parameter Gopher, which used a similar budget.

InteractiveSpend a compute budgetPick a model, or move the sliders

Loading visual…

Loss predicted by the Chinchilla scaling law (Hoffmann et al., 2022, as re-estimated by Besiroglu et al., 2024) for the paper’s training data: only the differences between runs are meaningful. Llama 3 8B deliberately trains a small model on far more data than is compute-optimal. Training costs more for the quality reached, but the model is much cheaper to serve to millions of users.

Stage 2: the base model

The output of pre-training is a base model. It is knowledgeable but not cooperative: it continues text rather than answering it. Base models are still useful to researchers and for some specialised tasks, but a chat product needs more.

Stage 3: supervised fine-tuning

In supervised fine-tuning (SFT) — also called instruction tuning — the model trains on a much smaller, carefully curated set of examples: an instruction and an ideal response. The objective is still next-token prediction, but now on text written the way an assistant should respond. A relatively small amount of high-quality data changes the model’s behavior dramatically, because the underlying knowledge is already there.

Stage 4: preference tuning

It is often easier for people to compare two answers than to write the perfect one. Preference tuning exploits this:

RLHF: Reinforcement learning from human feedback. Human rankings train a reward model that scores answers; the language model is then optimized to earn higher scores, while being kept close to its earlier behavior.
DPO: Direct preference optimization. Skips the separate reward model and adjusts the model directly from pairs of preferred and rejected answers — simpler and widely used.
AI feedback: Some pipelines use models, guided by written principles, to generate or judge comparisons at scale, with humans checking samples.

Follow one comparison through both approaches:

AnimationFrom preferences to a better model

Loading animation…

Two answersThe supervised model writes two answers to the same prompt. Both are fluent; only one actually helps the user.
People compareA person picks the better answer. Comparing is far easier than writing the perfect answer, so thousands of comparisons can be collected.
A reward modelThe comparisons train a reward model: a network that scores any answer, giving higher scores to the kind of answers people prefer.
Optimize the modelReinforcement learning (typically PPO) tunes the language model to earn higher scores. A penalty keeps it close to the supervised model, so it cannot drift into odd text that merely fools the reward model.
DPO: a shortcutDirect preference optimization skips the reward model. It adjusts the language model straight from the pairs, making preferred answers more likely and rejected ones less likely.

This stage shapes tone, helpfulness and safety: declining harmful requests, admitting uncertainty, following formatting instructions.

Stage 5: evaluation never stops

Models are evaluated on public benchmarks, private task suites and red-teaming — deliberate attempts to make them misbehave. But benchmarks only approximate real use. Once deployed, companies monitor quality, collect feedback and retrain. You will build your own evaluation practice in module 3.

What this means for your team

You will almost never pre-train a model. The practical choice is how to adapt an existing one:

Approach	Best for	Watch out for
Prompting	Most tasks — start here	Long prompts cost tokens on every request
Retrieval (RAG)	Company knowledge, documents that change, answers that need citations	Quality depends on search and chunking
Fine-tuning	A consistent style or format, specialised classification, shorter prompts at high volume	Needs good example data; doesn’t reliably add fresh facts

That completes module 1: you can now explain, end to end, how a language model is trained and how it generates text. Module 2 puts that knowledge to work — prompts, context windows, retrieval and tool use.

In this lecture you will