Tokens and embeddings

In the last lecture, the model predicted the next “token”. Now we look at what a token is, and at the question underneath every language model: how do you turn text into numbers that a network can compute with — without losing the meaning?

Step 1: split text into tokens

A tokenizer splits text into pieces from a fixed vocabulary, usually tens of thousands to a few hundred thousand entries. Most modern tokenizers use a method such as byte-pair encoding (BPE): start from single characters and repeatedly merge the most frequent neighboring pairs into new tokens. Common words end up as a single token; rare words are assembled from familiar pieces.

AnimationLearning a vocabulary with byte-pair encoding

Loading animation…

Start from charactersA tiny training corpus: “low” five times, “lower” twice, “newest” six times and “widest” three times. Every word starts as single characters plus an end-of-word marker.
Merge the most common pair“e s” appears 9 times — more than any other pair — so it becomes a new token, “es”.
RepeatWith “es” in the vocabulary, “es t” is now the most common pair, and becomes “est”.
Keep mergingThree more merges produce “est·”, “lo” and “low”. Each merge adds one entry to the vocabulary; real tokenizers perform tens of thousands.
Tokenizing new textThe unseen word “lowest” is split with the learned merges into two tokens: “low” and “est·”. Nothing is ever out of vocabulary — in the worst case, text falls back to characters.

InteractiveHow text becomes tokensTry the examples

Loading visual…

Splits and IDs are illustrative; every model family has its own tokenizer and vocabulary.

Two practical consequences follow:

Tokens are the unit of cost. API prices, rate limits and the context window are all counted in tokens. For English, a rough rule is about four characters, or three-quarters of a word, per token.
Not all text is equal. Code, numbers and many non-English languages need more tokens for the same meaning. A system that is cheap in English can cost noticeably more in other languages.

Estimate what a feature will cost before building it. Change the kind of text, the document size and the traffic.

InteractiveTokens, context and costChange the text, the size and the price tier

Loading visual…

Push the document size up until it no longer fits the context window. Then compare price tiers at 2,000 requests a day: the model choice often changes the monthly bill by a factor of 100. Ratios are rough averages; always measure with your model’s own tokenizer.

Step 2: turn each token into a vector

The model never sees the token text — only its ID, an integer like 18,442. The first layer of the model looks up each ID in a large table and replaces it with a list of numbers called an embedding — typically hundreds to several thousand numbers long.

These numbers are not assigned by hand. They start random and are adjusted during training, like every other parameter. Because the model is rewarded for predicting text well, tokens that behave similarly in language end up with similar vectors. Meaning becomes geometry.

Explore an embedding space

Real embeddings have far too many dimensions to draw, so this model shows an illustrative 3D projection. Step through it and rotate the space to see the structure from different angles.

3D modelMeaning as geometryDrag to rotate

Loading 3D model…

Tokens are just IDsBefore training, each token is an arbitrary number. #18442 has no more in common with #3921 than with any other ID — the model must learn what they mean.
Each ID becomes a pointThe embedding table maps every token to a vector — a point in a space with hundreds or thousands of dimensions. We show three of them here.
Similar meanings clusterAfter training, words used in similar contexts sit close together: animals near animals, places near places, programming terms near each other.
Directions carry relationshipsThe step from a country to its capital points in roughly the same direction every time. The space encodes relationships, not only similarity.
Vector arithmeticThe step from dog to puppy matches the step from cat to kitten. That is why “kitten” lands near cat + (puppy − dog) — the famous embedding analogies.
Search by meaningEmbed a query — “young cat” — and find the nearest vectors: kitten, cat, puppy. This nearest-neighbour search is the engine of semantic search and retrieval.

Measuring closeness

To compare two embeddings, systems usually use cosine similarity: the cosine of the angle between the two vectors. It ranges from −1 (opposite) to 1 (pointing the same way):

\cos(\theta) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}

Cosine similarity ignores the vectors’ length and looks only at direction, which works well for comparing meaning.

Token embeddings and text embeddings

There are two related ideas worth separating:

Token embeddings: The input vectors inside a language model — one per token. They are refined layer by layer as the model reads the context, as you will see in the next two lectures.
Text embeddings: A single vector for a whole sentence, paragraph or document, produced by a dedicated embedding model. These are what you store in a vector database for search.

Text embeddings power a surprising range of features: semantic search that finds documents by meaning rather than keywords, clustering of support tickets or reviews, deduplication, recommendations — and retrieval-augmented generation (RAG), where relevant documents are retrieved by embedding similarity and handed to an LLM. You will build that in module 2.

Each token now has a vector, but on its own that vector knows nothing about its neighbors. In “river bank” and “bank loan”, the token bank starts with the same embedding. The next lecture shows how attention lets context flow between tokens.

In this lecture you will