Word embeddings
Embeddings give words meaning by placing them in space, so that similar words sit close together and consistent directions encode relationships. Explore a word space in 3D, measure similarity with cosines, and see how word2vec learns from context.
In this lecture you will
- Explain what a word embedding is and why it beats one-hot or bag-of-words features
- Measure similarity between vectors with cosine similarity
- Describe how word2vec learns embeddings from context
- Recognize the limits of static word embeddings and how contextual embeddings fix them
In the last lecture every word was an independent column in a table. “Laptop” and “notebook” were as unrelated as “laptop” and “pizza”, so a model that learned about one learned nothing about the other. Word embeddings fix that by placing each word at a point in space, where distance means difference in meaning.
Words as points in space
An embedding is a dense vector — typically 100 to 1,000 numbers — for each word. Nobody assigns the numbers by hand: they are learned from how words are used. The 3D view below shows a small embedding built for this course, reduced from 15 dimensions to the three that vary most.
- Words as vectorsEach word is a point. The model sees 15 numbers per word; we show the three most informative directions, found with principal component analysis.
- Similar words clusterAnimals, vehicles, places, technology, food and people form separate neighborhoods — the geometry reflects meaning.
- Directions carry meaningThe arrows from man to woman, king to queen and boy to girl point the same way: one direction in the space corresponds to gender.
- Analogy arithmeticStart at king, subtract man and add woman. The nearest word to the result is queen. Relationships have become vector arithmetic.
- Other relationshipsCountry → capital is another consistent direction: France → Paris runs parallel to Italy → Rome and Japan → Tokyo.
Measuring similarity: the cosine
To compare two embeddings we compare their directions. The cosine of the angle between two vectors is 1 when they point the same way, 0 when they are unrelated (at right angles), and −1 when they point in opposite directions:
Make the arrows longer without changing their direction: the dot product grows, but the cosine stays the same. On the right, the nearest neighbors of real words in the course embedding.
Look at the neighbors of apple: food words, but also a hint of technology. A single vector has to represent every sense of a word — a limitation we come back to below.
- One-hot vector
- A vector with a single 1 for the word’s position in the vocabulary. Every pair of words is equally different.
- Embedding
- A dense, learned vector in which similar words have similar values.
- Cosine similarity
- Similarity of direction between two vectors, from −1 to 1.
- Nearest neighbors
- The vectors most similar to a query vector — the basis of semantic search.
- Dimensionality reduction
- Projecting high-dimensional vectors to 2D or 3D for display, for example with PCA or t-SNE.
Learning embeddings from context: word2vec
In 2013 a Google team showed that excellent embeddings can be learned with a simple trick: train a small network to predict a word’s neighbors. Words that appear in similar contexts are forced towards similar vectors.
- Meaning from companyThe idea behind word2vec: words used in similar contexts tend to have similar meanings.
- Slide a windowA window slides along the training text. For each center word, the words within two positions become its context, producing (center, context) training pairs.
- Billions of pairsRepeated over a large corpus, this produces billions of pairs. A shallow network learns to predict context words from the center word.
- Vectors drift togetherEach training step nudges the vectors. Words that keep appearing near the same neighbors — router, wifi, modem — move together.
- The result: embeddingsAfter training, the network’s input weights are the word embeddings. Unrelated words, such as pizza and invoice, stay far away.
From static to contextual embeddings
Word2vec and its cousin GloVe produce static embeddings: one vector per word, whatever the sentence. “Bank” gets the same vector in “river bank” and “bank account”. Modern transformer models such as BERT produce contextual embeddings: every occurrence of a word gets its own vector, computed from the whole sentence. The Generative AI course shows the attention mechanism that makes this possible.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2") # a small contextual embedding model
vectors = model.encode(["My laptop will not turn on", "Computer does not boot", "Pizza delivery"])
print(util.cos_sim(vectors[0], vectors[1])) # high: same meaning, different words
print(util.cos_sim(vectors[0], vectors[2])) # low: unrelatedEmbeddings tell us what words mean. Many applications also need to pull out specific facts — names, companies, dates, amounts. That is named-entity recognition, the next lecture.
Key takeaways
- An embedding represents each word as a dense vector; words with similar meanings get similar vectors.
- Cosine similarity compares directions and is the standard similarity measure for embeddings.
- Word2vec learns vectors by predicting a word’s context, so words used in similar contexts end up close together.
- Static embeddings give one vector per word; contextual embeddings from transformers give each occurrence its own vector.
Check your understanding
4 questions · answers explainedQ1Why is cosine similarity preferred over raw distance for comparing embeddings?
Q2In a good embedding, what does king − man + woman land closest to?
Q3How does word2vec decide that “router” and “modem” should have similar vectors?
Q4What is the main weakness of a static embedding such as word2vec for the word “bank”?