Semantic search

A help-desk search box receives “laptop won’t turn on”. The answer exists — an article titled “Computer will not start” — but it does not contain a single word of the question. Traditional search finds nothing; a frustrated employee opens a ticket. Semantic search closes this gap by comparing meaning instead of words. This lecture builds both kinds of search, shows where each one fails, and ends with the combination that production systems use.

Keyword search and BM25

Keyword search starts with an inverted index: for every word, the list of documents that contain it. A query looks up its words and ranks the matching documents. The standard ranking function for thirty years has been BM25, which improves on TF-IDF from the first lecture in two ways:

• Saturation (k1). The fifth mention of “printer” should not count as much as the first. The fraction above rises quickly and then levels off; it can never exceed k1 + 1.
• Length normalization (b). A 400-word guide mentions everything a few times. With b = 1 its score is scaled down in proportion to its length; with b = 0 length is ignored.

Inverted index

A map from each word to the documents that contain it, so a search looks up words instead of scanning documents.

BM25

The standard keyword ranking function: idf-weighted term frequency with saturation and length normalization.

Vector index

A data structure that finds the nearest vectors to a query vector quickly, such as HNSW or IVF.

Top-k

The k highest-scoring results a retriever returns.

Chunk

A passage of a longer document — often a few hundred words — that is embedded and retrieved on its own.

Semantic search with vectors

The previous lectures showed that embeddings place texts with similar meanings close together. Semantic search puts that to work: embed every document once, store the vectors in a vector index, and at query time embed the question with the same model and return the nearest documents.

Keyword, semantic or both?

Each approach fails where the other succeeds. Try the example queries below — or type your own — and compare the three result lists. The knowledge base has 16 help-desk articles; hover over a title to read the article.

The pattern is consistent:

Hybrid search runs both retrievers and fuses their rankings. The simplest robust method is reciprocal rank fusion (RRF): every list gives each document 1 / (k + rank), usually with k = 60, and the scores are added. RRF uses only ranks, so it never has to compare BM25 scores with cosine similarities.

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, util

articles = [a.title + " " + a.body for a in kb]                     # your knowledge base
bm25 = BM25Okapi([doc.lower().split() for doc in articles])
model = SentenceTransformer("all-MiniLM-L6-v2")
doc_vecs = model.encode(articles, normalize_embeddings=True)       # index once

def hybrid(query, k=60, top=5):
    kw = bm25.get_scores(query.lower().split()).argsort()[::-1][:20]
    sem = util.semantic_search(model.encode(query, normalize_embeddings=True), doc_vecs, top_k=20)[0]
    fused = {}
    for rank, i in enumerate(kw):
        fused[i] = fused.get(i, 0) + 1 / (k + rank + 1)
    for rank, hit in enumerate(sem):
        fused[hit["corpus_id"]] = fused.get(hit["corpus_id"], 0) + 1 / (k + rank + 1)
    return sorted(fused, key=fused.get, reverse=True)[:top]

Measuring search quality

Search is only as good as its evaluation. Collect a few hundred real queries from logs or tickets, mark the correct documents for each, and track:

• Recall@k — how often the right document appears in the top k. For RAG this matters most: a document the retriever misses can never reach the language model.
• Mean reciprocal rank (MRR) — how high the first correct result appears, on average.
• Click-through and ticket deflection — the business outcome: did people find the answer without asking a human?

This completes the text foundations: turning text into features, classifying it, representing meaning with embeddings, extracting entities and searching by meaning. The next module moves from single words and documents to sequences — language models, recurrent networks and the transformers behind today’s AI assistants.