Text as data

Most of an organization’s knowledge is written down: support tickets, contracts, emails, chat logs, product reviews, meeting notes. Natural language processing (NLP) is the set of techniques that let software search, sort, understand and generate this text. It powers spam filters, search boxes, document routing, translation and — at the far end of the scale — large language models.

Every one of these systems starts with the same problem: models work with numbers, and text is not numbers. This lecture covers the first step: turning raw text into something a model can use.

From characters to tokens

Text arrives as a string of characters. The first step is to split it into tokens — usually words, numbers and sometimes punctuation. Then, depending on the task, we normalize the tokens so that different spellings of the same idea look alike.

InteractiveA text-preparation pipelineType your own text

Loading visual…

Each step is a trade-off. Stemming merges “printers” and “printing”, which helps search — but can also merge words with different meanings. Removing “not” would flip the meaning of a review.

Token: A unit of text a system works with — usually a word, number or punctuation mark. (Language models use smaller subword tokens; see the Generative AI course.)
Normalization: Making equivalent text look the same: lowercasing, fixing Unicode quirks, expanding contractions.
Stop words: Very common words such as “the”, “and”, “of” that carry little topical meaning.
Stemming: Chopping word endings with simple rules: printing → print. Fast but crude.
Lemmatization: Reducing a word to its dictionary form using grammar: better → good, ran → run. More accurate than stemming.
n-gram: A sequence of n neighboring tokens. Bigrams (n = 2) capture short phrases such as “not happy”.

Documents as vectors: bag of words and TF-IDF

The simplest way to turn a document into numbers is to count its words. Build a vocabulary of every term in the collection, and represent each document as a vector of counts — a bag of words, because the order has been thrown away.

InteractiveFrom tickets to a document–term matrixHover a column

Loading visual…

Hover over a term to highlight it in the tickets. The matrix is mostly empty: real vocabularies have tens of thousands of terms, and each document uses only a few.

Raw counts over-reward common words. TF-IDF fixes that by multiplying each count by how rare the term is across the collection:

\text{tf-idf}(t, d) = \text{tf}(t, d) \times \log \frac{N}{\text{df}(t)}

where $\text{tf}(t, d)$ is how often term $t$ appears in document $d$ , $N$ is the number of documents and $\text{df}(t)$ is how many documents contain $t$ . A word in every document scores $\log 1 = 0$ .

Why a few words dominate: Zipf’s law

Count words in any large body of text and a striking pattern appears.

AnimationZipf’s law

Loading animation…

The most frequent wordsSort words by how often they appear. The bars show the idealized share of all words for the twenty most common English words.
A handful dominateThe top ten — the, of, and, to, a… — make up roughly a quarter of everything written. They are function words that carry grammar rather than topic.
The long tailTens of thousands of other words share the rest. In a typical collection, about half of the distinct words appear only once.
A straight line on log scalesPlot frequency against rank on logarithmic axes and the points fall on a straight line: frequency is roughly proportional to 1 / rank. This is Zipf’s law.
What it means in practicePipelines drop stop words because they add little meaning, and prune very rare words because they add noise and bloat the vocabulary.

Putting it together

A typical classical pipeline in Python with scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer

tickets = [
    "Printer not printing after the update",
    "Laptop battery not charging after the update",
    "Cannot log in, password reset needed",
    "Printer paper jam, printer shows offline",
]
vectorizer = TfidfVectorizer(lowercase=True, stop_words="english", ngram_range=(1, 2))
X = vectorizer.fit_transform(tickets)        # sparse matrix: 4 documents × vocabulary
print(X.shape)
print(vectorizer.get_feature_names_out()[:8])

With documents turned into vectors, we can train a model to sort them. The next lecture builds text classifiers — the engines behind spam filters and ticket routing.

In this lecture you will