1. Courses
  2. Natural Language Processing
  3. Lecture 2
Lecture 2 of 1417 minInteractiveFree lecture

Text classification

Spam filters, sentiment analysis and ticket routing all answer one question — which category does this text belong to? Follow a spam filter end to end, watch a Naive Bayes classifier weigh each word as you type, and choose a decision threshold that matches the business cost of mistakes.

In this lecture you will

  • Describe the pipeline from raw text to a predicted label
  • Explain how Naive Bayes and logistic regression score a document from its words
  • Choose a decision threshold using precision, recall and the cost of errors
  • Recognize when a simple classifier is enough and when to use a fine-tuned model

Text classification assigns a document to one or more categories. It is the most widely deployed form of NLP in business:

  • Spam and phishing filters — spam or not.
  • Sentiment analysis — positive, negative or neutral reviews and survey answers.
  • Ticket and email routing — send each message to billing, technical support or sales.
  • Content moderation and compliance — flag abusive posts or sensitive documents.

A classifier, end to end

Scroll through a spam filter at work. The same shape applies to every classifier in this lecture.

Scroll storyInside a spam filterStep 1 of 6
Loading visual…
  1. 1
    An email arrives

    The filter sees raw text: a subject line, a body and a sender. Nothing about it is a number yet.

  2. 2
    Tokenize

    The text is lowercased and split into tokens. Punctuation and case, like the shouting capitals, can be kept as extra features — spammers love them.

  3. 3
    Vectorize

    Each token becomes a count in a bag-of-words vector. Most of the 20,000 vocabulary entries are zero for any one email.

  4. 4
    Apply learned weights

    During training, the model learned a weight for every term: positive for words typical of spam (“prize”, “winner”), negative for words typical of real mail (“meeting”, “report”).

  5. 5
    Score and probability

    Multiply each count by its weight, add them up with a bias, and squash the score into a probability with the sigmoid function. Here P(spam) is 99%.

  6. 6
    Decide

    Compare the probability with a threshold. This email goes to the spam folder. Where to set that threshold is a business decision — more on that below.

This is logistic regression on bag-of-words features — the same model you met in Machine Learning Foundations, now reading text.

Naive Bayes: adding up the evidence

Naive Bayes is the classic text classifier, and still a strong baseline. From labeled examples it learns how much more often each word appears in one class than the other. To classify a new document, it adds up the evidence from every word:

log⁡P(pos∣words)P(neg∣words)=log⁡P(pos)P(neg)+∑wlog⁡P(w∣pos)P(w∣neg)\log \frac{P(\text{pos} \mid \text{words})}{P(\text{neg} \mid \text{words})} = \log \frac{P(\text{pos})}{P(\text{neg})} + \sum_{w} \log \frac{P(w \mid \text{pos})}{P(w \mid \text{neg})}

The classifier below was trained, in your browser, on 48 short product reviews. Type your own review.

InteractiveA Naive Bayes sentiment classifierType a review
Loading visual…

Teal words push towards positive, coral towards negative; the stronger the color, the stronger the evidence. Gray words were never seen in training and are ignored.

Notice both the strength and the weakness. The model is transparent — you can see exactly why it decided — but it has no idea about context: “not great” counts “great” as positive evidence, and sarcasm defeats it completely.

Features
The numbers a classifier sees: word counts, TF-IDF weights, n-grams, or embeddings from a language model.
Naive Bayes
Adds per-word log-likelihood ratios learned from counts. Assumes words are independent given the class.
Logistic regression
Learns a weight per feature by gradient descent and outputs a probability with the sigmoid.
Laplace smoothing
Adds one to every count so that unseen word–class combinations do not produce zero probabilities.
Multi-label
A document can belong to several classes at once — for example a ticket about both billing and login.

Choosing the threshold

A classifier outputs a probability; turning it into a decision needs a threshold. Where you put it decides which mistakes you make.

InteractiveSetting a spam filter’s thresholdDrag the threshold
Loading visual…

Scores for 1,000 legitimate emails and 1,000 spam emails. Everything right of the line goes to the spam folder.

With equal costs, a threshold near the middle minimizes total errors. Now set the cost of losing a good email to ten spam: the best threshold moves right — the filter lets more spam through to protect real mail. There is no universally correct threshold; there is only the right one for your costs.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2), LogisticRegression(max_iter=1000))
clf.fit(train_texts, train_labels)
probs = clf.predict_proba(new_texts)[:, 1]      # probability of spam
is_spam = probs >= 0.8                           # a threshold chosen from the costs

Bags of words treat “laptop” and “notebook” as unrelated. The next lecture gives words meaning with embeddings — vectors in which similar words sit close together.

Key takeaways

  • A text classifier turns a document into features, combines them with learned weights, and converts the score into a probability.
  • Naive Bayes adds up per-word evidence learned from labeled examples; it is fast, transparent and a strong baseline.
  • The decision threshold sets the balance between false positives and false negatives; pick it from the cost of each mistake.
  • Modern practice often fine-tunes a pretrained language model, but simple, explainable baselines remain valuable.

Check your understanding

4 questions · answers explained

Q1A spam filter moves 30 legitimate emails into the spam folder. What kind of error is that?

The positive class is “spam”. Legitimate emails wrongly predicted as spam are false positives. Spam that reaches the inbox would be false negatives.

Q2Why is Naive Bayes called “naive”?

It treats each word as independent evidence, ignoring that words like “New” and “York” go together. The assumption is wrong, yet the classifier often works surprisingly well.

Q3Losing a real customer email costs far more than letting one spam through. How should you move the threshold?

A higher threshold reduces false positives (good emails lost) at the cost of letting more spam through. When false positives are expensive, that is the right trade.

Q4What does add-one (Laplace) smoothing solve in Naive Bayes?

Without smoothing, a single word that never appeared in positive training reviews would force P(positive) to zero, whatever else the review says.