Classical text classification

TF-IDF, naive Bayes and linear models, and why a well-tuned non-neural baseline is the number you have to beat before reaching for a transformer.

Bag of words and TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

texts = [
    "the delivery was fast and the box was intact",
    "terrible service, my package arrived broken",
    "refund requested, item never shipped",
    "great quality, would order again",
]
labels = ["positive", "negative", "negative", "positive"]

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(
        ngram_range=(1, 2),        # unigrams and bigrams
        min_df=1,                  # drop terms appearing fewer than n times
        max_df=0.9,                # drop terms appearing in over 90% of docs
        sublinear_tf=True,         # 1 + log(tf) instead of raw counts
        strip_accents="unicode",
        lowercase=True,
    )),
    ("clf", LogisticRegression(max_iter=1000, C=4.0)),
])
pipeline.fit(texts, labels)
print(pipeline.predict(["the item arrived broken again"]))

TF-IDF downweights terms that appear everywhere. That is the whole point: the is frequent but useless, refund is rarer and decisive.
sublinear_tf=True compresses the effect of repeating a word twenty times, which is usually closer to how meaning scales.
Bigrams recover a little word order and catch negation such as not good. Trigrams add little and multiply the feature count.
Fit the vectoriser on the training split only. Fitting on the full corpus leaks term statistics and inflates test scores.

Naive Bayes versus linear models

from sklearn.naive_bayes import MultinomialNB, ComplementNB
from sklearn.svm import LinearSVC
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import cross_val_score

models = {
    "multinomial_nb": MultinomialNB(alpha=0.1),
    "complement_nb": ComplementNB(alpha=0.3),
    "logreg": LogisticRegression(max_iter=2000, C=4.0),
    "linear_svc": LinearSVC(C=1.0),
    "sgd_log": SGDClassifier(loss="log_loss", alpha=1e-5, max_iter=50),
}

for name, clf in models.items():
    scores = cross_val_score(
        Pipeline([("t", TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)),
                  ("c", clf)]),
        texts, labels, cv=2, scoring="f1_macro")
    print(f"{name:16s} {scores.mean():.3f}")

Model	Strengths	Weaknesses
MultinomialNB	Extremely fast, tiny data works	Features assumed independent
ComplementNB	Better on imbalanced classes	Less standard, still linear
LogisticRegression	Calibrated probabilities, interpretable weights	Needs scaling and regularisation tuning
LinearSVC	Often the best accuracy on sparse text	No probabilities without extra calibration
SGDClassifier	Streams data, fits in constant memory	Needs careful learning-rate tuning

Because the features are sparse and the decision boundary is close to linear in TF-IDF space, a regularised linear model is a genuinely strong baseline. On many short-text tasks it lands within a few points of a fine-tuned transformer, at a thousandth of the cost.

💡

Always compare against a majority-class baseline before celebrating any score. If 92% of the data is one class, a model reporting 92% accuracy has learned nothing at all — report macro F1 alongside accuracy.

Inspecting what the model learned

import numpy as np

vec = pipeline.named_steps["tfidf"]
clf = pipeline.named_steps["clf"]
terms = np.array(vec.get_feature_names_out())

for class_index, class_name in enumerate(clf.classes_):
    weights = clf.coef_[class_index]
    top = np.argsort(weights)[-12:][::-1]
    print(class_name, list(terms[top]))

from sklearn.metrics import classification_report, confusion_matrix
pred = pipeline.predict(texts)
print(confusion_matrix(labels, pred))
print(classification_report(labels, pred, zero_division=0))

Top-weighted terms are a quick sanity check. Leaked metadata (an order id, a date, a customer name) usually appears near the top and is the signature of a data problem.
Inspect the confusion matrix by class, not just the total. A model that never predicts the rare class can still look accurate.
Read a sample of misclassified rows by hand. Twenty examples tell you more than another hundredth of a point of accuracy.
Store the fitted vectoriser with the model: the vocabulary and IDF weights are part of the classifier, and re-fitting on new data silently changes the feature space.

FAQ

When should I move to a transformer?

When the linear baseline's errors are semantic rather than lexical: sarcasm, paraphrase, long context, or domain vocabulary that sparse features cannot generalise across. If the errors are typos and rare words, better preprocessing may be enough.

Does stemming or lemmatisation help?

Sometimes, modestly, and less than expected with ngrams and sublinear TF. Test it: stemming can merge unrelated terms (university and universe) and cost accuracy as often as it gains.

Text preprocessing and tokens Working with large language models for NLP tasks

Last refreshed 2026-09-18.

Classical text classification

Bag of words and TF-IDF

Naive Bayes versus linear models

Inspecting what the model learned

FAQ

Related