Text similarity, clustering and topic modelling

Document similarity, near-duplicate detection, clustering without a fixed k, and using LDA or BERTopic to get topics you can actually label.

Similarity and near-duplicates

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = [
    "How do I reset my password?",
    "I forgot my password and cannot log in",
    "Reset password instructions",
    "The refund policy for physical goods",
]

vec = TfidfVectorizer(ngram_range=(1, 2)).fit(docs)
sim = cosine_similarity(vec.transform(docs))
print(np.round(sim, 2))

# lexical near-duplicates: character shingles plus MinHash-style hashing
from datasketch import MinHash, MinHashLSH

def minhash(text, num_perm=128, k=5):
    m = MinHash(num_perm=num_perm)
    tokens = text.lower().split()
    shingles = [" ".join(tokens[i:i + k]) for i in range(max(0, len(tokens) - k + 1))]
    for s in shingles:
        m.update(s.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for i, d in enumerate(docs):
    lsh.insert(str(i), minhash(d))
print(lsh.query(minhash("password reset help")))

TF-IDF cosine similarity is fast, interpretable and has no training. It fails on paraphrase (car versus automobile) where embeddings succeed.
Rescale embeddings to unit length once, then similarity is a single matrix multiply — much faster than computing cosine per pair.
Near-duplicate detection is a different problem from semantic similarity: MinHash with LSH finds exact-ish copies at scale, which embeddings are bad at because a copy and a paraphrase score nearly the same.
Never deduplicate train and test separately. Cross-split duplicates are the most common way a text classifier gets a fake 99% score.

Clustering

from sklearn.cluster import KMeans, AgglomerativeClustering
from sklearn.metrics import silhouette_score
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2")
X = encoder.encode(docs, normalize_embeddings=True)

# k-means needs k; sweep it and look for an elbow and a silhouette peak
for k in range(2, min(8, len(docs))):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X)
    print(k, round(silhouette_score(X, labels), 3))

# when you do not know k, HDBSCAN finds clusters and labels the rest as noise
from sklearn.cluster import HDBSCAN
labels = HDBSCAN(min_cluster_size=2, metric="euclidean").fit_predict(X)
print(labels)      # -1 marks an outlier

# agglomerative with a distance threshold needs no k either
agg = AgglomerativeClustering(n_clusters=None, distance_threshold=0.6,
                              metric="cosine", linkage="average")
print(agg.fit_predict(X))

Cluster the reduced representation (UMAP to 5-50 dimensions), not the raw 384-dimensional vectors: density methods degrade badly in high dimensions.
Silhouette score on cosine distance is not valid; compute it on normalised vectors with Euclidean distance, which is equivalent up to a monotone transform.
k-means forces every document into a cluster, which is wrong for a support inbox where many messages are one-offs. HDBSCAN's noise label is often the more honest answer.
Name clusters by their most distinctive terms in TF-IDF, not by the most frequent terms, or every cluster will be labelled with the same stopwords.

Topic modelling

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

counts = CountVectorizer(
    max_df=0.95, min_df=2, stop_words="english", ngram_range=(1, 2))
X_counts = counts.fit_transform(docs)

lda = LatentDirichletAllocation(
    n_components=5, max_iter=20, learning_method="batch", random_state=0)
lda.fit(X_counts)

vocab = counts.get_feature_names_out()
for index, topic in enumerate(lda.components_):
    top = topic.argsort()[-8:][::-1]
    print(index, ", ".join(vocab[top]))

# BERTopic: embeddings plus clustering plus class-based term weighting
from bertopic import BERTopic
topic_model = BERTopic(min_topic_size=5, calculate_probabilities=False)
topics, probs = topic_model.fit_transform(docs)
print(topic_model.get_topic_info().head())

Method	Input	Needs k?	Best for
LDA	Document-term counts	Yes	Long documents, interpretable word distributions
NMF	TF-IDF matrix	Yes	Shorter texts, sharper topics than LDA
KMeans on embeddings	Dense vectors	Yes	When documents are short and semantic
BERTopic / HDBSCAN	Dense vectors	No	Mixed-quality corpora with outliers
Top2Vec	Dense vectors	No	Auto-sizing, fewer knobs than BERTopic

⚠️

Topic models do not produce labels, only word distributions. The numbers are a starting point for a human to name, and a topic whose top words are incoherent is almost always a sign of chunking or preprocessing, not of a bad model.

FAQ

Should I use LDA or embedding-based topics?

Use BERTopic-style pipelines on short, modern text and LDA or NMF when you need a stable, reproducible decomposition of long documents with term-level interpretability. Compare on whether the topics are actionable, not on a coherence number alone.

How do I handle duplicated reviews in training data?

Deduplicate before splitting, using both an exact hash and a near-duplicate check. Cross-split duplicates inflate every metric and create a model that has memorised rather than generalised.

Word and sentence embeddings Semantic search and vector databases

Last refreshed 2026-09-18.

Text similarity, clustering and topic modelling

Similarity and near-duplicates

Clustering

Topic modelling

FAQ

Related