Embeddings and semantic search
Turn text into vectors, compare them with cosine similarity, chunk documents sensibly, and build a working search over your own corpus.
What an embedding is
import numpy as np
from openai import OpenAI
client = OpenAI()
def embed(texts, model="text-embedding-3-small"):
resp = client.embeddings.create(model=model, input=texts)
return np.array([d.embedding for d in resp.data], dtype=np.float32)
docs = [
"Refunds are processed within five working days.",
"Cancel a subscription from the billing page.",
"The API rate limit is 600 requests per minute.",
]
E = embed(docs)
E /= np.linalg.norm(E, axis=1, keepdims=True) # normalise once
query = embed(["how long do refunds take?"])[0]
query /= np.linalg.norm(query)
scores = E @ query # cosine similarity
print(scores.round(3), docs[int(scores.argmax())])- An embedding maps text to a fixed-length vector; texts with similar meaning end up close together.
- Cosine similarity ignores magnitude, so normalising makes the dot product the cosine. Do it once at write time.
- Embeddings capture meaning, not freshness, permissions or exact identifiers. Those still need filters.
Chunking
| Strategy | Use when | Watch out for |
|---|---|---|
| Fixed token window | A first pass on unstructured text | Splits sentences and tables down the middle |
| By paragraph or heading | Documentation, policies, articles | Very uneven chunk sizes |
| Overlapping windows | Long prose where context spans boundaries | Duplicated text in the results |
| One row or record | Structured data, tickets, products | Chunks too short to carry meaning |
| Parent and child | Long documents needing precise hits | More storage and a lookup step |
def chunk(text, size=800, overlap=120):
words = text.split()
step = size - overlap
return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
chunks = chunk(open("handbook.txt").read())
print(len(chunks), len(chunks[0].split()))Include the document title and section heading in each chunk's text, not just in metadata. The heading is often the strongest signal of what the chunk is about, and it costs a few tokens.
Storing and querying
import json, numpy as np
def build_index(path, out="index.npz"):
chunks = []
for line in open(path):
rec = json.loads(line)
chunks.append(rec)
vectors = embed([c["text"] for c in chunks])
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
np.savez(out, vectors=vectors)
with open(out + ".meta.jsonl", "w") as f:
for c in chunks:
f.write(json.dumps(c) + "\n")
def search(query, k=5, min_score=0.25):
z = np.load("index.npz")
q = embed([query])[0]
q /= np.linalg.norm(q)
scores = z["vectors"] @ q
order = np.argsort(-scores)[:k]
return [(float(scores[i]), i) for i in order if scores[i] >= min_score]⚠️
Never mix vectors produced by two different models, or two different versions of the same model, in one index. The spaces are unrelated, so similarities become meaningless noise and retrieval quietly degrades rather than failing loudly. Record the model and version beside the index.
FAQ
How large should a chunk be?
Large enough to answer a question on its own, small enough that the answer dominates the vector. A few hundred tokens is a reasonable start for prose; measure retrieval quality on a small labelled set and adjust from there.
Do I need a vector database?
No, for up to a few hundred thousand vectors. A normalised matrix and a dot product are fast enough, and you avoid another service. Move to a dedicated index when you need filtering plus scale, incremental updates, or sub-millisecond latency.
Related
Retrieval-augmented generation Using a model API
Last refreshed 2026-09-18.