Question answering and reading comprehension

Extractive QA with SQuAD-style models, open-domain QA with retrieval, grounding answers in the passage, and abstaining when the answer is not present.

Extractive question answering

from transformers import pipeline

qa = pipeline("question-answering", model="deepset/roberta-base-squad2")

context = """The support plan costs 49 GBP per month and includes a four-hour
response target. Enterprise customers receive a one-hour response target and a
dedicated account manager."""
question = "What is the response time for enterprise customers?"

answer = qa(question=question, context=context)
print(answer)
# {'score': 0.94, 'start': 118, 'end': 124, 'answer': 'one-hour'}

# return the top candidates, not just the best one
answers = qa(question=question, context=context, top_k=5)
for a in answers:
    print(round(a["score"], 3), a["answer"], a["start"], a["end"])

• Extractive QA finds a span inside the provided context. It cannot say anything the context does not contain, which is its main safety property and its main limitation.
• The model returns character offsets, so you can highlight the source span and build a citation without a second model call.
• top_k lets you inspect alternatives. If the top answer is wrong and the third is right, the problem is ranking, not extraction.
• The context window still applies: a passage longer than the model's maximum is truncated silently, and the answer may be cut off. Truncate on sentence boundaries and document what was dropped.

Open-domain QA with retrieval

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

encoder = SentenceTransformer("all-MiniLM-L6-v2")
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve(question, chunks, vectors, k=8):
    q = encoder.encode([question], normalize_embeddings=True).astype("float32")
    scores = (vectors @ q[0])
    order = np.argsort(scores)[::-1][:k]
    return [(chunks[i], float(scores[i])) for i in order]

def answer(question, chunks, vectors, qa_pipe, min_score=0.05):
    candidates = retrieve(question, chunks, vectors, k=20)
    pairs = [(question, c) for c, _ in candidates]
    rerank_scores = reranker.predict(pairs)
    best = np.argsort(rerank_scores)[::-1][:5]

    for i in best:
        passage, dense = candidates[i]
        result = qa_pipe(question=question, context=passage)
        if result["score"] > min_score:
            return {
                "answer": result["answer"],
                "passage": passage[:200],
                "extraction_score": round(result["score"], 3),
                "retrieval_score": round(float(rerank_scores[i]), 3),
            }
    return {"answer": None, "reason": "no supported answer in the retrieved passages"}

print(answer("How much does the support plan cost?", chunks, vectors, qa))

Measure the two stages separately. Retrieval recall at k is a ceiling: if the right passage is never retrieved, no improvement in the reader can help. Once recall is high, look at the reader's exact-match score on passages you know contain the answer.

Making the system abstain

import re

def normalise(text):
    return re.sub(r"\s+", " ", text.lower()).strip()

def is_supported(answer, passage, threshold=0.6):
    """Cheap lexical check: does the answer appear in the passage at all?"""
    return normalise(answer) in normalise(passage)

def answer_with_abstention(question, passages, qa_pipe, min_extraction=0.1):
    best = {"answer": None, "confidence": 0.0, "reason": "no answer found"}
    for passage in passages:
        result = qa_pipe(question=question, context=passage)
        if result["score"] < min_extraction:
            continue
        if not is_supported(result["answer"], passage):
            continue                      # a hallucinated span is not a span
        if result["score"] > best["confidence"]:
            best = {"answer": result["answer"], "confidence": round(result["score"], 3),
                    "source": passage[:120], "reason": "supported"}
    return best

# and evaluate abstention explicitly: a good system says "I do not know" on
# unanswerable questions instead of returning the least-bad span
unanswerable = ["What is the refund period for gift cards?"]
for q in unanswerable:
    print(q, "->", answer_with_abstention(q, chunks[:5], qa)["reason"])

• Extractive models always return a span, even for an unanswerable question. Without an abstention rule the system is confidently wrong on exactly the questions where trust matters most.
• Calibrate the threshold on a set that contains unanswerable questions. A threshold tuned only on answerable ones will always be too permissive.
• A lexical support check catches the most common failure: the span comes from a passage that mentions the topic but not the answer. It is cheap and removes a whole class of error.
• Report the abstention rate alongside accuracy. A system that abstains 40% of the time is not the same product as one that answers everything and is right 85% of the time.

FAQ

Related

Semantic search and vector databases Evaluating NLP systems

Last refreshed 2026-09-18.