Summarisation

Extractive TextRank, abstractive sequence-to-sequence and transformer summarisers, length control, and checking that the summary is actually faithful.

Extractive summarisation

import re
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

def split_sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])s+", text) if len(s.strip()) > 30]

def textrank(text, n=3):
    sentences = split_sentences(text)
    if len(sentences) <= n:
        return sentences

    tfidf = TfidfVectorizer(stop_words="english").fit_transform(sentences)
    sim = cosine_similarity(tfidf)
    np.fill_diagonal(sim, 0)

    # row-normalise into a transition matrix, then power-iterate the stationary vector
    row_sums = sim.sum(axis=1, keepdims=True)
    row_sums[row_sums == 0] = 1
    transition = sim / row_sums

    scores = np.ones(len(sentences)) / len(sentences)
    for _ in range(100):
        scores = 0.85 * transition.T @ scores + 0.15 / len(sentences)

    chosen = np.argsort(scores)[::-1][:n]
    return [sentences[i] for i in sorted(chosen)]

print(textrank(article, n=3))

Extractive summaries cannot hallucinate: every sentence is copied from the source verbatim. That property alone makes them the right choice for compliance, medical and legal text.
TextRank ranks sentences by centrality in a similarity graph. The damping factor and the choice of similarity function dominate the output more than the iteration count.
Score sentences with position and length priors as well as centrality: the opening sentence and unusually long sentences are often informative.
Use Maximal Marginal Relevance to pick sentences so that they are relevant and mutually non-redundant rather than all about the same topic.

Abstractive summarisation

from transformers import pipeline

summariser = pipeline("summarization", model="facebook/bart-large-cnn", device=0)

def summarise(text, max_len=130, min_len=40):
    # BART has a 1024-token limit; split long inputs on paragraph boundaries
    result = summariser(text, max_length=max_len, min_length=min_len,
                        do_sample=False, truncation=True)
    return result[0]["summary_text"]

print(summarise(article[:4000]))

# length control: max_length is a ceiling, not a target.
# Asking for a 60-token summary of a 200-token document fights the model's training prior.
for target in (40, 80, 120):
    out = summariser(article[:2000], max_length=target, min_length=int(target * 0.6),
                     do_sample=False, truncation=True)[0]["summary_text"]
    print(target, len(out.split()), out[:80])

Set do_sample=False for a summary. Sampling produces fluent but different text on every run and makes evaluation impossible.
A model trained with a target length distribution will fight an extreme length request. Summarising a 200-word note in 10 words produces truncation rather than compression.
Long documents must be chunked. Summarise each part and then summarise the summaries, keeping the intermediate summaries so a citation can be traced back.
Sentence-level abstractive models frequently copy dates, names and numbers incorrectly. Restrict abstractive output to prose and keep identifiers in a structured field.

Evaluating faithfulness

from rouge_score import rouge_scorer
import numpy as np

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
scores = scorer.score(reference_summary, generated_summary)
print({k: round(v.fmeasure, 3) for k, v in scores.items()})

# entity-level factuality: do the numbers in the summary appear in the source?
import re

def numbers(text):
    return set(re.findall(r"\b\d+[\.,]?\d*\b", text))

def unsupported_numbers(summary, source):
    return numbers(summary) - numbers(source)

print(unsupported_numbers(generated_summary, source_article))    # should be empty

# embed the summary and each source sentence to check entailment-style support
from sentence_transformers import SentenceTransformer, util
enc = SentenceTransformer("all-MiniLM-L6-v2")

def max_support(sentence, source_sentences):
    a = enc.encode(sentence, normalize_embeddings=True)
    b = enc.encode(source_sentences, normalize_embeddings=True)
    return float(util.cos_sim(a, b).max())

print([round(max_support(s, split_sentences(article)), 3)
       for s in split_sentences(generated_summary)])

Metric	Measures	Blind to
ROUGE-1/2/L	N-gram and longest-common-subsequence overlap	Paraphrase, and any unsupported content
BERTScore	Semantic similarity to the reference	Faithfulness to the source
FactCC / entailment	Whether each claim follows from the source	Fluency and coverage
Entity overlap	Missing or invented names and numbers	Everything outside those entities
Human rating	Coherence, coverage, factual accuracy	Cost and scale

⚠️

A high ROUGE score says the summary overlaps with the reference, not that it is true. A summary containing a fabricated figure scores well because the rest of the sentence matches. Always run a separate unsupported-content check, and never ship an abstractive summary of a high-stakes document without one.

FAQ

Extractive or abstractive?

Extractive when faithfulness is non-negotiable or the source is long and technical. Abstractive when the input is conversational, the audience needs compression across many sources, or the source is already a transcript full of disfluency.

Why is my summary always roughly the same length?

The model's training data biases it toward a target length and the beam search stops when it produces an end-of-sequence token. Control length with explicit prompting for a transformer LLM, or post-process and re-generate rather than assuming max_length will be respected.

Evaluating NLP systems Working with large language models for NLP tasks

Last refreshed 2026-09-18.

Summarisation

Extractive summarisation

Abstractive summarisation

Evaluating faithfulness

FAQ

Related