Choosing between prompting, RAG and fine-tuning

Compare the three ways to change model behaviour by cost and effort, and know which problems fine-tuning genuinely solves.

The three options

Approach	Changes	Effort	Best for
Prompting	Instructions and examples	Minutes to hours	Tone, format, simple classification, one-off tasks
RAG	What the model can see at answer time	Days	Questions about private or changing facts, citations
Fine-tuning	The weights themselves	Weeks	Consistent format at scale, niche vocabulary, distilling a large model
Fine-tuning plus RAG	Both	Weeks	A specialised assistant grounded in your data

The order to try them

Improve the prompt: make the task, format and refusal behaviour explicit, and add two or three examples.
Add retrieval if the failure is about missing or changing knowledge rather than instruction-following.
Add tool calls if the model needs live data or an action rather than more text.
Fine-tune only when the first three are exhausted and the failure is specifically about style, format consistency or cost at high volume.

# a cheap test of whether fine-tuning would help at all
def diagnose(cases, baseline_fn):
    buckets = {"missing_knowledge": 0, "format": 0, "reasoning": 0, "refusal": 0}
    for case in cases:
        out = baseline_fn(case["input"])
        if not case["recall_hit"]:
            buckets["missing_knowledge"] += 1
        elif not is_valid_schema(out):
            buckets["format"] += 1
        elif out.strip().lower() != case["expected"].strip().lower():
            buckets["reasoning"] += 1
    return buckets

# mostly "missing_knowledge"  -> retrieval
# mostly "format"            -> prompting first, then fine-tuning
# mostly "reasoning"         -> a stronger model, not a fine-tune

⚠️

Fine-tuning teaches form, not facts. Training on your documents does not reliably make a model remember them, and it cannot keep up with data that changes. Use it to shape behaviour, and retrieval to supply knowledge.

When fine-tuning earns its keep

# a typical adapter configuration: small, cheap, reversible
from peft import LoraConfig

cfg = LoraConfig(
    r=16,                      # rank of the low-rank update
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

# the dataset is the expensive part: thousands of input/output pairs
# in the exact format you want at inference time

You have thousands of consistent examples, not dozens. A hundred rows will not move the model measurably.
The task is narrow and stable, such as always emitting a fixed JSON schema or a house writing style.
You are distilling: using a large model to generate training data so a small, cheap model can serve production traffic.
Cheaper inference matters more than the one-off training cost, which is usually the real justification at volume.

FAQ

Will fine-tuning make the model stop hallucinating?

No. It changes the distribution of outputs, not the model's ability to verify truth. If hallucinations are the problem, ground the answers with retrieval and add a refusal path.

How do I evaluate whether a fine-tune helped?

Hold out a test set that was never in training, measure task accuracy and format validity for both the base model with a good prompt and the tuned model, and compare cost and latency too. If the tuned model does not beat a well-prompted baseline, keep the baseline.

Retrieval-augmented generation Evaluating AI features

Last refreshed 2026-09-18.

Choosing between prompting, RAG and fine-tuning

The three options

The order to try them

When fine-tuning earns its keep

FAQ

Related