Designing an AI feature end to end
Take a feature from specification to production: choose the model, ground it, evaluate it, guard it, monitor it, and plan the rollback before you ship.
The stages and their deliverables
| Stage | Deliverable |
|---|---|
| Specification | The user task, the acceptable failure rate, and what happens when it fails |
| Model choice | A named model and version, with the reason recorded |
| Prompting | A versioned prompt with an explicit output schema |
| Grounding | Retrieval over the corpus, with permission filters |
| Evaluation | A frozen test set and a script that scores it |
| Guards | Validation, refusal behaviour, timeouts and cost caps |
| Monitoring | Metrics, sampled review, and an alert threshold |
| Rollback | A feature flag and the documented way to turn it off |
The failure path is part of the design, not an afterthought. Decide before you build what the user sees when the model is unavailable, when confidence is low, and when the output fails validation.
A shape that holds up
def feature(request, user):
# 1. cheap deterministic work first
if not request.text.strip():
return fallback("empty input")
# 2. retrieve with permissions applied before ranking
hits = search(request.text, k=5, tenant=user.tenant)
if not hits:
return fallback("no_source") # honest empty rather than invented
# 3. one model call, temperature 0, bounded output
try:
raw = call_model(build_prompt(hits, request.text), timeout=8, max_tokens=400)
except Timeout:
return fallback("timeout") # a timeout is a normal outcome
except RateLimited:
return fallback("busy")
# 4. validate before anything downstream touches it
parsed = parse_schema(raw)
if parsed is None:
log_invalid(raw)
return fallback("invalid_output")
# 5. return sources so the user can verify
return {"answer": parsed["answer"], "sources": [h["url"] for h in hits]}- Every external call has a timeout and a bounded output. An unbounded call is an outage waiting for traffic.
- Validation failure is a normal branch, not an exception. Log it and serve a safe fallback.
- Return the sources: it makes the feature auditable and turns retrieval bugs into visible, fixable problems.
Shipping and watching
METRICS = {
"requests": counter("ai.feature.requests"),
"fallbacks": counter("ai.feature.fallbacks", tags=["reason"]),
"invalid_rate": gauge("ai.feature.invalid_output_rate"),
"latency_ms": histogram("ai.feature.latency_ms"),
"cost_per_request": gauge("ai.feature.cost"),
}- Ship behind a flag to a small cohort, with the fallback path active from the first request.
- Watch fallback reasons before you watch answer quality: a spike in timeouts or invalid output is visible immediately and explains most user complaints.
- Sample real answers for human review weekly, and add the failures you find to the frozen test set.
- Re-run the evaluation on every model, prompt or retrieval change, and treat a drop beyond your threshold as a blocked release, not a discussion.
- Keep the previous prompt and model version deployed and switchable, so rollback takes one configuration change.
💡
A feature that cannot be turned off in one step is not ready to ship. The flag, the fallback path and the alert threshold are part of the first release, not follow-up work, because the moment you most need them is the moment you cannot add them.
FAQ
What is the minimum viable evaluation for a first release?
A hundred labelled cases from real traffic, scored automatically for format validity and task correctness, plus a manual read of twenty outputs. That catches most embarrassing failures and gives you a baseline to compare against.
How do I control cost before it becomes a problem?
Cap output tokens per call, truncate retrieved context, use the smallest model that passes evaluation for routine cases, cache identical requests, and alert on cost per request rather than on the monthly bill.
Related
Evaluating AI features AI safety, privacy and data governance
Last refreshed 2026-09-18.