The gap between an AI prototype and a production feature is mostly a measurement gap. In a prototype, quality is judged by whoever is demonstrating it. In production, quality has to be something the team can assert on a Tuesday afternoon after changing a prompt.
Build the evaluation set first
Before tuning anything, assemble a set of real inputs with known-good outputs. Thirty to fifty carefully chosen examples are more useful than a thousand arbitrary ones, provided they cover the range of what actually arrives.
- Ordinary cases that represent the bulk of real volume
- Known-hard cases that experienced staff find genuinely ambiguous
- Adversarial or malformed inputs, including prompt-injection attempts in user-supplied content
- Cases where the correct answer is "I do not know" or "insufficient information"
- Cases where the source material contradicts itself
Decide how outputs are graded
Grading is the hard part. Exact matching only works for narrow extraction tasks. Human review is accurate and slow. Model-based grading is fast and needs its own validation — check that the grader agrees with human judgement on a sample before trusting it at scale.
For most product features, a small number of specific criteria beats a single quality score: is it factually grounded in the source, does it follow the required format, does it avoid stating things not present in the evidence, and does it abstain appropriately.
Treat evaluation as a regression suite
Every change to a prompt, a retrieval strategy, a chunking rule, a model version or a system instruction should be scored against the evaluation set before it ships. Without that, teams accumulate undocumented prompt changes and lose the ability to explain why quality moved.
Name the failure modes explicitly
- Fabrication — asserting specifics not present in any source
- Retrieval miss — the right document existed but was not surfaced
- Stale grounding — the answer reflects superseded content
- Permission leakage — content surfaced to someone who should not see it
- Overconfidence — no signal of uncertainty where the evidence is weak
- Format drift — output that breaks downstream parsing or display
Permission leakage deserves particular attention. If retrieval is not filtered by the requesting user’s access rights, an assistant becomes a very efficient way to read documents you were not entitled to see.
Design abstention deliberately
A system that always answers is a system that will sometimes invent. Decide what evidence threshold justifies an answer, what the interface shows when that threshold is not met, and how a user escalates from there. Well-designed abstention increases trust rather than signalling weakness.
Monitor after launch
- Acceptance, edit and override rates — the clearest ongoing quality signal you have
- Retrieval quality: how often relevant sources were returned at all
- Abstention rate, watched for sudden movement in either direction
- Latency and cost per interaction
- A sampled human review on a fixed cadence
- Alerting on provider model or version changes
Source content drifts, user behaviour shifts and providers update models. Monitoring is what turns those from incidents into maintenance.