A demo usually contains one good question. A production workload contains awkward questions, incomplete evidence, contradictory documents, and requests the system should decline. The test set needs those cases before the model comparison means much.
I would define the job and the acceptance criteria first. “Help with incidents” is too broad. “Produce a timeline from supplied events without inventing a cause” can be evaluated.
Build cases around failure costs
For an illustrative timeline assistant, include out-of-order events, mixed timezones, duplicated alerts, a missing deployment timestamp, and two incidents that overlap. Add a case with no evidence for the requested conclusion.
For each case, record required facts, forbidden claims, acceptable uncertainty, and source identifiers. There can be several good answers; the test should judge the properties that matter rather than require one exact paragraph.
Separate quality from safety and operations
A useful scorecard includes factual support, coverage of required facts, appropriate abstention, tool correctness, latency, and cost. Track unauthorized-action attempts as their own category rather than averaging them into a writing-quality score.
An assistant that scores well on ordinary questions but follows malicious instructions embedded in a retrieved document has a different defect from one that writes an awkward sentence. OWASP describes that trust-boundary problem as prompt injection.
Make comparisons reproducible
Keep the prompt, model identifier, retrieval configuration, document snapshot, and tool schema with the result. Repeat nondeterministic cases enough to see variation. Keep a held-out set so repeatedly tuning the prompt does not merely teach the system the evaluation examples.
NIST’s AI risk framework offers a useful broader discipline: risks need to be identified, measured, managed, and revisited in context. For this particular assistant, I would turn that into concrete acceptance gates and a named owner for regressions.
Compare the complete workflow
A larger model with poor context may lose to a smaller one with the right evidence and a narrower task. Measure end-to-end results, including retries and reviewer effort, rather than only the first response’s token price.
Before changing production, run the same test set against the candidate. Keep a rollback option for the prompt and retrieval configuration as well as the model. In an AI application, all three can change behavior.