How to Write an Evaluation Set for an AI Feature Before You Build It

The eval set is the spec

Most teams write the evaluation set after they've built the thing, which means it quietly becomes a justification for what they shipped. Write the AI evaluation set first and it's a target instead — a concrete, agreed definition of what "working" means for this feature, in numbers, before anyone writes model code.

What goes in it

• Real inputs from your own data. Historical support tickets, real search queries from your logs, real transactions, real shelf photos from varied stores. Not synthetic examples — those test whether the model handles the cases you imagined, not the ones you actually get.
• Expected outputs or acceptance criteria. For each input, what a correct response looks like, or the rule for judging one.
• The hard cases on purpose. The ambiguous ticket, the misspelled query, the transaction that looks fraudulent but isn't, the shelf shot with glare. The easy cases won't differentiate two versions of your system.
• Known edge cases. Anything you already know trips people up.

How big

Dozens to low hundreds of examples. Big enough to catch a regression and estimate a rate; small enough that a person can review the whole set and trust it.

What to measure

Pick the metric that maps to the business outcome, not a generic accuracy number. Resolution rate for a support agent. Relevance (nDCG) for search. Precision and recall for a fraud model or a detector. Schema-valid rate for tool calls. Containment for a voice agent. The projects in our portfolio are each measured against a fixed set drawn from the client's real data — search on real queries, fraud on labelled historical transactions, shelf monitoring on photos from the messiest stores.

Freeze it

A moving evaluation set can't tell you whether a change helped. Lock it, version it, and only add to it deliberately — when a new failure mode shows up in production, it becomes a new test case, and you note when it was added.

Slice the results, don't just average

A single aggregate score hides the trade-offs that matter. Break the results down by category — question type, query intent, transaction segment, store condition — and look at each slice separately. A model that's 85% overall might be 95% on order-status questions and 40% on returns; the average tells you nothing useful, and the breakdown tells you exactly where to spend the next week.

The labelling problem

Some domains hand you labels for free — fraud chargebacks, search clicks, resolved-or-not on a ticket. Others need a human to adjudicate what "correct" means, cold, against a written rubric. Budget for that. An eval set with sloppy labels is worse than none, because it gives you false confidence.

Run it on every change

Once the set exists, it's a gate. Run it before and after every prompt tweak, model swap, retrieval change, or threshold adjustment. Watch for the change that improves the aggregate score while quietly regressing a subset — a prompt that lifts overall resolution but breaks refunds is not an improvement. A change that doesn't move the eval set, or moves it the wrong way, doesn't ship.

Start small and grow it

Ten to twenty cases on day one is enough to begin. The set grows from production: every time something goes wrong in a way the set didn't catch, that case gets added, with a note on when and why. Over a few months it becomes a precise map of the ways your feature can fail — and the strongest defence against fixing one thing and breaking another.

Where this stops being right

• A throwaway prototype doesn't need a formal set — you're exploring, not committing.
• Open-ended generation judged on taste (tone, style) needs review panels, not a scored set.
• A tiny, stable problem where you can eyeball every case and there's no regression risk.

FAQ

How many examples do I need? Dozens to low hundreds — reviewable by hand, large enough to spot a regression and estimate a rate.

Where do the examples come from? Your real historical data: tickets, queries, transactions, photos. Synthetic-only sets test your imagination, not your traffic.

Who writes the expected answers? A domain expert, adjudicating cold against a written rubric for what "resolved" or "relevant" means — and you keep that rubric so the judgement is repeatable.

ISTRALLEN scopes every build against a fixed evaluation set drawn from the client's real data — see what we do.