How many test cases is enough?.
Is trying a dozen examples enough before shipping an AI feature? Not even close. Here’s how many cases you need, and which ones to pick.
By Nizar Oukhchi, Solution architect
Published · 8 min read
Someone on the team tries a dozen examples. The answers look good. The AI feature ships.
A few weeks later, a customer finds the case that nobody tried.
What went wrong? It's tempting to say the team skipped testing. But they didn't! They tested. The problem is that a dozen examples can't tell you much, and I think it's worth knowing exactly how little.
A score is an estimate
Say you run your AI feature on a set of test cases, and 90% of the answers are correct.
What does that 90% mean?
It's tempting to read it as a property of the system. It isn't. It's an estimate, based on a sample, of how the system will behave on the inputs it meets in production. And like any estimate, it comes with a margin of error. That margin depends mostly on one thing: how many cases you tested.
Here's what that looks like:
| Number of test cases | Score around 50% | Score around 90% |
|---|---|---|
| 20 | ±22 | ±13 |
| 40 | ±15 | ±9 |
| 60 | ±13 | ±8 |
| 80 | ±11 | ±7 |
| 100 | ±10 | ±6 |
| 120 | ±9 | ±5 |
| 140 | ±8 | ±5 |
| 160 | ±8 | ±5 |
| 180 | ±7 | ±4 |
| 200 | ±7 | ±4 |
| 220 | ±7 | ±4 |
| 240 | ±6 | ±4 |
| 260 | ±6 | ±4 |
| 280 | ±6 | ±4 |
| 300 | ±6 | ±3 |
| 320 | ±5 | ±3 |
| 340 | ±5 | ±3 |
| 360 | ±5 | ±3 |
| 380 | ±5 | ±3 |
| 400 | ±5 | ±3 |
| 420 | ±5 | ±3 |
| 440 | ±5 | ±3 |
| 460 | ±5 | ±3 |
| 480 | ±4 | ±3 |
| 500 | ±4 | ±3 |
| 520 | ±4 | ±3 |
| 540 | ±4 | ±3 |
| 560 | ±4 | ±2 |
| 580 | ±4 | ±2 |
| 600 | ±4 | ±2 |
| 620 | ±4 | ±2 |
| 640 | ±4 | ±2 |
| 660 | ±4 | ±2 |
| 680 | ±4 | ±2 |
| 700 | ±4 | ±2 |
| 720 | ±4 | ±2 |
| 740 | ±4 | ±2 |
| 760 | ±4 | ±2 |
| 780 | ±4 | ±2 |
| 800 | ±3 | ±2 |
| 820 | ±3 | ±2 |
| 840 | ±3 | ±2 |
| 860 | ±3 | ±2 |
| 880 | ±3 | ±2 |
| 900 | ±3 | ±2 |
| 920 | ±3 | ±2 |
| 940 | ±3 | ±2 |
| 960 | ±3 | ±2 |
| 980 | ±3 | ±2 |
| 1,000 | ±3 | ±2 |
With twenty cases, a measured 90% is compatible with a true rate anywhere from the high 70s to 100%. That's a huge range.
With one hundred cases, the margin narrows to about six points either way. It takes around four hundred cases to get within three points.
Notice the pattern. The margin shrinks with the square root of the number of cases. So if you want to halve it, you need four times as many cases.
A score without the number of cases behind it doesn't tell you much.
This changes how I read a demo. A feature that answers ten questions out of ten hasn't shown that it's reliable. It has shown that it's not obviously broken.
That's not nothing. But it's not the same thing.
Comparing two versions is harder
In practice, the question people ask most often isn't "how good is it?"
It's "is the new version better?"
You try a new model, or revise the prompt, or change the retrieval setup. You get a new score. The temptation is to keep whichever number is higher. I've felt that pull too.
Not so fast.
Detecting a difference takes many more cases than measuring a single score. The smaller the improvement, the more cases you need, and the requirement grows quickly:
A ten-point improvement shows up clearly with about two hundred cases per version. A five-point improvement needs nearly seven hundred. A two-point improvement needs several thousand.
Now imagine your set has fifty cases.
Most of the differences you'll see between two versions are noise. If you act on them, you end up switching back and forth between changes that do nothing. It feels like progress. It isn't.
So what do you do? Two things:
- Score every version on the same set of cases. That makes the comparison paired, which removes much of the noise and lowers the number of cases you need.
- Decide in advance what size of improvement is worth acting on. Then size the evaluation set for it.
What to put in the set
Are we done once we have enough cases? Actually, we're not!
The number matters, but so does what the cases cover. A set made only of easy, typical questions will give you a high score. That score will say nothing about the cases that cause incidents.
I find it helps to mix four kinds of cases. Here's what a first set can look like:
- Common cases, from real data. Sample them from actual requests, documents or tickets, so the score reflects what users really send. Anonymize them first.
- Edge cases. Long inputs, missing information, ambiguous wording, other languages, unusual formats. This is where behavior changes, and where users lose trust.
- Past failures. Every incident and every bad answer a user reports becomes a test case. This is how the set grows over time, and it guarantees that a fixed problem stays fixed.
- Out-of-scope and adversarial requests. Questions the feature should decline, and attempts to make it reveal data or ignore its instructions. Here, the correct answer is often a refusal.
How to score answers
Each case needs a way to decide whether the answer is right.
Where you can, use checks that code can run. An exact value. A classification label. A valid schema. A citation that points to the right source document. These are fast, cheap and consistent. I'd reach for them first.
But what about open-ended answers, like summaries or drafted replies?
For those, write a rubric with a small number of explicit criteria, and have people grade a sample. A second model can apply the same rubric at scale. But only after you've checked, on a few dozen cases, that its grades agree with your reviewers'.
Why bother? Because an unchecked automated grader adds its own errors to the measurement. You'd be measuring with a ruler you never checked.
Make it part of delivery
An evaluation set is most useful when nobody has to remember to run it.
Run it automatically on every change: to the prompt, the model, the retrieval setup, or the surrounding code. Block a release when the score drops beyond the margin you've decided matters.
And report the score with its margin, not as a bare number. Otherwise someone (me included, on a good day) will celebrate a one-point gain measured on fifty cases.
Are we done after launch? Not quite.
Keep measuring. Sample production traffic, have it reviewed regularly, and add what you learn to the set. Inputs drift. Models get updated. A feature that scored well at launch can degrade without any change on your side.
Where to start
You don't need thousands of cases on day one.
Here's what I'd do:
- Collect one hundred to two hundred real examples.
- Write down what a correct answer looks like for each.
- Add the edge cases and failures you already know about.
That set will tell you more than any demo. And it'll grow with every problem you find.
That's really it. Start small, and keep it honest.
Our applied AI work starts with exactly this step. If you're planning a feature, I'd love to hear about it.