How to Tell If Your AI Feature Actually Works

Every AI product begins with a demo that works. Someone types a question into a prototype, the answer comes back fluent and correct, and the room relaxes. A second question, chosen by the same person, also lands. By the third, people are talking about launch dates. Nobody in the room is lying. They have simply mistaken a performance for a measurement.

A demo tells you the system can succeed. An eval tells you how often it does.

This article is about closing that gap. An eval is a repeatable test of what your AI system does on inputs that matter. You do not need a platform to start. You need a spreadsheet, an afternoon and a willingness to read what your system wrote. The quotations are from my book Evals in a Nutshell.

Why "it feels better" is not evidence

When someone says a new prompt feels better, they are reporting the results of an experiment. It is worth asking what the experiment was. Usually it was five or six inputs, typed by one person, read once, judged against a memory of how the old version behaved. That is not nothing. It is also not much. Five tries cannot separate a system that is right nine times in ten from one that is right seven times in ten.

Another shortcut is giving up because outputs vary. Ask a language model the same question twice and you may get two different answers. It is tempting to conclude that measurement is pointless. That is exactly backwards. One input run ten times gives you a rate, and a rate is something you can track. The trick is to test properties of the output, not exact strings.

It also helps to be clear about what you are testing. When something goes wrong, people tend to blame the model. Often the model was handed the wrong documents, a prompt with a contradictory instruction, a truncated conversation history, or a tool that returned an error it was never told how to handle. The user does not experience your model. They experience everything you wrapped around it. So evaluate the whole thing end to end, and zoom in on the parts when it fails.

Four parts, then twenty rows

Strip away the dashboards and the vocabulary and every eval has the same four parts. There are inputs. There is the system you are testing. There is a grader that looks at each output and decides how good it is. And there is a score that summarises the grader's decisions.

If you can name each of the four for an eval you are running, you understand it. If you cannot, you are probably looking at a number without knowing what it means.

Then sort your failures, because they are not equal. A support bot that answers in slightly stilted prose has failed in a minor way. One that tells a customer the wrong refund deadline has failed in a meaningful way. One that tells a customer to share their password in the chat has failed in a way that should stop a release. A single pass rate blends all three into one number, and the blend hides the thing you most need to know.

Now the practical part. There is a version of evaluation that requires infrastructure, budgets, annotation teams and a platform with a subscription. There is another version that requires a spreadsheet and an afternoon. Start with the second.

Make four columns: the input, the output, your verdict, and one sentence saying why. Fill in twenty rows. That is the whole first eval.

The first eval is not a measurement. It is a list of what you forgot to want.

The reading matters more than the counting. The most common mistake in evaluation is choosing a metric before looking at the data. A team decides it cares about helpfulness, builds a grader that scores helpfulness, runs it on a thousand examples and gets a number. Nobody has read the outputs. Nobody knows whether the failures are about helpfulness at all, or about something the team never thought to measure, such as the system confidently inventing order numbers. So read first, note the oddities, group them, count the groups, and only then decide what to measure.

From there, grow the dataset on purpose. Keep a small golden set, often somewhere between fifty and a few hundred examples, where each expected outcome has been checked by someone who knows the domain. Its value comes from trust, not volume. Once you have real users, sample from your logs, because real inputs have a texture invented ones never quite capture: the typos, the pasted fragments of emails, the requests that combine three things.

Choose the cheapest grader that can do the job

Every eval result is a joint product of the system and the grader. If the grader is too lenient, a weak system scores well. If it is too strict, a strong system scores badly. The grader deserves half of your attention.

Work up a ladder, cheapest first.

Start with code. A code-based check takes the output, does something deterministic with it and returns a verdict. It might parse JSON and validate it against a schema, check that a required field is present, confirm that a cited URL appears in an allowed list, count words or look for forbidden phrases. If a rule can be written as code, write it as code.

Next comes a model judge, for criteria code cannot handle: whether an answer is relevant, whether a tone is appropriate, whether an explanation would make sense to a beginner. A judge is fast and tireless, but it is only as trustworthy as its agreement with the humans whose standards it is meant to represent. Check that agreement before you rely on it. This is the step most often skipped, because the judge produces plausible verdicts from the start and nobody thinks to check.

People come last, on a small number of cases, because they are slow and expensive.

Whichever grader you use, prefer pass or fail to a scale. A score of seven tells you a grader felt fairly good. A failed check tells you what to fix.

Read the score as an estimate

When your eval reports that the system passed eighty per cent of examples, it is tempting to read that as a fact about the system. It is a fact about how the system did on these particular examples, on this particular run, with this particular grader.

The uncertainty is larger than most people expect. An 80 per cent score on 100 examples really means somewhere between 72 and 88. So a change that moves you from 80 to 83 on a hundred rows has not shown you much yet.

More examples help, but slowly. Halving the margin takes four times the examples. Small evals find big problems, and only big evals find small ones.

One cheap technique improves matters a lot. When comparing two versions, run both on exactly the same examples and compare them example by example. Only the examples where the two versions disagree carry signal, and looking at those directly can show differences that two separate runs of the same size would miss.

Make it a habit, not an event

An eval you run once is a snapshot. The value comes from running it on every change.

One habit does most of the work. Every time a real failure is found, by a user, a colleague, a reviewer or a monitoring alert, it becomes an example in the dataset before the fix is made. The test fails, the fix is applied, the test passes, and the test stays.

A bug you fixed without a test is a bug you have agreed to fix again.

Collect those cases, your core tasks and your critical cases into a regression suite, and run it automatically, so that a prompt change, a model swap or a retrieval tweak cannot reach production without the evidence having been gathered. Fast checks can run on every change, the full suite nightly, and the costly tier before a release.

After release, keep watching a sample of live traffic. Offline evals test the world you imagined. Online evals test the one you got.

What to take from this

• A demo shows the system can succeed. Only an eval shows how often.
• Name the four parts: inputs, system, grader, score.
• Start with twenty rows in a spreadsheet and read every output before choosing a metric.
• Use code checks first, a calibrated model judge second, people for a few cases.
• Prefer pass or fail, and treat every score as an estimate with a margin.
• Turn every real bug into a test, and run the suite on every change.

An eval, in the end, is a written-down opinion about what good means for your users. If you want help writing yours, Evals in a Nutshell: Measuring what your AI actually does covers the whole path in 100 short chapters, from a first golden set to regression suites, online evaluation and the weekly review. It is a 314-page PDF and costs £9 on Gumroad: https://matsiems.com/books/evals-in-a-nutshell.