How to Evaluate an AI Agent Before You Ship It

You evaluate an AI agent before shipping it by building a fixed test set of real tasks with known-good answers, running the agent against it on every change, and measuring task success, not vibes. A demo that works three times tells you nothing about the tenth thousand request. The teams whose agents survive production treat evaluation as an engineering discipline: a graded set of representative cases, a clear pass threshold, and adversarial tests for the ways an agent fails quietly. This guide covers what to measure, how to build the test set, and the failure modes a happy-path demo hides. Last updated July 2026.

Most agent projects die the same way. The prototype dazzles in a meeting, it ships, and then real users feed it inputs nobody tried, it confidently makes things up, and trust evaporates in a week. The fix is not a better model. It is knowing, before launch, how often the agent actually gets the job done and where it breaks. That knowledge comes from evaluation.

Why a demo is not an evaluation

A demo is a curated success. You pick the inputs, you know the answer, and you stop when it works. An evaluation is the opposite: a fixed, representative set of tasks you did not cherry-pick, scored the same way every time so you can compare versions honestly. The difference matters because language models are non-deterministic and sensitive to phrasing. An agent that answers one version of a question perfectly can fail a slight rewording, and you will never see it if you only ever test the version you wrote.

The goal of evaluation is a number you trust: on a set of tasks that look like real usage, this agent succeeds X percent of the time. Once you have that number, every change to the prompt, the model or the tools can be measured against it instead of argued about.

What to actually measure

Accuracy is the headline, but a useful evaluation tracks several dimensions, because an agent can be right and still unusable. These are the ones that predict whether an agent survives production.

Faithfulness deserves special attention for anything that retrieves information. An agent that pulls the wrong document and answers fluently from it is more dangerous than one that clearly says it cannot find the answer. Much of an agent's real-world accuracy comes down to retrieval quality, the same problem that good enterprise search over your internal knowledge is built to solve, so measure whether the agent found the right source before you measure whether it wrote a good answer.

How to build the test set

The test set is the whole game, and it does not need to be large to be useful. Fifty to a few hundred well-chosen cases beat thousands of random ones. Build it like this:

• Pull real examples, not invented ones. Mine actual user requests, support tickets or historical tasks so the set reflects how people really phrase things.
• Write the known-good answer for each. This is the labor, and it is where domain experts earn their keep. Without a reference answer you cannot grade automatically.
• Cover the edge cases on purpose. Include the ambiguous request, the out-of-scope question, the input with a typo, the case where the right answer is "I can't help with that."
• Include adversarial cases. Add inputs designed to trick the agent into ignoring its instructions or leaking data, so you measure resistance, not just competence.
• Freeze it. Once the set exists, do not quietly edit it to make a bad version look good. A fixed set is what lets you compare across changes.

Grading can be a mix. Exact-match works for structured outputs, a rubric works for open text, and an LLM-as-judge, one model scoring another's answer against the reference, scales the middle, as long as you spot-check the judge against human grades so you trust it.

The failure modes a happy-path demo hides

Evaluation exists to surface the failures you would otherwise ship. The recurring ones:

• Confident hallucination. The agent invents a plausible answer rather than admitting uncertainty. Only a faithfulness check against sources catches it.
• Silent tool errors. The agent calls the wrong tool or the right tool with bad arguments, and narrates success anyway. Tool-correctness scoring is the only way to see it.
• Brittleness to phrasing. Small rewordings flip a right answer to wrong. A test set with paraphrased variants exposes it.
• Prompt injection. Hidden instructions in a retrieved document redirect the agent. Adversarial cases measure resistance before an attacker does.

That last one is why evaluation and security overlap. An agent that can take real actions needs both a success score and a resistance score, a theme we cover in AI agent security and guardrails.

How to test an AI app before shipping

Test an AI app before shipping by running it against a fixed set of 50 to 200 real tasks with known correct outcomes, scoring task success rather than whether the output reads well, and setting a pass threshold you agreed before you saw the results. Re-run that set on every prompt, model or tool change. A demo proves the happy path works once, which is not evidence of anything.

The practical difference between testing an AI app and testing ordinary software is that the same input can produce a different output twice, so a single pass tells you very little. Run the set repeatedly and look at the distribution: an agent that succeeds 95 percent of the time on one run and 78 percent on the next has a consistency problem that a single green test run would have hidden completely. Include the awkward inputs deliberately, because real users supply them within the first day.

Three checks belong in every pre-ship run regardless of what the app does. Does it refuse cleanly when it should, instead of inventing an answer? Does it stay inside its instructions when a user tries to talk it out of them? And does every factual claim trace back to a source you supplied? Those three catch most of what reaches production and embarrasses people.

Make evaluation part of the build, not an afterthought

The cheapest time to build a test set is while the agent is being built, because the person building it already knows the edge cases. Ask for it as a deliverable: an evaluation set, a pass threshold and a report showing the agent clears it. That single requirement separates a specialist from someone who ships a demo and hopes. It is one of the questions to ask before hiring an AI development company, and it is exactly what a vetted builder expects to be asked.

The bottom line

Do not ship an agent because it worked in a demo. Ship it because it passed a fixed set of real tasks with a success rate you can live with, including the edge cases and the attacks. Build the test set from real usage, measure task success and faithfulness rather than impressions, and re-run it on every change. If you want that discipline built in from day one, describe your agent in the hire-brief demo and get matched to a generative AI developer who treats evaluation as part of the job.