How to evaluate an AI agent before production

Evaluating an agent is not reading a transcript and saying it “felt right.” An agent is a program that chooses tools under uncertainty. Evaluate it like a program that can also lie, loop, and overreach.

Score the task, not the prose

Define the unit of work: a ticket, a change request, a lookup, a draft that a human will approve. For each unit, decide what done means in a way a second engineer can check.

Prefer:

• Structured outputs you can validate (IDs, enums, schemas)
• Side effects you can inspect (did the tool run? with which arguments?)
• Human rubrics for judgment calls, applied to a fixed set

Avoid:

• “The model said it succeeded”
• BLEU-style scores on enterprise workflows
• A leaderboard of models with no tool or permission model attached

Build a set that includes how it fails

Your evaluation set should include:

• Happy paths you actually care about
• Missing data (empty retrieval, unknown ID)
• Conflicting instructions (policy vs user request)
• Tool failure (timeout, 403, partial result)
• Permission edges (the agent could call a tool it must not)
• Jailbreak-adjacent asks if the agent has write tools

Twenty cases beat two hundred synthetic ones if the twenty are real.

Measure the system, not only the model

Log, per run:

• Task success / fail / abstain
• Tools invoked and whether arguments were legal
• Steps or turns (looping is a reliability and cost bug)
• Human edits required, if a human is in the loop
• Latency and token or call cost

An agent that succeeds 94% of the time is unfinished until you know what the other 6% does — especially if those 6% are high-impact actions.

Permissions are part of the eval

If the agent has a god token in eval and least privilege in production, you evaluated a different system. Run eval with the same tool allow-list, identity, and approval gates you intend to ship.

When to stop “just one more prompt”

If success is flat across prompt revisions, the bottleneck is usually architecture: retrieval, tools, schema, or the decision to use an agent at all. Prompt iteration is cheap. Shipping a stochastic actor into a privileged workflow is not.