Evaluating agents

Lesson 3.3 · 8 min read · by Vinay C

We'll cover the following

An agent's final answer can look right while the process was wrong: it skipped a check, called the wrong tool, or got lucky. And the same agent can pass today and fail tomorrow on the same input. So you evaluate agents at three levels, and you repeat the runs.

The three levels

LevelQuestionExample
OutcomeDid it get the task done?The refund was issued, the right row was updated
TrajectoryDid it take a sensible path?Compared prices before booking; no repeated or useless calls
StepWas each single decision right?Picked the right tool with the right arguments
  • Start with outcome. It is the only level users care about, and it is the cheapest to automate.
  • Add trajectory when the outcome is right but cost, latency or safety depends on the path.
  • Add step evals to find *where* a failing run went wrong.

Check the world, not the words

Do not grade "Booked! Email sent." Grade the end state: is there a booking in the database, and was exactly one email sent? Check state with code (a query, an API call, a file diff) wherever you can. Use an LLM judge only for what code cannot check, such as tone or whether the explanation is faithful.

Tool-use accuracy

Score each tool call on:

  • Right tool selected (not a different tool, or none).
  • Right arguments (valid schema, correct values).
  • Right moment (not too early, and not called when it should have answered directly).
  • No forbidden calls (for example a write tool on a read-only task).

Also track the unnecessary-call rate. It is a direct cost and latency signal.

Trajectory scoring

Match the actual steps to the expected steps strictly (exact order), in order (extra steps allowed) or in any order. Pick by how rigid the task is. Do not over-constrain: if two valid paths exist, a strict match punishes a good agent. For open-ended tasks, score the path with a calibrated LLM judge against a rubric instead of one gold path.

Agents are non-deterministic: repeat the runs

  • Run each test case k times, not once.
  • pass@k: succeeded at least once in k tries. This is *capability* (can it ever do this?).
  • pass^k: succeeded on all k tries. This is *reliability*, what users feel. A 90% per-try agent has pass^5 of about 0.9^5 = 59%.
  • Find your noise floor first (see "Is your eval gain real or noise?") before claiming an improvement.

Building the test set

  • Start from real traces and real failures, not invented prompts.
  • Cover the happy path, ambiguous requests, broken or missing tools (timeouts, empty results, errors), permission-denied cases, and adversarial inputs (prompt injection inside a tool result).
  • Include cases where the right behavior is to refuse, ask a clarifying question, or hand off. Many teams only test "do the task".
  • Keep a small stable golden set and a larger rolling set fed by production failures.

Simulated users for multi-turn agents

For conversational agents a fixed input is not enough. Use an LLM-simulated user with a goal and a persona, and let the conversation run. Score whether the goal was reached, how many turns it took, and whether the agent stayed within policy. Script or seed the simulator where you can, or it becomes a second source of noise.

Make evals part of shipping

  • Run the suite on every prompt, model or tool change and gate the release on it: no drop in outcome pass rate, and no jump in cost or steps per task.
  • Track cost per task, steps per task and latency next to quality. An agent that is 2% better and 3x more expensive is often a regression.
  • After release, watch online signals: task-completion rate, escalation rate, retries, repeated tool calls, thumbs-down. Feed every failure back into the test set.

Common mistakes

  • Grading only the final message.
  • One run per case, so noise looks like a result.
  • A judge that was never calibrated (see "Can you trust an LLM judge?").
  • Testing only success cases.
  • A test set that never changes while production does.
Check yourselfAn agent ends with "Booked! Email sent." Is that enough to call the run correct?

No. Check the end state (a booking exists, exactly one email was sent) and whether it took the expected steps. A correct-looking message can hide a skipped step.

Check yourselfYour agent succeeds 90% of the time per run. What can you promise over 5 runs in a row?

About 59% (0.9^5). Per-run accuracy overstates reliability, so report pass^k.

Check yourselfTwo valid paths solve the task. Which trajectory match do you use?

In order or any order, or a rubric-based judge. A strict match would mark a good run as a failure.

Check yourselfQuality went up 2% but cost per task tripled. Do you ship?

Not automatically. Gate on quality, cost and steps together. This may be a regression.

Summary: say it in the interview

  • "I evaluate at three levels: outcome by checking the end state in code, trajectory for the path, and step level to localize failures. I run each case several times and report pass^k for reliability, gate releases on it, and feed production failures back into the test set."