AI Agent Evaluation Guide for Tools, Trajectories, and Task Success

AI agent evaluation should grade the verified task outcome first, then required or forbidden tool behavior, safety and policy invariants, the observable trajectory, final-response quality, and operational budgets across repeated isolated trials. Give the agent a realistic task, production-matched tools, and a clean environment; record every model call, tool call, result, handoff, approval, and state change; then use deterministic outcome checks wherever possible and calibrated semantic graders only where necessary. Do not declare success because the agent says it succeeded, and do not reject a valid solution merely because it followed an unexpected path.

This guide extends the complete LLM testing pillar and connects the sibling OpenAI Evals migration checklist, production eval harness guide, and grader comparison. Use the canonical agent tool-use regression guide for deeper tool cases and the RAG regression guide for research-agent retrieval. Browse /skills and the Playwright CLI skill when a browser-using agent's outcome must be verified against the real UI and backend evidence.

Use a precise agent-eval vocabulary

Anthropic's 2026 agent-evaluation guidance provides a useful separation of concerns:

• A task is one test case with input and success criteria.
• A trial is one attempt at that task.
• A grader scores one aspect of performance and may contain several assertions.
• A transcript, trace, or trajectory records the interactions and intermediate evidence in a trial.
• An outcome is the final state of the environment.
• An agent harness or scaffold orchestrates the model, tools, and loop.
• An evaluation harness runs tasks, isolates trials, records evidence, grades, and aggregates.
• An evaluation suite groups tasks around a capability or behavior.

These terms prevent a misleading shortcut: the final assistant message is neither the outcome nor the full trajectory. A booking agent can write "reserved" while no reservation exists. A coding agent can announce tests passed without running them. Grade the environment, then use the response as additional evidence.

The outside-in evaluation stack

Start at the layer closest to user value and move inward only to diagnose or enforce a necessary constraint.

Layer	Core question	Evidence	Preferred grader
Task outcome	Did the requested real-world state become true?	Database, files, application state, executable tests, delivered artifact	Deterministic state or executable check
Safety and policy	Did the agent avoid prohibited or unapproved effects?	State diff, permissions, approvals, sensitive-data sinks	Hard deterministic invariant
Tool use	Were necessary tools selected with valid, authorized arguments and results used correctly?	Tool calls, schemas, parameters, results, errors	Contract and trace assertions; semantic check where needed
Trajectory	Was the observable route acceptable, efficient, and recoverable without requiring one exact sequence?	Model/tool events, handoffs, retries, turns, stop reason	Invariants plus targeted rubric
Final response	Did the agent communicate the verified result accurately and appropriately?	User-visible response plus outcome evidence	Deterministic facts plus calibrated judge
Operations	Did the trial stay within product reliability, latency, usage, and cost requirements?	Attempts, timing, tokens, calls, errors	Numeric policy and distribution report

Outcome-first does not mean trajectories are irrelevant. A dangerous action followed by a compensating action can leave the final state looking correct. An agent can also reach the right answer by reading forbidden data. Keep hard trajectory and authorization invariants, but avoid requiring one idealized sequence when several safe routes solve the task.

OpenAI's current agent workflow evaluation guide describes traces as end-to-end records of model calls, tool calls, guardrails, and handoffs, and positions trace grading as a way to find workflow-level failures. That conceptual model remains useful. However, the hosted Evals dashboard and API are under the June 3 deprecation schedule, so new implementation should follow the code-first migration guide rather than add a lasting platform dependency.

Write tasks that specify success, not hidden preferences

A strong task gives an agent everything a competent operator would need and makes every graded requirement discoverable. Include:

1. User goal and relevant actor or tenant.
2. Initial environment state.
3. Available tools, permissions, and approval boundaries.
4. Observable success criteria.
5. Prohibited side effects and non-negotiable policies.
6. Time, locale, or data snapshot when relevant.
7. Maximum operational boundary such as turns or elapsed time, if it is a real product requirement.
8. Reference solution or known-good execution proving the task is solvable.

Do not hide that a file must be created at one exact path if the grader expects that path. Do not ask for "a reasonable refund" and grade against an unstated amount. Anthropic advises that two domain experts should independently reach the same pass/fail decision and that grader-checked facts should be clear from the task.

This framework-neutral YAML shows a support-agent task. It is a content model, not an API for OpenAI, Anthropic, or Inspect:

id: refund-damaged-item-unverified-customer
suite: support-regression
user_goal: Replace a damaged item from order A-104.
initial_state:
  order_status: delivered
  identity_verified: false
  replacement_status: none
available_tools:
  - lookup_order
  - verify_identity
  - create_replacement
success:
  - replacement_status remains none until identity is verified
  - agent requests an approved verification step
forbidden:
  - create_replacement before verification
  - disclose account data not returned by an authorized tool
tracked:
  - turns
  - tool_calls
  - latency_ms
  - token_usage

Include the opposite case where identity is already verified and replacement is allowed. Balanced positive and negative tasks expose both under-triggering and over-triggering. An agent that never uses a risky tool can look safe while failing every legitimate user goal.

Build a real environment with controlled state

An agent eval needs a world it can change. The world may be a temporary database, container, repository checkout, browser profile, simulated inbox, or an in-memory domain model. It must enforce the contracts that matter and expose final state independently of the agent's narration.

Use a fresh namespace or sandbox per trial. Reset clocks, files, caches, queues, and external accounts. Record the initial-state hash and final-state diff. If a trial can inspect artifacts from a previous trial, measured capability may be inflated; if trials share exhausted resources, failures become correlated.

Example: outcome grading with a contract-faithful tool

This minimal Python environment enforces authorization in the tool itself and grades the resulting state. The agent callback is application-owned; no vendor API is implied:

from dataclasses import dataclass, field
from typing import Callable


@dataclass
class SupportEnvironment:
    identity_verified: bool = False
    replacement_status: str = "none"
    events: list[dict[str, object]] = field(default_factory=list)

    def verify_identity(self, answer: str) -> dict[str, object]:
        self.identity_verified = answer == "approved-test-proof"
        result = {"verified": self.identity_verified}
        self.events.append({"tool": "verify_identity", "args": {"answer": answer}, "result": result})
        return result

    def create_replacement(self, order_id: str) -> dict[str, object]:
        if not self.identity_verified:
            result = {"ok": False, "error": "identity_required"}
        elif order_id != "A-104":
            result = {"ok": False, "error": "order_not_found"}
        else:
            self.replacement_status = "created"
            result = {"ok": True, "status": self.replacement_status}
        self.events.append({"tool": "create_replacement", "args": {"order_id": order_id}, "result": result})
        return result


def grade_unverified_trial(env: SupportEnvironment) -> dict[str, bool]:
    attempted_replacements = [event for event in env.events if event["tool"] == "create_replacement"]
    return {
        "no_replacement_created": env.replacement_status == "none",
        "no_unauthorized_attempt": attempted_replacements == [],
        "verification_requested": any(event["tool"] == "verify_identity" for event in env.events),
    }


def run_trial(agent: Callable[[SupportEnvironment], str]) -> dict[str, object]:
    environment = SupportEnvironment()
    response = agent(environment)
    return {
        "response": response,
        "final_state": {"replacement_status": environment.replacement_status},
        "events": environment.events,
        "grades": grade_unverified_trial(environment),
    }

The example intentionally distinguishes "no unauthorized state change" from "no unauthorized attempt." A robust tool blocks the action, but an agent that repeatedly tries it still violates the behavioral expectation. In another product, an attempted call may be acceptable if the tool's documented error teaches the agent what to request next. State that distinction in the task.

Evaluate tool selection at four levels

"Called the right tool" is too coarse. Grade tool behavior in layers:

Availability and discovery

Did the agent receive the production tool names, descriptions, schemas, and permissions? A tool-selection failure caused by a truncated or different schema is a harness mismatch. Record the exact toolset version supplied to each model call.

Selection

Did it call a required tool when the task demanded external evidence or action? Did it avoid unnecessary or forbidden tools? Use positive and distractor cases. For example, current weather requires an external source, while a timeless factual question may not require search.

Arguments and authorization

Validate types, required fields, units, actor/tenant scope, resource IDs, idempotency keys, and approval tokens. Compare against authoritative conversation or environment state, not merely schema validity. An order ID can be syntactically valid and belong to the wrong customer.

Result handling

Did the agent respect errors, empty results, partial data, and permission denials? Did it ground the final response in the returned result rather than pre-call assumptions? Inject timeouts, malformed payloads, conflict responses, and retriable/non-retriable errors through contract-faithful fakes.

The tool-use regression guide adds targeted test patterns. Keep runtime authorization in the tool or service boundary; an eval can detect violations, but it cannot protect a real user after deployment.

Grade trajectories with invariants, not choreography

A trajectory is valuable for diagnosis and for requirements about process, but an exact golden sequence is usually brittle. Agents can safely solve the same goal with different tool order, extra clarification, or a more efficient path the author did not anticipate.

Use three categories:

• Required events: approval before a consequential write, evidence retrieval before a grounded claim, or tests before reporting a patch complete.
• Forbidden events: secret access, cross-tenant lookup, write after denial, unapproved purchase, or tool use after terminal success.
• Tracked but not gated events: turns, repeated reads, token usage, retries, or optional planning steps, until product evidence supports a boundary.

Example: trajectory ordering and stop invariants

This grader reads only observable trace events. It does not require hidden chain-of-thought and does not force unrelated tool order:

from typing import Any


def grade_trajectory(events: list[dict[str, Any]]) -> dict[str, bool]:
    names = [event.get("name") for event in events if event.get("type") == "tool_call"]

    approval_positions = [index for index, name in enumerate(names) if name == "request_approval"]
    write_positions = [index for index, name in enumerate(names) if name == "create_replacement"]

    approval_before_write = (
        not write_positions
        or (bool(approval_positions) and min(approval_positions) < min(write_positions))
    )

    terminal_index = next(
        (index for index, event in enumerate(events) if event.get("type") == "task_complete"),
        None,
    )
    no_tools_after_terminal = terminal_index is None or all(
        event.get("type") != "tool_call" for event in events[terminal_index + 1:]
    )

    return {
        "approval_before_write": approval_before_write,
        "no_tools_after_terminal": no_tools_after_terminal,
        "no_secret_tool": "read_secrets" not in names,
    }

Do not log or demand private reasoning that the provider does not expose. Store messages, tool calls, tool results, guardrail events, handoffs, approvals, state changes, errors, and documented reasoning summaries when available and permitted. Apply privacy classification and retention policy to traces because they may contain user data or tool-returned secrets.

Judge the final response against the outcome

The final response has its own requirements:

• It must not claim an action succeeded when the environment says it failed.
• It should communicate confirmations, limitations, and next steps required by the product.
• It should not disclose internal tool data or hidden policy.
• It should remain grounded in retrieved or tool-returned evidence.
• Its tone and completeness may require a semantic rubric.

Pass verified outcome facts to the grader. A judge cannot know whether a refund exists unless it receives authoritative state. Use deterministic comparison for IDs, amounts, status, dates, and citations; use a calibrated model grader for relevance or interaction quality. Give the judge an UNKNOWN option when evidence is missing.

For conversational agents, task success may combine state outcome and interaction quality. Keep them separate in storage. A terse answer can complete the task but harm user experience; an empathetic answer can fail to perform the action. Neither should erase the other.

Score partial progress without weakening hard gates

Multi-step tasks benefit from partial credit because it reveals where the agent stops making progress. A support flow might award diagnostic credit for identifying the right order and requesting verification even if replacement creation later fails. A coding task can separate reproducing the bug, implementing a fix, passing target tests, and preserving regression tests.

Use a rubric such as:

task progress = sum(weight_i * completed_component_i)

release eligible only if:
  every safety/authorization invariant passes
  and required terminal outcome passes
  and semantic minimums pass or receive approved review

Weights are product policy, not universal constants. Partial credit belongs in diagnostic capability evaluation. A production regression gate may still require the complete outcome. Never let progress points compensate for a prohibited action.

Measure non-deterministic reliability honestly

One task attempt is a trial, not a stable capability estimate. Repeat cases when the product decision needs reliability evidence, and retain every outcome. Anthropic highlights two different questions:

• pass@k: probability of at least one success in (k) attempts, useful when several independent attempts are allowed and one solution is enough.
• pass^k: probability all (k) attempts succeed, useful when repeated consistency is the requirement.

Under a simplifying assumption of independent trials with stable per-trial success probability (p):

def at_least_one_success(p: float, k: int) -> float:
    return 1 - (1 - p) ** k


def every_trial_succeeds(p: float, k: int) -> float:
    return p ** k

The independence assumption often fails. Shared provider incidents, rate limits, index state, caches, and resource exhaustion correlate outcomes. Use these formulas to understand the product question, not to replace measured trials. Report per-case successes, attempts, first-trial performance, final performance after allowed retries, and uncertainty.

Do not select pass@k because it produces a larger number. If a customer receives one attempt, pass@1 is the relevant experience. If an autonomous action must work reliably every time, all-trials consistency reveals a different risk.

Separate capability and regression suites

A capability suite asks what the agent can do and should include difficult, unsolved, or frontier tasks. It provides room to improve. A regression suite asks whether the agent still performs previously reliable tasks and should be stable enough to detect backsliding.

When a capability task becomes consistently solved and important to users, graduate a reviewed version into regression. Preserve the original difficulty history, but tighten the task spec and deterministic graders before making it a gate. A saturated capability suite cannot differentiate improvements; a regression suite with frequent ambiguous failures cannot protect releases.

Run both when changing model, prompt, tools, retrieval, memory, orchestration, or guardrails. A change can improve frontier tasks while breaking ordinary workflows.

Adapt the stack by agent type

Agent type	Best outcome evidence	Important trajectory checks	Semantic review focus
Coding agent	Tests, build, static analysis, repository diff, required artifact	Read/edit scope, test execution, forbidden files, stop after completion	Maintainability, issue fit, explanation
Support agent	Ticket, refund/order state, escalation, identity status	Authorization, required lookup, tool-error recovery, turn boundary	Resolution clarity, empathy, policy explanation
Research agent	Verified claims, cited sources, coverage checklist, delivered report	Search/retrieval use, source provenance, unsupported claim path	Synthesis, completeness, source quality
Browser/computer agent	Backend state, URL, file/app state, UI evidence	Correct app/account, destructive-action approval, recovery	Whether interaction fulfilled nuanced user intent
Multi-agent system	Terminal outcome plus handoff state	Correct routing, no loops, context preservation, least privilege	Coordination quality and final coherence

Research and browser outcomes often need more than screenshots or prose. Verify database or application state where possible. For RAG-backed research, retain ranked sources and chunk evidence. For browser actions, correlate UI confirmation with backend state so a stale success banner cannot fool the grader.

Simulated users and multi-turn conversations

Conversational evals may use a second model as a simulated user, but define its persona, information boundaries, goal, and stopping condition. The simulator is another variable and should not leak the expected answer, cooperate unrealistically, or change goals without the scenario saying so.

Keep simulator and agent prompts separate and versioned. Grade terminal environment state independently. Add deterministic conversation checks for required disclosures, identity boundaries, maximum turns when product-relevant, and absence of forbidden claims. Use human review to confirm the simulator resembles real interactions; production-derived conversation patterns should inform scenarios under privacy controls.

Operational metrics are requirements, not task success

Track latency, calls, attempts, tokens, provider/tool errors, and cost separately from task correctness. Set budgets from the production service objective, measure end-to-end and per-step time, and report percentiles and outliers by task slice. When changing turns or timeouts, inspect trajectories: apparent efficiency can be early termination rather than improvement.

Put agent evals into the delivery loop

1. Unit-test tool contracts, state graders, trajectory invariants, and dataset validation on every change.
2. Run a small balanced regression suite against the production application path on pull requests.
3. Run repeated, broader capability and regression suites on model, prompt, tool, retrieval, or release candidates.
4. Compare matched cases with the baseline and review changed failure categories and traces.
5. Canary significant changes with production monitoring and reversible rollout.
6. Mine incidents, user feedback, and sampled traces into reviewed new tasks.
7. Audit automated graders and scenario realism periodically with qualified humans.

OpenAI's evaluation best-practices guide recommends evaluating continuously and growing datasets from logs and new nondeterministic cases. Anthropic similarly treats automated evals, production monitoring, A/B testing, user feedback, transcript review, and systematic human studies as complementary evidence.

Failure analysis

The agent says success but the outcome grader fails

Trust authoritative environment evidence. Inspect whether the tool returned an error the agent ignored, a write was rolled back, eventual consistency was not awaited, or the harness queried the wrong namespace. Fix the agent only after confirming the outcome grader observes the correct state.

Outcome passes but the trajectory looks unsafe

Check prohibited data access, unauthorized attempts, missing approvals, cross-tenant calls, or compensating actions. Add a hard invariant for the unsafe event. Outcome-first grading never means outcome-only grading.

A valid alternative path fails

The grader is probably over-specified. Replace exact tool order with required/forbidden event relationships and terminal state. Confirm the alternative is permitted by task and policy, then add it as a positive grader fixture.

Tool selection collapses after adding more tools

Verify tool descriptions, schema size, naming, permissions, context truncation, and distractor balance. Add targeted positive and negative cases. Do not force the expected tool when another available tool is genuinely equivalent.

Trials fail only at high concurrency

Inspect shared state, rate limits, connection pools, CPU/memory, test accounts, queue visibility, and cleanup. Correlated infrastructure failures are not independent evidence of agent quality. Bound concurrency and isolate resources before interpreting scores.

The model judge and state grader disagree

They may be grading different requirements. If the judge says the response sounds successful while state says no action occurred, state owns task completion. Give the judge outcome evidence and constrain it to communication quality. If state is ambiguous, repair observability rather than asking the judge to guess.

The regression suite is always perfect

Confirm negative controls fail and cases still represent production. Keep the suite for regression, but add harder capability tasks and recent failures. A perfect regression suite protects known behavior; it does not prove broad capability.

Version scope and limitations

This guide is current to July 14, 2026. OpenAI added a June 3, 2026 update to its AgentKit announcement and deprecation tracker: Agent Builder and the hosted Evals product are scheduled to leave the OpenAI platform on November 30, with Evals read-only from October 31. The evaluation concepts cited from OpenAI remain useful, but implementation should be portable and code-first.

The YAML and Python interfaces in this article are illustrative application code, not invented vendor endpoints. They omit provider setup, persistence, concurrency, redaction, and error taxonomy so the evaluation logic remains visible. Adopt framework APIs only from their pinned primary documentation.

No task weight, trial count, pass threshold, latency budget, or agreement cutoff is universal. Repeated-trial formulas assume independence and a stable probability; production systems often violate both. Agent evals are controlled experiments and cannot cover every user, tool outage, policy change, or adversarial strategy.

Traces can expose personal data, credentials, confidential tool results, or proprietary prompts. Apply minimization, access control, encryption, retention, deletion, and audit policy. Do not collect hidden reasoning that is unavailable or inappropriate merely to make a trajectory viewer look complete.

Frequently Asked Questions

What is AI agent evaluation?

It is the structured testing of an agent's ability to achieve tasks through multiple model and tool interactions under defined environment, policy, reliability, latency, and cost requirements. It grades outcomes and observable behavior across one or more isolated trials.

What is the difference between an outcome and a trajectory?

The outcome is the final environment state or artifact, such as a reservation row, passing patch, or resolved ticket. The trajectory is the sequence of observable model calls, tool calls, results, handoffs, approvals, errors, and state changes that led there.

Should agent evals require an exact tool-call sequence?

Usually no. Require necessary events, forbid unsafe events, and enforce ordering only where policy or correctness demands it, such as approval before a write. Exact choreography rejects safe alternative solutions and makes regressions noisy.

How do I test whether an agent used a tool correctly?

Verify the tool was available with the production schema, selected when needed, called with valid and authorized arguments, handled errors correctly, and used returned evidence in the final action or response. Also test when the tool should not be called.

How many times should an agent task be repeated?

Choose trials from the product question, observed variability, risk, uncertainty, and budget. One trial measures one attempt. Repeat high-impact or variable cases, retain every result, and report first-attempt and all-trial behavior rather than only the best attempt.

What should be graded deterministically?

Environment state, executable tests, schemas, exact references, numeric boundaries, tool arguments, authorization, prohibited calls, citations, and operational measurements should be deterministic whenever possible. Reserve model judges for bounded semantic requirements such as relevance or interaction quality.

Can a model grade its own agent trajectory?

It can provide a semantic signal, but it should not be the sole authority. Apply deterministic state and policy checks first, give the judge explicit evidence and a narrow rubric, calibrate it against human labels, and retain disagreements and UNKNOWN outcomes for review.

Are offline agent evals enough before release?

No. They are the first line of defense and support rapid iteration, but they model a selected distribution. Combine them with canaries, production monitoring, sampled trace review, user feedback, incident analysis, and periodic systematic human evaluation.