How to Build an LLM Eval Harness That Matches Production Behavior

An LLM eval harness matches production behavior when it invokes the same application entry point with the same prompt assembly, retrieval, tools, policies, model settings, output parsing, retries, and state boundaries that users encounter. The harness may replace external side effects with controlled fakes, but it must preserve their contracts and record every substitution. Build it around versioned cases, isolated trials, observable results, layered graders, and an immutable run manifest. If the eval calls a model directly while production calls an orchestrated application, it measures the model prompt, not the product you ship.

Place this implementation inside the LLM testing pillar. Before selecting runner code, review the sibling OpenAI Evals migration checklist, the grader decision guide, and the agent evaluation guide. Build datasets with the canonical golden dataset guide, and add retrieval slices from the RAG regression testing guide. Reusable instructions live in /skills; the Playwright CLI skill is useful when a production journey must be verified through its browser surface.

Begin with a parity contract

"Use the same model" is not a parity contract. An LLM application's behavior depends on every transformation before and after the provider call. Write down the production path as a sequence of observable boundaries, then identify which implementation the eval uses at each boundary.

Boundary	Production behavior to preserve	Controlled substitution that is usually safe	Evidence to record
Request intake	Authentication context, locale, tenant, feature flags, conversation state	Synthetic identity with the same authorization claims	Case ID, actor, flags, locale, initial state hash
Input processing	Validation, normalization, classification, policy checks	Fixed clock or deterministic ID generator	Parsed request and transformation versions
Context construction	Prompt version, retrieved chunks, memory, history truncation	Frozen retrieval snapshot for offline regression	Prompt hash, document IDs, ranks, chunk hashes
Model invocation	Provider, model identifier, generation options, timeout, retry policy	Recorded response only for component tests	Resolved config, attempt number, usage, latency
Tools and side effects	Tool schema, permissions, errors, idempotency, external state	Contract-faithful fake in an isolated namespace	Calls, arguments, results, state diff, approvals
Output processing	Structured parsing, fallback, citation mapping, policy filtering	None unless the parser itself is the unit under test	Raw provider output and final user-visible output
Delivery	Streaming, status, error mapping, persistence, analytics	In-memory sink with the same event contract	Events, finish reason, persisted record IDs

OpenAI's evaluation best-practices guide calls production-mismatched datasets an anti-pattern and recommends tests that reflect real-world distributions. Anthropic goes further for agents: its agent-eval guidance defines the agent harness as part of the evaluated system and says the eval harness should provide tools, execute tasks, capture steps, grade outputs, and aggregate results. Production parity therefore includes orchestration, not only inference.

Write the parity contract in the repository next to the harness. A reviewer should be able to answer: "Which production behavior is real, which is simulated, and why does the simulation preserve the requirement being graded?"

Separate four objects that teams often conflate

A maintainable harness has four independent objects:

1. Case: immutable input, initial state, references, tags, and success criteria.
2. Trial: one attempt by one resolved application configuration against one case.
3. Grade: one scorer's result for one observable requirement, including evidence and error state.
4. Run: a collection of trials plus environment, dependency, dataset, and code metadata.

This separation matters because a case may have several trials, a trial may have several grades, and a run can fail without any product requirement failing. If a provider timeout is stored as a score of zero, reliability appears worse even though the agent never produced an answer. If a judge parser crashes and the harness records "fail," product and evaluation defects become indistinguishable.

Use explicit statuses such as completed, product_failed, grader_error, infrastructure_error, and cancelled. Scores belong only to completed grades. Operational policy can decide whether too many incomplete trials block a release, but the data model must retain the distinction.

Design a versioned case format

Keep the case format framework-neutral and narrow. Business requirements should survive a future runner change. JSON Lines works well for reviewable records; larger payloads can be referenced by immutable URI and hash.

{
  "id": "support-refund-identity-required",
  "suite": "support-regression",
  "input": {
    "message": "Refund order A-104. I no longer have access to my email.",
    "actor": "customer-17"
  },
  "initial_state": {
    "order_status": "delivered",
    "identity_verified": false,
    "refund_status": "none"
  },
  "expected": {
    "refund_status": "none",
    "required_action": "request_identity_verification"
  },
  "tags": ["refund", "authorization", "negative-path"],
  "schema_version": 1
}

The case states outcomes, not a script that forces one sentence or one exact tool sequence. The negative-path tag is important: an eval that tests only successful refunds can reward an agent that refunds everything. Anthropic recommends balanced tasks that cover when a behavior should and should not occur, and reference solutions that prove cases and graders are solvable.

Validate case files before calling any model. Reject duplicate IDs, missing references, unknown schema versions, invalid initial states, and impossible success criteria. Dataset defects are cheaper to fix at load time than after a costly run.

Route evals through one application port

Create a narrow interface representing the production application, not the provider SDK. Production adapters can attach web or queue transport; eval adapters call the same service function directly. This avoids HTTP overhead without bypassing business logic.

The following is application-owned sample code. ApplicationPort, EvalCase, and TrialResult are illustrative interfaces, not APIs from a vendor:

from __future__ import annotations

from dataclasses import asdict, dataclass
from hashlib import sha256
from time import perf_counter
from typing import Any, Protocol


@dataclass(frozen=True)
class RunConfig:
    model: str
    prompt_version: str
    toolset_version: str
    max_attempts: int = 1


@dataclass(frozen=True)
class EvalCase:
    case_id: str
    request: dict[str, Any]
    initial_state: dict[str, Any]
    expected: dict[str, Any]


@dataclass(frozen=True)
class AppResult:
    output: dict[str, Any]
    trace: list[dict[str, Any]]
    final_state: dict[str, Any]
    usage: dict[str, int]


class ApplicationPort(Protocol):
    def invoke(
        self,
        request: dict[str, Any],
        initial_state: dict[str, Any],
        config: RunConfig,
    ) -> AppResult: ...


def stable_hash(value: object) -> str:
    import json

    encoded = json.dumps(value, sort_keys=True, separators=(",", ":")).encode()
    return sha256(encoded).hexdigest()


def execute_trial(app: ApplicationPort, case: EvalCase, config: RunConfig) -> dict[str, Any]:
    started = perf_counter()
    result = app.invoke(case.request, case.initial_state, config)
    return {
        "case_id": case.case_id,
        "config": asdict(config),
        "input_hash": stable_hash(case.request),
        "output": result.output,
        "trace": result.trace,
        "final_state": result.final_state,
        "usage": result.usage,
        "latency_ms": round((perf_counter() - started) * 1000, 3),
    }

The function records the resolved configuration and returns raw evidence; it does not grade. That makes trials replayable through a new grader without paying for generation again. In production code, catch and classify exceptions at this boundary, record attempts separately, and redact sensitive fields before durable storage.

Use dependency injection for controlled external systems

Do not patch deep provider functions differently in each test. Construct production services from explicit dependencies: model client, retriever, tool registry, clock, identity service, and persistence port. The eval assembles the same service with controlled implementations.

Controlled does not mean simplistic. A fake tool must enforce the same input schema, authorization, idempotency, and error categories that matter to the test. If the production payment tool rejects duplicate idempotency keys but the fake accepts them, the harness can approve a retry bug. Add contract tests that run against both the fake and a safe real test environment.

Pytest's official fixture guidance supports managed setup and teardown, while its monkeypatch fixture safely restores changed attributes, mappings, and environment variables. Prefer constructor injection for core boundaries and use monkeypatch for narrow process-level state such as a fixed environment flag.

Here is a deterministic outcome test against the same application port:

def test_unverified_customer_cannot_trigger_refund(app_with_fake_tools):
    case = EvalCase(
        case_id="support-refund-identity-required",
        request={
            "message": "Refund order A-104. I cannot access my email.",
            "actor": "customer-17",
        },
        initial_state={
            "order_status": "delivered",
            "identity_verified": False,
            "refund_status": "none",
        },
        expected={"refund_status": "none"},
    )
    config = RunConfig(
        model="resolved-by-test-environment",
        prompt_version="support-v7",
        toolset_version="support-tools-v3",
    )

    trial = execute_trial(app_with_fake_tools, case, config)

    assert trial["final_state"]["refund_status"] == "none"
    refund_calls = [
        event for event in trial["trace"]
        if event.get("type") == "tool_call" and event.get("name") == "process_refund"
    ]
    assert refund_calls == []

This test intentionally makes the safety invariant strict: no refund side effect and no attempted refund tool call. A separate semantic grader can judge whether the response explains the verification step helpfully, but tone must not compensate for an unauthorized action.

Reproduce production prompt and context assembly

Prompt text is only one input. The harness should call the production renderer with the same developer instructions, message roles, tool schemas, memory rules, retrieval filters, context budget, and output schema. Record a normalized prompt/context hash and enough non-sensitive metadata to explain a difference.

For RAG, offer two modes:

• Frozen retrieval mode injects a versioned list of document IDs and chunks. It isolates generation and grading from index drift.
• Live retrieval mode queries a production-like index snapshot. It measures the assembly and detects ranking, filtering, or chunking regressions.

Run fast frozen cases on pull requests and live retrieval cases on a scheduled or pre-release job. Compare retrieved IDs and ranks before scoring answer quality. The RAG regression guide explains how retrieval and generation failures can otherwise cancel each other in an aggregate score.

For conversations, preserve production history construction. Do not hand a judge the full untruncated transcript if production summarizes or compacts it before the model call. Record both the canonical conversation and the exact context sent for each turn.

Model settings, retries, and time

Record the provider-visible model identifier, not a friendly alias alone. Also record temperature or sampling controls where supported, maximum output limits, reasoning settings, seed where supported, request timeout, retry policy, and fallback route. Do not claim temperature zero guarantees determinism; provider implementation, model updates, tool results, and concurrency can still vary.

Retries create a measurement choice. A user-facing service that retries a transient failure should be evaluated with that retry policy for end-to-end reliability. But retain each attempt so you can separately see first-attempt quality, recovered infrastructure errors, and final task success. Never overwrite a failed attempt with the successful retry.

Use a fixed clock for expiration, scheduling, and date-sensitive logic, then include that timestamp in the case. If a test says "today" without a recorded date and locale, it is not replayable.

Isolate every trial

Anthropic's guidance says each trial should start clean because shared files, caches, exhausted resources, database state, or history can create correlated failures and artificial performance. Implement isolation at every mutable boundary:

• Unique trial ID and namespace for database rows, queues, object storage, and external test accounts.
• Fresh filesystem or sandbox when agents can read and write files.
• Empty application cache, or a deliberately recorded warm-cache mode.
• Fixed dependency snapshots for search indexes, policies, schemas, and tools.
• Bounded concurrency aligned with production limits and test-resource capacity.
• Idempotent teardown that records cleanup failure rather than hiding it.

Run the same case twice in different namespaces as an isolation test. Then deliberately contaminate one namespace and prove the other trial is unchanged. Isolation is a harness feature and deserves its own tests.

Layer graders over retained evidence

Grade the cheapest objective facts first. Parse output, validate schema, inspect state, execute code, check tool authorization, and verify citations before invoking a model judge. A deterministic prerequisite failure may make a later semantic grade irrelevant; mark it skipped with a reason instead of assigning an arbitrary zero.

A useful grade record contains:

{
  "case_id": "support-refund-identity-required",
  "trial_id": "run-481-trial-03",
  "grader": "refund-state-invariant",
  "grader_version": "2",
  "status": "completed",
  "score": 1,
  "passed": true,
  "evidence": {
    "refund_status": "none",
    "process_refund_calls": 0
  }
}

Model judges should receive only the evidence needed for one rubric dimension, return structured labels, and have an abstain path for insufficient evidence. Calibrate them against adjudicated human examples and store judge prompt and model versions. Human review should sample disagreements, high-risk cases, and new failure clusters rather than acting as an invisible cleanup queue.

Repeat trials for the product question

Repeated trials answer different questions depending on product behavior. For a system that gets one chance, first-trial success matters. For a workflow allowed several independent proposals, at-least-one success may matter. For a customer-facing agent expected to work repeatedly, all-trials reliability matters.

If a task has an estimated independent per-trial success probability (p), two simple planning formulas are:

at least one success in k trials = 1 - (1 - p)^k
all k trials succeed              = p^k

Anthropic calls these perspectives pass@k and pass^k and warns that they move in opposite directions as (k) grows. Independence is an assumption, not a fact. Shared outages, cached state, rate limits, and common retrieval snapshots correlate trials. Report the raw per-case outcomes and confidence uncertainty; do not present a formula-derived number as measured reliability when trials are correlated.

Choose trial counts from decision risk, expected variability, and budget. A pull-request smoke suite may run one trial per stable regression case; a scheduled evaluation can repeat the volatile or high-impact slices. Never hide that the two jobs answer different questions.

Build an immutable run manifest

A result without provenance cannot support a regression decision. Write one manifest before execution and finalize it after completion. Include:

• Run ID, parent/baseline run, trigger, commit SHA, dirty-state flag, and CI job URL.
• Dataset name, schema version, content hash, included case IDs, and slice counts.
• Application build, prompt versions, retrieval snapshot, tool schema, policy version, and feature flags.
• Provider/model identifiers, generation options, timeout, retry, and fallback settings.
• Grader names, versions, judge configuration, rubrics, thresholds, and aggregation policy.
• Runtime image, dependency lock hash, region, clock, concurrency, and environment substitutions.
• Start/end timestamps, completed/incomplete counts, usage, latency, and artifact locations.

Inspect AI's eval-log format is a useful primary-framework example: its log model preserves task and model details, execution plan, aggregate results, usage, errors, and per-sample data, and its current tooling can export run configuration for replay. Promptfoo's output documentation similarly distinguishes rich result exports from compact JUnit output. Even a custom harness should meet that provenance bar.

Aggregate by requirement and slice

Do not average unrelated criteria into one number too early. Report schema validity, task outcome, groundedness, unauthorized-action rate, grader errors, p50/p95 latency, and cost separately. Then slice by domain-relevant tags: intent, language, tenant policy, tool, retrieval difficulty, adversarial class, or conversation length.

An overall pass rate can improve while a small safety slice collapses. Require minimum coverage and explicit critical invariants. For weighted scores, publish the weights and show the component values. If a task allows partial credit, retain the assertions that earned it.

Compare candidate and baseline on matched cases. Added or removed cases change the denominator, so show common-case comparison separately from full-suite health. Do not update the baseline automatically when a candidate passes; baseline promotion should be a reviewed event with a preserved predecessor.

CI topology that stays useful

Use several jobs rather than one giant gate:

Job	Trigger	Typical scope	Decision
Harness unit tests	Every code change	Loaders, fakes, graders, aggregation, negative controls	Harness itself is trustworthy
Deterministic component evals	Pull request	Frozen outputs/retrieval and fast invariants	Parsing, policy, state, and grader regression
Production-path smoke eval	Pull request or merge	Small balanced suite, one controlled trial	Candidate can proceed
Repeated full suite	Scheduled or release	Representative slices and repeat policy	Reliability trend and release evidence
Shadow production eval	Sampled live traffic under policy	Real distribution with delayed/automated labels	Drift and missing-case discovery

Cap concurrency, fail on missing cases, and enforce a maximum infrastructure-error budget. Upload artifacts even when the job fails. Keep secrets out of result exports and apply retention controls to production-derived inputs.

Failure analysis by layer

Eval is green but production is failing

Diff the parity contract. Common causes are a copied prompt, missing retrieval filters, different feature flags, direct provider calls, fake tools with weaker authorization, stale policy data, or a dataset that no longer represents traffic. Add the production failure as a case only after reproducing the actual path.

Results change with no application diff

Compare manifests for model alias resolution, dependency versions, data snapshots, judge changes, clock, region, and concurrency. Then inspect infrastructure errors and raw outputs. A mutable dependency is still a change even when Git is clean.

Failures cluster under parallel execution

Suspect shared state or resource exhaustion before blaming the model. Run the same cases serially, inspect namespace collisions, and monitor rate limits, CPU, memory, connection pools, and external test accounts. Reducing workers diagnoses the symptom; isolation and capacity controls fix it.

A judge rejects outputs humans accept

Review the rubric, evidence supplied, output parser, judge model, and boundary examples. Split compound criteria, add an insufficient-evidence label, and recalibrate on adjudicated cases. Do not raise the threshold blindly or edit the candidate output before judging.

Latency improves while task quality drops

Check truncation, timeouts, early stopping, skipped tools, reduced retrieval, and fallback routing. Latency is a tracked requirement, not a substitute for task success. Compare traces and finish reasons for matched cases.

Retries make the final pass rate look healthy

Report first-attempt and final outcomes separately. Also report attempts, recovered errors, and added latency/cost. A user may tolerate a transparent retry, but an evaluation should not erase the reliability problem it recovered from.

Version scope and limitations

This guide is current to July 14, 2026 and is framework-neutral. The Python types are intentionally local examples, not published APIs. Adapt them to your application while preserving the separation among cases, trials, grades, and runs. Promptfoo, Inspect AI, and pytest details should be checked against the pinned versions in your lockfile and the linked official documentation.

A production-matched harness is still a model of production. Safe fakes cannot reproduce every outage, permission drift, third-party response, search-index update, or user interaction. Combine offline evals with production monitoring, feedback, A/B tests where appropriate, transcript review, and periodic human studies. Anthropic explicitly presents these as complementary methods rather than replacements.

The formulas assume a stable probability and independent trials. Real agent trials can be correlated. Thresholds, trial counts, sample sizes, and confidence requirements must come from your risk and observed variance; none in this article is an industry default.

Frequently Asked Questions

What is an LLM eval harness?

It is the infrastructure that loads test cases, configures and invokes the LLM application, isolates trials, captures outputs and traces, runs graders, aggregates results, and stores provenance. It is different from the agent harness, which orchestrates model and tool behavior inside one trial.

Should an eval harness call the model provider directly?

Only when the component under test is the provider call or prompt itself. An end-to-end product eval should invoke the same application port production uses; otherwise it bypasses retrieval, policies, tools, parsing, retries, fallbacks, and state that can determine user-visible behavior.

Can external services be mocked without losing production parity?

Yes, when the fake preserves the contract relevant to the requirement: schemas, authorization, errors, idempotency, timing behavior where important, and state changes. Run contract tests against the fake and a safe real environment. Document every substitution in the run manifest.

How do I keep LLM evals from becoming flaky?

Isolate mutable state, pin dependencies and data snapshots, record model configuration, classify infrastructure errors separately, use deterministic graders for objective requirements, repeat only where the product question needs it, and compare distributions rather than expecting identical prose. Do not hide variance with unlimited retries.

What should every eval result record?

At minimum: case and trial IDs, application commit/build, dataset and prompt hashes, model/provider configuration, retrieval and tool versions, raw output, final state, observable trace, grader versions, evidence, status, timing, usage, and environment substitutions. Apply redaction and retention policy before storage.

How many trials should each case run?

There is no universal number. Use one for fast deterministic smoke checks, and repeat cases when variability or the release decision requires a reliability estimate. Base the count on risk, observed variance, desired uncertainty, and budget. Report the policy and raw outcomes.

Should CI fail when the provider times out?

The run may need to block because evidence is incomplete, but record the event as an infrastructure error, not a product-quality failure. Set a reviewed tolerance for incomplete trials, preserve attempts, and ensure the report tells owners whether to debug the application, grader, or environment.

How often should the dataset change?

Add real failures, new requirements, distribution shifts, and adversarial cases through review. Keep stable IDs and version every change. Compare candidate and baseline on the same case set, then report new-case health separately so denominator changes cannot masquerade as quality movement.