OpenAI Evals Platform Shutdown: Migration Checklist for November 2026

OpenAI announced the Evals platform deprecation on June 3, 2026. Existing evals become read-only on October 31, 2026, and the Evals dashboard and API are scheduled to shut down on November 30, 2026. An OpenAI Evals shutdown migration should therefore export runnable configurations and historical results before October 31, move execution into version-controlled code or configuration, validate grader parity on frozen outputs, and switch CI before November. OpenAI's documented migration path is Promptfoo; a custom harness or another code-first framework is reasonable when production behavior cannot be represented by a declarative prompt matrix.

Start with the complete LLM testing guide, then use the sibling guides to build a production-matched harness, divide work among deterministic, model, and human graders, and evaluate agent tools and outcomes. The canonical OpenAI grader reference explains the source concepts being migrated, while the LLM CI quality-gates guide covers the destination release policy. Browse reusable QA instructions in /skills, including the Playwright CLI skill when browser evidence is part of an end-to-end eval.

The official timeline, without extrapolation

The controlling source is OpenAI's API deprecations page. It lists three Evals milestones and says that graders documented for eval workflows are included in the transition. The revised AgentKit announcement independently carries a June 3 update saying Agent Builder and Evals will no longer be available on the OpenAI platform from November 30 onward.

Date	Official platform state	Operational consequence
June 3, 2026	OpenAI notified Evals users that the product is deprecated	Freeze new platform-specific architecture and assign migration owners
October 31, 2026	Existing evals become read-only	Complete runnable-config, dataset, grader, and result exports before this date
November 30, 2026	Evals dashboard and API are scheduled to shut down	No CI, script, or application may still depend on the hosted Evals surfaces

Treat October 31 as the export deadline and November 30 as the dependency-removal deadline. A read-only project may still be viewable, but it cannot be your recovery plan for a missing grader or an incomplete dataset. If your organization has a Q4 release freeze, set an earlier internal cutover and leave time for parallel runs.

What is shutting down, and what is not

The notice names the Evals platform, including its dashboard and API. Do not rewrite that as "OpenAI is ending evaluation" or "the Responses API is shutting down." Evaluation remains an engineering practice, and model inference APIs are separate products. OpenAI's current evaluation best-practices guide still recommends task-specific tests, production-representative data, comprehensive logging, continuous evaluation, and human calibration. Those design principles survive the hosted product.

The June 3 notice also covers Agent Builder, but its recommended destinations differ: OpenAI points code-managed workflows to the Agents SDK and natural-language use cases to Workspace Agents. ChatKit remains available according to the notice. If one project combines Agent Builder workflows and hosted Evals, run two linked migrations: port the agent runtime first, then point the replacement eval runner at the ported runtime. Otherwise you can mistakenly validate the legacy workflow while production moves to different code.

Reusable prompt objects were announced in the same deprecation batch, with their own November 30 shutdown. Inventory them separately. A prompt embedded in an exported eval is test input; a reusable prompt fetched by production at runtime is an application dependency. Both need versioned ownership, but replacing one does not automatically replace the other.

Choose the destination before exporting

OpenAI's migration cookbook recommends Promptfoo and explains the conceptual mapping: prompts, providers, test data, and grading criteria become portable configuration and CLI-driven runs. That is the shortest supported route for an eval the platform can export.

Destination	Choose it when	Main migration work	Main limitation to test
Promptfoo configuration and CLI	The eval is primarily prompts, provider calls, variables, and assertions	Export, validate, pin dependencies, add CI and artifact retention	Unsupported graders or custom agent/provider behavior need manual setup
Application-owned pytest or test runner	Production behavior lives behind stable code interfaces and deterministic fixtures	Build adapters, result schema, repeat policy, reports, and grader orchestration	Your team owns concurrency, retries, observability, and maintenance
Inspect AI	Tasks need model-independent datasets, solvers/agents, scorers, sandboxes, and rich logs	Re-express tasks and scorers, configure models, preserve run metadata	It is a reimplementation, not an automatic OpenAI export target
Another evaluated framework	Existing organizational tooling already owns traces, datasets, or experiments	Map every artifact and prove equivalent decisions	Similar names do not imply equivalent scoring or run semantics

Do not select a replacement from a feature checklist alone. Select one representative eval from each shape you operate: simple classification, retrieval-grounded answer, model judge, custom code grader, tool-using agent, and external provider. Prove those vertical slices before migrating the full catalog. This reveals unsupported behavior while the source remains writable.

Phase 1: inventory every dependency

An inventory should connect each hosted object to a repository, runtime owner, data owner, and release consumer. Search code and CI definitions for Evals endpoint calls, SDK methods, stored eval IDs, dashboard URLs, and scripts that parse run output. Also search operational documentation; a manual approval checklist can depend on a dashboard even when no code does.

This shell example is intentionally a discovery aid, not a claim that every match is an Evals dependency:

rg -n --hidden   '(/v1/evals|client[.]evals|eval_id|platform[.]openai[.]com/.+eval|prompt_id)'   .github packages scripts docs

rg -n --hidden   '(OPENAI_API_KEY|OPENAI_PROJECT|OPENAI_ORG|promptfoo|inspect eval)'   .github packages scripts

For each true dependency, record:

1. Eval name and stable internal owner.
2. Dataset source, update process, privacy classification, and row count at export time.
3. Prompt or application version that produces candidate outputs.
4. Model/provider configuration, including any non-default generation settings.
5. Every grader, threshold, weight, and aggregation rule.
6. Historical baseline used for a release decision.
7. Consumers: pull-request check, nightly job, dashboard, analyst notebook, or manual review.
8. Required secrets, network access, tools, and test-environment state.
9. Retention and deletion obligations for inputs, outputs, traces, and annotations.

An eval with no owner should not silently migrate. Decide whether to retain, merge, or retire it, and preserve the decision. Migration is an opportunity to remove dead gates, but not to delete unexplained evidence.

Phase 2: export two different assets

OpenAI's cookbook distinguishes a runnable Promptfoo configuration from historical results. They are separate downloads with separate purposes. For a supported eval, select a completed run, choose Download runnable Promptfoo config, and review export warnings. If there is no completed run, the cookbook says to run one before export. Download results separately when past runs need to remain available for comparison or audit.

The distinction prevents two common mistakes:

• Importing old results does not create the configuration for future runs.
• Running an exported configuration creates a new Promptfoo run; it does not continue the hosted OpenAI run.

Store the raw exports immutably, then create a reviewed working copy. Add a manifest with source eval ID, source run ID, export timestamp, exporter, file hashes, warnings, dataset provenance, and any omitted component. Keep raw sensitive data out of a repository when policy forbids it; store a pointer and checksum instead.

Run the documented commands against the reviewed export:

promptfoo validate config -c evals/support-routing/promptfooconfig.yaml
promptfoo eval -c evals/support-routing/promptfooconfig.yaml --no-cache
promptfoo view

# Historical results are imported separately from runnable configuration.
promptfoo import exports/openai-evals/support-routing-results.json
promptfoo view

The OpenAI cookbook uses --no-cache for the first fresh run so cached outputs cannot disguise a provider or prompt mismatch. Keep that behavior during parity testing. Later, adopt caching only with an explicit cache key and a clear understanding of which inputs, prompts, models, and grader versions it covers.

Phase 3: make the destination reviewable

An exported file is a starting point, not a reviewed quality contract. Move prompts to named files, give tests stable IDs, separate secrets from configuration, and pin the Promptfoo version used in CI. The Promptfoo configuration guide defines prompts, providers, tests, variables, transforms, and assertions as the core configuration pieces.

The following small classification suite is an illustrative destination config, not a reproduction of OpenAI's export format. It uses documented Promptfoo fields and keeps the expected answer deterministic:

description: support-routing-regression

prompts:
  - file://prompts/route-support.txt

providers:
  - id: openai:gpt-5-mini
    config:
      temperature: 0

tests:
  - description: damaged item routes to returns
    vars:
      message: The mug arrived cracked. I need a replacement.
    assert:
      - type: equals
        value: returns

  - description: password failure routes to account support
    vars:
      message: My reset link expired and I cannot sign in.
    assert:
      - type: equals
        value: account_support

The prompt file must constrain the output to the approved labels. If production adds preprocessing, retrieval, policy logic, fallback models, or output normalization, calling a model directly is not parity. Use Promptfoo's documented custom or HTTP provider support, or place the suite behind the same application entry point production uses. The goal is not to preserve a dashboard-shaped test; it is to preserve the behavior and decision the test was meant to protect.

Phase 4: map graders by observable requirement

Do not translate grader names mechanically. Write down what each grader observed and what decision its score controlled. Then choose the destination assertion that sees the same evidence.

Source intent	Preferred code-first expression	Required parity evidence
Exact label or required string	Equality, membership, or normalized string assertion	Same frozen outputs produce the same pass/fail labels
Structured output	JSON parse plus schema or field invariants	Malformed and boundary fixtures fail for the same reason
Numeric tolerance	Explicit formula with units and inclusive/exclusive boundary	Values immediately below, at, and above the threshold
Semantic similarity	Documented similarity assertion with pinned model/config	Distribution comparison; exact scores may differ across systems
LLM judge	Versioned rubric, judge configuration, structured verdict	Agreement against frozen human labels and disagreement review
Python or custom logic	Repository-owned function with unit tests	Golden fixtures exercise exceptions, missing fields, and partial credit
Tool or agent workflow	Run production-like agent and grade outcome, state, and allowed actions	Equivalent environment, tools, permissions, stop conditions, and traces

OpenAI explicitly warns that similarity-based scores may not be numerically identical and that manually recreated graders, especially LLM judges, must be validated before they drive regression decisions. It also lists tools, agents, custom providers, and omitted graders as cases that may require manual setup. Treat an export warning as an unresolved test gap, not informational noise.

Phase 5: prove parity with frozen outputs

Provider sampling makes live A/B runs noisy. First compare graders on the same frozen candidate outputs. This isolates scorer migration from generator variation. Build a neutral fixture containing case ID, input, reference, candidate output, old verdict, old score when meaningful, and reviewer notes. Run the new grader over those candidates.

This application-owned Python example compares decisions without assuming either platform's result schema:

from dataclasses import dataclass


@dataclass(frozen=True)
class Decision:
    case_id: str
    passed: bool


def agreement(old: list[Decision], new: list[Decision]) -> dict[str, object]:
    old_by_id = {row.case_id: row.passed for row in old}
    new_by_id = {row.case_id: row.passed for row in new}
    if old_by_id.keys() != new_by_id.keys():
        raise ValueError("case IDs differ; compare identical fixtures")

    disagreements = [
        case_id
        for case_id in sorted(old_by_id)
        if old_by_id[case_id] != new_by_id[case_id]
    ]
    total = len(old_by_id)
    return {
        "cases": total,
        "agreement": (total - len(disagreements)) / total if total else 0.0,
        "disagreements": disagreements,
    }

Do not define an acceptable agreement threshold from this article. Set it from the consequence of a changed verdict and inspect every disagreement during migration. For deterministic graders, unexplained disagreement should normally block cutover. For model judges, compare both systems with adjudicated human labels; agreement with the old judge is not proof that either judge is correct.

After scorer parity, run old and new generators in parallel on the same dataset and environment. Separate generator differences, grader differences, infrastructure errors, and expected nondeterminism in the report. A single aggregate pass rate cannot tell you which layer moved.

Phase 6: rebuild CI as a controlled release dependency

The replacement job should run from a locked dependency graph, use least-privilege secrets, record the evaluated commit, and upload a full result artifact. Promptfoo's output documentation notes that rich exports can contain configuration, prompts, variables, raw outputs, reasons, and provider errors, while JUnit output intentionally omits much of that detail. Use JUnit for a compact check annotation and a restricted JSON or native artifact for diagnosis.

A migration-ready CI job should make these states distinct:

• Product failure: the application ran and a requirement failed.
• Grader failure: the candidate exists, but scoring could not complete.
• Infrastructure failure: provider, network, sandbox, or dependency prevented a valid trial.
• Insufficient evidence: a judge abstained or required human escalation.
• Policy exception: an authorized reviewer accepted a documented, time-bounded waiver.

Never convert all five to "red eval." Teams quickly learn to rerun an opaque red check until it becomes green. Preserve failure class, attempt count, error, case ID, configuration hashes, timing, and cost evidence so the owner can act.

Code-first alternatives when export is incomplete

A small application-owned harness is valid when it deliberately shares production adapters and has a stable result contract. Pytest can own fixtures and assertions; your code must own model invocation, tracing, repeated trials, concurrency limits, and reports. This route minimizes framework semantics but maximizes maintenance responsibility. The production harness sibling includes a complete architecture.

Inspect AI is another code-first option for task, solver or agent, scorer, sandbox, and log workflows. Its official evaluation log documentation describes logs containing task/model configuration, plan, aggregate results, usage, errors, and per-sample records. Current Inspect logs also support exporting run configuration for replay. Those capabilities are useful, but they do not make an OpenAI-hosted eval automatically portable; tasks and graders still need an explicit mapping and parity test.

Whichever option you choose, retain a framework-neutral case ID and dataset format. A future runner migration is far easier when business requirements are not encoded only in one vendor's object IDs.

Failure analysis during migration

The exported configuration will not validate

Read every export warning and identify omitted providers, graders, variables, or transforms. Do not patch until the source behavior is documented. Compare against Promptfoo's current schema, then make the smallest reviewed correction. Record the manual mapping in the migration manifest.

The first new run is unexpectedly perfect

Check caching, dataset selection, empty test discovery, and whether assertions were actually attached. Confirm case count and force a known-bad candidate through each grader. A quality gate that cannot fail its negative control is not active.

Scores move but pass/fail mostly agrees

Similarity implementations and model judges can produce different scales. Recalibrate thresholds against labeled fixtures instead of copying the old number. Preserve score distributions and boundary cases; an unchanged average can hide changed verdicts near the gate.

Agent evals fail only in the replacement

Compare tools, schemas, credentials, approvals, network, initial state, timeouts, maximum turns, and stop logic. The agent harness is part of the evaluated system. If the replacement exposes different tools or truncates results, it is not running the same task.

Parallel trials contaminate one another

Give every trial a clean namespace, database state, filesystem, cache prefix, and external resource allocation. Anthropic's agent-eval guidance warns that leftover state and shared resource exhaustion can create correlated failures or artificial advantages.

Historical results import but cannot rerun

That is expected when only result history was exported. Import preserves reference evidence; the runnable configuration is a separate asset. Return to the source project and export a completed run's configuration before the read-only date, or manually reconstruct and validate it.

Cutover and rollback checklist

1. Finish inventory and assign one owner per eval and destination repository.
2. Export raw runnable configs, historical results, data, grader definitions, and warnings.
3. Hash and archive raw exports under the applicable retention policy.
4. Recreate unsupported components and add direct unit tests for custom graders.
5. Compare old and new graders on frozen outputs, including negative and boundary fixtures.
6. Run source and destination in parallel using the same production-matched application version.
7. Review disagreements, traces, latency, cost, and infrastructure-error rates by case.
8. Pin runner and grader dependencies; provision least-privilege CI secrets.
9. Switch the required check to the replacement while keeping the old run non-blocking briefly.
10. Remove all Evals API calls, IDs, secrets, dashboard runbooks, and scheduled jobs before November 30.
11. Preserve an application-version rollback, not a dependency on the retiring hosted service.
12. Test restore instructions from the archived dataset, config, and result artifacts.

Rollback means returning to the last known-good code-first configuration and application version. It must not mean re-enabling the hosted Evals API after its shutdown date. Practice the rollback before disabling the old gate.

Version scope and limitations

This guide is current to July 14, 2026 and follows the dates and replacement path published by OpenAI. Deprecation schedules can change, so the official deprecations page remains authoritative. The article does not claim that the OpenAI model APIs, Responses API, Agents SDK, ChatKit, evaluation as a practice, or every AgentKit component are shutting down.

Promptfoo commands and fields here match the linked primary documentation available on the update date. Pin and verify the version used by your repository; do not put @latest into a controlled quality gate. The example configuration is deliberately small and does not promise byte-for-byte equivalence with an exported file. Custom graders, external providers, tools, agents, retrieval, and stateful workflows require case-specific work.

No threshold in this guide is a universal release standard. Preserve your organization's labeled evidence, risk analysis, and reviewer decisions. Migration parity proves that a replacement measures the intended contract under a recorded setup; it does not prove the contract covers every production failure.

Frequently Asked Questions

When did OpenAI announce the Evals platform shutdown?

OpenAI says it notified developers on June 3, 2026 that the Evals platform was being deprecated. The official schedule makes existing evals read-only on October 31 and schedules the dashboard and API to shut down on November 30, 2026.

Does the OpenAI Evals API continue after November 30, 2026?

No, not according to the current official schedule. The Evals dashboard and API are both scheduled to shut down on November 30. Remove CI and script dependencies rather than assuming API-only use survives the dashboard retirement.

Is Promptfoo the official migration path?

OpenAI's deprecations page links to its cookbook titled "Moving from OpenAI Evals to Promptfoo," and that cookbook says OpenAI recommends Promptfoo for continuing and extending eval workflows. Supported evals can export runnable Promptfoo configuration; unsupported pieces need manual setup.

Do historical result exports include a runnable eval?

No. OpenAI documents runnable configuration and historical results as separate exports. Importing results makes past evidence available for reference, but it does not define prompts, providers, test variables, and assertions for a future run.

Must every team use Promptfoo?

No. It is the documented OpenAI path and should be evaluated first. A custom pytest harness, Inspect AI, or an established internal framework can be a better fit when the eval must execute a complex production application. The burden is explicit mapping, parity evidence, CI ownership, and durable logs.

How should we migrate an LLM-as-a-judge grader?

Freeze candidate outputs, recreate the rubric and judge configuration, collect new verdicts, and compare both old and new judges with adjudicated human labels. Review disagreements rather than merely copying a numeric threshold. OpenAI specifically warns that manually recreated judges require validation.

What should be completed before October 31?

Complete exports while projects remain writable: runnable configs, historical results, datasets, graders, warnings, metadata, and any completed source runs needed for export. Ideally, finish parity testing and CI cutover too. Waiting until read-only leaves no room to repair the source project.

Does this shutdown mean teams should stop writing evals?

No. It means teams should move eval ownership out of the retiring hosted product. OpenAI and Anthropic both continue to recommend eval-driven development, production-representative tasks, multiple grader types, logging, repeated trials where appropriate, and human calibration.