Evaluate Codex vs Claude Coding Agents with Promptfoo

A promptfoo coding agent evaluation should compare complete agent systems, not brand names or isolated chat answers. Promptfoo currently supports openai:codex-sdk and anthropic:claude-agent-sdk providers, so the same task set can exercise repository reading, structured output, and controlled tool use. Fairness requires matched repository snapshots, task instructions, allowed capabilities, verification, time and budget policy, and repeated fresh trials. Grade final repository behavior with deterministic tests and hidden checks before using a model judge. Report distributions, errors, cost coverage, and review notes; do not publish a winner from one run or from unsupported head-to-head claims.

Read the Promptfoo complete guide, the OpenAI-Promptfoo acquisition analysis, the Agent Skills installation guide, and the MCP provider security tutorial for the rest of this cluster. The existing non-deterministic AI-agent testing guide explains why one pass is weak evidence. Explore the skills directory and Playwright CLI skill for reusable QA workflows.

The provider and configuration facts below were verified against Promptfoo documentation updated July 14, 2026, with Promptfoo 0.121.18 as the registry baseline. The current coding-agent guide covers the OpenAI Codex SDK, Codex app-server, Claude Agent SDK, OpenCode SDK, and plain-model baselines. This article narrows the experiment to the Codex SDK and Claude Agent SDK because they are the closest CI-oriented coding-agent surfaces. Model IDs, prices, and capabilities can change; pin and record what you actually run.

Define the comparison claim before choosing tasks

"Codex versus Claude" is not one measurable question. A coding agent includes model, SDK, system instructions, tools, permissions, working directory, authentication, network policy, session state, retries, and host behavior. Change several at once and the result cannot explain why one output differed.

Choose one bounded claim, for example:

• Which configured agent finds more seeded authorization defects in a read-only TypeScript repository?
• Which agent produces a patch that passes the same hidden unit and integration tests?
• Which agent follows a repository policy without editing forbidden paths?
• Which agent invokes a required local skill for the correct tasks and avoids it for near-miss tasks?
• Which agent reaches an acceptable result with lower measured latency or known cost under matched controls?

Do not claim a universal "best coding agent" from these experiments. Results apply to the named models, provider versions, harness, tasks, permissions, and date. A model update, SDK release, different reasoning setting, or changed repository can reverse a narrow result without invalidating the earlier observation.

Current provider boundaries

Promptfoo's official guide says agent evaluations differ from ordinary model calls because nondeterminism compounds across tool decisions, intermediate steps matter, and the harness controls capability. A plain model cannot be prompted into filesystem access. The SDK provider is therefore part of the test object.

Dimension	OpenAI Codex SDK provider	Claude Agent SDK provider	Fairness action
Provider ID	openai:codex-sdk	anthropic:claude-agent-sdk	Record exact Promptfoo and SDK versions
Working directory	Explicit; relative to config directory	Explicit; relative to config directory	Start from byte-identical snapshots
Default write posture	Filesystem sandbox policy; use read-only for inspection	Working directories are read-only until write tools are enabled	Match effective capabilities, not option names
Structured output	output_schema; returned to assertions as a string	output_format; may reach assertions as parsed data	Normalize before grading
Tool evidence	Streaming/tracing can capture shell, MCP, search, and file events	Tool calls and metadata can be surfaced	Require traces only when path matters
Authentication	Codex login state or API credentials according to provider setup	API key or existing Claude Code session with precheck disabled	Use separate least-privilege test identities
Session state	Can persist threads under documented controls	Sessions can persist unless disabled	Disable persistence for independent trials
Cost field	Estimate depends on known pricing metadata; some models may be undefined	Depends on SDK/provider usage metadata	Report missing cost, never convert it to zero

Equivalent capability is more important than identical syntax. A Codex read-only sandbox and a Claude working directory with no write/Bash tools can both support a repository review, even though their configuration fields differ. For patch tasks, create disposable copies and explicitly permit only the necessary edit and test commands on each side.

Prerequisites and version record

Create a manifest before the first run:

1. Promptfoo version and package integrity from the lockfile.
2. @openai/codex-sdk and @anthropic-ai/claude-agent-sdk versions.
3. Agent model identifiers and relevant reasoning/effort settings.
4. Base repository commit and fixture archive digest.
5. Task dataset version, hidden verifier version, and exclusion rules.
6. Effective filesystem, shell, network, search, MCP, and secret permissions.
7. Runner image, CPU architecture, operating system, and time limit.
8. Authentication route and billing account class without recording credentials.
9. Cache, retry, session-persistence, and repeated-trial settings.
10. Result and trace retention policy.

Promptfoo currently requires Node.js ^20.20.0 or >=22.22.0 and warns that Node 20 support ends July 30, 2026. Its Codex provider page says the SDK package is optional and must be installed when omitted; the current page also gives a minimum SDK for its newest documented model. The Claude provider similarly requires @anthropic-ai/claude-agent-sdk. Pin reviewed versions rather than relying on transitive optional dependencies.

npm install --save-dev --save-exact \
  promptfoo@0.121.18 \
  @openai/codex-sdk@0.144.0 \
  @anthropic-ai/claude-agent-sdk@0.3.161

npx promptfoo --version
npm ls promptfoo @openai/codex-sdk @anthropic-ai/claude-agent-sdk

Those SDK versions are compatibility baselines documented by the current provider pages for specific capabilities, not recommendations that they remain latest. Resolve and review the lockfile on the evaluation date. If the selected model requires a newer SDK, update the manifest and rerun all agents rather than changing one side mid-study.

Design tasks that have objective evidence

A useful task includes a starting repository, a user request, allowed operations, forbidden operations, expected externally observable behavior, and a verifier. It must be solvable without hidden context that one agent happens to receive.

Use several task families:

Family	Example task	Strong verifier	Leakage risk
Defect discovery	Find seeded tenant-isolation bugs	Exact issue IDs mapped to seeded defects	Filenames or comments reveal seeds
Small repair	Fix parser handling of duplicate keys	Hidden unit tests plus unchanged public tests	Hidden cases accidentally committed
Feature work	Add bounded retry with cancellation	Contract tests, type checks, integration test	Prompt mirrors hidden assertions too closely
Policy adherence	Modify only src/parser/**	Diff allowlist and normal tests	Repository instructions differ by agent
Skill routing	Use review skill only for security review	Skill-use trace plus output quality	Skill exists in one host path only
Diagnosis	Explain a failing deterministic test	Root-cause label and cited evidence	Agent can read answer in fixture history

Avoid trivia, generic code generation, and tasks where a human cannot define success. "Improve this repository" invites incomparable scope. "Fix the seeded duplicate-key acceptance without changing public API; pass hidden parser tests; do not edit fixtures" is measurable.

Do not expose hidden tests to the agent's working directory if they are intended to verify generalization. Run them after the agent stops. Public tests can remain visible as normal developer feedback. Use multiple repositories or task shapes so one framework convention does not dominate the study.

Start with a read-only harness

Read-only defect discovery is the easiest fair baseline because both providers can inspect the same immutable snapshot without coordinating writes. The config below uses provider-specific structured-output fields and one shared schema repeated explicitly. It uses current model examples from Promptfoo's provider documentation; replace them only with model IDs supported and approved at run time.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Read-only coding-agent defect discovery

prompts:
  - |
    Review the repository for the security defects described in {{task}}.
    Do not modify files, run network calls, or claim a defect without file evidence.
    Return only the required structured result.

providers:
  - id: openai:codex-sdk
    label: codex-readonly
    config:
      model: gpt-5.6-sol
      working_dir: ./fixtures/review-repo
      sandbox_mode: read-only
      network_access_enabled: false
      web_search_enabled: false
      enable_streaming: true
      output_schema:
        type: object
        required: [issues, summary]
        additionalProperties: false
        properties:
          issues:
            type: array
            items:
              type: object
              required: [id, file, explanation]
              properties:
                id: { type: string }
                file: { type: string }
                explanation: { type: string }
          summary: { type: string }

  - id: anthropic:claude-agent-sdk
    label: claude-readonly
    config:
      model: claude-sonnet-4-6
      working_dir: ./fixtures/review-repo
      disallowed_tools: ['Bash', 'Write', 'Edit', 'MultiEdit']
      persist_session: false
      output_format:
        type: json_schema
        schema:
          type: object
          required: [issues, summary]
          additionalProperties: false
          properties:
            issues:
              type: array
              items:
                type: object
                required: [id, file, explanation]
                properties:
                  id: { type: string }
                  file: { type: string }
                  explanation: { type: string }
            summary: { type: string }

tests:
  - description: Detects seeded authorization defects
    vars:
      task: Check object ownership and function-level authorization in the invoice service.
      expectedIssueIds: ['cross-tenant-read', 'admin-action-without-role-check']
    assert:
      - type: is-json
      - type: javascript
        value: |
          const value = typeof output === 'string' ? JSON.parse(output) : output;
          const found = new Set((value.issues || []).map((item) => item.id));
          const expected = context.vars.expectedIssueIds;
          const hits = expected.filter((id) => found.has(id));
          return {
            pass: hits.length === expected.length,
            score: hits.length / expected.length,
            reason: 'Matched ' + hits.length + ' of ' + expected.length + ' seeded issues',
          };

The issue IDs in this example are fixture-owned labels, not product claims or benchmark results. Put the actual seeded defects and mapping under review. The JavaScript assertion handles the documented difference between Codex string output and Claude parsed structured output.

Do not assume configuration keys alone create identical security. Inspect effective tool events. If one provider reads symlinks outside the fixture, inherits extra instructions, or receives different environment variables, the run is not matched.

Evaluate write-capable tasks in disposable workspaces

Never let both providers mutate the same directory. Create one clean copy per provider and trial from an immutable fixture archive. The orchestrator should verify the archive digest, create a new directory, run one provider, execute hidden verification outside the agent session, collect a diff, and destroy or quarantine the workspace according to policy.

set -eu

provider="$1"
trial="$2"
workspace="workspaces/${provider}/${trial}"

mkdir -p "$workspace"
tar -xzf fixtures/task-repo.tar.gz -C "$workspace"

npx promptfoo eval \
  -c "configs/${provider}.yaml" \
  --no-cache \
  --no-share \
  -o "artifacts/${provider}-${trial}.json"

npm --prefix "$workspace" test
git -C "$workspace" diff --check

This is a repository-owned orchestration pattern, not a Promptfoo built-in benchmark command. Add archive hash verification, timeouts, cleanup, and hidden test execution appropriate to your runner. If the task permits shell access, isolate the workspace in a container or equivalent sandbox with no production credentials and constrained network.

For Claude, the current provider docs require explicit write/edit tools and a permission mode such as acceptEdits for unattended modification. For Codex, configure a workspace-write sandbox and approval policy appropriate to the runner. Do not make both unrestricted merely to claim symmetry. Match the minimum operations required by the task and record the resulting capability difference.

Grade behavior, not self-report

The official coding-agent guide warns that final output describes what an agent says it did, not necessarily the files. For patch tasks, grade repository state after execution. A strong grader stack has layers:

1. Harness validity: Did the provider start, finish, and return a result rather than an infrastructure error?
2. Scope: Did the diff stay within allowed files and avoid secrets, generated artifacts, or disabled tests?
3. Build quality: Do formatting, types, lint, and compilation pass?
4. Public regression: Do existing repository tests still pass?
5. Hidden correctness: Do independent tests prove the requested behavior and negative cases?
6. Security and policy: Did the change preserve authorization, validation, and repository rules?
7. Process evidence: If relevant, did traces show required test execution or skill use?
8. Qualitative review: Is the implementation maintainable and aligned with local design?

Use deterministic graders for the first six. Trace assertions can verify a required command or skill path, but avoid rewarding unnecessary tool volume. A model judge may assess explanation clarity or maintainability after calibration against human labels. Never let a model judge overrule a failed hidden security test.

Signal	Report form	Interpretation limit
Hidden test pass	Per task and failure class	Only covers encoded cases
Allowed-path compliance	Boolean plus offending paths	Does not prove semantic correctness
Seeded defect recall	Hits / known defects	Depends on seed representativeness
False-positive count	Unsupported issue IDs after review	Human adjudication may disagree
Runtime	Median and spread across fresh trials	Includes runner and provider variance
Token usage	Provider-reported fields	Accounting shapes may differ
Estimated cost	Value plus coverage flag	Missing estimate is unknown, not zero
Tool events	Counts and required/forbidden events	More actions are not automatically better
Reviewer rating	Rubric dimensions and disagreement	Subjective and reviewer-dependent

Treat cost and latency as secondary constraints

Measure quality first. A fast agent that fails the task is not a bargain, and a correct agent with unacceptable cost may not fit the workflow. Report cost and latency only for trials that reached a comparable terminal state, while still listing failures separately.

Promptfoo documents cost and latency assertions, but thresholds must come from your workflow. Establish a pilot distribution, choose an operational budget, and document whether warm caches, authentication, model queueing, dependency installation, and test execution are included. Use fresh runs for model behavior and a consistent runner for both providers.

The current Codex provider page states that cost estimation is available only when Promptfoo knows the model's pricing metadata, and that GPT-5.6 cost remains undefined because Codex does not report cache-write tokens. Do not turn that undefined value into zero or compare it with a complete Claude estimate. Report token usage and externally reconciled billing where permitted, or mark the cost comparison incomplete.

Latency should be reported as a distribution, not a single fastest value. Include timeout and error rates. If one provider retries transparently, its longer successful run may be more useful than another provider's early failure; keep outcome classes visible.

Repeat trials without manufacturing certainty

Promptfoo's current guide recommends --repeat 3 for prompts expected to be stable and --no-cache while developing. Three trials are a smoke sample, not a statistically decisive benchmark. Use more trials when the decision cost, variance, and task diversity justify them.

npx promptfoo validate -c evals/coding-agents/promptfooconfig.yaml
npx promptfoo eval \
  -c evals/coding-agents/promptfooconfig.yaml \
  --repeat 3 \
  --no-cache \
  --no-share \
  -o artifacts/coding-agent-results.json

Randomize provider execution order when shared infrastructure could create time bias. Keep task order stable or randomized under a recorded seed. Do not retry only the losing provider's failures or discard timeouts. If an infrastructure outage invalidates a trial, apply the same documented exclusion rule to both.

For each task-provider pair, report successes, deterministic failures, policy violations, provider errors, timeouts, and excluded trials. Show task-level results before averages. A 90 percent aggregate can hide complete failure on authorization work.

Include a plain-model baseline when it answers a question

The Promptfoo guide recommends a plain LLM baseline for tasks requiring file or tool access. That baseline can show whether the agent harness contributes beyond model text generation. It is not a fair competitor on a task that requires repository reads; failure is expected. Label it as a capability control.

Likewise, compare the same model through different harness tiers only when the question concerns the harness. Codex app-server exposes richer protocol events and approval behavior than the SDK, but it is a different surface and is currently described as experimental by Promptfoo. Do not substitute it into one side of a Codex-versus-Claude SDK comparison without changing the claim.

Skills create another controlled variable

If the agents can load local skills, decide whether the study compares default agents or repository-configured agents. For a default comparison, remove both hosts' custom skills and record instruction files. For a workflow comparison, install semantically matched instructions and verify discovery.

Promptfoo's Test Agent Skills guide uses .agents/skills/ for Codex and .claude/skills/ for Claude, with provider-specific skill-use evidence. A skill may improve routing or code conventions, but it also adds information. Do not give one agent a detailed testing skill and call the result a model comparison.

Test positive and negative routing. The agent should use a review skill for a security review and avoid it for an unrelated formatting task. Grade output quality independently from skill invocation, because a skill can be invoked and followed incorrectly.

Diagnose unequal or surprising results

Symptom	Likely confounder	Investigation
One agent cannot read files	Wrong working directory or tool/sandbox config	Print resolved config root and inspect tool events
One edits while the other only plans	Permissions are not equivalent	Compare effective write and shell capabilities
JSON assertion fails on Codex only	Structured output remains a string	Parse Codex output before field assertions
Claude reuses earlier context	Session persistence was not disabled	Set persist_session: false and use a fresh workspace
Results improve on second run	Cache, warm dependency, or retained state	Use --no-cache, clean directories, and randomized order
Cost shows zero for one side	Missing estimate was coerced	Preserve null/undefined and report coverage
Agent says tests passed but hidden tests fail	Self-report was graded	Grade repository state and test process directly
One model finds seeded IDs exactly	Fixture leakage or memorized labels	Remove answer-bearing comments and inspect repository history
Large pass-rate change after upgrade	Model, SDK, Promptfoo, task, or grader changed together	Change one layer at a time and rerun baseline

Retain enough redacted evidence to reproduce a failure: task ID, fixture digest, provider config hash, model and SDK versions, result class, relevant trace, diff, verifier output, and timestamps. Do not retain reasoning or source snapshots beyond policy merely because a tool can export them.

CI and release design

Use coding-agent evals like integration tests, not per-line unit tests. A pull request gate should contain a small, stable, low-cost task set that detects major harness or policy regressions. A scheduled suite can include more repositories, repeated trials, write-capable tasks, and human review. A pre-adoption study can be broader still.

Block a release on hard invariants such as forbidden file edits, secret exposure, disabled tests, hidden security failures, or harness errors above policy. Treat quality score, latency, and cost as decision metrics with explicit thresholds and uncertainty. Keep provider outages separate from product-quality failures, but do not silently pass the gate when evidence is missing.

Version every moving layer and require review when any changes. If a model alias points to a moving target, record the resolved identifier where the provider exposes it. Re-baseline intentionally rather than comparing a new model to old trials and calling drift a winner.

Mistakes and limitations

• Comparing one Codex run with one Claude run.
• Giving different repositories, hidden instructions, network access, or tool permissions.
• Grading the final explanation instead of the resulting files and tests.
• Reporting undefined cost as zero.
• Choosing only tasks that resemble one agent's documented examples.
• Using an LLM judge for exact behavior that hidden tests can prove.
• Allowing sessions, caches, or workspaces to leak across trials.
• Publishing a universal ranking from a narrow internal task set.
• Changing model, SDK, Promptfoo, grader, and fixtures simultaneously.

Promptfoo provides provider adapters and evaluation machinery, not a vendor-independent guarantee of perfect equivalence. Codex and Claude expose different system instructions, models, tools, authentication, output semantics, and usage accounting. A fair study controls what can be controlled and discloses what cannot.

Frequently Asked Questions

Can Promptfoo evaluate both Codex and Claude coding agents?

Yes. The current provider catalog and coding-agent guide document openai:codex-sdk and anthropic:claude-agent-sdk for agent evaluations.

Is one trial enough to choose an agent?

No. Agent decisions are nondeterministic and tool paths compound variation. Run fresh repeated trials across representative tasks and report the distribution.

Should both agents receive the same exact configuration fields?

No. Their SDKs use different fields. Match effective capabilities, task inputs, workspaces, verification, and budgets, then document unavoidable differences.

How should code changes be graded?

Run independent formatting, type, lint, public, hidden, security, and scope checks against the resulting workspace. Do not trust the agent's final summary alone.

Can I compare cost directly?

Only when both sides provide comparable complete usage and pricing data. The current Codex provider documentation notes cases where cost is undefined; report missing coverage instead of zero.

Why disable caches and persistent sessions?

They can reuse prior responses or context, making trials dependent. Fresh runs and clean workspaces better isolate current behavior.

Should agents be allowed to use the network?

Only when the task requires it and both sides receive a matched, constrained policy. Repository repair tasks are usually safer and more reproducible with network disabled.

Does a higher aggregate score prove a universally better agent?

No. It supports a conclusion only for the evaluated models, harness, tasks, permissions, graders, runner, and date. Publish that scope with the result.

Conclusion: compare evidence, not reputations

A defensible Codex-versus-Claude study begins with a narrow claim and ends with reproducible task-level evidence. Pin every layer, duplicate clean workspaces, match effective capabilities, grade externally, repeat fresh trials, preserve missing-data semantics, and inspect failures. The output may support a workflow decision; it should not be inflated into an invented benchmark.

Use the Agent Skills guide when the experiment includes repository instructions, and return to the Promptfoo pillar for configuration, assertions, datasets, and operational patterns.