Install Promptfoo Agent Skills in Codex and Claude Code

Promptfoo agent skills can be installed as the official four-skill plugin bundle or copied into a project as Agent Skills. In Claude Code, the verified marketplace commands are /plugin marketplace add promptfoo/promptfoo followed by /plugin install promptfoo@promptfoo. In Codex, Promptfoo publishes the same bundle through .agents/plugins/marketplace.json; use the Codex plugin import available in your workspace, or copy the reviewed skill directories into the project's .agents/skills/ directory. Do not invent a Codex slash command when your installed surface does not expose one. After either installation, inspect every SKILL.md, constrain agent permissions, generate one disposable config, run promptfoo validate, and review the diff before enabling broader work.

The Promptfoo complete guide explains the evaluation framework. Also read what the OpenAI acquisition announcement means, how to red-team an MCP server, and how to evaluate Codex and Claude agents. The LLM evaluation CI gate guide provides the release-policy layer. Browse QA skills, including the Playwright CLI skill, for related repository-owned instructions.

This tutorial reflects the official Promptfoo Agent Skills page updated July 10, 2026, the current Promptfoo source tree, Anthropic's Claude Code plugin documentation, and OpenAI's July 2026 Codex plugin guidance. Promptfoo 0.121.18 was the npm registry baseline checked July 14. The skill bundle is source content in the Promptfoo repository; the CLI package version and skill source commit should be recorded separately because they can change on different schedules.

Know what the official bundle contains

Promptfoo currently publishes one plugin bundle with four focused skills. It intentionally does not include a meta-selector skill; the host routes from each skill's description. The bundle is represented under plugins/promptfoo, with Claude and Codex marketplace metadata in the repository.

Skill	Documented purpose	Typical artifact to review
promptfoo-evals	Author non-red-team suites, assertions, test cases, and result inspection	Config, prompts, test files, assertions, run command
promptfoo-provider-setup	Connect HTTP targets and JavaScript or Python custom providers	Target adapter, secret references, smoke call
promptfoo-redteam-setup	Build focused red-team configuration from endpoints, specs, or code	Purpose, inputs, plugins, strategies, exclusions
promptfoo-redteam-run	Execute generated probes, triage failures, and prepare narrow reruns	Result artifact, failure classification, rerun scope

The official guidance says the bundle teaches deterministic assertions before model-graded ones, file-based tests, explicit grader providers, Nunjucks environment syntax such as '{{env.API_KEY}}', and no-share handling for internal systems. Those instructions reduce common configuration mistakes; they do not certify the generated suite, grant credentials, or replace a domain reviewer.

Prerequisites and version baseline

Before installation, choose the ownership and scope:

1. Decide whether the skills are project-level, personal, or distributed as a plugin.
2. Confirm the repository permits agent instruction files and name their code owners.
3. Select the Promptfoo CLI version the generated configs must target.
4. Prepare a disposable eval target with synthetic data and least-privilege credentials.
5. Ensure the host can read the skill path but cannot write or execute beyond the test's needs.
6. Establish the normal formatter, secret scanner, config validator, and CI commands.

Promptfoo's current package documentation requires Node.js ^20.20.0 or >=22.22.0 and announces the end of Node 20 support on July 30, 2026. For a new runner, use a supported maintained Node line approved by your organization. The examples pin Promptfoo 0.121.18 because that is the verified July 14 version; update only after reviewing a later release.

Install the CLI dependency independently from the skills:

npm install --save-dev --save-exact promptfoo@0.121.18
npx promptfoo --version
npx promptfoo validate -c evals/promptfooconfig.yaml

The final command assumes a config already exists. On a new repository, use it after the agent creates the first draft. A successful schema validation does not prove that the target is reachable or that assertions encode the right requirement.

Install the full bundle in Claude Code

Run the following commands inside Claude Code, not in a normal shell:

/plugin marketplace add promptfoo/promptfoo
/plugin install promptfoo@promptfoo

Promptfoo says this installs all four current skills. The former marketplace package name promptfoo-evals represented only the eval skill; its current page instructs users of that older package to reinstall promptfoo@promptfoo for the complete bundle and future updates. Claude Code plugin skills are namespaced, so an explicit invocation can look like:

/promptfoo:promptfoo-evals Create a small eval suite for the JSON support endpoint.
Use deterministic schema and authorization assertions first. Write files only under
evals/support-json, use environment references for secrets, validate the config,
and stop before running any paid provider calls.

Claude can also select a skill from the task context. Explicit invocation is useful for installation smoke testing because it removes routing ambiguity. After installation or enablement changes, follow the host's documented plugin reload flow if the current session does not see the namespace.

Anthropic's plugin documentation explains that installed marketplace plugins are copied into a local cache rather than executed in place. Review the plugin source before installation, then review the installed version or recorded source revision under your supply-chain policy. A trusted marketplace name is not a code review.

Install in Codex through the published marketplace bundle

Promptfoo exposes the same plugin through its Codex marketplace file. That file names the promptfoo plugin and points at ./plugins/promptfoo. Current OpenAI guidance describes installing and managing plugins through the Codex plugin directory or workspace plugin controls; availability and installation can depend on plan, workspace, role, surface, and admin policy.

Use the marketplace import supported by your Codex workspace and select the Promptfoo plugin from that imported source. If an administrator manages plugins, ask for the skill-only bundle to be made available under the repository's normal installation policy. Review included skills before enabling them. The bundle itself does not grant app or repository permissions, but the Codex session can still have shell, filesystem, browser, MCP, or connected-app access from other configuration.

There is no portable Codex terminal command published on the Promptfoo Agent Skills page that is equivalent to Claude's two slash commands. That absence matters. Do not publish a guessed codex plugin install command. Use the installed Codex UI/workspace flow, or use the manual standards-based path below.

Manual project installation for Codex

The official page says Codex and other Agent Skills tools discover project skills under .agents/skills/. It also says that copying only promptfoo-evals creates an eval-only setup, while copying the whole plugins/promptfoo/skills directory preserves sibling handoffs for provider and red-team work.

This shell sequence implements those documented paths from a reviewed source commit:

git clone https://github.com/promptfoo/promptfoo.git vendor/promptfoo-source
git -C vendor/promptfoo-source checkout "$PROMPTFOO_SKILLS_COMMIT"

mkdir -p .agents/skills
cp -R vendor/promptfoo-source/plugins/promptfoo/skills/. .agents/skills/

find .agents/skills -mindepth 2 -maxdepth 2 -name SKILL.md -print
git diff -- .agents/skills

Set PROMPTFOO_SKILLS_COMMIT to a commit the team reviewed; do not leave it empty in automation. The commands are an implementation recipe based on the official repository layout, not a Promptfoo installer command. If the organization vendors third-party source elsewhere, adjust only the staging location, not the Codex discovery path.

For an eval-only copy, select plugins/promptfoo/skills/promptfoo-evals and place that directory at .agents/skills/promptfoo-evals. Do not copy just SKILL.md if its references are needed. The current skill directories include reference material, and provider/red-team skills can include scripts. Preserve the complete reviewed directory.

Manual project installation for Claude Code

Claude Code discovers project-level standalone skills under .claude/skills/. A manual full-bundle copy uses the same reviewed source but a different destination:

mkdir -p .claude/skills
cp -R vendor/promptfoo-source/plugins/promptfoo/skills/. .claude/skills/

find .claude/skills -mindepth 2 -maxdepth 2 -name SKILL.md -print
git diff -- .claude/skills

Project installation is normally preferable for a team because source control can record the exact instructions and reviewers can see changes. The Promptfoo page also documents ~/.claude/skills/ for personal installation, but a personal copy is harder for a repository owner to inventory and keep consistent. Do not depend on an unrecorded global skill for a release-critical workflow.

Installation choice	Update mechanism	Team visibility	Best use
Claude marketplace plugin	Marketplace/plugin refresh	Plugin settings plus source review	Complete bundle with namespaced invocation
Codex marketplace plugin	Codex workspace/plugin refresh	Workspace policy and plugin details	Managed bundle where the surface supports import
Project .agents/skills/	Reviewed source-copy diff	High; committed with repository	Codex projects requiring explicit instruction ownership
Project .claude/skills/	Reviewed source-copy diff	High; committed with repository	Claude Code projects requiring explicit instruction ownership
Personal ~/.claude/skills/	Individual manual update	Low	Private experimentation, not shared release policy

Review the skill as executable influence

A skill is Markdown plus references and sometimes scripts, but it can influence an agent that has powerful tools. Treat it like code. Review frontmatter descriptions, file references, suggested commands, provider examples, environment-variable handling, output-sharing behavior, generated paths, and any bundled scripts. Confirm that examples align with the CLI version pinned in the repository.

Apply four boundaries:

• The skill may propose files only in an approved eval directory.
• It may reference secrets only through approved environment-variable names.
• It may validate configs without making provider calls unless the task permits cost and data transfer.
• It may not weaken assertions, broaden red-team targets, enable sharing, or change CI release thresholds without review.

An agent can follow a good skill badly, and a good result can occur without skill use. Installation verification must cover discovery, routing, artifact quality, and normal repository checks separately.

Run a zero-cost activation smoke test

Ask each host to create a config that uses a local fixture or an intentionally non-executed provider stub. Stop before a remote evaluation. The prompt should name scope, output, and prohibitions:

Use the promptfoo-evals skill. Inspect existing conventions first. Create an eval draft
under evals/skill-smoke for a JSON response with fields status and requestId. Use an
is-json assertion and a JavaScript assertion for the two required fields. Reference
secrets only as Nunjucks environment variables. Do not run a provider, do not share
results, and do not edit application code. Validate the config and summarize the diff.

Then verify the files yourself:

npx promptfoo validate -c evals/skill-smoke/promptfooconfig.yaml
git diff --check
git diff -- evals/skill-smoke .agents/skills .claude/skills

The current Promptfoo command reference says promptfoo validate validates the configuration and test suite and exits with code 1 on validation failure. That makes it suitable for CI, but it remains structural validation. Review whether the assertion can fail for the intended defect and whether the target adapter sends the correct inputs.

Test skill routing, not just skill presence

The official Test Agent Skills guide distinguishes two questions: whether the agent used the relevant skill and whether that skill produced better work. It also recommends testing routing boundaries in bundles. A request to connect an HTTP target should activate provider setup, not silently become a broad eval authoring task. A request to triage existing scan failures should not regenerate probes by default.

Create four positive prompts, one per skill, plus near-miss prompts that belong to siblings. Inspect host traces where available. Promptfoo documents a skill-used assertion for supported agent providers, but repository installation does not need to wait for a full meta-eval. A human can first inspect the selected SKILL.md, output path, and generated commands.

Test request	Expected skill	Wrong-route signal
Add deterministic assertions to an existing suite	promptfoo-evals	Rewrites provider transport
Connect a multi-field invoice HTTP endpoint	promptfoo-provider-setup	Flattens IDs and message into one prompt
Design RBAC and BOLA probes from an endpoint contract	promptfoo-redteam-setup	Launches a broad scan before boundary review
Triage existing red-team failures and prepare a narrow rerun	promptfoo-redteam-run	Regenerates all cases or edits target code

Routing quality is part of safety. A provider setup task can expose credentials; a red-team run can incur cost and send adversarial content; an eval edit can change a release oracle. The skill description should narrow behavior, not grant blanket autonomy.

Repository policy and ownership

Commit project skills only after third-party review. Add code ownership for .agents/skills/**, .claude/skills/**, evals/**, and CI files. Require a source URL, reviewed commit, import date, expected CLI range, and local modifications record. If your team customizes a skill, keep the patch small and explain why it exists.

A useful inventory file can be ordinary Markdown or JSON; it does not need a new tool. Record:

• Source repository and exact commit.
• Installed skill names and destination paths.
• Files changed locally after import.
• Promptfoo CLI version used by examples.
• Reviewer and review date.
• Permitted commands, directories, providers, and secret names.
• Upgrade and rollback procedure.

Do not allow an automatic dependency bot to overwrite agent instructions and merge based only on formatting. Skill changes can alter command choice, data handling, or assertion philosophy without changing executable package code.

Upgrade without instruction drift

Marketplace installs should be refreshed through the host's documented plugin mechanism. Manual installs should be updated from a new reviewed commit into a temporary staging directory, then diffed against both upstream and local customizations. Never copy over the active directory before preserving the previous source identity.

Review these changes especially closely:

1. New scripts or executable references.
2. New provider, network, sharing, or telemetry instructions.
3. Changed secret names or inline credential examples.
4. Changed red-team plugins, strategies, test counts, or target assumptions.
5. Changed validation and run commands.
6. Changed handoff behavior among the four skills.
7. Any instruction that permits broader file edits or shell access.

After an update, repeat the activation and routing suite. Run a known config through both the old and new instruction versions in disposable workspaces if the change is material. The test should compare artifacts and verification outcomes, not style alone.

Diagnose installation and activation failures

Symptom	Likely boundary	Next check
Claude cannot find /promptfoo:promptfoo-evals	Marketplace not added, plugin not installed/enabled, or session not reloaded	Inspect plugin manager and namespace, then reload through documented flow
Codex does not select a copied skill	Wrong discovery path, invalid frontmatter, or task does not match description	Confirm .agents/skills/<name>/SKILL.md and inspect the task wording
Only eval authoring appears	Old eval-only package or only one directory was copied	Install current promptfoo@promptfoo bundle or copy all four directories
References cannot be opened	Only SKILL.md was copied	Restore the complete skill directory including references/scripts
Agent writes invalid environment syntax	Skill not loaded or local edits are stale	Confirm loaded source; Promptfoo configs use {{env.NAME}} syntax
Validation passes but provider call fails	Schema is valid; target/auth/network is not	Run promptfoo validate target -c <config> only with approved credentials
Agent changes application code	Prompt scope or host permissions are too broad	Stop, inspect diff, narrow allowed paths, and rerun in a disposable branch
Upgrade removes local policy	Upstream copy overwrote customization	Restore from source control and adopt a staged three-way review

Do not fix discovery by placing duplicate skills in every global and project directory. Duplicate names make source selection harder to explain. Choose one authoritative project path per host, or document precedence explicitly.

CI and release controls

CI should validate artifacts created with the skill, not execute the skill itself on every merge. Pin the CLI, run promptfoo validate, execute an approved small suite, retain controlled output, and run the repository's normal tests. Agent generation belongs in a reviewed change; deterministic verification belongs in the gate.

name: eval-config-gate

on:
  pull_request:
    paths:
      - evals/**
      - .agents/skills/**
      - .claude/skills/**

jobs:
  validate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx promptfoo validate -c evals/promptfooconfig.yaml

Add live eval execution only when its data, cost, provider credentials, and output retention are approved. A skill update should not silently increase the number of calls or add model-graded assertions to a per-commit gate.

Mistakes and limitations

• Confusing Agent Skills with the Promptfoo CLI; both must be installed and versioned appropriately.
• Publishing a guessed Codex installation command instead of using the current plugin UI or documented .agents/skills/ path.
• Installing the former eval-only Claude package and expecting red-team handoffs.
• Copying only SKILL.md while omitting references or scripts.
• Giving the host unrestricted shell and write access because the skill text looks safe.
• Accepting generated llm-rubric assertions when deterministic checks can prove the requirement.
• Letting an agent run paid or sensitive targets during an installation smoke test.
• Updating skills from a moving branch without recording a commit and reviewing the diff.

Agent Skills improve context and workflow consistency; they do not make model output deterministic, prove test completeness, or authorize actions. Host behavior, plugin availability, discovery paths, and permission prompts can differ by Codex or Claude Code version and workspace policy. Re-check the official pages before distributing installation instructions.

Frequently Asked Questions

How many Promptfoo agent skills are in the current bundle?

Four: eval authoring, provider setup, red-team setup, and red-team execution/triage. The current bundle intentionally has no selector skill.

What are the verified Claude Code install commands?

Use /plugin marketplace add promptfoo/promptfoo, then /plugin install promptfoo@promptfoo inside Claude Code.

Where should Promptfoo skills live in a Codex project?

For a manual project install, place each complete skill directory under .agents/skills/<skill-name>/ so its SKILL.md and references remain together.

Is there an official Promptfoo Codex shell install command?

The July Promptfoo page documents the Codex marketplace bundle and manual .agents/skills/ destination, but it does not publish a portable shell command equivalent to Claude's slash commands. Use the installed Codex plugin workflow or manual copy.

Should the skills be committed to source control?

Promptfoo recommends committing project skills so teammates receive them. Do so only after source review, with the upstream commit, owner, and local changes recorded.

How do I verify the skill was actually used?

Start with explicit invocation in Claude or a task that clearly matches the Codex skill description. For automated comparison, Promptfoo's supported agent providers expose skill-use evidence and a skill-used assertion.

Can a skill safely receive API keys?

The generated config should reference approved environment variables, not contain keys. The host and CI still need least-privilege secret injection and output controls.

How should I update a manual installation?

Fetch a reviewed upstream commit into staging, diff it against the installed source and local changes, rerun routing and validation checks, then merge through normal code review.

Conclusion: install instructions and controls together

The reliable installation is not merely a copied folder. It is a reviewed source revision, the correct host discovery path, constrained permissions, an activation test, valid Promptfoo configuration, CI verification, and an upgrade owner. Use the full marketplace bundle when you need eval, provider, and red-team handoffs; use a manual project copy when repository-level visibility is more important.

Continue with the Promptfoo MCP provider tutorial for a concrete security target, or use the Codex versus Claude evaluation guide to test whether the installed skill changes agent behavior under controlled conditions.