Test and Red-Team an MCP Server with Promptfoo's MCP Provider

The promptfoo mcp provider treats an MCP server itself as the target under test. Configure id: mcp, point config.server at a local command or remote URL, and send each prompt as a JSON tool call containing tool and args. Start with deterministic contract, authorization, and error assertions. Then add a focused Promptfoo red-team configuration for risks supported by the server's actual tools and data. Keep credentials in environment references, filter destructive tools, run against an isolated tenant, and retain restricted result artifacts. The provider does not emulate a complete model-driven agent: it directly invokes MCP tools, and the official documentation lists JSON tool-call input and JSON-string responses among its current limits.

Start with the Promptfoo pillar, then review the OpenAI-Promptfoo acquisition facts, Promptfoo Agent Skills installation, and the Codex versus Claude agent harness. The existing Promptfoo red-team guide covers broader application testing. Browse the QA skills directory and the Playwright CLI skill for adjacent automation workflows.

This tutorial follows the official MCP provider page updated July 7, 2026, Promptfoo 0.121.18 as the July 14 package baseline, the current Promptfoo CLI and CI documentation, and the November 25, 2025 MCP authorization specification. Configuration names below come from those sources. Threat selection, environment design, approval rules, and release thresholds are recommendations for your system; Promptfoo cannot infer them from the protocol alone.

Understand the provider boundary

Promptfoo exposes two related MCP capabilities. The dedicated mcp provider makes a server the direct target and expects JSON tool-call prompts. A separate MCP integration can give another model provider access to MCP tools. Use the dedicated provider when you need repeatable server-level tests for tool discovery, argument handling, authorization, output, errors, and red-team probes. Use a model-plus-MCP integration when the decision-making behavior of the agent and its tool orchestration are part of the test object.

Boundary	Dedicated mcp provider	Model provider with MCP integration
Target under test	MCP server and its tools	Model or agent using MCP tools
Test input	JSON object with tool and args	Natural-language or provider-specific messages
Best evidence	Tool result, error, metadata, authorization behavior	Tool choice, sequence, model output, trace
Main use	Contract, negative, access-control, server red-team tests	Agent planning, tool selection, end-to-end behavior
Important limit	Does not prove an agent will choose the right tool	Adds model nondeterminism and a larger diagnostic surface

Do not claim an agent is secure because direct tool calls pass. An agent may select a dangerous tool, construct different arguments, reveal data in its final response, or combine several individually permitted operations. Conversely, a model refusal does not prove the MCP server enforces authorization. Test both layers when both carry risk.

Prerequisites and safe baseline

You need a running local or remote MCP server, Promptfoo, representative test identities and objects, and a disposable environment. The current Promptfoo package requires Node.js ^20.20.0 or >=22.22.0; its documentation also announces the end of Node 20 support on July 30, 2026. Pin the CLI and choose a supported CI runtime intentionally.

Before a first call, inventory:

1. Server transport and startup command or HTTPS URL.
2. Tool names, JSON schemas, side effects, and data classifications.
3. User roles, tenant identifiers, object ownership, and denied operations.
4. Authentication method, token audience, scopes, and expiry behavior.
5. Seed/reset procedure and proof that the target is non-production.
6. Result fields safe to retain, redact, or discard.
7. Timeout and concurrency limits that protect downstream systems.

Use synthetic object IDs that still preserve ownership relationships. A BOLA test needs an object belonging to a different test user; replacing every identifier with the same dummy value removes the boundary the test is meant to exercise.

Configure a local MCP server

The smallest official shape sets id: mcp, enables the provider, and gives a command plus arguments. This example adds a narrow tool allowlist and deterministic cases. Replace the server file and tool schema with your implementation.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Local invoice MCP contract tests

providers:
  - id: mcp
    label: invoice-mcp-local
    config:
      enabled: true
      server:
        command: node
        args: ['dist/mcp-server.js']
        name: invoice-test-server
      tools: ['get_invoice', 'list_invoices']
      exclude_tools: ['delete_invoice', 'refund_invoice']
      timeout: 60000

prompts:
  - '{{toolCall}}'

tests:
  - description: Owner can read a seeded invoice
    vars:
      toolCall: '{"tool":"get_invoice","args":{"user_id":"USER-A","invoice_id":"INV-A-100"}}'
    assert:
      - type: is-json
      - type: contains
        value: 'INV-A-100'

  - description: Unknown tool fails without invoking another tool
    vars:
      toolCall: '{"tool":"not_a_real_tool","args":{}}'
    assert:
      - type: contains
        value: 'not found'

The unknown-tool message is based on the provider's documented error class, but exact error text can evolve. Prefer a structured error transform or a stable error field if your release gate must survive wording changes. A contains assertion against prose is acceptable for a smoke check, not a durable authorization oracle.

Tool filtering reduces what Promptfoo can call through this provider. It is not server-side authorization. Keep destructive tools disabled in the test credential and target environment even when exclude_tools is configured. Defense must survive a different client.

Connect a remote server without embedding secrets

For a URL-based server, Promptfoo supports headers and an auth block. The current provider documentation covers bearer, basic, API-key, and OAuth configurations. Prefer the method the server officially implements. This example uses a short-lived bearer token injected through the environment:

providers:
  - id: mcp
    label: invoice-mcp-remote
    config:
      enabled: true
      server:
        url: https://mcp-test.example.internal/mcp
        name: invoice-test-remote
        auth:
          type: bearer
          token: '{{env.MCP_BEARER_TOKEN}}'
        headers:
          X-Test-Tenant: '{{env.MCP_TEST_TENANT}}'
      tools: ['get_invoice', 'list_invoices']
      timeout: 60000
      pingOnConnect: true

Do not commit a token, API key, Basic password, OAuth client secret, or copied session cookie. Masking a value in console output does not remove it from a config, process environment, child process, debug log, result artifact, shell history, or crash report. Use a CI secret store, test-only account, minimal scopes, and a token lifetime bounded to the run.

The MCP authorization specification requires audience-bound tokens and forbids token passthrough to upstream APIs. A server must validate that an access token was issued for that server. Promptfoo can exercise valid, missing, expired, wrong-audience, and insufficient-scope cases, but the harness cannot prove server implementation merely by reading configuration. Seed credentials that represent each condition and assert the observed denial.

Turn tool contracts into deterministic cases

Build a matrix from the tool schema and business policy. Every test should state actor, object, operation, input boundary, expected result class, and expected state change. Test success and failure separately.

Case family	Example	Primary oracle
Required arguments	Omit invoice_id	Structured invalid-arguments error; no side effect
Type and range	Negative refund amount	Schema or domain rejection
Unknown fields	Add unrecognized admin flag	Field rejected or ignored according to contract
Object ownership	User A requests User B invoice	Authorization denial and no sensitive fields
Function authorization	Read-only role calls refund tool	Function-level denial and no mutation
State transition	Refund already-refunded invoice	Idempotent/domain error; ledger unchanged
Injection boundary	SQL-like string in search field	Treated as data; no expanded result set
Output minimization	List invoices for a user	Only approved fields and owned objects returned
Timeout/cancellation	Controlled slow fixture	Bounded error with no duplicate side effect
Replay/idempotency	Repeat request with same key	Contract-defined result and single mutation

Use is-json, schema assertions, exact field comparisons, and JavaScript assertions before a model grader. Authorization, ownership, amount, state, and tool name are deterministic facts. An LLM rubric can help judge a free-text explanation, but it should not decide whether User A was allowed to read User B's record.

When an MCP result contains nested content blocks or structuredContent, the provider supports transformResponse. A transform can promote a stable result field into output and retain selected metadata. Keep the transform in a reviewed file when it becomes more than a short expression, and unit-test it independently.

// evals/mcp/normalize-result.js
module.exports = (result, content, context) => ({
  output: JSON.stringify({
    tool: context.toolName,
    data: result.structuredContent || null,
    text: content,
  }),
  metadata: {
    toolName: context.toolName,
  },
});

Reference it as transformResponse: 'file://evals/mcp/normalize-result.js'. The documented transform receives result, normalized content, and a context with tool name, arguments, and original payload. Avoid copying raw arguments into retained metadata when they may contain secrets or personal data.

Add focused authorization assertions

For each protected tool, create at least four identities: authorized owner, authenticated non-owner, authenticated wrong role, and unauthenticated or invalid-token caller. If the transport cannot vary credentials per test in one provider instance, define separate provider configurations whose secrets map to distinct CI jobs or configs. Do not simulate authorization only by changing a user_id argument while reusing an administrator token; that tests input trust, not real identity enforcement.

Assert both response and state. A tool that returns "forbidden" after performing the mutation is still vulnerable. Read back the affected object through an independent test fixture or service API, or inspect an audit sink designed for testing. Keep this verifier outside the MCP tool under test when feasible, so one broken implementation does not validate itself.

Denied responses should not reveal whether a cross-tenant object exists unless policy permits it. Compare error shape and timing cautiously; exact timing equality is rarely realistic, but gross differences can identify an enumeration channel worth investigation. Do not invent a universal latency threshold. Establish a controlled baseline on your infrastructure.

Red-team the real server boundary

Promptfoo's red-team configuration names targets, purpose, plugins, strategies, and test count. The MCP provider page currently recommends considering PII, broken function-level authorization, broken object-level authorization, and SQL injection for MCP systems where those risks apply. Select plugins from actual tools and data, not from a generic checklist.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Invoice MCP focused red team

targets:
  - id: mcp
    label: invoice-mcp-redteam
    config:
      enabled: true
      server:
        url: https://mcp-test.example.internal/mcp
        auth:
          type: bearer
          token: '{{env.MCP_BEARER_TOKEN}}'
      tools: ['get_invoice', 'search_invoices']
      exclude_tools: ['delete_invoice', 'refund_invoice']

redteam:
  purpose: >-
    Test-only invoice tools for signed-in customers. A caller may read only invoices
    owned by the caller's test tenant. The server must not expose another customer's
    invoice, hidden fields, credentials, or administrative functions.
  plugins:
    - pii
    - bfla
    - bola
    - sql-injection
  strategies:
    - basic
  numTests: 25

The plugin names and numTests example match the current provider documentation; they are not a claim that 25 probes provide comprehensive coverage. Review generated probes before execution. A red-team generator may produce irrelevant, malformed, duplicate, or unsafe inputs. Keep the target isolated and remove destructive tools rather than relying on generated text to stay harmless.

Run the current workflow with a pinned local dependency:

npx promptfoo validate -c evals/mcp/redteam-config.yaml
npx promptfoo redteam generate -c evals/mcp/redteam-config.yaml
npx promptfoo redteam eval -c evals/mcp/redteam.yaml --no-cache --no-share -o artifacts/mcp-redteam.json

Promptfoo documents redteam run as a shortcut that generates and evaluates. Splitting generation and evaluation is useful when policy requires probe review. Confirm command options against the pinned CLI help, because generated-file arguments and defaults can change. Never interpret an attack success rate without separating target failures, grader errors, connection errors, and excluded or malformed cases.

Threat cases beyond generated probes

Generated red teaming complements, but does not replace, hand-authored protocol and domain tests. Add cases for:

• Tool-name confusion and near-collision.
• Oversized or deeply nested arguments.
• Unicode normalization in identifiers.
• Duplicate keys and ambiguous serialization at HTTP boundaries.
• Cross-tenant IDs, stale sessions, and revoked roles.
• Wrong-audience and expired access tokens.
• Concurrent mutations and replayed idempotency keys.
• Tool output containing instructions that could influence an upstream agent.
• Error responses that echo secrets, stack traces, filesystem paths, or SQL.
• Downstream timeout after the operation commits but before the response returns.

The direct provider can observe server response, but agent prompt-injection resistance requires a model-plus-MCP test. A malicious tool result may be harmless in a direct JSON assertion and dangerous when an agent treats it as instructions. Preserve that distinction in the threat model.

Authentication and secret controls

For static test credentials, prefer a purpose-built account with no production access. For OAuth, use explicit token endpoints where possible, short-lived tokens, minimal scopes, and a resource/audience bound to the MCP server. Promptfoo's provider supports OAuth client-credentials and password-grant configuration, but protocol support does not make the password grant a recommended design. Follow the server's current authorization architecture and organizational security standard.

Avoid query-parameter API keys unless the server requires them; URLs can appear in logs and intermediaries. Keep debug and verbose off in normal CI. If diagnostics require them, run a minimal case with synthetic inputs and inspect logs before retention or sharing. Rotate a credential after any uncertain exposure.

CI and release controls

Use two lanes. A small deterministic contract suite can run on every relevant change. Broader red-team scans should run on a schedule, before high-risk releases, and after tool, authorization, retrieval, or policy changes. Both lanes need explicit target identity and a reset step.

name: mcp-security-gate

on:
  pull_request:
    paths:
      - mcp-server/**
      - evals/mcp/**

jobs:
  contract:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run seed:mcp-test
      - run: npx promptfoo validate -c evals/mcp/promptfooconfig.yaml
      - run: npx promptfoo eval -c evals/mcp/promptfooconfig.yaml --no-cache --no-share -o artifacts/mcp-contract.json
        env:
          MCP_BEARER_TOKEN: ${{ secrets.MCP_BEARER_TOKEN }}

npm run seed:mcp-test is a repository-owned illustrative script, not a Promptfoo command. Implement it to fail closed unless the target proves it is the intended disposable environment. The example uses GitHub's encrypted-secret expression for the bearer token; adapt secret injection to the selected CI platform. Add artifact retention only after reviewing content.

Release criteria should be case-based: no unauthorized state change, no cross-tenant disclosure, no unexpected tools, and no unclassified execution errors. Aggregate scores can supplement those invariants but should not average away a critical authorization failure.

Diagnose failures by layer

Symptom	Likely layer	Next action
Connection refused	Server startup, URL, transport, or network	Run server health check and inspect command/URL from the same runner
Invalid JSON prompt	Harness input	Validate the exact tool and args serialization
Tool not found	Name mismatch, filter, or server discovery	Inspect available tools with safe debug logging; compare exact name
Every call is unauthorized	Missing/expired token, audience, scope, or test tenant	Decode only non-secret token metadata safely; inspect server auth logs
Cross-tenant call succeeds	Server authorization defect or admin credential misuse	Stop the run, preserve minimal evidence, verify credential role and state
Timeout after progress	Tool duration or timeout policy	Use documented timeout/reset controls and inspect whether side effect committed
Assertion cannot parse output	Response shape or transform defect	Save a redacted sample and unit-test transformResponse
Red-team score changes with no code change	Generated probes, grader, provider, data, or environment changed	Compare probe IDs, grader config, target revision, and errors before concluding regression
CI passes while local fails	Cached outputs, different config root, secrets, or server build	Use --no-cache, print versions, and compare normalized non-secret configuration

Promptfoo documents MCP_DEBUG, MCP_VERBOSE, and provider debug/verbose controls. Enable them only for a narrow diagnostic run. Detailed tool arguments and results can be sensitive.

Mistakes and limits

• Sending natural language to the dedicated provider instead of the required JSON tool-call shape.
• Treating client-side exclude_tools as server authorization.
• Testing User B's object with an administrator token and calling the result a BOLA test.
• Enabling destructive tools in a shared environment for generated red teaming.
• Using an LLM grader for exact ownership, amount, state, or schema facts.
• Committing bearer tokens or API keys in YAML.
• Running redteam run without reviewing generated probes and target scope.
• Comparing aggregate scores while ignoring connection and grader errors.
• Claiming direct MCP tests prove agent tool-selection or prompt-injection safety.

The current provider requires standard MCP servers, uses JSON tool-call prompts, depends on remote server implementation for URL support, and returns tool responses as JSON strings according to its documented limitations. Protocol and provider features can evolve; re-check the official page and pinned CLI help during upgrades.

Frequently Asked Questions

What input format does the Promptfoo MCP provider require?

Each prompt must be JSON with a tool name and an args object. Natural-language agent instructions belong in a model provider test, not this direct provider.

Can it start a local MCP server?

Yes. Configure server.command and server.args. Promptfoo also supports a remote server.url configuration.

How should remote credentials be supplied?

Use the documented auth or header configuration with environment references, least-privilege test credentials, and restricted logs. Never commit the value.

Does exclude_tools secure the MCP server?

No. It limits tools exposed through that provider configuration. The server must independently authenticate, authorize, validate, and audit every call.

Which red-team plugins should I use?

Choose from the actual threat model. PII, BFLA, BOLA, and SQL injection are current MCP documentation examples, not a mandatory universal set.

Should I use redteam run in CI?

It combines generation and evaluation. Use separate generate/review/eval stages when probe approval matters, and keep broad scans outside the fastest pull-request lane.

How do I test OAuth audience validation?

Prepare a token issued for the MCP server and a controlled wrong-audience token, then assert acceptance and rejection respectively. Do not pass production tokens through the test.

Does a passing direct provider suite prove the whole MCP agent is safe?

No. It proves observed server behavior for those tool calls. Test the model or agent layer separately for tool selection, sequencing, prompt injection, and final-response handling.

Conclusion: test the server and the agent as different systems

The dedicated MCP provider is strongest when used as a precise server harness: explicit tool, explicit arguments, explicit identity, deterministic result and state assertions, and controlled negative cases. Red-team generation adds adversarial breadth after the boundary is defined. Neither layer excuses weak server authorization or unrestricted test credentials.

Build the small contract suite first, add focused threats, preserve generated-probe provenance, and gate critical invariants rather than averages. Then use the coding-agent evaluation guide if a model's choice and handling of MCP tools also need evidence.