Run the Official MCP Conformance Suite Against Your Server

An MCP conformance suite server run answers a narrow but important question: does a running MCP server behave as the official conformance scenarios expect for the selected protocol release? With the stable @modelcontextprotocol/conformance@0.1.16 runner, you start and health-check your own Streamable HTTP server, point the runner at its MCP endpoint, select the published 2025-11-25 protocol baseline, and retain the generated check records. The runner is the test client; your application remains responsible for server startup, fixture state, credentials, and cleanup.

Start with the MCP server testing pillar, then connect this workflow to the GitHub Actions conformance baseline, MCP Inspector CLI tutorial, and MCP tool contract testing guide. Teams can browse reusable instructions in the QASkills directory, install the Playwright CLI skill, and use the existing Playwright MCP browser automation guide when an MCP workflow also needs browser evidence.

What the official suite proves, and what it does not

The official conformance repository describes two distinct modes. Client mode starts a scenario server and launches a client implementation. Server mode connects as an MCP client to a server that is already running. This tutorial covers server mode only. There is no official server-adapter interface that the application must implement. The integration boundary is a reachable MCP URL; any startup wrapper around that URL belongs to your test harness.

Keep four kinds of statements separate during review:

Statement class	Authority	Example in this workflow	How to treat it
MCP specification requirement	Published MCP specification	Streamable HTTP has one MCP endpoint and lifecycle messages follow the negotiated protocol	A failure can indicate protocol nonconformance
Conformance implementation behavior	Runner version 0.1.16	server --url connects to an already running endpoint; active excludes pending scenarios	Pin and re-check when upgrading the runner
Security recommendation	MCP security or transport guidance	Bind local fixtures to 127.0.0.1, validate Origin, and protect nonlocal endpoints	Test it, but do not mislabel every recommendation as a suite assertion
Team policy	Your release process	Active suite must pass before release; evidence is retained for 14 days	Document owner, exception path, and expiry

The suite is not a substitute for product semantics, authorization abuse cases, tenant isolation, load testing, secret scanning, or destructive-operation controls. It also does not certify a client, because server and client modes exercise different implementations. The stable active server suite excludes scenarios marked pending. Do not say "all MCP behavior passed" when you ran only active, one scenario, or one transport.

Version and environment baseline

As of July 14, 2026, the MCP documentation identifies 2025-11-25 as the current published protocol version. The conformance repository contains work for a later 2026 draft, but draft scenarios are not published requirements. The commands below therefore pin the stable npm release 0.1.16 and explicitly select 2025-11-25. A future migration should be a reviewed change, not an automatic consequence of using latest.

Before the first run, record:

• the conformance package version and lockfile digest;
• the MCP protocol version the server negotiates;
• server SDK and application commit identifiers;
• endpoint transport and authentication mode;
• fixture dataset revision;
• runner operating system and Node version;
• the exact suite or scenario selector.

Install the runner as an exact development dependency so local and CI runs resolve the same package:

npm install --save-dev --save-exact @modelcontextprotocol/conformance@0.1.16
npm exec -- conformance --version
npm exec -- conformance list --server --spec-version 2025-11-25

The final command is deliberately part of setup. Scenario membership changes between runner releases, so the command output is better evidence than an article that freezes a count. Review the names before adding a failure baseline. The tagged official guide names server-initialize, tools-list, tool-call scenarios, resource scenarios, and prompt scenarios as representative server coverage; the actual list from the pinned binary is authoritative for the run.

Build a server adapter without inventing an API

For conformance server mode, "adapter" should mean a repository-owned launcher that exposes your real implementation through the transport expected by the runner. It should not reimplement protocol behavior merely to make tests pass. A useful adapter selects test configuration, starts the production server entry point, exposes a separate health route, seeds deterministic dependencies, and forwards termination signals.

The official SDK integration guide says the server must already be running and that your workflow manages startup, readiness, and cleanup. A minimal shell harness can enforce that lifecycle:

#!/usr/bin/env bash
set -euo pipefail

export NODE_ENV=test
export DATABASE_URL='file:./artifacts/conformance-fixture.db'
export MCP_TEST_CLOCK='2026-07-14T00:00:00Z'

mkdir -p artifacts/mcp-conformance
node dist/conformance-server.js --host 127.0.0.1 --port 3001 &
SERVER_PID=$!
trap 'kill "$SERVER_PID" 2>/dev/null || true' EXIT INT TERM

for attempt in 1 2 3 4 5 6 7 8 9 10; do
  if curl --fail --silent http://127.0.0.1:3001/healthz >/dev/null; then
    break
  fi
  if ! kill -0 "$SERVER_PID" 2>/dev/null; then
    echo 'MCP fixture server exited before readiness' >&2
    exit 1
  fi
  sleep 1
done

curl --fail --silent http://127.0.0.1:3001/healthz >/dev/null
npm exec -- conformance server   --url http://127.0.0.1:3001/mcp   --suite active   --spec-version 2025-11-25   --output-dir artifacts/mcp-conformance

The health route is a harness choice, not an MCP requirement. It avoids treating an expected GET /mcp status as a generic readiness contract. The Streamable HTTP specification defines behavior for the MCP endpoint, including POST and GET handling, but a deployment can still benefit from a separate health probe that does not initialize a protocol session.

Do not point the suite at a production endpoint. A conformance scenario may invoke tools and exercise state. Bind locally, use synthetic credentials, deny external egress unless a fixture needs it, and replace irreversible integrations with test doubles at the same application boundary used by ordinary integration tests.

Design fixtures that make failures interpretable

Conformance checks become noisy when tool data changes between scenarios. Create a fixture contract before running the suite:

Fixture concern	Deterministic design	Failure if omitted
Tool catalog	Fixed names, descriptions, schemas, and ordering for the test build	A catalog change looks like a transport regression
Tool results	Stable IDs, timestamps, and resource contents	Assertions vary by wall clock or random seed
Side effects	Transaction rollback, disposable database, or per-run namespace	One scenario contaminates the next
External APIs	Local fake with recorded request assertions	Network availability masks server behavior
Authentication	Dedicated principal with minimum fixture permissions	A credential problem is confused with protocol failure
Logs	JSON logs on stderr or a file, never protocol noise on stdio stdout	Diagnostics can corrupt stdio framing in other test modes

A fixture manifest gives reviewers a human-readable inventory:

{
  "fixtureVersion": "2026-07-14.1",
  "protocolVersion": "2025-11-25",
  "transport": "streamable-http",
  "mcpEndpoint": "http://127.0.0.1:3001/mcp",
  "healthEndpoint": "http://127.0.0.1:3001/healthz",
  "tools": ["orders.lookup", "orders.quote_refund"],
  "externalEgress": false,
  "destructiveOperations": false
}

Reset fixture state before the suite, not only before the workflow. If a scenario crashes midway, the next retry should begin from a known state. A database migration plus seed script is preferable to a hand-maintained database snapshot when schema evolution matters. If the server caches its tool list, restart it after fixture generation so the test process observes the intended catalog.

Run one scenario before the active suite

Begin with initialization. It gives a small diagnostic surface for endpoint, transport, lifecycle, and capability problems:

npm exec -- conformance server   --url http://127.0.0.1:3001/mcp   --scenario server-initialize   --spec-version 2025-11-25   --output-dir artifacts/mcp-conformance-init   --verbose

npm exec -- conformance server   --url http://127.0.0.1:3001/mcp   --suite active   --spec-version 2025-11-25   --output-dir artifacts/mcp-conformance-active

The first command is a diagnostic run. The second is the release candidate. Do not replace the suite with the initialization scenario after initialization passes. Conversely, do not start with the full suite when every request fails at the same transport boundary; isolate the lifecycle problem first.

The --spec-version filter is runner behavior, not protocol negotiation by itself. Your server still participates in MCP initialization and must reject unsupported negotiation appropriately. Capture the server log line that records the negotiated version. If the runner filter and server behavior disagree, treat that as a test configuration defect before classifying application behavior.

Interpret checks without turning the suite into a score

For server runs with an output directory, the stable runner writes per-scenario checks.json evidence below the chosen directory. A check includes an identifier, status, description, timestamp, and optional error details. Preserve raw files before creating summaries. A count such as "48 of 50" is not durable because scenario and check membership can change; scenario IDs plus runner version are durable enough for comparison.

Use this triage order:

1. Harness failure: the process never became ready, exited unexpectedly, used the wrong URL, or lacked a fixture dependency.
2. Transport or lifecycle failure: initialization, protocol version, headers, session handling, or HTTP framing failed before feature logic ran.
3. Capability mismatch: the server declared a feature but did not implement the corresponding behavior, or the test build exposed a different catalog.
4. Feature conformance failure: a request reached the intended handler but the response violated the scenario check.
5. Runner question: behavior appears consistent with the published specification but conflicts with a check. Reproduce narrowly and review the pinned scenario source before filing an upstream issue.

Never edit a result file to make a release green. Fix the server, fix the harness, or add a reviewed expected-failure entry with an issue and expiry through the baseline mechanism. The companion CI guide explains why a stale baseline must also fail.

Review evidence across reruns

Keep the first failing run when a retry passes. The pair can reveal nondeterministic fixture state, startup races, expiring credentials, or shared service instability that a final green status conceals. Compare scenario identifier, check identifier, server commit, fixture revision, negotiated protocol version, and sanitized request correlation data. A rerun is new evidence; it does not rewrite the earlier observation.

Classify reproducibility separately from severity. A repeatable low-impact catalog mismatch and an intermittent authorization leak require different release decisions. The specification defines protocol behavior, and the pinned runner defines how its scenarios observe that behavior, but neither defines your retry allowance or risk acceptance. Those are team policies. Record who reviewed the discrepancy, which evidence was compared, whether the same binary and fixture were used, and what event will force another review.

When a retry uses a changed image, seed, credential, dependency, or runner version, label it as a different configuration rather than a rerun. This prevents an environment repair from being misreported as proof that the original server build conformed.

Add positive and negative product tests beside conformance

The official suite validates implemented scenarios against the specification; it does not know your business rules. Keep a second layer of server contract tests with explicit positive and negative fixtures.

Positive cases should prove that a representative tool appears in tools/list, accepts a valid fixture argument, returns the expected content kind, and performs the expected isolated side effect. Negative cases should cover an unknown tool, a malformed request, schema-invalid input, business-invalid input, unauthorized access, dependency failure, and output that fails the declared schema. The MCP tools specification distinguishes protocol errors from tool execution errors, so assert the channel as well as the message.

Test	Expected evidence	Owned by
Unknown tool name	JSON-RPC protocol error	Protocol adapter
Missing structurally required request member	JSON-RPC protocol error	Protocol adapter
Date has correct JSON type but violates business rule	Tool result with isError: true and actionable content	Tool implementation
Valid lookup	Successful content and, when declared, schema-valid structuredContent	Tool implementation
Unauthorized refund quote	Access denial with no protected data	Security integration
Duplicate idempotency key	Product-defined repeat behavior	Product contract

This matrix prevents a common error: claiming that a passing conformance suite proves refund policy, tenant filtering, or safe retry behavior. Those are product contracts and team policies, even when transported through MCP.

Use the result in CI and release review

A practical cadence has three layers. Run a single initialization scenario during rapid local development. Run the stable active suite on every pull request that changes the MCP server, SDK, transport, schemas, or deployment configuration. Run supplemental security and product tests before release, plus draft conformance in a nonblocking research job when planning the next protocol migration.

The blocking release record should include:

• commit and fixture identifiers;
• exact conformance package and protocol versions;
• selected suite and scenario inventory output;
• raw checks.json directories;
• server logs with secrets redacted;
• approved baseline file, if any;
• links to product and security test runs;
• reviewer and exception expiry.

The suite's exit code is a gate signal, not a full release decision. A zero exit after expected failures means only known baseline entries failed and no baseline entry became stale under the runner's algorithm. It does not mean every scenario passed. Display both the process outcome and baseline contents in release review.

Troubleshooting by boundary

The runner cannot connect. Confirm the server process is alive, the URL ends at the actual MCP endpoint, and the fixture binds to 127.0.0.1. Check whether a container uses a different network namespace. Test the dedicated health endpoint first, then inspect MCP request logs.

Every scenario fails after initialization. Compare declared capabilities with enabled handlers. A test build that advertises tools, resources, or prompts but disables their backing modules creates a false capability contract.

Only later scenarios fail. Suspect state leakage. Reset storage, clear in-memory caches, make generated IDs deterministic, and inspect whether a previous tool invocation changed shared fixture data.

The server works in Inspector but fails conformance. Inspector is an exploratory client, while a conformance scenario asserts particular behavior. Save the exact Inspector request, compare negotiated protocol and transport, then run the single failing conformance scenario with --verbose.

A draft scenario contradicts the published release. Do not relabel draft behavior as a current requirement. Keep draft runs separate and nonblocking until the protocol revision is published and the migration is approved.

No artifact files appear. Pass an explicit --output-dir, verify the process can write it, and upload the directory even when the test step fails. The runner prints results to the console regardless, but console output is weaker evidence than the raw check files.

Limitations to state in every report

The stable suite is a work in progress, according to its own tagged README. It exercises the scenarios shipped in the pinned runner, not every sentence in the specification. Server mode reaches a running HTTP endpoint and does not validate a separate stdio deployment unless you create an appropriate HTTP test surface that faithfully uses the same implementation. It cannot establish production authorization correctness with synthetic credentials, nor can it prove operational resilience, scalability, privacy, or safe model behavior.

Conformance is also version-relative. A server can conform to one published protocol revision and intentionally reject another. Record the selected revision instead of using an unqualified "MCP compliant" label.

Release checklist

• Pin @modelcontextprotocol/conformance@0.1.16 and commit the lockfile.
• Record published protocol baseline 2025-11-25 separately from draft work.
• Start the real server implementation through a thin repository-owned launcher.
• Use a dedicated health route and deterministic fixture state.
• List server scenarios from the pinned runner before accepting a baseline.
• Run initialization first, then the complete selected active suite.
• Preserve raw check files and sanitized server logs.
• Classify failures as harness, protocol, capability, feature, or runner questions.
• Keep product semantics and security abuse tests outside the conformance claim.
• Require an owner, issue, rationale, and expiry for every expected failure.

Frequently asked questions

Does the official runner start my MCP server?

No. In server mode, your harness starts, health-checks, and stops the server. The official runner connects to the URL passed with --url. Client mode has a different lifecycle and should not be used as evidence for the server implementation.

Is a custom server adapter required by the conformance package?

No adapter interface is required. A team may create a launcher that configures the real server for deterministic tests, but the official server-mode integration boundary is the live MCP URL. Keep that launcher thin so it does not hide production protocol behavior.

Should I run the active, all, or pending suite?

Use active as the stable default for a blocking baseline, because the runner documents it as excluding pending scenarios. Run pending or broader exploratory selections separately. Record the exact command and generated scenario list rather than implying that one selection covers every implementation concern.

Why select protocol version 2025-11-25 explicitly?

It is the current published MCP version on July 14, 2026. Explicit selection prevents draft scenario work from silently changing a release claim and makes results comparable. Upgrade the protocol baseline through a deliberate compatibility review.

Can an expected failure make a nonconforming release acceptable?

That is a team policy decision, not an MCP rule. The runner can classify a named failure as expected, but a reviewer must assess impact, affected users, workaround, owner, and expiry. Never baseline an unexplained failure merely to obtain exit code zero.

Does a passing suite prove that tool inputs are safe?

No. The tools specification requires input validation and access controls, but the conformance scenarios do not know every domain invariant or attacker path. Add negative contract, authorization, injection, rate-limit, and side-effect tests for your actual tools.

Can I run the suite against production?

Do not use production as the default. Scenarios may call features and mutate state. Use an isolated endpoint with synthetic identities and disposable data. A separately approved production smoke test should be read-only, narrowly scoped, and governed by operational policy.

Conclusion

The official MCP conformance suite is most valuable when its claim stays precise. Pin the runner, target the published protocol revision, expose the real server through a thin lifecycle harness, run the complete selected server suite, and retain raw evidence. Then place product semantics, authorization, security, and operational tests beside conformance rather than pretending the suite replaces them. That produces a release signal another engineer can reproduce and defend.