Add MCP Conformance Tests to GitHub Actions with Failure Baselines

A reliable MCP conformance GitHub Actions job does more than run a command. It locks the conformance runner and Node runtime, starts a disposable server fixture, records the protocol and suite scope, preserves raw checks even when the gate fails, and treats the expected-failures file as reviewed technical debt. The stable @modelcontextprotocol/conformance@0.1.16 baseline has a useful safeguard: an unlisted failure exits nonzero, and a listed scenario that starts passing also exits nonzero so the stale exception must be removed.

Use the MCP server testing pillar for the full test architecture, the official conformance server tutorial for local runner setup, the MCP Inspector CLI tutorial for focused diagnostics, and the MCP tool contract guide for product-specific assertions. The QASkills catalog, Playwright CLI skill, and canonical Playwright MCP browser guide cover reusable QA and browser automation workflows around the protocol gate.

Define the gate before writing YAML

The job should have one auditable claim: a specific server build ran a named conformance suite from a pinned runner against a published MCP version, with only a reviewed set of expected failures. Everything else is supporting evidence. Do not label the job "MCP certified" or "fully compliant" because the official repository describes the framework as a work in progress and because application security and product behavior sit outside its scenario set.

Separate authorities in the pull request description:

Decision	Classification	Source of truth
MCP messages use JSON-RPC and negotiate a protocol version	Specification requirement	Published MCP specification
--expected-failures fails on a new failure and a stale baseline entry	Runner behavior	Pinned conformance implementation
Raw results should be retained after a failing job	Engineering recommendation	GitHub artifact capability and incident needs
Retain evidence for 14 days and require two reviewers for baseline changes	Team policy	Repository governance

That distinction matters when the suite evolves. A runner upgrade can change scenarios without changing the protocol. A security review can add stricter policy without claiming the specification mandates that exact threshold. A product team can temporarily accept a known gap without rewriting history to call it conformant.

Pin every moving part that can change the result

As of July 14, 2026, the stable npm conformance package is 0.1.16, the published MCP protocol version is 2025-11-25, and the maintained Node 22 line is 22.23.1. The workflow below uses those exact values. It also references immutable commits corresponding to GitHub's official July 2026 action releases: Checkout v7.0.0, Setup Node v7.0.0, and Upload Artifact v7.0.1.

Pin the npm dependency and commit the generated lockfile:

{
  "engines": {
    "node": "22.23.1"
  },
  "devDependencies": {
    "@modelcontextprotocol/conformance": "0.1.16"
  },
  "scripts": {
    "mcp:scenarios": "conformance list --server --spec-version 2025-11-25",
    "mcp:conformance": "conformance server --url http://127.0.0.1:3001/mcp --suite active --spec-version 2025-11-25 --output-dir artifacts/mcp-conformance --expected-failures ./conformance-baseline.yml"
  }
}

Run npm ci, not a fresh install, in CI. The exact top-level package plus lockfile constrains transitive resolution. Dependabot or Renovate can propose upgrades, but each upgrade should regenerate the scenario inventory, run without a baseline in a research branch, and receive the same review as a protocol-facing code change.

An immutable commit SHA is stronger than a moving major tag for actions. Keep the human-readable release in a comment so maintainers can trace it. A policy bot can verify SHA pinning, but the policy itself is a repository decision rather than an MCP requirement.

Create a baseline that cannot normalize new failures

The official runner accepts a YAML file keyed by mode. For a server job, only the server list affects the baseline. The scenario names below come from the official README's format example; they are not a recommendation to accept those failures in your project.

server:
  # MCP-412, owner: protocol-team, expires: 2026-08-15
  - tools-call-with-progress
  # MCP-427, owner: resources-team, expires: 2026-07-31
  - resources-subscribe

client: []

Do not create this file by copying every current failure. Establish each entry through this sequence:

1. Run the pinned suite without --expected-failures and save all artifacts.
2. Re-run the exact failing scenario with verbose output.
3. Decide whether the defect belongs to the server, fixture, environment, or runner.
4. Link a tracked issue containing impact, reproduction, intended fix, and owner.
5. Assess whether accepting the gap exposes data, side effects, or interoperability risk.
6. Add only the exact scenario identifier after approval, with an expiry comment.
7. Re-run the full suite with the baseline and verify no unlisted failure remains.

The runner evaluates the baseline bidirectionally:

Scenario result	Listed in baseline	Runner outcome	Required response
Fails	Yes	Exit 0 for that expected gap	Keep issue active; review expiry
Fails	No	Exit 1	Treat as regression or classify before review
Passes	Yes	Exit 1	Remove stale entry in the same change
Passes	No	Exit 0	Normal passing scenario

This behavior prevents the baseline from becoming an ever-growing ignore list. Never add continue-on-error: true to the conformance step, because that bypasses the runner's distinction between known, new, and stale outcomes. Use if: always() only on evidence-upload steps.

Use one step for server lifecycle and runner execution

Background process behavior is easier to control when startup, readiness, conformance, and cleanup share one shell. The following workflow uses a separate /healthz route for readiness and the real /mcp route for the runner. Replace the build and server command with your repository's commands; do not replace the protocol handler with a test-only implementation.

name: MCP conformance

on:
  pull_request:
    paths:
      - 'src/mcp/**'
      - 'package.json'
      - 'package-lock.json'
      - 'conformance-baseline.yml'
      - '.github/workflows/mcp-conformance.yml'
  push:
    branches: ['main']

permissions:
  contents: read

jobs:
  server-conformance:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - name: Check out repository
        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

      - name: Set up pinned Node
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version: '22.23.1'
          cache: npm

      - name: Install locked dependencies
        run: npm ci

      - name: Build server
        run: npm run build

      - name: Run published MCP conformance suite
        shell: bash
        run: |
          set -euo pipefail
          mkdir -p artifacts/mcp-conformance
          npm run mcp:scenarios > artifacts/server-scenarios.txt
          npm exec -- conformance --version > artifacts/conformance-version.txt
          node dist/conformance-server.js             --host 127.0.0.1             --port 3001             > artifacts/server.log 2>&1 &
          SERVER_PID=$!
          trap 'kill "$SERVER_PID" 2>/dev/null || true' EXIT INT TERM

          for attempt in 1 2 3 4 5 6 7 8 9 10; do
            if curl --fail --silent http://127.0.0.1:3001/healthz >/dev/null; then
              break
            fi
            kill -0 "$SERVER_PID"
            sleep 1
          done

          curl --fail --silent http://127.0.0.1:3001/healthz >/dev/null
          npm run mcp:conformance

      - name: Preserve conformance evidence
        if: always()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: mcp-conformance-server
          path: artifacts/
          if-no-files-found: error
          retention-days: 14

The action versions and SHAs are verified facts for July 14, 2026. The ubuntu-24.04 runner choice, 15-minute timeout, ten readiness attempts, and 14-day retention are example team policies, not universal MCP thresholds. Adjust them from measured startup time, artifact sensitivity, storage policy, and support needs. Keep those choices explicit so a future maintainer knows which numbers are negotiable.

Preserve evidence even when the command exits nonzero

GitHub defines workflow artifacts as files retained after a job for later inspection. The official workflow artifact documentation lists test results and logs as common examples. The workflow's if: always() ensures upload is attempted after a failing test step. if-no-files-found: error catches a broken output path rather than producing a green upload step with no evidence.

Retain at least these files:

Artifact	Why it matters	Redaction rule
conformance-version.txt	Proves the runner binary selected by the lockfile	No secrets expected
server-scenarios.txt	Records suite inventory for the pinned version	No secrets expected
Runner checks.json files	Provides scenario-level status and diagnostic details	Inspect tool content for sensitive fixture data
server.log	Correlates runner failures with application behavior	Remove tokens, credentials, personal data, and raw production payloads
conformance-baseline.yml from the commit	Explains accepted gaps at execution time	Keep issue references but no secret context
Fixture manifest	Records data revision, transport, and disabled integrations	Use synthetic values only

The current Upload Artifact action supports retention-days and can fail when a path is empty. It also notes that hidden files are excluded by default. Do not enable hidden-file upload broadly; an environment file can contain credentials. Package only the reviewed artifact directory.

For a failed release build, add the artifact URL to the defect or release record before rerunning. A successful retry does not erase the evidence needed to diagnose the original failure.

Review baseline changes as production changes

Protect conformance-baseline.yml with CODEOWNERS or an equivalent approval rule. A baseline addition changes what the release gate accepts, so it deserves review from the server owner and, for authorization or data-handling scenarios, a security owner.

A strong review template asks:

• What exact scenario failed, on which pinned runner and protocol version?
• Can the failure be reproduced with --scenario?
• Is it a server defect, fixture defect, environment issue, or disputed check?
• Which users, tools, or transports are affected?
• Does the gap permit unauthorized access, data leakage, or destructive behavior?
• What issue, owner, and expiry control remediation?
• Which compensating test or deployment restriction reduces risk?
• What evidence will prove the baseline can be removed?

Do not accept wildcard matching or generated baseline updates in the blocking job. The stable file format lists scenario names. An automation may propose a diff, but a human should approve every added entry. Removal can be encouraged aggressively because the runner already identifies stale entries, but it should still accompany evidence that the passing behavior is real and not a fixture regression.

Turn every exception into a bounded decision

The runner baseline records scenario names because that is what it needs to evaluate expected and unexpected outcomes. It does not carry your business context. Keep the decision record in the pull request, issue tracker, or release system and link it to the exact baseline diff. Record the observed check, affected server surface, reproduction command, impact assessment, owner, target removal date, and compensating control. This additional record is team governance, not an MCP or conformance file-format requirement.

Review additions and removals differently. An addition broadens what the gate accepts and needs evidence that shipping is safer than blocking, plus a clear remediation path. A removal narrows the exception set, but still needs a clean run from the same pinned runner, protocol selector, fixture revision, and server commit under review. If a scenario passes only after changing fixture data, investigate whether the contract was repaired or the test stopped reaching the defective branch.

Treat a runner upgrade as a baseline migration, not routine dependency maintenance. First capture the scenario inventory from the old and new pinned versions. Then run the new version without silently extending the baseline and classify each difference as a newly active scenario, a changed check, a server regression, or a harness incompatibility. Preserve both result sets. A scenario renamed upstream should not be copied into the baseline until reviewers confirm that it represents the same known gap; matching names are implementation details of the runner, not stable protocol identifiers.

Retries also need an explicit policy. Automatic retry can be useful for diagnosing runner infrastructure, but a second attempt must not replace the first status in the blocking decision. Preserve artifacts from each attempt under distinct paths and compare server logs before calling a result flaky. If the same scenario alternates between pass and fail with identical recorded inputs, open a reliability defect and decide whether the affected release surface can be constrained. Do not add an intermittent failure to the expected-failure file merely to make the final attempt green.

A baseline review should answer three independent questions: what behavior is currently nonconforming, why the release is allowed despite that behavior, and how the team will know the exception can be removed. Combining these questions into a generic approval such as "known issue" makes the file durable while its rationale disappears. A bounded decision keeps the technical result visible without pretending that the conformance runner made the product-risk decision.

Add positive and negative controls around the gate

A CI job can appear healthy while discovering no meaningful work. Add controls that prove the harness can fail:

1. Validate that server-scenarios.txt is nonempty and contains the expected initialization scenario.
2. Run a unit test against the fixture health contract before launching conformance.
3. Keep a separate test that supplies an invalid product input and expects a tool execution error.
4. Verify artifacts exist after a deliberately failing test in a nonblocking harness validation workflow.
5. Confirm that a temporary bogus baseline entry causes the runner to report a stale entry and exit nonzero; do this during harness development, not on every production pull request.

The negative-control exercise should be documented, then removed. Do not commit a fabricated failure to main merely to prove the gate reacts.

Split pull request, nightly, and release coverage

One workflow cannot optimize for speed, breadth, and migration research at the same time.

Cadence	Selection	Blocking?	Purpose
Pull request	Stable active server suite for published 2025-11-25	Yes for MCP-affecting changes	Detect protocol regressions before merge
Main branch	Same stable suite plus product contracts	Yes	Confirm merged dependency graph and fixture
Nightly	Stable suite on supported deployment variants	Team decision	Detect environmental drift
Protocol research	Draft or pending scenarios with separate artifacts	No until migration approval	Estimate future work without redefining current requirements
Release	Stable suite, security tests, product semantics, and evidence review	Yes	Produce a defensible release record

If the server supports multiple SDK versions or transports, use a matrix only when each combination is an actual support promise. A large matrix that nobody owns wastes capacity and creates ignored failures. Give every cell a fixture, artifact name, and responsible team.

Troubleshoot common workflow failures

npm ci changes the resolved runner. It should not. Confirm the lockfile is committed, the exact top-level version is 0.1.16, and no workflow step runs npm update or an unpinned npx ...@latest command.

The server never becomes ready. Read server.log, confirm the build output path, and check fixture dependencies. Do not increase the retry loop until you understand normal startup distribution. A longer timeout can hide a crash-restart loop.

Artifacts are missing after failure. Ensure the upload step has if: always(), the runner receives --output-dir, and the path is under the workflow workspace. Keep if-no-files-found: error so evidence failures remain visible.

A new failure passes because the job uses continue-on-error. Remove that setting from the conformance step. Expected failures belong in the reviewed baseline, where the runner can distinguish them from regressions.

A fixed scenario still fails the job. This is intentional stale-baseline behavior. Remove the scenario from conformance-baseline.yml, rerun the entire selected suite, and close the tracked issue with artifacts.

A runner upgrade introduces many changes. Revert only the upgrade branch's dependency change if needed, not application fixes. Compare scenario inventories, read upstream release notes and scenario source, classify each change, and update the baseline only after review.

Limitations and release language

GitHub Actions proves behavior in its runner environment, not every production topology. The conformance framework covers its shipped scenarios, not all security, load, model, or business requirements. An expected-failure baseline explicitly records known gaps; its presence means a zero process exit is not identical to all checks passing.

Release notes should say: "The server ran the active server suite from @modelcontextprotocol/conformance@0.1.16 against protocol 2025-11-25; raw results and any expected failures are attached." Avoid "MCP certified" unless an actual recognized certification process exists and was completed.

Implementation checklist

• Pin Node, the conformance npm package, GitHub actions, and the lockfile.
• Keep the published protocol selector separate from draft research.
• Use a disposable server fixture on 127.0.0.1 with a dedicated health route.
• Capture scenario inventory and runner version before execution.
• Run startup, readiness, suite, and cleanup in one controlled shell step.
• Upload artifacts with if: always() and fail when expected files are absent.
• Redact secrets before writing logs to the artifact directory.
• Require issue, owner, impact, reviewer, and expiry for every baseline addition.
• Let new failures and stale baseline entries fail the job.
• Pair conformance with negative product, authorization, and output-schema tests.

Frequently asked questions

Should the conformance step use continue-on-error?

No. That setting erases the useful distinction between passing scenarios, approved expected failures, new regressions, and stale entries. Keep the test step blocking and apply if: always() only to diagnostics and artifact upload.

Why does a passing scenario in the baseline fail CI?

The runner treats it as a stale exception. This prevents fixed scenarios from remaining silently ignored. Remove the entry, run the complete suite, and retain the new evidence.

Is 14-day artifact retention required by MCP?

No. It is an example team policy. Choose retention from incident response needs, data sensitivity, legal requirements, and storage budget. The action accepts a retention period within the limits configured for the repository.

Can a bot update the failure baseline automatically?

A bot can prepare a proposed diff and attach evidence, but automatically accepting every current failure normalizes regressions. Require human review for additions. Automated removal suggestions are safer, yet reviewers should still confirm the pass is reproducible.

Why pin action commit SHAs instead of major tags?

Commit SHAs are immutable references, while some tags can move. Comments preserve the release label for readability. This is a CI supply-chain policy, not an MCP protocol rule.

Should draft conformance run in the same blocking job?

Not by default. Keep draft or pending scenarios in a separately named, nonblocking research job until the new protocol is published and your migration plan is approved. Otherwise a future proposal can redefine a current release gate.

What if the server requires real external services?

Prefer contract-faithful local fakes or isolated sandbox accounts with synthetic data. If an external service is unavoidable, classify outages separately, restrict credentials and egress, and avoid baselining environmental failures as protocol exceptions.

Conclusion

An MCP conformance workflow is trustworthy when it is reproducible and difficult to weaken accidentally. Pin the runner and execution environment, make the server fixture deterministic, preserve raw results after every outcome, and govern expected failures as temporary release exceptions. The stable runner's regression and stale-baseline behavior then becomes an asset instead of an ignore mechanism.