Fail-Fast vs continue-on-error in CI Test Jobs: When to Use Each

Two CI settings that both appear to "keep the pipeline moving" can produce opposite outcomes. Fail-fast stops related work after a decisive failure. continue-on-error allows work or a workflow to continue despite a failure. One saves compute by ending low-value execution. The other collects evidence from a failure that is not allowed to block delivery.

For QA teams, the distinction is a release-policy decision, not YAML decoration. Apply the wrong one and a pull request may look green despite a broken contract test, or a 30-shard browser run may stop after the first failure and hide 12 unrelated regressions. This guide turns both controls into explicit test policies using documented GitHub Actions behavior and portable decision rules.

Separate early termination from failure tolerance

Fail-fast answers: "After one member fails, should the remaining members still run?" In GitHub Actions, strategy.fail-fast controls jobs generated by a matrix. It defaults to true. When enabled, a non-tolerated matrix job failure cancels in-progress and queued jobs in that matrix.

continue-on-error answers: "Should this failure make the job or workflow fail?" At the step level, it allows a job to continue when that step fails. At the job level, it prevents that job's failure from failing the workflow run. It does not make the underlying assertion pass.

Control	Primary decision	Appropriate example	Main danger
Matrix fail-fast	Stop sibling matrix work after failure	Compiler compatibility smoke tests where one failure is decisive	Losing diagnostic coverage from canceled combinations
Fail-fast disabled	Let every matrix member finish	Browser or shard matrix used to map the regression surface	Higher compute use after an obvious blocker
Step continue-on-error	Continue later steps in the same job	Temporary migration probe followed by evidence collection	Hiding a required test behind a green conclusion
Job continue-on-error	Keep an allowed job failure from failing the workflow	Experimental runtime axis outside the release contract	Stakeholders interpreting green workflow status as full support

These controls can coexist. For example, an experimental matrix member may tolerate failure while stable members remain blocking. The policy must be visible in job names, branch protection, and reporting.

Classify tests by what a failure means

Before editing CI, give each suite an enforcement class. Do not start by asking whether developers dislike waiting. Start by asking what shipping with a failure would mean.

Test class	Example	Failure policy	Execution policy
Release blocker	Authentication contract, migration check, checkout E2E	Must fail required check	Fail early only if remaining results add little value
Diagnostic suite	Cross-browser compatibility matrix	Usually blocking for supported targets	Often finish all targets to expose scope
Informational canary	Upcoming runtime or browser channel	Allowed temporarily with owner and expiry	Finish and publish result
Cleanup/reporting	Trace upload, JUnit publication, environment teardown	Must run after test failure	Use an always-run condition, not failure tolerance on tests

This classification exposes a frequent misuse: adding continue-on-error so reports upload after a test command fails. Report publication should use a condition that runs after success or failure. The test step should retain its real result.

An AI coding agent changing pipeline YAML should be given the enforcement class and required-check names. "Make CI pass" is unsafe because the shortest edit may simply tolerate the failure. Ask the agent to preserve blocking semantics, show the resulting status graph, and explain every tolerated failure.

Use fail-fast for fast, decisive matrix feedback

Consider a small API smoke suite against supported Node.js versions. If every matrix member runs the same contract checks and the first stable target fails because a shared endpoint returns 500, canceling siblings may save time. GitHub Actions expresses that with the matrix strategy.

jobs:
  api-smoke:
    name: API smoke on Node ${{ matrix.node }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: true
      matrix:
        node: [20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci
      - run: npm run test:api-smoke

This is valuable when one failure is enough to reject the change and sibling output is mostly repetitive. It is less useful when matrix members exercise different behavior. Chromium, Firefox, and WebKit failures are not interchangeable evidence. Nor are database versions, regional endpoints, or independent test shards.

Use fail-fast: false when the matrix is your diagnostic map:

jobs:
  browser-tests:
    name: E2E ${{ matrix.browser }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        browser: [chromium, firefox, webkit]
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps ${{ matrix.browser }}
      - run: npx playwright test --project=${{ matrix.browser }}

Now every browser job can finish even if one fails. The workflow still fails when a required browser job fails. You spend more compute, but receive the full compatibility picture in one run.

Reserve continue-on-error for explicitly non-blocking risk

A tolerated job should have three things: a reason, an owner, and an exit condition. "Flaky" is not enough. A concrete example is a canary runtime the team does not yet claim to support. It can reveal future incompatibility without blocking today's merge.

GitHub Actions allows continue-on-error to depend on a matrix value:

jobs:
  unit-tests:
    name: Unit tests, Node ${{ matrix.node }}
    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.experimental }}
    strategy:
      fail-fast: true
      matrix:
        include:
          - node: 22
            experimental: false
          - node: 24
            experimental: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci
      - run: npm test

Here the supported target is blocking, while the experimental target is permitted to fail. GitHub's matrix documentation also describes the interaction: a tolerated experimental job does not trigger cancellation of the other matrix jobs under fail-fast.

Name the canary clearly in the UI and create a review date in the repository's normal tracking system. When the runtime becomes supported, change the value to blocking. If nobody reads the result, delete the job. A permanently ignored red test consumes compute without reducing risk.

Continue a job without erasing the original step outcome

Step-level tolerance is narrower. It can be useful during a migration when a legacy comparison is informative but not yet gating. GitHub exposes both outcome, the result before continue-on-error, and conclusion, the result after it. That lets later steps record the truth.

      - name: Compare legacy snapshots
        id: legacy
        continue-on-error: true
        run: npm run test:legacy-snapshots

      - name: Record legacy probe result
        if: ${{ always() }}
        run: |
          echo "Legacy outcome: ${{ steps.legacy.outcome }}"
          echo "Legacy conclusion: ${{ steps.legacy.conclusion }}"

Do not use this pattern for a required security, contract, or regression suite. A later echo does not restore merge protection. If the comparison must block, remove tolerance and keep evidence upload under an always-run condition.

A good migration probe also has a removal condition, such as "delete after the new serializer matches all approved fixtures." Add a test owner and report its failure rate. Otherwise the exception becomes invisible infrastructure.

Preserve traces, JUnit, and cleanup after failure

QA jobs often need more execution after the main command fails. Playwright traces, screenshots, server logs, and JUnit XML make the failure actionable. Keep the test blocking, then use if: ${{ always() }} on evidence and cleanup steps.

      - name: Run Playwright tests
        run: npx playwright test

      - name: Upload Playwright report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          if-no-files-found: ignore

      - name: Stop test environment
        if: ${{ always() }}
        run: ./scripts/stop-test-environment.sh

The test step remains failed, the artifact step gets a chance to run, and cleanup does not depend on a green assertion. This is the cleanest answer when someone proposes continue-on-error only because teardown was being skipped.

Artifact upload itself can fail for operational reasons. Decide whether missing evidence should block the workflow. For a regulated audit trail it may need to. For a best-effort screenshot from an already failed test, the original failure is usually the primary signal. That is a separate policy from tolerating the tests.

Choose a policy for shards, retries, and flaky tests

Sharded E2E jobs are the hardest case. With fail-fast enabled, one shard can cancel other shards before they reveal independent failures. With it disabled, all shards complete and provide a reliable failure count, but a doomed run continues consuming browser minutes.

Use these questions:

Question	Favor fail-fast	Favor complete execution
Does one failure make the commit unmergeable?	Yes	Not sufficient by itself
Do matrix members cover the same risk?	Yes	No, they cover distinct targets
Is a full failure map needed for one repair cycle?	No	Yes
Is the suite extremely costly after the first decisive result?	Yes	Only with a compute budget
Are artifacts from every shard needed to diagnose flakiness?	No	Yes

Retries do not convert flaky tests into non-blocking tests. A bounded framework-level retry can gather evidence and label intermittent behavior, but the policy should still say whether a test that passes only after retry blocks delivery. Keep retry counts in reports and track the first-attempt failure rate.

For persistent instability, publish structured results and assign ownership. GitLab CI JUnit reporting for flaky tests shows how test evidence can support that process in GitLab. The general lesson applies elsewhere: tolerance without visibility becomes silent quality debt.

Prevent duplicate work before tuning cancellation

Matrix fail-fast only responds after a matrix job fails. It does not address a newer commit making the entire older workflow irrelevant. For pull requests with frequent pushes, add workflow concurrency so obsolete runs do not compete with the newest candidate. The implementation details are covered in canceling stale E2E runs on a new commit.

Then choose within-run behavior separately:

# Policy review checklist for a CI change
# 1. Is this suite required for merge?
# 2. Which sibling jobs are canceled after failure?
# 3. Which failures remain visible but non-blocking?
# 4. Do reports and cleanup run after failure?
# 5. Who owns each temporary exception, and when is it removed?

This order prevents three policies from being conflated: obsolete-run cancellation, matrix early termination, and failure tolerance. They save time in different situations and change confidence in different ways.

Audit the status users actually see

After changing YAML, test the policy on a branch with controlled failures. Make one supported matrix member fail, then one experimental member fail. Confirm which siblings are canceled, which jobs show failure, whether required checks block merging, and whether artifacts appear.

Do not review only the workflow's final color. Inspect individual job and step results. A tolerated failure may leave the workflow successful, exactly as configured, while still representing a canary defect. Dashboards and notifications should surface that distinction.

Document the final policy beside the workflow: required targets, diagnostic targets, canaries, and always-run evidence steps. This gives both humans and coding agents a stable specification. A future refactor can then prove it preserves enforcement instead of merely producing syntactically valid YAML.

Frequently Asked Questions

Does fail-fast make a failing CI matrix pass?

No. Fail-fast controls whether sibling matrix jobs continue after a qualifying failure. It does not change that failed job into a success. The workflow still reflects the blocking failure. What changes is the amount of additional evidence collected from queued or running matrix members, so teams should enable it only when those canceled results are less valuable than the saved time.

Is continue-on-error a reasonable permanent fix for flaky tests?

Usually not. It removes merge pressure without removing the defect, and the tolerated result is easy to ignore. Use bounded retries to collect flake evidence, publish structured results, assign an owner, and repair or quarantine the test under a written policy. If temporary tolerance is necessary, give it a narrow scope, visible job name, measured failure rate, and expiry condition.

How can artifacts upload after a blocking test failure?

Leave the test command blocking and place if: ${{ always() }} on the upload step. The same pattern can run teardown or log collection after success, failure, or cancellation conditions that allow the step to start. This preserves the original test result while retaining diagnostics. There is no need to mark the assertion step continue-on-error merely to reach report publication.

Should browser test shards use fail-fast?

Choose based on the value of unfinished shard results. Enable it for a quick required signal when any failure rejects the commit and full coverage would add little diagnostic value. Disable it when shards hold distinct tests, teams need the complete regression map, or flake analysis depends on every report. Many teams use a fail-fast smoke stage followed by complete sharded execution for candidates that pass.