by igrlk
A symptom-to-fix checklist for Playwright visual regression tests that come back changed without a real change: feature flags, live data, the clock, third-party widgets and dynamic layouts. Written for UI Verify's @uiverify/playwright capture, where the fix is to pin the variation before the snapshot. Official skill from UI Verify, visual regression testing for coding agents (uiverify.ai).
npx @qaskills/cli add playwright-visual-regression-testing-stop-flaky-screenshot-diffs-ui-verifyAuto-detects your AI agent and installs the skill. Works with Claude Code, Cursor, Copilot, and more.
@uiverify/playwright captures each test's final DOM + every resource the page loaded into an
archive; UI Verify re-renders and pixel-diffs that archive server-side. So:
Whatever the page was at the instant of capture is baked into the archive forever. Your job is to drive the page to one canonical state before the snapshot. UI Verify can't un-see a random A/B variant, a live feed, or a piece of lazy content that never loaded.
The snapshot fires automatically at the end of every test that passes or fails (a failed one too, so a broken
page shows up as a visual change rather than vanishing from the build; a test that times out is skipped), so your beforeEach + navigation is the
determinism surface. Integration is one import:
- import { test, expect } from '@playwright/test';
+ import { test, expect } from '@uiverify/playwright';
What UI Verify handles server-side (don't hand-fix these): it re-renders the archived DOM with
prefers-reduced-motion emulated and fonts and images settled. So pixel-level settling — waiting on
document.fonts.ready, sticky-header animation, image decode — is not your job. Your job is
narrower: make sure the content and state the archive records is the same every run.
One caveat on "animations": the freeze is replay-side, via reduced-motion emulation — the archive
does not bake a frozen frame. A CSS animation that honors prefers-reduced-motion freezes for free;
an infinite CSS/JS animation that ignores it keeps running on replay and lands on a random frame
(recipe 8), and a <canvas>/requestAnimationFrame loop is beyond the media query entirely (recipe 9).
Those live inside your app, so the fix does too.
Everything below is the same move — something rendered differently run-to-run and it wasn't a real change; find what injected the variation and pin it before the snapshot.
| Symptom in the diff | Cause class | Lever |
|---|---|---|
| Whole page reflows; height differs between runs | Feature flag / A-B experiment rendering a different variant per load (Optimizely, LaunchDarkly, VWO, your own) | Stub the flag calls so one fixed variant renders → recipe 1 |
| A list / feed / counts differ | Live data from an API | Mock the endpoint with a fixed response → recipe 2 |
| Timestamps, "today", a year, a chart's day axis drift | The clock (Date.now) at render time | Freeze Date before navigation → recipe 3 |
| A banner / chat / consent overlay appears or shifts | Third-party widget injecting DOM | Stub or dismiss it → recipe 1 + 4 |
| Blank/placeholder where an image should be | Lazy content that never loaded, so its bytes aren't in the archive | Trigger it at record time → recipe 5 |
| Same content, packed or ordered differently each run | Dynamic layout that reflows or reorders on its own | Mask it (or pin a fixed order/size) → recipe 6 |
An infinite JS animation the capturer can't stop (a charting lib's loop, a <canvas> spinner) | App-driven animation with no final frame | Branch the component on capture and render the end state → recipe 7 |
| A different logo/mascot/hero or list order each run — a whole-element diff, not a pixel shift | Math.random() in the render body, run at SSR time in Next.js/Remix | Pin the variant on isUIVerify() + UI_VERIFY=1 on the app server → recipe 10 |
| An infinite CSS keyframe animation on a different frame | CSS animation that ignores prefers-reduced-motion | Honor the media query in the app's CSS → recipe 8 |
A <canvas> (particles, chart, background) diffs every run | requestAnimationFrame loop the media query can't reach | Draw one static frame on isUIVerify() → recipe 9 |
| The capture is a skeleton / spinner, not the loaded page | Snapshot fired before the data arrived | Wait for the real content (networkidle) → recipe 12 |
Prefer removing a whole class at the source: test a build/staging with flags and third-party scripts off and rows 1 and 4 disappear before you write any code.
1. Stub feature-flag / third-party scripts (register in beforeEach, before goto):
await page.route(/optimizely|optly|launchdarkly|segment|intercom/i, (route) =>
route.fulfill({ status: 204, body: '' }).catch(() => undefined),
);
Use fulfill({ status: 204 }), not route.abort() — an aborted request reads as pending network
activity and can hang the page's readiness wait. A 204 completes it empty: the script "loads," resolves
no variant, and the page renders its control DOM. Identify the culprit by loading the page a few times
and watching document.body.scrollHeight flip between values.
2. Mock live data: intercept the API route and fulfill a fixed JSON fixture — same shape every run.
await page.route('**/api/items', (route) =>
route.fulfill({ json: fixtures.items }),
);
3. Freeze the clock (before navigation — it changes what the app renders into the DOM). Use
Playwright's built-in clock so Date, Date.now(), and Date()-without-new all stay correct:
await page.clock.setFixedTime(new Date('2020-01-01T00:00:00Z'));
4. Dismiss overlays (best-effort; absent = fine):
await page.getByRole('button', { name: 'Accept All' }).click({ timeout: 4000 }).catch(() => {});
5. Trigger lazy content so its bytes land in the archive. This is the one "settle" that is yours — not for pixel-stability (UI Verify handles that) but because content that never loaded was never recorded. Scroll it into view before the snapshot:
await page.evaluate(async () => {
for (let y = 0; y < document.body.scrollHeight; y += innerHeight) {
scrollTo(0, y);
await new Promise((r) => setTimeout(r, 60));
}
scrollTo(0, 0);
});
6. Mask what can't be pinned. Last resort for content that is supposed to differ and can't be
pinned at the source (a third-party embed, a map, an ad slot, a live feed): add data-uiverify-ignore
to the element. UI Verify paints its box solid black on every capture, so its content never reaches the
diff. The box's size and position still diff (give it a fixed size), and nothing inside it is checked, so
mask only what you can't pin and fix what you own. Docs.
<aside data-uiverify-ignore style={{ width: 320, height: 256 }}><AdSlot /></aside>
On a page you don't own, set the attribute from the test before the snapshot; the archive keeps it and replay masks the box:
await page
.locator('[data-testid="dynamic-region"]')
.evaluateAll((els) => els.forEach((el) => el.setAttribute('data-uiverify-ignore', '')));
7. Freeze an app-driven JS animation the capturer can't reach — an infinite charting-library loop, a <canvas> spinner with no final frame. Detect the capture in the component and render the end state. @uiverify/playwright sets a UIVerify user-agent marker and a window.__UI_VERIFY__ global at record time, so a client-side branch works on its own; if the component decides this during server-side rendering (no window yet), also set a UI_VERIFY env var when you boot the app for the record run and check it:
// in the component (your app code, not the test):
const isUIVerify = () =>
(typeof process !== 'undefined' && !!process.env.UI_VERIFY) || // SSR / build time
(typeof window !== 'undefined' &&
(navigator.userAgent.includes('UIVerify') || '__UI_VERIFY__' in window)); // browser
<Chart isAnimationActive={!isUIVerify()} />
Boot the app with the env var for the capture run only — e.g. UI_VERIFY=1 npm start — so a server-rendered branch can see it; the browser markers cover client-side animation without it. This is the record-side of the same freeze the Storybook/Vitest skills apply in-browser.
This is the first part of the skill. The complete, current version: https://github.com/uiverify/uiverify/blob/main/packages/skills/skills/playwright-visual-testing/SKILL.md
- name: Install QA Skills
run: npx @qaskills/cli add playwright-visual-regression-testing-stop-flaky-screenshot-diffs-ui-verify4 of 29 agents supported
Go from zero to Playwright pro: Page Object Model, fixtures, and CI/CD on real projects.