How to Test AI-Generated Code: A Practical SDET Review Playbook

Testing AI generated code should treat every model-produced diff as untrusted implementation: recover the requirement, inspect the complete change, run cheap deterministic gates, add independent positive and negative tests, apply security controls proportional to risk, and require a human merge decision. Do not trust polished syntax, an agent summary, coverage alone, or tests written by the same agent as proof. The AI test automation pillar explains where coding agents fit; this playbook gives SDETs a repeatable review path from prompt to production evidence.

The process is deliberately stricter than "run the tests." OWASP's current Secure Coding with AI Cheat Sheet warns that agents can delete tests, weaken assertions, add mocks that bypass the unit under test, or assert faulty behavior to make CI green. DORA's 2025 research characterizes AI as an amplifier of existing organizational strengths and weaknesses. Fast generation needs faster, higher-quality feedback, not a lower bar.

Establish the Review Contract Before Reading the Diff

An SDET review needs four inputs: intended behavior, allowed scope, risk classification, and verifiable completion criteria. If the task says only "fix checkout," stop and recover the missing contract. An agent can produce a plausible change for an ambiguous request, but no reviewer can decide correctness without knowing which behavior was authorized.

Write a compact review contract:

• Intent: the user or system behavior that must change.
• Non-goals: behavior, files, data, and interfaces that must not change.
• Invariants: authorization, compatibility, financial, privacy, and reliability rules that remain true.
• Evidence: tests, static checks, traces, screenshots, migration proof, or threat checks required for acceptance.
• Rollback: how to disable or revert the behavior if production evidence is wrong.

Preserve the original request and any agent plan with the change record where policy permits. They help explain why code exists, but they are not the oracle. Requirements, contracts, threat models, and independently reviewed expected outcomes outrank the model's interpretation.

The AI4Testing versus Testing AI guide clarifies that this workflow is AI4Testing governance: AI helps develop conventional software. If the generated code implements an AI feature, add the lifecycle and data/model controls in the ISTQB CT-AI v2 guide. If an agent repairs the generated tests later, apply the separate self-healing governance guide.

Route the Change by Consequence

Change class	Examples	Minimum independent evidence	Merge authority
Low	Copy, non-executable docs, isolated styling	Scope diff, render or link check, existing CI	Normal owner after review
Moderate	Pure business function, UI state, non-sensitive API client	Unit boundaries, integration behavior, regression test, static checks	Code owner plus normal QA gate
High	Authentication, authorization, money, personal data, migrations, concurrency	Threat-based negative tests, security requirements, integration/e2e, rollback proof	Domain and security owners as defined by policy
Critical	Deployment credentials, CI trust, cryptography, destructive data path, safety control	Specialist design review, isolated validation, recovery exercise, explicit approval	Named accountable specialist; no agent-only approval

This table routes effort; it does not prove safety. A one-line authorization change can be higher risk than a thousand-line generated fixture. Classify by possible impact and reversibility, not line count or model confidence.

Gate 0: Prove Provenance and Scope

Start from the repository, not the assistant's prose summary. Inspect every changed file, including tests, lockfiles, workflow files, generated artifacts, configuration, migrations, and deleted files. Compare the actual diff with the allowed scope. Out-of-scope edits are separate review items even when harmless.

A practical first pass is tool-neutral:

git status --short
git diff --stat
git diff --name-status
git diff --check
git diff

Ask concrete questions:

1. Did the agent modify its own instruction or rules files?
2. Did a dependency or lockfile change without an explicit need?
3. Were tests deleted, skipped, broadened, or made less specific?
4. Did permissions, network access, logging, telemetry, or secret handling change?
5. Is generated code copied from a package or API that actually exists at the pinned version?
6. Did formatting noise obscure a small semantic change?

OWASP identifies out-of-scope edits and review anchoring as a specific AI-assisted development risk. Review each file rather than approving from the pull-request description. If the diff is too large to understand, split it. Smaller batches improve both human review and diagnostic feedback.

Gate 1: Reconstruct Behavior Before Running Anything

Read callers, data flow, error paths, tests, configuration, and external contracts around the change. AI output often looks locally coherent while violating a repository convention or distant invariant. Search for equivalent logic rather than accepting a duplicate helper. Verify units, time zones, encoding, rounding, nullability, transaction boundaries, and async cancellation explicitly.

For each changed branch, write the expected behavior in plain language before observing the implementation's output. This reduces anchoring. Include:

• normal path and boundary values;
• invalid, absent, malformed, oversized, and repeated input;
• unauthorized and cross-tenant access;
• dependency timeout, partial failure, duplicate delivery, and retry;
• concurrent requests or stale state where relevant;
• old clients, stored data, and rollback behavior;
• logs and errors that must not expose secrets or personal data.

If no one can explain the new behavior without reading generated tests, the oracle is not independent enough.

Gate 2: Run Deterministic, Cheap Feedback First

Use the repository's documented commands for formatting, linting, type checking, build, tests, dependency checks, secret scanning, and static security analysis. Do not paste a generic CI stack into every project. A TypeScript compiler cannot validate a Python service, and a package audit cannot prove authorization.

Run narrow checks while iterating, then the required full gate before merge. Record tool versions and exact commands in CI. A clean result means only that the configured check found no blocking issue. It does not mean the behavior is correct or secure.

NIST's final Secure Software Development Framework 1.1 recommends integrating secure practices into the SDLC, reviewing designs against security requirements, testing executable code, and identifying residual vulnerabilities. Apply those practices to AI output through the existing engineering system rather than creating a weaker "generated code" lane.

Gate 3: Add Tests the Generator Did Not Choose

Tests produced with the implementation are useful scaffolding, not independent assurance. The reviewer should add or select cases based on requirements and risk without asking the same generation context what it forgot.

Use multiple test levels deliberately:

• Unit tests isolate calculations, state transitions, and boundary rules.
• Property or invariant tests cover a range wider than hand-picked examples.
• Contract tests protect API, event, schema, and consumer expectations.
• Integration tests expose database, queue, cache, identity, and transaction behavior.
• End-to-end tests prove a few critical user outcomes through deployed boundaries.
• Exploratory checks target ambiguity, interruption, abuse, and surprising state combinations.

Coverage is navigation evidence, not oracle evidence. A line can execute under an assertion that checks nothing meaningful. Mutation testing can help reveal weak tests when supported by the project, but mutation score is also not a universal release threshold.

Example 1: Independently Test a Generated Refund Function

Assume the approved rule says: amounts are integer cents; the refund is the captured amount minus a non-refundable fee; the result cannot be negative; invalid money input is rejected. An agent generates the implementation and its happy-path test. The SDET adds requirement-derived boundaries using the project's existing Vitest stack:

import { describe, expect, it } from 'vitest';
import { calculateRefund } from './calculate-refund';

describe('calculateRefund', () => {
  it.each([
    { captured: 10_00, fee: 1_00, expected: 9_00 },
    { captured: 1_00, fee: 1_00, expected: 0 },
    { captured: 50, fee: 1_00, expected: 0 },
  ])('applies the approved floor for $captured and $fee', ({ captured, fee, expected }) => {
    expect(calculateRefund(captured, fee)).toBe(expected);
  });

  it.each([
    { captured: -1, fee: 0 },
    { captured: 100.5, fee: 0 },
    { captured: Number.MAX_SAFE_INTEGER + 1, fee: 0 },
  ])('rejects invalid integer-cent input: %o', ({ captured, fee }) => {
    expect(() => calculateRefund(captured, fee)).toThrow();
  });
});

The table is illustrative, not a universal refund policy. The key practice is that cases come from approved rules and domain risks. The reviewer should also inspect callers for currency mixing, duplicate refunds, authorization, persistence, and idempotency. A perfect pure-function test cannot prove the payment workflow.

Gate 4: Apply Security Requirements, Not a Generic "Security Prompt"

Threat-model changed trust boundaries. Trace attacker-controlled input to interpreters, databases, HTML, files, shell commands, network requests, deserializers, templates, logs, and AI tool calls. Check authentication separately from authorization. Test object-level access, tenant isolation, role changes, expired sessions, replay, rate limits, and safe failure.

The OWASP Application Security Verification Standard is a primary, versioned source for web application security verification requirements. OWASP lists ASVS 5.0.0 as the current stable version and recommends version-qualified requirement identifiers because identifiers can change. Select requirements relevant to the application's verification level and architecture; do not claim "OWASP compliant" from a scanner pass.

For adjacent application techniques, the canonical security testing for AI-generated code guide expands threat-driven checks, while the QA engineer's AI code review guide covers review workflow. Those articles supplement, rather than replace, the primary controls cited here.

AI-specific development risks also need direct controls:

• Verify every new dependency, package name, version, license, and maintainer source rather than trusting a plausible import.
• Treat issues, pull-request comments, repository documents, fetched pages, and tool output as possible indirect prompt-injection inputs to an agent.
• Prevent secrets, personal data, proprietary code, and production records from entering unapproved model context.
• Restrict agent credentials, filesystem scope, network destinations, and destructive commands to the minimum task need.
• Require explicit review when tests, CI, deployment, access control, or agent rules change.

Security scanners provide findings, not final risk decisions. Triage false positives with evidence and preserve accepted residual risk through the organization's normal process.

Example 2: Add a Negative Authorization Journey in Playwright

Assume an AI agent adds an invoice details page and a happy-path browser test. The requirement says a workspace member must never learn whether another workspace's invoice exists. In a test project whose authenticated fixture already signs in a normal member, add an independent cross-tenant journey:

import { test, expect } from './fixtures';

test('member cannot read another workspace invoice', async ({ memberPage }) => {
  await memberPage.goto('/workspaces/acme/invoices/invoice-from-rival');

  await expect(memberPage.getByRole('heading', { name: 'Not found' })).toBeVisible();
  await expect(memberPage.getByText(/invoice total/i)).toHaveCount(0);
  await expect(memberPage.getByText(/rival workspace/i)).toHaveCount(0);
});

The fixture and URLs must be adapted to the real project; they are not Playwright product APIs. The Playwright calls shown are current documented APIs. Official Playwright best practices recommend testing user-visible behavior, isolating tests, and using user-facing locators. The API or database layer still needs direct authorization tests because a UI journey alone cannot cover every object access path.

Gate 5: Review AI-Generated Tests as Production Code

Inspect test intent, not just pass status. Reject tests that:

• copy implementation branches into expected logic;
• assert only status "not 500," truthiness, element presence, or non-null output when stronger behavior is known;
• use broad snapshots without reviewing meaningful change;
• mock the component or dependency whose contract is under test;
• depend on execution order, shared accounts, sleeps, or current wall-clock time;
• swallow errors, add retries to deterministic defects, or skip the broken scenario;
• hard-code secrets, unstable IDs, generated element references, or production data;
• pass alone but fail under repetition, parallelism, different order, or clean state.

Compare assertion strength before and after the diff. Review deleted and skipped tests explicitly. If an assertion changed, ask whether the requirement changed and where that decision is recorded.

For reusable agent guidance, browse QA skills or use the Playwright CLI skill for observable browser work. A skill can encode review expectations, but it cannot independently approve the code it helped create.

Use Playwright Test Agents Without Creating a Closed Assurance Loop

The official Playwright Test Agents provide planner, generator, and healer roles. Their artifact structure is useful: human-readable plans under specs/, generated tests under tests/, and seed tests for setup. Keep the plan reviewable and align tests one-to-one with approved scenarios where feasible.

Playwright documents that the generator can produce initial errors and the healer can replay failures, inspect the current UI, suggest patches, and rerun until success or guardrails stop the loop. It also says the healer's output can be a passing test or a skipped test if it believes functionality is broken. Therefore "the healer finished" is not a merge gate. Inspect whether the scenario was skipped, assertions changed, data altered, or product behavior masked.

A safer loop is:

1. Human approves requirement and planned scenarios.
2. Generator creates a candidate test.
3. CI executes unchanged acceptance assertions.
4. Healer proposes, but does not merge, a minimal patch.
5. Reviewer compares the patch with intent and checks all files.
6. CI reruns the target, neighboring tests, and required full suite.
7. A human records the merge or reject decision.

Regenerate agent definitions when Playwright is upgraded, as the official docs require, then review changes to those definitions like other tooling changes.

Make the Merge Decision Explicit

The change is ready only when the reviewer can answer yes to all applicable questions:

• The complete diff matches authorized scope.
• The intended behavior and invariants are independently stated.
• Static, build, and test gates pass with recorded versions.
• New tests cover normal, boundary, error, and abuse paths proportional to risk.
• Existing tests were not silently weakened, deleted, or skipped.
• Dependencies and generated APIs are real, supported, and pinned appropriately.
• Security requirements and threat paths have evidence.
• Data, migration, compatibility, observability, and rollback are addressed.
• Residual risks and limitations have named owners.

Do not merge when the only explanation is "the agent says it fixed it." Ask for a smaller diff, stronger requirement, specialist review, or reproducible evidence.

Concrete Failure Paths and Recovery

The agent changes tests until CI passes

Freeze the product requirement and compare test assertions before and after. Revert unauthorized test changes through the normal review process, add an independent regression test, and fix the product or formally change the requirement. Passing CI after oracle drift is a false green.

A plausible package does not exist or is not the intended dependency

Stop installation. Verify the package through the official registry and project source, inspect ownership and release history, and confirm the exact API for the pinned version. Treat a lookalike package as a supply-chain incident, not a typo to work around.

The full suite is too slow, so only generated tests run

Use risk-based test selection for fast feedback but retain a required broader gate before merge or release. Track what was not run. Optimize the suite rather than redefining a narrow green run as complete evidence.

Reviewers cannot understand the generated diff

Do not approve it. Split the change, remove unrelated refactors, require design notes, or replace generated sections with a maintainable implementation. Reviewability is a production quality attribute because future responders must diagnose and change the code.

Production behavior fails despite high coverage

Reconstruct the missing condition and determine whether the oracle, environment, data, integration, or monitoring was absent. Add a test at the lowest effective level plus a production detection or recovery control. Do not respond by chasing a coverage percentage alone.

Measure the System, Not Generated Lines

DORA's 2026 Impact of Generative AI in Software Development advises organizations to reinforce automated testing and fast code reviews as AI increases code output. Track whether the review system handles changes safely: batch size, review latency, rework, change failure, recovery, escaped defects, security findings, and test signal quality.

Generated lines, prompts, accepted completions, and test count are activity measures. They can help operate a tool but should not be presented as product value. Compare outcomes over enough time to see delayed rework, and segment high-risk generated changes from low-risk assistance.

Version and Limitation Notes

This playbook is current to July 14, 2026. It cites OWASP ASVS 5.0.0, the current OWASP Secure Coding with AI Cheat Sheet, NIST SSDF 1.1 as the final publication, DORA's 2025 report and 2026 impact guidance, and current Playwright Test Agent documentation. NIST published an initial draft of SSDF 1.2 in late 2025; do not represent draft language as the final 1.1 standard, and check for a later final before adopting policy.

The TypeScript and Playwright examples illustrate review technique, not drop-in application APIs. Risk classification, approval, data handling, and security requirements must follow the product, jurisdiction, and organization. No combination of automated checks proves absence of defects or vulnerabilities.

Frequently Asked Questions

1. Is AI-generated code inherently worse than human code?

This playbook does not assume a universal defect rate. It treats provenance as a reason for specific controls: rapid volume, context gaps, plausible nonexistent APIs, review anchoring, and tests generated by the same system. Human code still requires the normal quality and security gates.

2. Should every line of AI-generated code receive human review?

Executable changes should follow the organization's review policy, with human accountability and greater scrutiny for high-impact paths. Low-risk assistance can use lighter routing, but no agent summary should substitute for inspecting the actual diff.

3. Can tests written by the same AI agent be used?

Yes, as candidate tests. Their expected results, scope, assertion strength, mocks, deletions, and skips need independent review. Add cases derived from requirements and threats outside the generation context.

4. What is the first check an SDET should run?

Inspect status and the complete diff before running generated commands. Scope violations, deleted tests, lockfile changes, and CI edits can change the risk and the safe execution plan.

5. Does high code coverage make generated code safe?

No. Coverage shows execution, not that expectations are correct, security properties hold, or important conditions were represented. Combine it with requirement-based, negative, integration, and threat-driven evidence.

6. How should Playwright's healer be governed?

Treat its output as a proposed patch. Review intent, every changed file, locator and assertion strength, test data, and skipped status. Rerun the unchanged acceptance criteria and broader required suite before a human merge decision.

7. Which standards are useful for security review?

OWASP ASVS supplies versioned web security verification requirements, OWASP's Secure Coding with AI Cheat Sheet addresses AI-assisted development risks, and NIST SSDF provides secure-development practices. Select applicable controls through the product threat model rather than claiming blanket compliance.