Blog
QA testing insights, AI agent tips, and skill development guides
All Articles
Page 9 of 36
DeepEval TaskCompletionMetric: Trace Setup and Failure Analysis
Implement DeepEval TaskCompletionMetric with complete agent traces, calibrated judges, CI gates, outcome evidence, and systematic failure diagnosis.
Deterministic Graders vs LLM Judges vs Human Review
Choose deterministic graders, calibrated LLM judges, or human review using task risk, objective evidence, agreement checks, cost, and escalation rules.
Evaluate Codex vs Claude Coding Agents with Promptfoo
Build a fair Promptfoo coding-agent evaluation for Codex and Claude using controlled tasks, sandboxes, graders, repeated trials, and review.
Generate Synthetic RAG Testsets with Ragas and Your Documents
Generate, validate, and govern synthetic RAG testsets with current Ragas concepts while controlling leakage, sampling bias, privacy, and held-out use.
High Answer Relevance but Low Faithfulness: Diagnose Wrong RAG Answers
Diagnose fluent, on-topic RAG answers that are unsupported by retrieved evidence using atomic claim tracing, controlled context tests, and release gates.
How to Build an LLM Eval Harness That Matches Production Behavior
Build an LLM eval harness that exercises the production application path, isolates state, records versions and traces, repeats trials, and gates releases.
How to Install Playwright CLI Skills in Codex and Claude Code
Install Playwright CLI skills for Codex and Claude Code, verify agent discovery, run a browser smoke test, and fix common skill setup failures.
How to Install the DeepEval Skill in Codex, Claude Code, and Cursor
Install and verify the official DeepEval skill in Codex, Claude Code, and Cursor, then govern permissions, eval loops, updates, CI, and rollback.
How to Tell Whether a RAG Failure Comes from Retrieval or Generation
Localize RAG defects with controlled context substitutions, claim evidence, retrieval labels, and release checks that separate retrieval from generation.
How to Test AI-Generated Code: A Practical SDET Review Playbook
Review and test AI-generated code with a risk-based SDET workflow covering diff scope, independent oracles, security, Playwright checks, CI, and merge gates.
Install Promptfoo Agent Skills in Codex and Claude Code
Install Promptfoo agent skills in Codex and Claude Code, verify routing and config output, and govern permissions, upgrades, and repository policy.
ISTQB CT-AI v2.0 Guide for QA Engineers: What Changed in 2026
Understand the ISTQB CT-AI v2.0 scope, syllabus changes, exam facts, practical exercises, migration choices, and a focused study plan for QA engineers.
LLM Testing Complete Guide: Evals, Agents, RAG, and Quality Gates
Build reliable LLM tests for prompts, agents, RAG, multi-turn workflows, security, monitoring, CI quality gates, cost, and eval-platform migration.
MCP Server Testing Complete Guide for Protocol, Tools, and Security
Test MCP servers across lifecycle, transports, contracts, conformance, Inspector, authentication, resilience, observability, and security in 2026.
npx playwright init-agents Setup Guide for Agentic Test Loops
Set up Playwright test-agent definitions for VS Code, Claude Code, Codex, or OpenCode, verify the generated files, and diagnose setup failures.
OpenAI Evals Platform Shutdown: Migration Checklist for November 2026
Migrate OpenAI Evals before the November 2026 shutdown with export, Promptfoo, code-first parity, CI, grader validation, and rollback checklists.
Playwright 1.61 WebAuthn Passkey Testing with Virtual Authenticators
Test passkey registration and sign-in with the cross-browser Credentials virtual authenticator added in Playwright 1.61, including reuse and failure diagnosis.
Playwright BrowserContext Guide for Isolation and Parallel Sessions
Use Playwright BrowserContext for clean test isolation, independent multi-user sessions, reusable auth state, context-wide controls, and safe parallel execution.
Playwright CLI Accessibility Snapshots and Element References Explained
Understand Playwright CLI accessibility snapshots, use short-lived element refs safely, scope agent context, and troubleshoot stale or missing references.
Playwright CLI Complete Guide for Browser Automation and AI Agents
Use Playwright CLI for agent-driven browser automation, snapshots, sessions, debugging, traces, video, secure CI workflows, and MCP decisions in 2026.
Playwright Generator Agent Guide for Maintainable Test Code
Turn reviewed Markdown plans into maintainable Playwright tests with the Generator agent, live verification, fixture reuse, and disciplined code review.
Playwright Healer Agent Guide for Repairing Failed Browser Tests
Use the Playwright Healer agent to replay a named failure, inspect current UI behavior, review a minimal patch, and reject repairs that hide regressions.
Playwright localStorage and sessionStorage API Guide for Version 1.61
Use Playwright 1.61 page.localStorage and page.sessionStorage to inspect, seed, clear, and diagnose origin-scoped browser state without page.evaluate boilerplate.
Playwright Locators Best Practices: Roles, Strictness, and Stability
Choose stable Playwright locators with roles, labels, scoped filters, strictness, and web-first assertions, then diagnose ambiguity without brittle shortcuts.
Playwright MCP Complete Guide for Browser Automation with AI Agents
Configure and use Playwright MCP for AI browser automation, testing, profiles, security, HTTP, Docker, CI, and reliable agent workflows in 2026.
Playwright MCP Persistent, Isolated, and Browser Extension Profiles
Choose Playwright MCP persistent, isolated, or browser-extension state for QA, including storage-state setup, parallel sessions, and profile risks.
Playwright MCP Security Best Practices for Files, Origins, and Secrets
Harden Playwright MCP file access, browser origins, profiles, secrets, artifacts, transports, sessions, and authorization with official security guidance.
Playwright MCP Server Configuration Reference for QA Teams
Configure Playwright MCP for QA with documented CLI flags, environment variables, JSON schema, capabilities, browsers, timeouts, output, and network controls.
Playwright MCP Testing Capability: Assertions and Test Generation
Enable Playwright MCP testing tools, verify elements, text, lists, and values, generate locators, and convert browser exploration into reviewable tests.
Playwright Planner Agent Guide for High-Coverage Markdown Test Plans
Use the Playwright Planner agent to explore bounded user flows, design risk-based scenarios, and produce precise Markdown plans ready for human review.
Playwright Test Agents Complete Guide: Planner, Generator, and Healer
Use Playwright test agents safely from setup through planning, generation, trace-driven healing, review, and CI with current Playwright 1.61 guidance.
Playwright Testing Complete Guide for Reliable E2E Automation in 2026
Build reliable Playwright E2E automation with current setup, locators, fixtures, isolation, auth, API testing, mocking, debugging, CI, and Playwright 1.61 guidance.
Promptfoo Complete Guide for LLM Evals, RAG, and Red Teaming
Use Promptfoo to design LLM evals, test RAG and coding agents, run red teams, enforce CI gates, preserve evidence, and govern reproducible releases.
RAG QA Testing Guide for Retrieval, Generation, and Citation Quality
Build a rigorous RAG testing strategy for retrieval, context, answers, citations, security, cost, latency, regression data, CI, and production monitoring.
Run Parallel Playwright CLI Sessions with PLAYWRIGHT_CLI_SESSION
Run coding agents in isolated Playwright CLI browser sessions, monitor them in the dashboard, attach to existing browsers, and clean up state safely.
Run the Official MCP Conformance Suite Against Your Server
Run the official MCP conformance suite against a live server with a pinned runner, deterministic fixtures, scoped results, and release-ready evidence.
Self-Healing Test Automation Governance for Reliable QA Suites
Govern self-healing test automation with eligibility rules, human review, audit evidence, stop conditions, safe examples, rollout phases, and reliability metrics.
Test and Red-Team an MCP Server with Promptfoo's MCP Provider
Configure Promptfoo MCP provider tests for local and remote servers, add authorization and threat cases, and gate safe results in CI.
Test MCP Tool Schemas, Defaults, Invalid Inputs, and Error Types
Test MCP tool schemas, omitted defaults, invalid inputs, protocol and execution errors, structured output, and product semantics under the current spec.
Use MCP Inspector CLI to Automate tools/list and tools/call Tests
Automate MCP tools/list and tools/call checks with the pinned Inspector CLI, typed arguments, JSON assertions, negative cases, and CI-safe evidence.
What OpenAI's Promptfoo Acquisition Means for Open-Source LLM Testing
Separate verified OpenAI-Promptfoo acquisition facts from inference, then protect open-source LLM eval continuity with practical release controls.
Add a PostgreSQL Service-Container Health Check in GitHub Actions
Add a PostgreSQL service-container health check in GitHub Actions so migrations and tests wait for readiness instead of racing database startup.
AI Testing Engineer Salary and Skills in 2026
Compare AI testing engineer salary benchmarks for 2026, identify high-value technical skills, and build evidence that supports a stronger compensation offer.
API Contract-Testing Interview Scenarios
Practice API contract-testing interview scenarios on consumer needs, provider verification, schema evolution, messaging, deployment gates, and failure triage.
Artillery Load Testing in Node.js: Complete 2026 Guide
A complete Artillery load testing guide for Node.js in 2026: install, write YAML scenarios, test HTTP and WebSocket, add JS processors, assert thresholds, and run in CI.
Assert a Specific Log Level with pytest caplog
Assert specific Python log levels with pytest caplog, isolate logger records, verify exception metadata, and avoid tests coupled to formatted output.
Assert That a Playwright List Has No Duplicate Items
Detect duplicate list items in Playwright with stable keys, normalized text, retrying assertions, and diagnostics that identify every repeated record.
Assert That a Table Column Is Sorted in Playwright
Assert sorted table columns in Playwright across numbers, dates, nulls, and locale-aware text with stable extraction and diagnostic failures.
Braintrust vs Langfuse: LLM Eval and Observability (2026)
Braintrust vs Langfuse compared for 2026: eval workflows, tracing, datasets, scoring, CI, self-hosting, pricing shape, and TS + Python SDK examples to pick the right platform.
Build a Rerun-Failed-Tests Workflow in CircleCI
Build a CircleCI rerun-failed-tests workflow that persists Playwright failure state, retries only failed tests in a follow-up job, and reports honestly.