Agentic Testing

Use AI agents — Claude Code, OpenAI Agents SDK, Cursor — to autonomously write, run, and fix tests.

What Is Agentic Testing?

Traditional AI assistance: you prompt, AI responds, you act.

Agentic testing: the AI takes a goal, plans steps, uses tools (file read/write, shell commands, browser), executes, observes results, and iterates — with minimal human intervention.

Goal: "Write e2e tests for the checkout flow"

Agent loop:
1. Read existing test structure           (filesystem tool)
2. Navigate the app                       (browser tool)
3. Inspect DOM / accessibility tree       (browser_snapshot)
4. Write test file                        (write file tool)
5. Run test                              (shell: npx playwright test)
6. Read failure output                   (stdout)
7. Fix selector mismatch                 (edit file tool)
8. Re-run → green                        (shell)
9. Open PR                               (git tool)

Claude Code for Test Automation

Claude Code is a CLI AI agent with file system access, shell execution, and MCP server support. It can autonomously implement test plans.

# Install
npm install -g @anthropic-ai/claude-code

# Run in your test project
cd e2e-tests/
claude

# Prompt examples:
# "Write Playwright tests for all API endpoints in src/routes/"
# "Fix all failing tests — run them first to see what's broken"
# "Add authentication fixtures to reduce test setup duplication"

What Claude Code does automatically:

• Reads your existing test files for patterns
• Runs the test suite and reads failures
• Edits files and re-runs until tests pass
• Commits with descriptive messages

OpenAI Agents SDK

import { Agent, tool } from '@openai/agents';
import { z } from 'zod';
import { execSync } from 'child_process';
import { readFileSync, writeFileSync } from 'fs';

const runTests = tool({
  name: 'run_playwright_tests',
  description: 'Run Playwright tests and return results',
  parameters: z.object({
    testFile: z.string().optional(),
    grep: z.string().optional(),
  }),
  execute: async ({ testFile, grep }) => {
    const cmd = ['npx', 'playwright', 'test'];
    if (testFile) cmd.push(testFile);
    if (grep) cmd.push('--grep', grep);
    try {
      return execSync(cmd.join(' '), { encoding: 'utf8' });
    } catch (err: any) {
      return err.stdout + err.stderr;
    }
  },
});

const testEngineerAgent = new Agent({
  name: 'TestEngineer',
  instructions: `You are a senior test engineer. When asked to write tests:
    1. Read the existing test structure first
    2. Write tests following established patterns
    3. Run tests and fix any failures
    4. Ensure all tests pass before finishing`,
  tools: [runTests, readFileTool, writeFileTool],
  model: 'gpt-4o',
});

const result = await testEngineerAgent.run(
  'Write Playwright tests for the login page at src/pages/login.tsx'
);

Cursor and Windsurf

IDE-native AI agents that understand your codebase:

Cursor (composer mode):
  @codebase "Write tests for the UserProfile component"
  → Reads component code
  → Reads existing test patterns  
  → Generates matching test file
  → Runs tests via terminal integration

Windsurf (cascade mode):
  Similar: multi-file edits, test execution, iterative fixing

Strengths of IDE agents:

• Full codebase context (not just the files you show)
• Live terminal integration (run and observe)
• Diff preview before applying changes
• Works with existing project config (playwright.config.ts)

Agentic Testing Guardrails

Agents need constraints — unbounded agents write untestable tests:

// CLAUDE.md — agent instructions for this project
// Test writing rules:
// - Use getByRole over CSS selectors
// - No waitForTimeout — use waitFor* assertions
// - storageKey must be unique: run grep before adding QuizCard
// - All tests must pass before committing
// - Run npm run build after adding MDX files

// .claudeignore or .cursorignore
// Prevent agents from touching:
dist/
node_modules/
.env
playwright/.auth/

Human-in-the-Loop vs Fully Autonomous

Most production use today is semi-autonomous: agent does the work, engineer reviews the diff.