Evaluate complete agent behavior across task outcomes, tool effects, authorization, cost, and failure recovery before releasing changes.
An agent regression suite is a versioned set of tasks, environments, assertions, and review rubrics that tests the system's observable behavior. It extends answer evaluation to the trajectory: which tools were called, what state changed, whether permission boundaries held, and whether unfinished work was reported accurately.
The unit under evaluation is a release bundle containing prompts, models, tools, retrieval configuration, policies, and runtime code. A good final answer cannot compensate for an unauthorized intermediate action. Separate deterministic invariants from subjective quality judgments, and calibrate model-based graders against human-reviewed examples.
A proof of concept uses a small curated task set and manual inspection to discover failure categories rather than claim broad reliability.
Production gates releases on versioned fixtures, prohibited-action assertions, outcome checks, and representative human-reviewed cases with explicit acceptance thresholds.
Enterprise scale adds cohort analysis, repeated stochastic runs, production-derived cases with privacy controls, evaluator drift checks, and ownership for failed release gates.
Stores task intent, input evidence, permissions, expected outcomes, and known failure categories.
Provides resettable tool state, fault injection, synthetic identities, and repeatable external responses.
Captures observable tool calls, state transitions, approvals, timing, and usage without requiring private model reasoning.
Checks exact invariants and separately scores qualitative outputs against calibrated rubrics.
Compares candidate and baseline results by cohort and blocks unacceptable regressions.
flowchart LR
R["Versioned release bundle"] --> E["Resettable evaluation environment"]
C["Cases and permissions"] --> E
E --> T["Observable action trace"]
T --> I["Deterministic invariants"]
T --> Q["Calibrated quality rubric"]
I --> G["Release comparison gate"]
Q --> G
G --> H["Accountable release decision"]Translate workflow requirements into outcome, authorization, resource, and recovery assertions before running a candidate.
Record the model, prompt, tool, policy, corpus, and evaluator versions so results can be interpreted later.
Reset tool state and run representative normal, adversarial, ambiguous, and dependency-failure scenarios.
Check receipts and state changes deterministically; use calibrated reviewers or graders only for aspects requiring judgment.
Review success, violations, cost, and latency by cohort across repeated runs, including uncertainty and sample size.
Block material regressions, document accepted tradeoffs, and convert confirmed production incidents into privacy-reviewed regression cases.
E-commerce · Illustrative enterprise scenario, not a reported customer outcome.
Illustrative architecture: A merchant changes the model used to prepare and execute approved refund workflows.
A stronger answer-writing model may still duplicate a refund or accept instructions embedded in a customer attachment.
Evaluate financial side effects, customer ownership, partial-order handling, and communication accuracy together.
A sandbox payment service records exact operations while curated cases cover split orders, timeouts, stale approvals, and malicious document text.
The candidate reports success before a payment receipt exists; the state assertion fails even if a text grader likes the response.
Track task success, unauthorized effects, duplicate operations, p95 latency, and cost per accepted case; report sample sizes and avoid extrapolating small suites to all traffic.
| Decision | Benefit | Cost |
|---|---|---|
| Deterministic assertions | Catch exact policy and state violations reliably. | Cannot fully judge usefulness, tone, or contextual completeness. |
| Model-based grading | Scales review of nuanced outputs. | Requires calibration and can reproduce evaluator bias or drift. |
| Repeated stochastic runs | Reveal outcome variability hidden by one pass. | Increase evaluation cost and release lead time. |
Detect: Production-derived holdout cases regress while familiar benchmark cases remain strong.
Recover: Refresh the holdout with independent cases and separate development examples from release evaluation.
Detect: Human calibration finds systematic disagreement on citation and uncertainty cases.
Recover: Revise the rubric, audit affected results, and pause the unreliable quality gate until recalibrated.
Assert tool permissions, actual side effects, and unresolved-state reporting from observable traces.
Keep hard violations and critical cohorts as separate release blockers.
Version the full evaluation bundle and rerun the baseline when fixtures change.