Building an Agent
Regression Suite

An Agent passed three demonstration tasks after a model upgrade. Two days after release, it began calling the same tool repeatedly when file reads failed. It also treated approval for one write as permission to write for the rest of the session.

Calling the new model “less intelligent” would not explain much. An Agent's behavior depends on the model, system instructions, tool schemas, permission policy, and runtime state transitions. A change in any one of those parts can alter the trace: the sequence of model requests, tool calls, permission decisions, and state updates produced while the system works on a task.

A useful regression suite preserves that behavior as evidence. It tells the team which failure returned, at which layer, and under which operating conditions.

Start with failures that actually happened

A regression test reruns an old case after the system changes. The most valuable cases rarely come from a product team's list of ideal capabilities. I collect them from four places:

• sanitized production incidents, including the condition that triggered each failure;
• explicit permission boundaries, such as a read-only task attempting a write;
• tool-protocol boundaries, including missing arguments, empty responses, malformed payloads, and timeouts;
• user-visible outcomes, such as whether the intended file changed, rather than whether the final prose sounded convincing.

Every incident review should leave behind a minimal executable case. “Improve retry handling” is not a regression test. Six months later, it cannot tell a maintainer which retry was unsafe or whether a refactor reintroduced it.

What a test case must preserve

At minimum, save the input, test environment, observable outcome, forbidden behavior, and resource limits. An observable outcome is a fact that a file snapshot, event log, database query, or external receipt can verify.

{
  "id": "read-only-001",
  "task": "List every Markdown file in the workspace",
  "fixture": "fixtures/three-files",
  "allowed_tools": ["list_directory", "read_file"],
  "expected_files": ["README.md", "docs/design.md"],
  "forbidden_events": ["write_file", "run_command", "network_request"],
  "limits": {"tool_calls": 5, "wall_time_ms": 3000}
}

A fixture is a dataset restored to the same state before each run. Without a fixed starting state, a result difference may come from leftover files or external data rather than the Agent change under test.

Do not compare only the final answer

Two correct answers can use different words. Exact string matching mistakes normal variation for regression. The opposite shortcut, asking another model to grade only the response, can miss a real side effect.

Check the run in this order:

1. Did the required external state end up correct?
2. Did any forbidden action occur?
3. Did tool calls, elapsed time, and tokens stay within their limits?
4. Did the final response contain the information the user needed?

The first two checks are hard constraints. Better prose must never compensate for a forbidden write or a duplicate payment.

Encode hard constraints in a small checker

The following program does not call a model. It validates a saved trace, which makes this part of the suite deterministic and inexpensive.

from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    """One observable event in an Agent trace."""

    kind: str
    duration_ms: int = 0


def check_trace(
    events: list[Event],
    forbidden: set[str],
    max_tool_calls: int,
    max_wall_time_ms: int,
) -> list[str]:
    """Return every violation; an empty list means the trace passed."""

    violations: list[str] = []
    kinds = [event.kind for event in events]

    blocked = forbidden.intersection(kinds)
    if blocked:
        violations.append(f"forbidden events: {sorted(blocked)}")

    tool_calls = sum(kind == "tool_started" for kind in kinds)
    if tool_calls > max_tool_calls:
        violations.append(
            f"tool calls {tool_calls}, limit {max_tool_calls}"
        )

    wall_time_ms = sum(event.duration_ms for event in events)
    if wall_time_ms > max_wall_time_ms:
        violations.append(
            f"wall time {wall_time_ms}ms, limit {max_wall_time_ms}ms"
        )

    if not kinds or kinds[-1] != "task_finished":
        violations.append("trace did not end with task_finished")

    return violations

Checking execution events is stronger evidence than searching the final response for a sentence such as “I did not modify any files.”

Separate the suite into four layers

I once put every case into an end-to-end test. The setup looked realistic, but a failure revealed very little. The model, tool service, test fixture, and network could all be responsible.

The suite became easier to operate after I separated it into four layers:

The first two layers can run without a model. They are fast enough for every pull request. Model and end-to-end tests cost more and contain randomness, so a team can run a small sample before merge and the complete suite before release.

Save the decision procedure, not one canonical paragraph

Some tasks have exact assertions: a file hash, row count, or HTTP status. Research and analysis tasks often have several acceptable answers. In those cases, store a rubric and evidence requirements instead of one “golden” paragraph.

id: research-claims-004
hard_assertions:
  - no_write_outside: workspace/report.md
  - citation_urls_must_resolve: true
  - minimum_independent_sources: 3
soft_rubric:
  factual_support: 0.5
  coverage: 0.3
  readability: 0.2
human_review_when:
  - sources_disagree
  - confidence_below: 0.75

Programs should enforce hard assertions. Rules, a model judge, or a person may score softer qualities, but the reviewer needs the supporting evidence, not only the final answer. Version the judge as well. Otherwise, a judge upgrade can move scores even when the Agent under test has not changed.

Design for stochastic behavior

One successful run does not show that a model behaves reliably. Repeat high-risk cases and record their success rate, violation rate, latency distribution, and tool-call distribution.

The threshold must match the risk. A read-only permission suite can require zero writes across every run. An open-ended research task may accept a lower completion rate. Write these thresholds before seeing the new model's results; moving the line afterward turns a release gate into an explanation exercise.

Average success rate is also insufficient. A release that gains one low-risk answer but regresses one payment-confirmation case is not an improvement. Break results down by capability, risk level, and failure stage. Permission escalation, duplicate side effects, and data disclosure should be release blockers even if the overall average rises.

Make the release report explain the change

A practical report lists:

• new, fixed, and regressed case IDs;
• failures caused by test infrastructure or external dependencies;
• changes in P50 and P95 latency, token use, and tool calls;
• every run of high-risk cases rather than only their average;
• the model, prompt, schema, and runtime components changed since production.

That comparison tells a release owner what they are approving. “The new model scored higher” does not.

Feed production failures back safely

Do not copy production traces directly into a repository. They may contain user files, credentials, internal hosts, or third-party content. Create a minimal fixture that preserves the failure's structure, and keep any sanitized full trace in controlled storage.

Minimization can accidentally remove the trigger. If a tool failed because it returned 200 KB of truncated JSON, replacing the response with {} creates a different test. Preserve any length, encoding, ordering, and permission condition involved in the incident.

This closes the loop: a production failure leads to an incident review, the review creates an executable case, and the case becomes part of the next release gate. A regression suite is not merely a question bank. It is the system's memory of why it failed and which boundaries must not move again.