AI Agent Testing: A Runnable Pytest and LLM-Judge Harness (2026)

Quick answer

AI agent testing means checking that an agent picks the right tools, produces the right structured output, and gives good enough answers, on every code change, before it ships. You cannot test an agent the way you test a pure function, because the model output is non-deterministic. The working pattern in 2026 is a two-layer harness: deterministic assertions for the parts that must be exact (which tool ran, what arguments, whether the JSON parses), and an LLM-as-judge grader for the open-ended parts (was the answer correct and on-topic). You run both under pytest, freeze a small labeled dataset, and gate merges in CI. This tutorial builds that harness end to end with runnable Python and no eval framework required.

Every agent tutorial shows you how to build the agent. Almost none show you how to prove it still works after you change the prompt. This is the missing half. By the end you will have a real pytest suite that catches two classes of regression that break agents in production: wrong tool selection, and quietly worse answers.

Why AI agent testing is different

A unit test for add(2, 2) expects exactly 4. An agent asked to "triage this support ticket" can answer correctly in a hundred different wordings. So the naive approach, asserting output == "expected string", fails on the first rephrasing even when the agent is right.

Two things ARE deterministic enough to assert on directly:

• Control flow: which tool the agent called, and with what arguments. This is structured data, not prose. Assert on it exactly.
• Schema: if you ask for JSON, it either parses against your schema or it does not.

For everything genuinely open-ended (tone, correctness of a summary, whether a refusal was appropriate), you grade with a second model. That is the LLM-as-judge layer. Keep the two layers separate so a judge flake never hides a real control-flow bug.

Prerequisites

You need Python 3.10+ and an Anthropic API key. Install the SDK and pytest.

Set up a clean environment first:

python -m venv .venv && source .venv/bin/activate
pip install "anthropic>=0.40" "pytest>=8.0"
export ANTHROPIC_API_KEY="sk-ant-..."

All code below uses the current Anthropic model ids as of July 2026: claude-opus-4-8 for the agent under test and claude-haiku-4-5 as the cheap grader. Swap in your own ids if Anthropic ships newer ones; the harness does not care which model it wraps.

the agent under test

Put the agent in its own module so tests can import it. This one triages a support ticket and must call exactly one tool, route_ticket, with a queue and a priority.

# agent.py
import json
from anthropic import Anthropic

client = Anthropic()

TOOLS = [{
    "name": "route_ticket",
    "description": "Route a support ticket to a queue with a priority.",
    "input_schema": {
        "type": "object",
        "properties": {
            "queue": {"type": "string",
                      "enum": ["billing", "technical", "account", "spam"]},
            "priority": {"type": "string",
                         "enum": ["low", "normal", "high", "urgent"]},
        },
        "required": ["queue", "priority"],
    },
}]

def triage(ticket: str) -> dict:
    """Return the tool call the agent made: {'queue': ..., 'priority': ...}."""
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=512,
        tools=TOOLS,
        tool_choice={"type": "tool", "name": "route_ticket"},
        messages=[{"role": "user", "content": f"Triage this ticket:\n\n{ticket}"}],
    )
    for block in resp.content:
        if block.type == "tool_use":
            return block.input
    raise AssertionError("agent did not call route_ticket")

tool_choice forces the tool call, which makes the control-flow layer testable. The Anthropic Python SDK exposes each tool_use block on the response; see the anthropic-sdk-python repo for the full response shape (2026). If you have not built the underlying agent yet, start with build-first-ai-agent-from-scratch and wire the tools per claude-agent-sdk-custom-tools first, then come back here to test it.

deterministic assertions

These tests are ordinary pytest. No model grades anything; we assert on the structured tool input directly.

# test_control_flow.py
from agent import triage

def test_billing_ticket_routes_to_billing():
    out = triage("I was charged twice for my subscription this month.")
    assert out["queue"] == "billing"
    assert out["priority"] in {"normal", "high", "urgent"}

def test_outage_is_urgent():
    out = triage("The whole dashboard is down and my customers cannot log in.")
    assert out["queue"] == "technical"
    assert out["priority"] == "urgent"

def test_output_matches_schema():
    out = triage("How do I change my email address?")
    assert set(out) == {"queue", "priority"}
    assert out["queue"] in {"billing", "technical", "account", "spam"}

Notice what we assert and what we do not. We pin queue because it must be exact. We allow a SET of acceptable priorities where more than one is defensible. Over-tight assertions on non-deterministic output are the number one reason agent test suites get deleted; assert the invariant, not the exact token.

the LLM-as-judge grader

For open-ended answers, a rubric-driven grader scores the output. Use a cheaper model than the one under test so the eval bill stays small; Haiku is Anthropic's fastest, lowest-cost tier and is plenty for pass/fail grading.

# judge.py
import json
from anthropic import Anthropic

client = Anthropic()

RUBRIC = """You are grading an AI support agent's answer.
Return ONLY JSON: {"pass": true|false, "reason": ""}.
Pass only if the answer is factually correct, on-topic, and safe."""

def judge(question: str, answer: str) -> dict:
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=256,
        system=RUBRIC,
        messages=[{"role": "user",
                   "content": f"Question:\n{question}\n\nAnswer:\n{answer}"}],
    )
    text = resp.content[0].text.strip()
    return json.loads(text)

The judge is itself a model call, so it can be wrong. Two guardrails keep it honest: give it a narrow rubric with a single pass condition, and make it return a reason so failing runs are debuggable instead of a bare False. This mirrors how reproducible scoring works on any serious benchmark; the neutral, documented rubric is the whole game. OpenAI's open-source evals framework formalizes the same idea with model-graded templates (2026) if you outgrow a hand-rolled judge.

freeze an eval dataset

A handful of hard-coded asserts is a start; a dataset is a suite. Store labeled cases as data and parametrize.

# test_dataset.py
import json, pytest
from agent import triage

with open("evals/tickets.jsonl") as f:
    CASES = [json.loads(line) for line in f]

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_routing_dataset(case):
    out = triage(case["ticket"])
    assert out["queue"] == case["expected_queue"], case["id"]

evals/tickets.jsonl, one JSON object per line:

{"id": "double-charge", "ticket": "I was billed twice.", "expected_queue": "billing"}
{"id": "login-outage", "ticket": "Nobody on my team can log in.", "expected_queue": "technical"}
{"id": "delete-account", "ticket": "Please close my account.", "expected_queue": "account"}

Every time the agent gets a routing case wrong in production, add it to this file as a new line. The dataset becomes a regression net that grows tighter over time, which is exactly the property you want. The pytest parametrize docs cover ids, marks, and fixtures if you want per-case xfail while you fix a known gap (2026).

gate merges in CI

A test suite that only runs on your laptop protects nobody. Run it on every pull request and fail the build on regression. Keep model calls off the free-for-all path by only running the live suite when the API key is present.

# .github/workflows/agent-evals.yml
name: agent-evals
on: [pull_request]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install "anthropic>=0.40" "pytest>=8.0"
      - run: pytest -q
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Because model output varies run to run, expect the occasional judge flake. Handle it by making judged tests tolerant (grade three samples, require two passes) rather than by rerunning the whole job until it goes green. A suite you trust to block a merge is worth more than a suite that is green because you keep hitting retry.

Verify your setup

Run the whole thing. From the project root:

pytest -q

Expected output on a healthy agent:

.....                                                         [100%]
5 passed in 6.42s

Now break the agent on purpose: change the system framing in agent.py to "route everything to billing" and rerun. You should see test_outage_is_urgent and the login-outage dataset case fail with the queue mismatch. If they fail, your harness is doing its job: it catches a bad prompt change before your users do.

Limitations and open questions

• Judge cost and drift: model-graded evals cost tokens and can drift as the judge model updates. Pin the judge model id and re-baseline when you bump it.
• Coverage is only as good as the dataset: an agent can pass every frozen case and still fail on a distribution you never wrote down. Mine real logs for new cases weekly.
• Non-determinism has a floor: even a correct agent will occasionally pick a defensible-but-different tool. Design assertions around invariants, and accept that a small flake rate is the cost of testing probabilistic systems, not a bug in your suite.

Test the control flow exactly, grade the prose with a rubric, freeze the cases you have already seen fail, and let CI hold the line.