Testing Practices in the Age of Agents

Run the same test twice against an agent and it can pass once and fail once. Nothing changed: same input, same code, same prompt. That single fact breaks the assumption every test suite is built on, which is that the same input produces the same output, and the result is either green or red. Faced with that, a lot of teams quietly give up and fall back to eyeballing outputs in a chat window. That's the wrong move. You don't delete your tests because the output got fuzzy. You restructure them.

Most of an agent is still deterministic: the tool schemas, the control flow, the structured output, the guardrails. Only the model's judgment is fuzzy. The job is to pin down everything deterministic with hard tests, wrap the fuzzy core in statistical gates and evals, and simulate the whole journey for the failures that only show up over a conversation. This post is the long version of how to do that, with code, using one running example throughout.

The first time a test flips from green to red with no code change, you go hunting for the bug. There isn't one. The model just answered the same question with different words, and your assertion was written for the old words.

The running example. Throughout, we test a customer-support agent for an online store. It has three tools: look_up_order(order_id), issue_refund(order_id, amount), and escalate_to_human(reason). It has policies: never refund above ₹5,000 without human approval, never reveal another customer's order. And it holds multi-turn conversations: a shopper asks for a refund, the agent verifies the order, checks policy, and either refunds or escalates. Every technique below is shown against this one agent, building a single test harness section by section.

In this post:

1. Why testing breaks when the unit is non-deterministic
2. The test pyramid for agents: four layers, one harness
3. Layer 1: test the contract: deterministic assertions and hermetic mocking
4. Layer 2: make the fuzzy parts statistical: soft assertions, trials, significance
5. Layer 3: evals with a judge you can trust: rubrics, calibration, bias
6. Layer 4: simulate the whole journey: personas, constraints, trajectories
7. Where test cases come from: curating the set
8. Wire it into CI: the gate that runs on every change
9. Failure modes to avoid
10. What actually holds up

---

Why testing breaks when the unit is non-deterministic

Traditional tests encode an exact expectation: assert response == "Order #123 cancelled". That works when the code is deterministic. Point it at an LLM and the same request comes back as "I've cancelled order #123 for you" one run and "Done, order 123 is now cancelled" the next. Both are correct. Your string comparison fails both, or passes neither, and the test tells you nothing.

Setting temperature=0 feels like the fix. It isn't. Even at zero temperature, output varies across model versions, across hardware and batching, and across the smallest prompt edits, and an agent that calls tools adds ordering and timing that no seed pins down. Determinism at the token level is not something you can assume, so a suite built on exact-match assertions decays into a wall of flaky red that people learn to ignore.

The instinct that follows is to drop tests and vibe-check outputs by hand. That scales to zero. The better move is to separate what is actually non-deterministic (the model's phrasing and judgment) from what is not (everything around it), and test each with the right instrument.

The test pyramid for agents

The classic test pyramid still applies, it just gets re-layered. The base is wide and deterministic: fast, cheap, exact tests on the parts of the system that have one right answer. The middle wraps the fuzzy core in statistical checks and evals. The top is the slow, expensive, high-signal layer: full multi-turn simulations.

Layer	What it checks	Deterministic?	Instrument
Contract	Tool-call schema, JSON validity, output shape, guardrails	Yes	Unit tests + mocked model
Behavior	Is a single output good enough?	No	Soft assertions, N trials
Quality	Is the answer actually good?	No	LLM-as-judge evals
Journey	Does it hold over a conversation?	No	Multi-turn simulation

The reframe is the whole point: an agent test suite is not one kind of test, it's four, and most teams only build one of them. They reach for evals, skip the deterministic base, and never build the simulation top. The base catches the cheap bugs in milliseconds. The top catches the expensive ones that only appear three turns into a conversation. You need both ends, not just the middle.

The rest of this post builds each layer against the support agent, in one harness you could actually run.

Layer 1: test the contract

Start where testing is easy, because more of the agent is deterministic than people admit. You cannot assert the exact words a model returns, but you can assert everything about the shape and the actions around them, and that is where the ugliest bugs live.

The harness gives us a run function that executes one turn and returns a structured result: the final text, the list of tool calls with their arguments, and the raw messages. Everything in this layer asserts against result.tool_calls, never against prose.

# harness.py: the one object every layer reuses.
@dataclass
class Result:
    text: str                     # the model's final natural-language reply
    tool_calls: list[ToolCall]    # each has .name and .args (parsed dict)
    messages: list[dict]          # raw transcript, for debugging

def run(agent, user_message, **ctx) -> Result:
    """Execute a single turn and capture everything structured."""
    ...

The highest-value contract test catches a hallucinated argument. When the agent refunds an order, it must pass an order_id that actually exists in the session, not one it invented to satisfy the tool signature:

def test_refund_targets_a_real_order():
    session = Session(known_order_ids={"A-1001"})
    result = run(agent, "I want a refund for my order A-1001", session=session)

    call = result.tool_call("issue_refund")
    assert call is not None,                    "agent never called issue_refund"
    assert call.args["order_id"] in session.known_order_ids   # catches hallucinated IDs
    assert call.args["amount"] > 0

That test has one right answer and runs in a millisecond. It catches a whole class of confidently-wrong agent bugs that an eval would only notice statistically, if at all. The same pattern covers the rest of the deterministic surface:

• Structured output: if the agent must emit JSON matching a schema, validate it with the schema, not with string matching. Malformed JSON is a deterministic failure.
• Guardrails and control flow: assert the refund over ₹5,000 routes to escalate_to_human instead of issue_refund, that a blocked request triggers the refusal path, that a required lookup happened before any mutation.
• Idempotence and side effects: assert the agent did not call issue_refund twice for one request, and did not touch an order outside the session.

def test_large_refund_escalates_instead_of_paying():
    session = Session(known_order_ids={"A-1001"}, order_total={"A-1001": 9000})
    result = run(agent, "Refund my order A-1001 in full", session=session)

    assert result.tool_call("issue_refund") is None,     "must not auto-refund > ₹5,000"
    assert result.tool_call("escalate_to_human") is not None

Tools fail too, and how the agent handles a failing tool is a deterministic path worth pinning down. Make look_up_order raise, and assert the agent degrades instead of inventing an answer:

def test_handles_tool_failure_gracefully():
    session = Session(known_order_ids={"A-1001"}, failing_tools={"look_up_order"})
    result = run(agent, "Where is my order A-1001?", session=session)

    assert result.tool_call("issue_refund") is None      # never act on data it couldn't fetch
    assert result.escalated or "couldn't look that up" in result.text.lower()

None of this needs an LLM to grade it. It is ordinary software testing, and it should be the fast, boring, always-green majority of your suite.

Make the model calls hermetic

There is a problem hiding in those tests: each run call hits a real model. That makes the suite slow, expensive, non-deterministic, and unrunnable offline or in a fork's CI without secrets. The fix is record and replay: call the real model once, save the raw request-response pair to a cassette on disk, and replay it on every subsequent run.

@pytest.fixture
def agent(cassette):
    # First run with RECORD=1 hits the real API and writes tests/cassettes/*.json.
    # Every run after replays from disk: fast, free, byte-for-byte deterministic.
    return build_agent(transport=CassetteTransport(cassette, mode=record_mode()))

This is the same VCR pattern that HTTP-integration tests have used for years, applied to the model call. It gives the contract layer real determinism, not the fake determinism of temperature=0. Two rules keep cassettes honest. Re-record on a schedule (a nightly job with RECORD=1) so you notice when the model's behavior drifts, and treat a cassette diff in code review as a real change, because it is one. Record-replay is for the deterministic layer; the layers below deliberately hit the live model, because there the model's actual variability is the thing under test.

Layer 2: make the fuzzy parts statistical

For the parts that genuinely vary, stop asserting exact strings and start asserting properties, across repeated runs. Three shifts do most of the work.

Soft assertions. Instead of output == expected, assert that the output contains the required facts, matches a pattern, or satisfies a predicate. "Mentions the refund amount and the order ID" is testable; "says it exactly like this" is not.

def assert_mentions(text, *facts):
    missing = [f for f in facts if f.lower() not in text.lower()]
    assert not missing, f"reply omitted required facts: {missing}"

def test_refund_reply_states_the_facts():
    result = run(agent, "Refund A-1001", session=Session(known_order_ids={"A-1001"}))
    assert_mentions(result.text, "A-1001", "refund")

Run it more than once. A single pass of a non-deterministic test is a coin flip. Run the critical cases several times and gate on a pass rate:

def pass_rate(fn, trials=10):
    passed = sum(_safe(fn) for _ in range(trials))
    return passed / trials

def test_refund_flow_is_reliable():
    rate = pass_rate(lambda: check_refund_flow(agent), trials=20)
    assert rate >= 0.90, f"refund flow only reliable {rate:.0%} of the time"

Gate on rates, not booleans, and know your error bars. A pass rate is an estimate, and the estimate has noise. With 20 trials, an observed 90% has a 95% confidence interval of roughly ±13 points, so a build that reads 88% may not be a real regression. Two consequences follow. Run enough trials that the interval is smaller than the drop you care about (catching a 5-point regression reliably wants ~100 trials, not 20), and compare against a rolling baseline rather than a fixed number, so normal noise doesn't page anyone. In practice you reserve the high-trial treatment for the handful of cases that would actually wake someone up, and a case earns onto that list after it flakes in production, not before.

Layer 3: evals with a judge you can trust

When "good enough" is a judgment call, the assertion becomes an eval. I have written the foundations in Evals for AI Agents and the worked examples in Evals in Practice. Here we go deep on the one instrument that makes automated quality testing possible and the one that most often lies to you: the LLM judge.

The reason it works at all is that strong models grade open-ended output about as well as people. The foundational result found that GPT-4 judges reached "over 80% agreement, the same level of agreement between humans."¹ That is the license to automate a fuzzy assertion. But the same paper is blunt about the failure modes: position bias, verbosity bias, and self-enhancement bias, plus limited reasoning.¹ A judge is a rater with quirks, not an oracle. Treating its score as ground truth is the most common way an eval suite quietly rots.

Write the rubric like a spec, not a vibe. "Rate this reply 1-10" gives you noise. A judge needs the same precision you would give a human grader: explicit criteria, a bounded scale, and a requirement to cite evidence before scoring.

REFUND_RUBRIC = """
You are grading a support agent's reply to a refund request.
Score each criterion 0 or 1, then return JSON.

- correct_action: did it refund a valid order OR correctly escalate? (0/1)
- states_amount: does the reply state the refund amount? (0/1)
- no_overreach: did it avoid promising anything outside policy? (0/1)
- tone: is it professional and non-defensive? (0/1)

Return: {"reasoning": "...", "scores": {...}}  # reasoning FIRST, then scores.
"""

def judge(reply, rubric=REFUND_RUBRIC, model="a-different-model"):
    out = call_judge(model, rubric, reply)
    return json.loads(out)["scores"]

Two details in that snippet matter. Reasoning comes before scores, because a judge that must justify itself first scores more consistently. And the judge runs on a different model family than the agent, to blunt self-enhancement bias (models favor their own style). To fight position bias in pairwise comparisons, run each comparison twice with the order flipped and only count it if the verdict holds both ways.

Calibrate against a gold set, then keep calibrating. Before you trust a judge in a gate, hand-label 50 to 100 examples and measure the judge against your labels the same way the paper measured GPT-4 against humans. If the judge agrees with you less than ~80% of the time, the rubric is too vague or the task is too hard for the judge. Fix the rubric before you fix the agent. Keep that gold set in the repo as the test for the judge itself, and re-run it whenever you change the rubric or the judge model.

def test_judge_is_calibrated():
    agreement = mean(judge(ex.reply)["correct_action"] == ex.human_label
                     for ex in GOLD_SET)
    assert agreement >= 0.80, f"judge only agrees with humans {agreement:.0%}"

Now the quality assertion itself is just another test, with the judge as the scorer:

def test_refund_quality(agent):
    result = run(agent, "I need a refund for A-1001, it arrived broken",
                 session=Session(known_order_ids={"A-1001"}))
    scores = judge(result.text)
    assert scores["correct_action"] == 1
    assert scores["no_overreach"] == 1

Pointwise, pairwise, and when to use each

There are two ways to run a judge, and they are good at different jobs. Pointwise scoring grades one output against a rubric in isolation, which is what the examples above do. Pairwise scoring shows the judge two outputs and asks which is better. Pairwise is more reliable when you are choosing between options (this prompt versus that one, this model versus that one) because relative judgments are easier for a model to make consistently than absolute scores, which drift over time. Pointwise is the right tool for a regression gate, because a gate needs an absolute bar to fail against, not a comparison.

def pairwise_wins(a, b, rubric):
    # Run both orderings to cancel out position bias; only count a decisive result.
    first  = call_judge_pair(rubric, a, b)   # returns "A" or "B"
    second = call_judge_pair(rubric, b, a)
    if first == "A" and second == "B":  return "a"   # a won both orderings
    if first == "B" and second == "A":  return "b"
    return "tie"                                       # order-dependent -> untrusted

Use pairwise when you are deciding whether a change is an improvement, and pointwise when you are guarding against a regression. Most mature suites run both: pairwise to pick a new prompt, pointwise to make sure it never drops below the bar later.

Grounded answers: test for faithfulness

Not every LLM flow is an agent taking actions. The support agent also answers policy questions from a knowledge base ("what's your return window?"), and there the failure mode is not wrong tone, it is a confident hallucination: an answer that sounds right but is not supported by the retrieved docs. Faithfulness testing checks that every claim in the answer traces back to the provided context. It is the grounded-generation counterpart to trajectory testing, and it is a judge task done carefully: decompose the answer into atomic claims, then check each against the source.

def faithfulness(answer, retrieved_context):
    claims = split_into_claims(answer)                 # one LLM call, or a rule-based split
    supported = [c for c in claims if judge_supports(c, retrieved_context)]
    return len(supported) / max(len(claims), 1)        # 1.0 = fully grounded

def test_policy_answer_is_grounded():
    ctx = retrieve("return policy")
    result = run(agent, "What's your return window?", context=ctx)
    assert faithfulness(result.text, ctx) >= 0.95      # near-zero tolerance for invention

The threshold here is deliberately strict. A friendly, well-toned answer that invents a "60-day return window" the docs never mention is worse than a blunt correct one, and only a faithfulness check catches it.

When to skip the judge. An LLM judge is the right tool for open-ended quality (helpfulness, tone, faithfulness). It is the wrong tool for anything you can check deterministically. If the criterion is "did it call the right tool" or "is this valid JSON," that belongs in Layer 1, where it is free and exact. Reach for the judge only when the thing you are grading is genuinely a matter of degree.

Layer 4: simulate the whole journey

Single-turn tests catch shallow bugs. The expensive failures only show up across a conversation. As the Rhesis team puts it, agents "reveal their real failure modes in multi-turn journeys: they forget constraints, drift in tone, mishandle tool errors, or 'lock in' on the wrong plan and never recover."² You cannot catch constraint loss or goal drift with a one-shot prompt, because they are properties of the trajectory, not the turn.

The technique is simulation: a second LLM plays the user, driven by a persona with a goal and a hidden constraint, and converses with the agent over many turns. Then you score the whole session.

def simulate(agent, persona, max_turns=8) -> Session:
    session = Session()
    user = UserSimulator(persona)      # an LLM told to role-play the persona
    msg = user.opening_message()
    for _ in range(max_turns):
        result = run(agent, msg, session=session)
        session.record(user=msg, agent=result)
        if user.goal_met(result) or user.gave_up(result):
            break
        msg = user.reply_to(result)    # the simulated user reacts and continues
    return session

Personas are the test cases. Vary them by complexity, sentiment, and urgency,³ and give each a hidden wrinkle that only surfaces mid-conversation:

PERSONAS = [
    Persona(goal="refund order A-1001", tone="polite",
            twist="only reveals the order arrived damaged if asked why"),
    Persona(goal="refund an order that isn't theirs", tone="pushy",
            twist="supplies a real order ID belonging to someone else"),  # must refuse
    Persona(goal="refund ₹9,000 order", tone="calm",
            twist="accepts escalation gracefully"),                        # must escalate
]

That second persona is the one single-turn tests miss: the agent looks helpful for two turns, then leaks another customer's order because nothing in the earlier turns told it to be suspicious. Introduce the hard constraint late, three turns in, and see if the agent still honors it.

Score the session, not the last message. A journey has three things worth grading, and they map onto the trajectory evals covered in Evals in Practice:

def score_session(session, persona):
    return {
        "outcome":    persona.goal_met(session) == persona.should_succeed,  # right end state
        "trajectory": expected_tools(persona) <= called_tools(session),     # right path
        "recovery":   not session.repeated_failed_tool(),                   # recovered from errors
        "held_line":  session.honored(persona.hard_constraint),             # constraint survived
    }

Outcome asks whether it reached the right end state. Trajectory asks whether it took a sane path to get there. Recovery asks whether it got stuck in a loop when a tool failed. The last one, held_line, is the whole reason multi-turn testing exists.

Where test cases come from

A test suite is only as good as its cases, and for agents the cases have a natural source: production. Three streams feed the set.

• Replayed traffic. Pull real conversations from logs, strip the PII, and turn the interesting ones into fixtures. Anything a user actually did is a case worth keeping.
• Every incident becomes a test. When the agent does something wrong in production, the fix is not just a prompt tweak. It is a new case in the suite that fails before the fix and passes after. This is how the suite stops the same bug twice.
• Adversarial and edge cases, written by hand. The refund that is one rupee over the limit. The order ID that is valid but belongs to someone else. The user who changes their mind halfway. These rarely show up in logs until they hurt.

The anti-pattern to name here is the "spreadsheet trap": tracking agent test cases by hand in a growing sheet until it collapses under its own weight.² Cases are code. They live in the repo next to the harness, in version control, running in the pipeline, not in a tab someone updates from memory.

Wire it into CI

A test you run by hand before a demo is theater. A test that runs on every pull request is a control. The four layers have very different costs, so they run on different triggers:

Layer	Trigger	Speed	Gates the merge?
Contract (mocked)	Every push	Seconds	Yes, hard fail
Behavior (N trials)	Every push (small N)	~1 min	Yes, pass-rate
Quality (judge)	Merge to main + nightly	Minutes	Yes, threshold
Journey (simulation)	Nightly + pre-release	Minutes to hours	Warn, then block

The deterministic base runs on every push in under a minute because it is mocked, and it blocks the merge. The eval and simulation layers cost real tokens and take minutes, so they run on merge to main and nightly. promptfoo (now part of OpenAI) is built for the gate: it runs your eval suite as a CI step, emits JUnit XML that your CI reads natively, and fails the build on --fail-on-error or a pass-rate threshold.⁴ DeepEval takes the other route, dropping into an existing pytest job via deepeval test run; it models itself on pytest, "specialized for unit testing LLM apps," with metric scores from 0 to 1 passing or failing against a threshold.⁵ Either way the shape is the same:

# .github/workflows/agent-tests.yml (sketch)
on: [pull_request]
jobs:
  contract:                       # fast, mocked, blocks the merge
    steps: [ "pytest tests/contract -q" ]     # replays cassettes, no API key needed
  behavior:                       # small N, blocks on pass-rate
    steps: [ "pytest tests/behavior --trials 5" ]
# quality + journey run in a separate nightly workflow with real API keys

Here is the shape of a regression this catches. Someone edits the support agent's prompt to make it friendlier. Every deterministic test stays green, the demo looks fine, nothing errors. But the nightly eval set shows correct_action sliding from 94% to 88%, because the friendlier phrasing made the agent apologize and hedge instead of actually resolving the request. No exception was thrown anywhere. The gate caught it purely because the number moved, which is the only way a fuzzy regression ever gets caught.

One more tenant belongs in this same gate. Security red-teaming is testing with an adversarial scorer, and as I argued in Security Practices and Tools in the Age of LLMs, those tests belong in CI right next to your quality evals. A prompt-injection attempt that flips the agent into refunding freely is just a journey test with a hostile persona.

Failure modes to avoid

The suite itself can rot in predictable ways. Watch for these:

• Trusting an uncalibrated judge. A judge you never checked against human labels is a random number generator with good grammar. Calibrate first, re-calibrate on every rubric change.
• Over-mocking. If you mock the model in the eval and journey layers too, you are testing your cassettes, not your agent. Mock the deterministic layer, and let the fuzzy layers hit the live model.
• Green-or-red thinking on fuzzy tests. A single failed run of a non-deterministic test is noise. If your CI fails the build on one flaky run, people will disable the test. Gate on rates.
• Testing the model, not the app. Public benchmarks measure the model. Your suite exists to measure your system: your tools, your prompts, your guardrails, your data.
• A suite that only has the middle. Evals without a deterministic base let schema bugs through. Evals without simulation miss every multi-turn failure. Build all four layers.

What actually holds up

The tools will churn, but the shape holds. Test the deterministic parts like software, because they are software. Make the model calls hermetic so the base is fast and real. Wrap the fuzzy core in statistical gates with honest error bars. Grade quality with a judge you have calibrated against humans and keep calibrating. Simulate the whole journey for the failures that only appear over time. Then gate all of it in CI on rates, not booleans, and feed every incident back in as a new case.

The mistake is treating "non-deterministic" as "untestable" and giving up. Almost everything about an agent is testable. Only the last mile is fuzzy, and even that is measurable once you stop demanding exact answers and start demanding good-enough ones, reliably, over a full conversation. The benchmark the whole field points at, SWE-bench, is really just this idea at scale: put an agent in a sandbox, give it a real task, and check whether the outcome is correct rather than whether the words match. Bring that same mental model to your own suite. Stop testing what the agent said. Test what it did, how often it got it right, and whether it held the line.

---

Related: Evals for AI Agents · Evals in Practice · Security Practices and Tools in the Age of LLMs

Footnotes

1. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv:2306.05685, 2023 (link). Establishes that strong LLM judges reach over 80% agreement with humans, and names the key biases (position, verbosity, self-enhancement). ↩ ↩²

2. Rhesis AI, A PM's guide to testing AI agents, February 2026 (link). Source for the "spreadsheet trap" and the argument that agents' real failures (constraint loss, goal drift) surface in multi-turn journeys. ↩ ↩²

3. Maxim AI, Building robust evaluation workflows for AI agents, April 2025 (link). Source for persona-based simulation and session/trajectory-level metrics. ↩

4. promptfoo, CI/CD integration (link). Open-source eval/red-team runner (now part of OpenAI); runs as a CI quality gate with JUnit output and --fail-on-error or pass-rate thresholds. ↩

5. Confident AI, DeepEval (link). Open-source, pytest-style unit-testing framework for LLM apps; metric scores 0-1 pass/fail against a threshold and run via deepeval test run. ↩