Agent Evaluation

Statistical regression testing for LLM agents: run version A and version B 50x each and get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

pip install agent-regress-cli

(uv add and the npm/npx wrapper are covered in Install below.)

The test that every eval framework skips

You changed a prompt. Your evals still pass. But your agent's tool accuracy dropped from 84% to 70%.

Is that a real regression? Or is it LLM run-to-run noise?

Threshold testing cannot answer that question. agent-eval can.

Run your agent 50x on version A, 50x on version B. Get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

p=0.003, Cohen's d=-0.61 -> REGRESSED (deploy blocked)
p=0.410, Cohen's d=0.021 -> STABLE (safe to ship)

[!NOTE] Promptfoo, one of the most widely used open-source LLM eval frameworks, was acquired by OpenAI in March 2026, staying open source but folding its team into OpenAI's Frontier platform. agent-eval is Apache 2.0-licensed, self-hostable, and has no commercial dependency. The statistical core (Mann-Whitney U, bootstrap CI, Cohen's d) will never be paywalled.

Install

pip install agent-regress-cli
# or
uv add agent-regress-cli
# or, from Node/npx (thin wrapper around the same Python CLI)
npx agent-regress-cli

Quickstart

In 30 seconds (CLI)

Already have per-run scores from your own harness? Point the CLI at two JSON arrays of scores, one per version:

pip install agent-regress-cli

agent-regress compare \
  --version-a-results v1_scores.json \
  --version-b-results v2_scores.json \
  --metric tool_accuracy

# ============================================================
# agent-regress Report -- tool_accuracy
# ============================================================
# Verdict:    REGRESSED
# p-value:    0.0000
# Cohen's d:  -2.193
# 95% CI:     [-0.213, -0.148]
#
# Version A:  0.8470 +/- 0.0525  (n=50)
# Version B:  0.6685 +/- 0.1025  (n=50)
# Delta:      -0.1786
# ============================================================

Add --json --fail-on-regression to get clean, parseable output and a non-zero exit code on REGRESSED, for wiring straight into CI.

All agent-regress compare flags:

The top-level agent-regress --version flag prints the installed version and exits.

Exit codes:

In your code (Python API)

Driving the agent yourself instead of pre-computing scores? Use the Python API:

from agent_regress import compare

# Any callable that takes a test case dict and returns a score 0.0-1.0
def agent_v1(test_case: dict) -> float:
    ...  # your existing agent

def agent_v2(test_case: dict) -> float:
    ...  # your updated agent

test_suite = [
    {"query": "find SKU for order 8823", "expected": "SKU-4492"},
    # ... more test cases
]

report = compare(
    version_a=agent_v1,
    version_b=agent_v2,
    test_suite=test_suite,
    n_runs=50,
    metric="tool_accuracy",  # use any name except "accuracy" when agents return floats
)

print(report)           # structured output with p-value, CI, effect size
report.assert_stable()  # raises AssertionError if behavior regressed

Agent returns text? Pass a scorer or use the built-ins:

from agent_regress import compare, exact_match_scorer, f1_scorer

# exact_match_scorer: 1.0 if str(output).strip() == str(expected).strip()
# f1_scorer: token-level F1 (multiset — handles repeated tokens correctly)
report = compare(
    version_a=agent_v1,
    version_b=agent_v2,
    test_suite=test_suite,
    n_runs=50,
    scorer=exact_match_scorer,  # test_case must have an "expected" key
)

Or write your own:

def my_scorer(output: str, test_case: dict) -> float:
    return 1.0 if output.strip() == test_case["expected"] else 0.0

report = compare(..., scorer=my_scorer)

MCP Server

agent-eval ships a Model Context Protocol server so an AI agent (Claude, Cursor, or any MCP-compatible client) can run statistical regression tests directly, without a human invoking the CLI by hand.

Install the extra:

pip install "agent-regress-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop, claude_desktop_config.json):

{
  "mcpServers": {
    "agent-eval": {
      "command": "uvx",
      "args": ["--from", "agent-regress-cli", "agent-regress-mcp"]
    }
  }
}

The server exposes one tool, run, that shells out to the agent-regress CLI with the given subcommand and arguments plus --json, and returns the parsed JSON result:

run(["compare", "--version-a-results", "a.json", "--version-b-results", "b.json", "--metric", "accuracy"])

Transport is stdio, so there is nothing to host: the MCP client spawns the server as a local subprocess. Source: src/agent_regress/mcp_server.py.

Why not DeepEval, Promptfoo, or Braintrust?

†Promptfoo acquired by OpenAI, March 2026; remains open source under its current license.

DeepEval tests whether an individual agent response clears a quality bar. Agent Evaluation tests whether behavior changed significantly between two agent versions, a different statistical question that threshold testing cannot answer. The scipy Mann-Whitney U call at the core is one line, so any SaaS eval platform can add it. What accumulates over time through production use is version-specific regression history and a community-maintained benchmark leaderboard with independent result verification.

Real regressions statistical testing catches that threshold testing misses