The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Ami Survey listing page.
Measures what a completed agent workflow cost to run, by reading the runtime's
own session log, and submits the result to the AMI survey at
survey.agentbenchmark.dev.
The cost of a single API call is easy to obtain. The cost of one finished piece of work is not: one triaged ticket, one screened CV, one drafted reply, across every call, retry and tool round-trip the agent made getting there. This measures that figure.
Every number is read from the runtime's records rather than reported by the agent. An agent asked how many tokens it has just used will estimate, and will present the estimate with confidence.
A real run of six support tickets, triaged and answered by Claude Opus 5 in Claude Code:
$0.12 per ticket, 21 seconds per ticket. That is the figure this exists to produce, and it is the one most teams cannot currently state about their own work.
The findings record where the scorecard's own numbers are soft. A cost reference that is still a placeholder is reported as a placeholder rather than folded into the score.
There are two routes. What separates them, and what separates the hosts below,
is whether anything can read your runtime's logs. That determines whether the
result is recorded as measured or unmeasured.
One block of configuration, then restart the agent.
Claude Code, in ~/.claude.json, or through the Developer settings in
Claude Desktop:
Runs are measured: the client reads Claude Code's own session log, so the token counts and the cost come from the runtime rather than from the agent.
In ~/.codex/config.toml. Codex uses TOML here, not JSON:
The client reads Codex's rollout log. This also covers a local or self-hosted model driven through Codex, because the log is written either way.
In .vscode/mcp.json, or your user profile through the MCP: Open User
Configuration command. The key is servers here, not mcpServers:
The tools work and the survey submits, but Copilot writes no session log the
client can read. The run carries the workflow, the grade and the stage timings
without token counts or cost, and is recorded as unmeasured.
Then, once the agent finishes a piece of work, ask it:
Take the AMI survey regarding the ticket triage you just did
There is nothing to clone and nothing to keep updated. No token needs to be
supplied: the first call that requires one registers the machine and stores the
token at ~/.ami-survey/token.
uvx is part of uv, the tool the MCP
documentation uses for Python servers. If uv is not installed,
GETTING-STARTED.md
gives a pipx form and a route that requires neither.
If you already hold a token, set it as AMI_API_TOKEN in that block's env and
it will be used instead of registering a new one.
In claude.ai, open Settings, then Connectors, then Add custom connector, and supply:
Nothing is installed and no token is required. These runs are recorded as
unmeasured and are never compared against measured ones: a remote server
cannot read your runtime's logs, so token counts and cost are absent rather than
estimated.
MIT. Use it, modify it, redistribute it, include it in your own work.
The licence covers this client only. It grants no rights in the survey service the client submits to, or in the data that service collects.
Full terms in LICENSE.
Token counts, timings, model names, the stage names the workflow declared, and the grade. Not your files, not your prompts, not your shell commands. GETTING-STARTED.md sets this out in full.
Submissions go to survey.agentbenchmark.dev and nowhere else. That destination
is a constant in the source rather than a setting, so a stale environment
variable cannot redirect a submission onto your own disk. That is the one
failure which would make a run appear successful while collecting nothing.
Python 3.9 or newer. No dependencies; the standard library only.