The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Plzebo listing page.
Cost-parity bench for agent loops. Point it at two or more ways of getting the same task done (a cheaper model mix, a different setup, a tool on vs off) and it measures which one delivers the outcome for less, in dollars and in minutes, and whether the gap is real or noise. Runs locally, on your own keys or your own subscription. An agent can call it on itself over MCP.
The headline metric is cost per successful outcome: total metered spend divided by successes, compared across arms with a 95% confidence interval. Time per successful outcome is reported beside it. When the data cannot support a claim, plzebo says so instead of making it.
claude-* call to each arm's model.sk-placebo-...) and a redirected base URL; the gateway swaps in the real key upstream and records provider-reported token usage per call. A call that bypasses the gateway fails at the provider instead of billing invisibly, and a trial with zero metered calls is flagged, never compared.plzebo verify receipt.json recomputes every derived number from the raw rows and reports any divergence.Node 22.6 or later (the CLI runs its TypeScript sources through Node type stripping; the launcher supplies the flag on versions that need it). Git, plus whatever your arms and oracle need (for the examples here: python3).
A spec:
Every run prints its cost estimate before the first trial. A run without a budget is refused. Start with a small K: the report on an inconclusive run prints the exact K that would resolve it and the estimated cost, and plzebo extend <receipt.json> --to-k K reuses the trials you already paid for. A receipt can be extended once, and the extension is disclosed inside it.
auth: placebo (default): you export ANTHROPIC_API_KEY or OPENAI_API_KEY; arms only ever see placebo keys. Strongest metering guarantee: a bypassing call fails loudly.auth: passthrough: the arm uses its own login. No API credits are spent; USD is imputed from metered tokens at list rates and the receipt discloses this. Tokens are the ground truth either way. Verified for Claude Code, which honors ANTHROPIC_BASE_URL under subscription auth. Codex under ChatGPT auth routes to its own backend and ignores OPENAI_BASE_URL, so subscription-authed Codex is unmeterable; run Codex arms in API-key mode via the codex-cli template instead.Two caveats for passthrough arms. A trial that bypasses the meter is excluded rather than falsely counted, but the tokens it consumed upstream are real; the receipt cannot see them. And a personal login carries personal agent configuration: instructions, hooks, and memory files travel with it, which can inflate costs and, for agents that act on standing instructions, cause writes outside the trial workspace. Prefer isolated-config templates when comparing loops rather than your own setup.
Every cost node in a receipt records two models: role, the model the loop REQUESTED, and model, what the arm's rewrite rules resolved it to. A loop that requests different models for different jobs (a main session on one model, subagents on another) therefore encodes its role structure into the receipt, and arms that share the same command can bind those roles differently through rewrites alone:
The packaged scripts/arms/subagent-mix.sh shows the Claude Code shape: a headless session with an inline --agents definition whose worker requests a different model than the main session. Adapt the agent prompt and models to your task.
A model-mix comparison is only valid when the loop actually delegated: a trial whose cost tree has no worker-role calls measured a solo run, not a mix.
fidelity exits 1 if any completed trial never made a worker-role call, and prints each trial's call counts, tokens, and imputed spend grouped by requested and resolved model, so the mix an arm claims is checkable from its receipt.
The gateway speaks two wire formats: Anthropic and OpenAI-compatible, which covers most open-weight serving (Ollama, vLLM, OpenRouter, Together, and similar). Arms can override providers individually:
Constraint: plzebo can only meter loops that authenticate with a key or token and honor a redirectable base URL (ANTHROPIC_BASE_URL / OPENAI_BASE_URL).
npx plzebo-mcp serves four tools on stdio: measure_start, measure_extend, measure_status, and measure_result. Starting a measurement is two-phase: the first call validates the request and returns a cost estimate plus a one-time confirm token while spending nothing; only a second call presenting that token runs trials. A budget is mandatory, and MCP arms come only from named templates (claude-cli, claude-subscription, codex-cli), never raw shell. Job state and receipts persist under .plzebo/, so verdicts survive the session that produced them.
A receipt is a single-task result under a stated model mapping. It is not a model capability ranking and does not generalize beyond its fixture; the estimand text inside the receipt says exactly this. USD figures on passthrough arms are imputed at pricing-table rates from metered tokens, not billed spend, and the pricing table version travels in the receipt so the numbers stay recomputable after prices change.
MIT