Tokentrust

TokenTrust

Install • What it measures • Commands • Agent-native / MCP • FAQ

Vendor-neutral CLI that independently verifies the token and cost savings AI-coding-agent context-reduction proxies actually deliver, by running the proxy for real against a labeled task corpus instead of trusting the maintainer's own number.

Install

TokenTrust ships as two complementary, equally first-class distributions: an npm package for Node.js toolchains and a PyPI package for Python toolchains. Both install a tokentrust command with the identical CLI surface, the same TT01-TT05 verification categories, and the same bundled task corpus, so pick whichever matches your existing stack.

npm (Node.js):

npx tokentrust-cli verify --proxy rtk

No clone, no local build. npx fetches the published package and runs it directly. To install it as a dependency instead: npm install -g tokentrust-cli.

pip (Python):

pip install tokentrust-cli
tokentrust verify --proxy rtk

See python/README.md for the Python package's full documentation, including a note on the one real behavioral difference between the two: the npm package's js-tiktoken dependency bundles its tokenizer data for fully offline use, while the Python package's tiktoken dependency fetches and caches that same public data on first use.

Real output from that exact command, run against this repo's own bundled task corpus:

$ npx tokentrust-cli verify --proxy rtk

TokenTrust v0.1 -- Token/Context-Reduction Claims Verification
Proxy: rtk 0.43.0 | Repo: TokenTrust | Task corpus: 23 labeled tasks

[MEASURED] TT01 Compression Ratio
  Claimed (rtk README): up to 70% context reduction
  Measured (this repo, this corpus): 60.6% average reduction across 23 tasks
  Range: 0.0% ("verify-go-build-filter") to 95.4% ("verify-git-log-filter")

[MEASURED] TT02 Cost-Savings Delta
  Baseline (uncompressed): $0.02 across 23 tasks @ claude-5-sonnet pricing
  Compressed (rtk-proxied): $0.00 across 23 tasks
  Actual savings: 77.0% ($0.01) -- vs. claimed 70% ceiling

[FAIL]  TT03 Never-Worse Output Guard
  2/23 tasks regressed in task-completion diff vs. uncompressed baseline

[PASS]  TT05 Version-Drift Regression Check
  No prior verified baseline for rtk on this repo -- this run establishes the first baseline.

Summary: 77.0% measured cost savings (claimed: up to 70%) -- see full report

That's a real run's output, not a hand-typed example -- npx tokentrust-cli invokes the exact same dist/cli.js entry point (the tokentrust command name is unchanged), so it reproduces on your machine with no clone required.

What it measures

• TT01: Compression Ratio. Actual token reduction, measured with a local tokenizer (js-tiktoken), against every task in the corpus.
• TT02: Cost-Savings Delta. Dollar-cost savings computed from TT01's measured token delta at published model pricing. Optional --live mode verifies the estimate against a real, provider-billed sample (opt-in, your own API key, gated behind --confirm-cost, capped at 5 tasks by default).
• TT03: Never-Worse Output Guard. Checks whether a proxy's compressed output dropped content a task marks as required to survive compression.
• TT04: Cross-Tool Comparative Benchmark. Pass --proxy more than once and TokenTrust runs the identical task corpus through every named proxy side by side.
• TT05: Version-Drift Regression Detection. Compares a run's measured savings against the last-verified baseline for the same proxy/repo pair, so a silent regression across a version bump (like rtk#582) gets caught automatically.

Commands

tokentrust verify --proxy <name> [options]

This table (and the tokentrust mcp reference below) is verified against the actual --help output of the published tokentrust-cli package, not an old copy. One known gap between that live --help text and the table above is called out directly in the FAQ, instead of silently repeating it.

--format json gives every category's claimed-vs-measured numbers as structured data, so a script or agent can pull the comparison straight out with jq instead of parsing terminal text:

Exit code is 0 when the run completes with no gated failure, non-zero otherwise. The bundled GitHub Action's --fail-on-regression maps that straight to a failed CI step, so a version-drift regression breaks the build instead of shipping silently.

Add it to CI with the bundled GitHub Action (action/action.yml) so verification reruns automatically whenever a proxy's version bumps:

- uses: RudrenduPaul/TokenTrust-CLI@main
  with:
    proxy: rtk
    fail-on-regression: 'true'
    cli-version: '<pin to the exact tokentrust-cli version you have verified against>'

Pin cli-version explicitly rather than relying on the Action's default. The Action deliberately does not support latest for this input: pinning is a supply-chain safeguard, so a compromised npm publish can't reach every workflow using this Action on its very next run without a deliberate version bump on your part. The Action's own default for this input is an early pre-TT04/TT05/MCP release -- omitting cli-version runs that old version, not the one this README documents.

Agent-native / MCP

TokenTrust ships in the same dual CLI + MCP-server mode as Semgrep, Trivy, Snyk, and SonarQube: one binary, one underlying verification engine, and a second, thin front door for agents that speak MCP (Model Context Protocol) instead of a shell. tokentrust mcp starts an MCP server over stdio, exposing a single tool, verify_proxy_savings, that calls straight into the same runVerify() engine tokentrust verify uses -- no verification logic is duplicated, and the tool returns the exact structured JSON report --format json already produces.

npx tokentrust-cli mcp

Register it with an MCP client

Point the client's server config at this binary with the mcp argument. For Claude Code, Claude Desktop, or any other client that reads an mcpServers block:

{
  "mcpServers": {
    "tokentrust": {
      "command": "npx",
      "args": ["tokentrust-cli", "mcp"]
    }
  }
}

The tool

This is the tool's real, unedited tools/list schema, captured from a running tokentrust mcp server (inputSchema trimmed of per-field descriptions here for length; the live server returns them in full):

{
  "name": "verify_proxy_savings",
  "title": "Verify proxy token/cost savings",
  "inputSchema": {
    "type": "object",
    "properties": {
      "proxy": { "anyOf": [{ "type": "string", "enum": ["rtk", "headroom"] }, { "type": "array", "items": { "type": "string", "enum": ["rtk", "headroom"] }, "minItems": 1 }] },
      "repo": { "type": "string" },
      "tasks": { "type": "string" },
      "live": { "type": "boolean" },
      "confirmCost": { "type": "boolean" },
      "liveMaxTasks": { "type": "integer", "exclusiveMinimum": 0 }
    },
    "required": ["proxy"]
  }
}