Truthroute

What truthroute MCP server does

The truthroute MCP server compares answers from multiple LLM providers to the same prompt. It supports OpenAI, Anthropic, and Gemini, with the provider list supplied by the caller. Results include the provider responses, completion status, and a divergence score from 0.0 to 1.0. Lower values indicate more similar answers; higher values indicate greater semantic difference.

This is a comparison primitive rather than a fact-checker, model router, or consensus system. It reports how much responses differ, not which response is correct. The implementation is stateless: it does not provide accounts, persistent storage, debate workflows, failover, or cost optimization.

How it works

The truthroute MCP server calls the requested providers in parallel, using temperature=0 for each provider request. It strips markdown and other formatting before scoring so presentation and verbosity have less influence. Refusals are removed from the score because a refusal should not be treated as substantive disagreement with an answer.

Scoring happens locally with the fastembed package and the BGESmallENV15 sentence-embedding model. The divergence value is calculated as one minus the average pairwise cosine similarity across usable responses. The embedding choice was checked against a hand-labeled validation set containing agreement, paraphrase, negation, and disagreement examples.

A comparison is complete when all requested providers succeed. It can be partial when at least two usable responses remain but a provider fails or a refusal is excluded. With fewer than two usable responses, the result is failed and no divergence score is reported. Repeated runs can produce a confidence band with --repeats; deterministic settings reduce variation but do not guarantee identical vendor responses.

Setup and configuration

Install the CLI globally with npm install -g truthroute-cli, or invoke it through npx. The MCP interface is started with truthroute mcp, which runs the server over standard input and output. An MCP client can then call the typed compare tool directly.

Set API keys as environment variables for the providers you intend to use:

• OPENAI_API_KEY
• ANTHROPIC_API_KEY
• GEMINI_API_KEY

Only requested providers need configured keys. Every real comparison sends billed requests to those vendor APIs. Use the CLI's --dry-run option to estimate the call count and cost before making provider requests. The --json option is intended for scripts and returns structured result data; it is separate from MCP protocol discovery.

Tools and capabilities

The MCP surface exposes one typed tool:

• compare: sends a prompt to a selected comma-separated provider list and returns responses, statuses, and divergence data.

The corresponding CLI supports human-readable output, structured JSON, dry-run estimates, and repeated comparisons with a confidence band. The JSON result can identify failed providers, responses excluded because of refusal, the overall status, and the providers included in the calculation.

Limitations and notes

Prompts are transmitted to every provider selected for a comparison, so sensitive input receives the same vendor exposure as direct API calls. TruthRoute has no hosted service or free request tier; users provide their own provider credentials and pay vendors directly.

The score measures semantic response difference, not factual accuracy. Cosine-based values may occupy a narrower practical range than an intuitive 0-to-1 interpretation suggests, so relative comparisons are more useful than treating a single number as an absolute truth measure. Vendor inference differences can also cause small run-to-run changes even with temperature set to zero.