Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.
Quick Install
Automated & IDE SetupCopy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.
Manual Client & Custom JSON ConfigExpand JSON ▾
Client Config & Setup
Choose your client or environment
Target File:
💡 Paste the JSON block into your client's configuration file under mcpServers, then restart the application.
Documentation Overview
LLMTest MCP Server
MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.
Quick Start
1. Get your API key
Sign up at llmtest.io and grab your API key from the dashboard.
2. Add to your tool
Claude Code:
Then set your key:
Cursor / Windsurf / Other MCP clients:
Add to your MCP config file:
3. Talk to your AI
Just ask in natural language:
- "Check my LLMTest status"
- "Find cheaper models for my AI calls"
- "Run a benchmark on my blog-writer flow"
- "What models are trending?"
How It Works
LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.
This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.
Available Tools
| Tool | Description |
|---|---|
status | Show proxy status and activity summary |
list_flows | List all AI flows with cost and latency stats |
get_suggestions | Get pending model-switch recommendations |
update_suggestion | Accept or dismiss a suggestion |
run_benchmark | Benchmark a flow against challenger models |
optimize_prompt | Rewrite a flow's prompt and find a cheaper model that still works |
seed_samples | Add test prompts for pre-launch benchmarking |
list_samples | Show stored test samples per flow |
list_new_models | Show new and trending models |
get_account | Check credit balance and usage |
get_autopilot_status | Check whether autopilot is on and whether the account is eligible |
enable_autopilot | Turn on weekly auto-optimization with safety gates + drift-based auto-revert |
disable_autopilot | Turn off autopilot (existing optimizations stay active) |
list_active_optimizations | List auto-accepted optimizations still inside their 24h revert window |
revert_optimization | Roll an auto-accepted optimization back to the previous prompt |
Autopilot
Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.
To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enable_autopilot. Use get_autopilot_status to confirm prerequisites.
Prerequisites (checked per flow each cycle):
- Autopilot enabled on the account
- Email verified
- Account age ≥ 14 days (trust ramp)
- Flow has ≥ 20 real calls in the last 7 days
- Flow not optimized by autopilot in the last 14 days (cooldown)
- Positive credit balance (~$1–2 per run)
Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.
Revert: 24h window after auto-accept. After that, only drift detection can roll back.
Typical Workflow
Pre-launch (no traffic yet):
- Tell your AI: "I'm building a support chatbot using gpt-4o"
- It seeds realistic test samples with
seed_samples - It runs
run_benchmarkto compare models - It shows you
get_suggestionswith cheaper alternatives
Post-launch (with real traffic):
- Route your AI calls through
https://llmtest.io/v1 - LLMTest monitors usage and auto-benchmarks when flows hit 50+ calls
- Ask "any cost-saving suggestions?" to see recommendations
- Accept a suggestion and update your code
Environment Variables
| Variable | Required | Description |
|---|---|---|
LLMTEST_API_KEY | Yes | Your API key from llmtest.io/dashboard |
LLMTEST_BASE_URL | No | Custom API URL (defaults to https://llmtest.io) |
Links
License
MIT
Related MCP Servers
Reviews
No reviews yet — be the first to share how this listing worked for you.
Frequently Asked Questions about Llmtest MCP
AllMCPs Directory Badge
Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.
Badge Style:
Live Dynamic SVG Preview
Markdown (GitHub README)
[](https://allmcps.com/mcp/llmtest-mcp)HTML Embed
<a href="https://allmcps.com/mcp/llmtest-mcp"><img src="https://allmcps.com/api/badge/llmtest-mcp?style=directory" alt="Llmtest MCP on AllMCPs" /></a>Technical Specs & Signals
CategoryDeveloper Tools
27Quality signal: Emerging · 27/100How this signal is calculated ▾
Server availabilityNot measured
Not scored for repo-hosted servers — we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.
Verified ownership8/20
Documentation & tools11/30
Adoption & activity1/15
Community engagement0/10
A guidance signal from public completeness & health data — not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.
Own this project?
This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to unlock edit access and the Official badge — proof is checked automatically, then reviewed by our team.
Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.
Claim & get free dofollowShare & Embed
Add our SVG badge (dark/light directory styles) or embeddable widget to your site.