Monitor LLM endpoints with synthetic probes for TTFT, latency, throughput, errors, and deployment regressions.
Copy the AI prompt to install this server into Claude Code, Cursor, or another agent β or use 1-click editor setup below.
One-click editor setup isnβt available for this listing yet β we donβt have a confirmed install command, and weβd rather show nothing than point your editor at the wrong package or host. Follow the projectβs own setup instructions, linked above.
Inspect callable tools, capabilities, and parameters exposed to AI agents by Llmprobe.
![]()
Synthetic monitoring and CI smoke tests for LLM inference endpoints. Measure TTFT, latency, throughput, and errors. Single binary, zero SDKs.
llmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or OpenAI-compatible inference servers, then reports the metrics that matter for production user experience: time to first token (TTFT), total latency, generation throughput (tokens/sec), and error rates.
Use it as a one-off health check, a continuous monitor, or a CI gate that blocks deploys when your LLM provider is degraded.

llm-bench uses llmprobe to run a continuous public benchmark of major LLM APIs. It publishes a live dashboard at bench.jonathanwrede.de and raw JSONL data in Jwrede/llm-bench-data.
This is the intended use case: repeated synthetic probes that make LLM latency, TTFT regressions, throughput drops, and provider degradation visible before users report them.
Download a prebuilt binary from the latest release (Linux, macOS, Windows; amd64 and arm64).
Or install from source:
Install as a Claude Code plugin for /llmprobe skill and MCP tools:
Or register the MCP server directly:
llmprobe runs locally and only contacts LLM endpoints you configure. See PRIVACY.md for details.
llmprobe works with OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI, Fireworks, DeepSeek, and Mistral.
Create a probes.yml (or copy the included example):
Run a probe:
| Metric | What it means |
|---|---|
| TTFT | Time from request send to first content token. This is what users feel as "lag" before the response starts streaming. |
| Latency | Total time from request to stream close. |
| Tok/s | Generation throughput: tokens produced per second after the first token. Calculated as token_count / (latency - ttft). |
| Tokens | Total output tokens. Prefers provider usage metadata when available, falls back to SSE event counting. |
| Status | healthy if all thresholds pass, degraded if any threshold is exceeded, error if the request failed. |
llmprobe probeOne-off health check. Probes all configured endpoints and prints results.
Exit codes for CI:
--fail-on | Exit 0 | Exit 1 |
|---|---|---|
error (default) | healthy or degraded | any error |
degraded | healthy only | degraded or error |
none | always | never |
llmprobe watchContinuous monitoring. Probes all endpoints on an interval and prints a summary line per iteration.
The --tui flag launches a live terminal dashboard with a TTFT chart,
color legend, and statistics table. Use --load to import historical
JSONL data (from llmprobe watch -f json > data.jsonl).

llmprobe reportGenerate a Markdown summary from JSONL probe data with p50/p95/p99 percentiles for TTFT, latency, and throughput per endpoint.
Output:
llmprobe baselineCreate a baseline file from historical JSONL data for regression detection.
Reference the baseline in your config to use multiplier-based thresholds:
This lets you detect regressions relative to your own historical data rather than setting absolute thresholds.
llmprobe versionPrint the installed binary version.
Use llmprobe probe as a pre-deploy gate:
This blocks the deploy if any LLM provider is experiencing degraded performance right now.
When a probe fails, the output shows only the failing endpoints:
llmprobe includes a built-in Model Context Protocol server, allowing Claude Code and other MCP hosts to check LLM API health directly from an agent workflow.
This starts the MCP server over stdio.
Once registered, Claude Code can call llmprobe tools during any conversation.
| Tool | Description |
|---|---|
probe_all | Probe all configured endpoints from probes.yml. Returns TTFT, latency, throughput, and health status for every model. Accepts an optional config parameter for a custom config path. |
probe_model | Probe a single model without a config file. Requires provider, model, and api_key_env. Supports optional base_url for OpenAI-compatible endpoints and optional label for display. |
list_providers | List all providers and models in the config file with their thresholds. Use this to discover available models before probing. |
get_config | Return the full parsed configuration including defaults, providers, models, and thresholds. |
Example use case: An agent calls list_providers to see what models
are configured, then probe_all to verify they are healthy before
deploying changes.
Factual signals from GitHub, npm, and our automated checks β not a rating.
No reviews yet β be the first to share how this listing worked for you.
Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.
[](https://allmcps.com/mcp/jwrede-llmprobe)<a href="https://allmcps.com/mcp/jwrede-llmprobe"><img src="https://allmcps.com/api/badge/jwrede-llmprobe?style=directory" alt="Llmprobe on AllMCPs" /></a>