Skip to main content
AllMCPs
BrowseBestCategoriesStackCompareToolsGuidesBlog
Log in Submit MCP

Stay in the loop

Get new MCP servers and top picks in your inbox.

AllMCPs

The open directory for discovering and installing Model Context Protocol servers.

AllMCPs on GitHub (opens in a new tab)
Launched onTiny Startupstinystartups.com
Explore
  • Browse servers
  • Best MCP servers
  • Categories
  • MCP clients
  • Agent prompts
  • Stack Builder
  • Compare servers
  • Random discovery New
  • Submit a server
  • Pricing & Boost Boost
Learn
  • Guides hub
  • What is MCP?
  • Install guide
  • Build an MCP server
  • Deploy an MCP server
  • Security guide
  • Troubleshooting
  • MCP for SEO & AEO
  • Protocol versioning
  • Blog & updates
Tools
  • All developer tools
  • Config generator
  • Config validator
  • Config auditor
  • MCP playground
  • Token calculator
  • OpenAPI β†’ MCP
  • Badge generator
For agents
  • REST API docs
  • Trust & traffic Live
  • Remote MCP server SSE β†— (opens in a new tab)
  • llms.txt β†— (opens in a new tab)
  • Catalog JSON β†— (opens in a new tab)
Company
  • About
  • Advertise Sponsor
  • Contact
  • GitHub β†— (opens in a new tab)
  • Terms
  • Privacy
AllMCPs VerifiedAllMCPs VerifiedFeatured on Nick LaunchesFeatured on Nick LaunchesLaunch Llama NewsletterLaunch Llama NewsletterVerified DR - allmcps.comVerified DR - allmcps.comFeatured on SaaSGrowFeatured on SaaSGrowFeatured on Twelve ToolsFeatured on Twelve ToolsFeatured on Saaspa.geFeatured on Saaspa.geFeatured on Findly.toolsFeatured on Findly.toolsFeatured on Startup FameFeatured on Startup FameFeatured on LaunchKiwiFeatured on LaunchKiwiFeatured on ScrollLaunchFeatured on ScrollLaunchFeatured on DailyPingsFeatured on DailyPingsFazier badgeFazier badgeFeatured on NewTool.siteFeatured on NewTool.siteFeatured on saasfame.comFeatured on saasfame.comDR Checker - Domain RatingDR Checker - Domain RatingListed on Turbo0Listed on Turbo0Launched on LaunchBoard - Product Launch PlatformLaunched on LaunchBoard - Product Launch PlatformList on SimilarlabsList on Similarlabshttps://codetrendy.comhttps://codetrendy.comListed on DevTool.ioFeatured on BuildlistFeatured on BuildlistLaunched on Tiny StartupsFeatured on ShowMeBestAIFeatured on ShowMeBestAIFind us on LaunchZoneFind us on LaunchZoneAllMCPs VerifiedAllMCPs VerifiedFeatured on Nick LaunchesFeatured on Nick LaunchesLaunch Llama NewsletterLaunch Llama NewsletterVerified DR - allmcps.comVerified DR - allmcps.comFeatured on SaaSGrowFeatured on SaaSGrowFeatured on Twelve ToolsFeatured on Twelve ToolsFeatured on Saaspa.geFeatured on Saaspa.geFeatured on Findly.toolsFeatured on Findly.toolsFeatured on Startup FameFeatured on Startup FameFeatured on LaunchKiwiFeatured on LaunchKiwiFeatured on ScrollLaunchFeatured on ScrollLaunchFeatured on DailyPingsFeatured on DailyPingsFazier badgeFazier badgeFeatured on NewTool.siteFeatured on NewTool.siteFeatured on saasfame.comFeatured on saasfame.comDR Checker - Domain RatingDR Checker - Domain RatingListed on Turbo0Listed on Turbo0Launched on LaunchBoard - Product Launch PlatformLaunched on LaunchBoard - Product Launch PlatformList on SimilarlabsList on Similarlabshttps://codetrendy.comhttps://codetrendy.comListed on DevTool.ioFeatured on BuildlistFeatured on BuildlistLaunched on Tiny StartupsFeatured on ShowMeBestAIFeatured on ShowMeBestAIFind us on LaunchZoneFind us on LaunchZone
Β© 2026 Jackalope Digital LLC. All rights reserved.
  1. Home
  2. πŸ“Š Monitoring
  3. Llmprobe
Llmprobe logo
Health: ActiveRecent health check succeeded.Last checked 9/9/2026, 3:50:07 PM

Llmprobe

User RatingsBe the first to rate and review this MCP server!
View Repository5 GitHub StarsTotal stargazers on GitHub for the source repository (5 stars).Visit Website
monitoringllmsynthetic-monitoringprometheusopentelemetry

Monitor LLM endpoints with synthetic probes for TTFT, latency, throughput, errors, and deployment regressions.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent β€” or use 1-click editor setup below.

One-click editor setup isn’t available for this listing yet β€” we don’t have a confirmed install command, and we’d rather show nothing than point your editor at the wrong package or host. Follow the project’s own setup instructions, linked above.

Manual Client & Custom JSON ConfigExpand JSON β–Ύ
No confirmed setup config for this listing yet. We only publish a config block when the install details come from the project itself β€” its README, its docs, or a verified owner. We haven’t found those for Jwrede/llmprobe, and we’d rather show nothing than a guess you’d paste into your client. Follow the project’s own setup instructions for the current steps.
Install Directory Badge Claim listing AlternativesπŸ“Š More in Monitoring

Overview

llmprobe is a CLI and MCP server for probing hosted LLM APIs and OpenAI-compatible inference servers. Use it for one-off health checks, continuous monitoring, or CI gates based on endpoint thresholds. It exports results through JSON, Prometheus, and OpenTelemetry.

Use cases

β€’Measure TTFT and total latency across configured models
β€’Detect provider degradation with recurring synthetic probes
β€’Block deployments when probe thresholds fail
β€’Compare throughput and error rates across endpoints
β€’Export monitoring data to Prometheus or OpenTelemetry

Key features

β€’Measures TTFT, total latency, tokens per second, output tokens, and errors
β€’Supports OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints
β€’Runs one-off probes or continuous monitoring intervals
β€’Provides table, JSON, JSONL, and terminal dashboard output
β€’Generates Markdown reports with p50, p95, and p99 percentiles
β€’Exports Prometheus and OpenTelemetry metrics

Capabilities & Tool Schemas

Inspect callable tools, capabilities, and parameters exposed to AI agents by Llmprobe.

Extracted Tool Capabilities
Measures TTFT, total latency, tokens per second, output tokens, and errors
Supports OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints
Runs one-off probes or continuous monitoring intervals
Provides table, JSON, JSONL, and terminal dashboard output
Generates Markdown reports with p50, p95, and p99 percentiles
Exports Prometheus and OpenTelemetry metrics

Documentation Overview

llmprobe

llmprobe

Synthetic monitoring and CI smoke tests for LLM inference endpoints. Measure TTFT, latency, throughput, and errors. Single binary, zero SDKs.

CI Go License: MIT llmprobe MCP server

llmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or OpenAI-compatible inference servers, then reports the metrics that matter for production user experience: time to first token (TTFT), total latency, generation throughput (tokens/sec), and error rates.

Use it as a one-off health check, a continuous monitor, or a CI gate that blocks deploys when your LLM provider is degraded.

demo

Public benchmark

llm-bench uses llmprobe to run a continuous public benchmark of major LLM APIs. It publishes a live dashboard at bench.jonathanwrede.de and raw JSONL data in Jwrede/llm-bench-data.

This is the intended use case: repeated synthetic probes that make LLM latency, TTFT regressions, throughput drops, and provider degradation visible before users report them.

Install

Download a prebuilt binary from the latest release (Linux, macOS, Windows; amd64 and arm64).

Or install from source:

bash
go install github.com/Jwrede/llmprobe@latest
llmprobe version

Claude Code plugin

Install as a Claude Code plugin for /llmprobe skill and MCP tools:

Terminal
claude plugin install Jwrede/llmprobe

Or register the MCP server directly:

Terminal
claude mcp add --transport stdio llmprobe -- llmprobe mcp

llmprobe runs locally and only contacts LLM endpoints you configure. See PRIVACY.md for details.

Quick start

llmprobe works with OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI, Fireworks, DeepSeek, and Mistral.

Create a probes.yml (or copy the included example):

yaml
providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
    models:
      - name: gpt-4o
        thresholds:
          max_ttft: 2s
      - name: gpt-4o-mini
        thresholds:
          max_ttft: 500ms

  - name: anthropic
    api_key: ${ANTHROPIC_API_KEY}
    models:
      - name: claude-sonnet-4-20250514
        thresholds:
          max_ttft: 1s

Run a probe:

Code
$ llmprobe probe

Provider   Model                    Status    TTFT    Latency  Tok/s  Tokens  Error
--------   -----                    ------    ----    -------  -----  ------  -----
openai     gpt-4o                   healthy   312ms   2100ms   68.4   42
openai     gpt-4o-mini              healthy   98ms    814ms    112.3  56
anthropic  claude-sonnet-4-20250514 healthy   420ms   2831ms   52.1   38
azure      gpt-4o                   healthy   289ms   1950ms   71.2   44
bedrock    anthropic.claude-3-5...  degraded  1820ms  4510ms   28.1   38

4 healthy, 1 degraded, 0 errors

What it measures

MetricWhat it means
TTFTTime from request send to first content token. This is what users feel as "lag" before the response starts streaming.
LatencyTotal time from request to stream close.
Tok/sGeneration throughput: tokens produced per second after the first token. Calculated as token_count / (latency - ttft).
TokensTotal output tokens. Prefers provider usage metadata when available, falls back to SSE event counting.
Statushealthy if all thresholds pass, degraded if any threshold is exceeded, error if the request failed.

Commands

llmprobe probe

One-off health check. Probes all configured endpoints and prints results.

bash
llmprobe probe                        # table output
llmprobe probe -f json                # JSON output
llmprobe probe --fail-on degraded     # exit 1 if any endpoint is degraded
llmprobe probe -c custom-config.yml   # custom config path

Exit codes for CI:

--fail-onExit 0Exit 1
error (default)healthy or degradedany error
degradedhealthy onlydegraded or error
nonealwaysnever

llmprobe watch

Continuous monitoring. Probes all endpoints on an interval and prints a summary line per iteration.

server.ts
llmprobe watch                          # default 60s interval
llmprobe watch --interval 30s           # custom interval
llmprobe watch --tui                    # live terminal dashboard with TTFT chart
llmprobe watch --tui --load data.jsonl  # load historical data into the dashboard
llmprobe watch -f json                  # JSONL output (one line per result)
llmprobe watch --prometheus :9090       # expose Prometheus metrics
llmprobe watch --otel localhost:4317     # export OpenTelemetry metrics via OTLP/gRPC

The --tui flag launches a live terminal dashboard with a TTFT chart, color legend, and statistics table. Use --load to import historical JSONL data (from llmprobe watch -f json > data.jsonl).

llmprobe

Code
$ llmprobe watch --interval 30s

Watching 4 endpoints every 30s (Ctrl+C to stop)

[14:01:02] All 4 endpoints healthy.
[14:01:32] All 4 endpoints healthy.
[14:02:02] 3 healthy, 1 degraded, 0 errors. DEGRADED: openai/gpt-4o (TTFT 1820ms)
[14:02:32] All 4 endpoints healthy.

llmprobe report

Generate a Markdown summary from JSONL probe data with p50/p95/p99 percentiles for TTFT, latency, and throughput per endpoint.

bash
llmprobe report data.jsonl

Output:

Code
| Provider | Model | Probes | Errors | TTFT p50 | TTFT p95 | ... | Tok/s p50 | ...
|----------|-------|--------|--------|----------|----------|-----|-----------|----
| openai   | gpt-4o | 100  | 2      | 115ms    | 188ms    | ... | 46.9      | ...

llmprobe baseline

Create a baseline file from historical JSONL data for regression detection.

bash
llmprobe baseline data.jsonl -o baseline.json

Reference the baseline in your config to use multiplier-based thresholds:

yaml
baseline: baseline.json

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
    models:
      - name: gpt-4o
        thresholds:
          max_ttft_multiplier: 2.0       # fail if TTFT > 2x baseline p50
          max_latency_multiplier: 2.5    # fail if latency > 2.5x baseline p50

This lets you detect regressions relative to your own historical data rather than setting absolute thresholds.

llmprobe version

Print the installed binary version.

bash
llmprobe version

CI integration

Use llmprobe probe as a pre-deploy gate:

yaml
# .github/workflows/deploy.yml
- name: Check LLM providers
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    go install github.com/Jwrede/llmprobe@latest
    llmprobe probe --fail-on degraded

This blocks the deploy if any LLM provider is experiencing degraded performance right now.

When a probe fails, the output shows only the failing endpoints:

Code
Failed endpoints (1/4):
  openai/gpt-4o  DEGRADED  TTFT=280ms  Latency=950ms  Tok/s=32.1

MCP server

llmprobe MCP server

llmprobe includes a built-in Model Context Protocol server, allowing Claude Code and other MCP hosts to check LLM API health directly from an agent workflow.

Running the server

bash
llmprobe mcp

This starts the MCP server over stdio.

Registering with Claude Code

Terminal
claude mcp add --transport stdio llmprobe -- llmprobe mcp

Once registered, Claude Code can call llmprobe tools during any conversation.

Available tools

ToolDescription
probe_allProbe all configured endpoints from probes.yml. Returns TTFT, latency, throughput, and health status for every model. Accepts an optional config parameter for a custom config path.
probe_modelProbe a single model without a config file. Requires provider, model, and api_key_env. Supports optional base_url for OpenAI-compatible endpoints and optional label for display.
list_providersList all providers and models in the config file with their thresholds. Use this to discover available models before probing.
get_configReturn the full parsed configuration including defaults, providers, models, and thresholds.

Example use case: An agent calls list_providers to see what models are configured, then probe_all to verify they are healthy before deploying changes.

Configuration

yaml
defaults:
  prompt: "Hello"                                # probe prompt
  max_tokens: 20                                 # max output tokens
  timeout: 30s                                   # per-probe timeout
  concurrency: 5                                 # max parallel probes

Read the full README β†’View source on GitHub β†’

Related MCP Servers

View all in Monitoring View all alternatives
  • Byok Observability MCP logoByok Observability MCP

    Comprehensive MCP server for Grafana, Prometheus, Kafka UI, and Datadog with a secure "Bring Your Own Key" or BYOK model.

    πŸ“Š Monitoring2 views
    Compare vs Byok Observability MCP β†’
  • Heimdall MCP logoHeimdall MCP

    Transparent proxy for any MCP server that intercepts all JSON-RPC messages, measures latency, and stores traces in SQLite, PostgreSQL, or MySQL. Exports OpenTelemetry (OTLP) spans to Jaeger, Tempo, or Grafana. Supports stdio, HTTP, and SSE transports. npx @cardor/heimdall-mcp

    πŸ“Š Monitoring2 views
    Compare vs Heimdall MCP β†’
  • World Monitor logoWorld Monitor

    Live global intelligence: real-time markets, conflicts, country risk, chokepoints, energy. 39 tools.

    πŸ“Š Monitoring5 views
    Compare vs World Monitor β†’
  • Fastmcp Sonarqube Metrics logoFastmcp Sonarqube Metrics

    A Model Context Protocol (MCP) server that provides a set of tools for retrieving information about SonarQube projects like metrics (actual and historical), issues, health status.

    πŸ“Š Monitoring8 views
    Compare vs Fastmcp Sonarqube Metrics β†’

Adoption & maintenance

Factual signals from GitHub, npm, and our automated checks β€” not a rating.

GitHub stars
5
Stargazers on the source repository.
Directory activity
2 views
Config copies, upvotes, and views on AllMCPs.

Reviews

No reviews yet β€” be the first to share how this listing worked for you.

Frequently Asked Questions about Llmprobe

It supports OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible servers such as vLLM, Ollama, OpenRouter, Groq, Together AI, Fireworks, DeepSeek, and Mistral.

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:
Live Dynamic SVG PreviewLlmprobe AllMCPs Directory Badge
Markdown (GitHub README)
[![AllMCPs](https://allmcps.com/api/badge/jwrede-llmprobe?style=directory)](https://allmcps.com/mcp/jwrede-llmprobe)
HTML Embed
<a href="https://allmcps.com/mcp/jwrede-llmprobe"><img src="https://allmcps.com/api/badge/jwrede-llmprobe?style=directory" alt="Llmprobe on AllMCPs" /></a>

Technical Specs & Signals

CategoryπŸ“ŠMonitoring
PricingFree
More technical detailsExpand β–Ύ
AuthAPI key
LicenseMIT
Last updatedAug 9, 2026
Views2
Unique ViewsTotal visits recorded for this listing page on AllMCPs.
Installs0
Installs & Copy ActionsTotal times users copied install commands or configuration snippets for this server.
GitHub stars5
GitHub Star CountTotal stargazers on GitHub representing community popularity (5 stars).
43Quality signal: Fair Β· 43/100How this signal is calculated β–Ύ
Server availabilityNot measured

Not scored for repo-hosted servers β€” we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.

Verified ownership10/20
Documentation & tools20/30
Adoption & activity2/15
Community engagement0/10

A guidance signal from public completeness & health data β€” not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.

β˜… FeaturedAllMCPs Server logo

AllMCPs Server

The official MCP server for AllMCPs.com - submit and manage tools directly from your AI. The open directory for MCP servers. Connect Claude, Cursor, Windsurf, and AI agents to databases, tools, files, and APIs. Explore 10,000+ servers. AllMCPs is the premier, open directory for discovering, evaluating, and installing Model Context Protocol (MCP) servers to equip AI agents and LLMs with real-world superpowers.

Explore Server β†’

Own this project?

This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to unlock edit access and the Official badge β€” proof is checked automatically, then reviewed by our team.

Free dofollow backlink: add your website and place the AllMCPs badge on it β€” no claim needed. We detect it automatically and keep it verified as long as the badge stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Explore more

More in πŸ“Š Monitoring β†’Best MCP servers for Monitoring & Observability β†’Alternatives to Llmprobe β†’Install in Claude DesktopInstall in CursorInstall in VS Code