tracehub-mcp

Also listed on the official MCP registry as io.github.mcpsmiths/tracehub-mcp (the registry's own ?search= endpoint can surface an outdated version first; this exact-name endpoint always reflects the current isLatest release).

Give your AI assistant a direct line into your observability backend. tracehub-mcp is an MCP (Model Context Protocol) server that lets Claude, Cursor, Windsurf, Gemini CLI, or any MCP client query OpenTelemetry traces from your LLM/GenAI application and reason about them — find expensive calls, debug errors, compare model performance, track token usage — without you copy-pasting trace JSON into a chat window.

It speaks OpenTelemetry's gen_ai.* semantic conventions natively, so it understands prompts, completions, token usage, and finish reasons as first-class concepts, not just generic span attributes.

tracehub-mcp started as a fork of traceloop/opentelemetry-mcp-server (Apache 2.0) — full attribution and fork history are in NOTICE. It's grown into a 5-backend, security-hardened server maintained independently under mcpsmiths; see What's Different From Upstream below for the parts that are new here.

Table of Contents

• Quick Start
• Supported Backends
• What's Different From Upstream
• Installation
• Configuration
• Diagnostics
• Security Considerations
• Privacy Policy
• MCP Client Setup
• Tools Reference
• Generic Filter System
• Example Queries
• Common Workflows
• Development
• Troubleshooting
• Roadmap
• License
• Support

Quick Start

tracehub-mcp is on PyPI. No install step needed — uvx fetches and runs it in one shot:

// claude_desktop_config.json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

Or from Claude Code directly:

claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp

That's it. Ask your assistant: "Show me traces with errors from the last hour."

See MCP Client Setup for Cursor, Windsurf, VS Code, and Gemini CLI, and Installation for pip/pipx/from-source alternatives.

Supported Backends

• Jaeger — local/self-hosted, the most common open-source trace backend. No auth required.
• Grafana Tempo — local or Grafana Cloud, TraceQL-native search.
• Traceloop — cloud LLM observability platform, API-key auth.
• Datadog — cloud APM, requires an API key and an Application key.
• Sentry — cloud or self-hosted, requires an auth token and an organization slug.
• AWS X-Ray — SigV4-signed via boto3, requires an AWS region and standard AWS credentials (not an API key). Because OTel span attributes land in unindexed X-Ray segment metadata by default (only annotations are queryable), native server-side filtering is narrower here than for the other backends — see the Backend Support Matrix.
• New Relic — NerdGraph GraphQL API, requires a User API key (not an Ingest/License key) and an account ID. Built without a live account to verify against — see the New Relic setup section below and the module docstring in backends/newrelic.py for exactly which assumptions are unverified.
• Honeycomb — Query Data API, requires a Configuration Key and a dataset slug. Built without a live account to verify against, and the research behind it found the programmatic Query Data API (the only way to actually run a query and get results back) is documented as Enterprise-plan exclusive — see the Honeycomb setup section below and the module docstring in backends/honeycomb.py.

All eight implement the same BaseBackend interface, so every MCP tool works identically regardless of which one you point the server at. See Configuration for per-backend setup.

What's Different From Upstream

Upstream opentelemetry-mcp-server shipped Jaeger, Tempo, and Traceloop. tracehub-mcp adds Datadog and Sentry as full backends — not thin wrappers, but complete implementations of every tool (search, span search, trace hydration, service discovery, health checks). Along the way, all five backends — including the three inherited from upstream — were hardened to a consistent bar:

• HTTPS-only enforcement on cloud backends. Datadog and Sentry both refuse to start against a plain http:// URL, because their auth is a bearer token / API+App key pair that has no business going out over plaintext.
• Query-injection-safe escaping. Every value spliced into a Datadog span-search query or a Sentry Discover query is escaped and exact-quoted; field names (which are less obviously untrusted, since they come from the MCP tool's filters parameter) are validated against an allowlist pattern before being spliced into the query string, closing off structural injection through a crafted field name.
• Bounded pagination on every backend that paginates via cursor (Datadog, Sentry) — a search stops at the requested limit or when the backend stops returning a continuation cursor, whichever comes first, so a single tool call can't degrade into an unbounded crawl.
• Exact-ID re-verification. Where a backend's search API can return neighbors instead of an exact match (notably Datadog's trace reconstruction from grouped spans), every result is re-checked against the exact ID that was asked for before being returned.
• No fabricated data on malformed responses, in the backends we built. Datadog and Sentry reject a span outright — rather than substituting a placeholder like now() for a missing timestamp or a literal "unknown" for a missing service_name/operation_name — since a fabricated value would silently corrupt trace ordering, duration aggregation, and any tool that groups by service or operation. (The three backends inherited from upstream — Jaeger, Tempo, Traceloop — predate this discipline and haven't been retrofitted; that's deliberate scope discipline, not an oversight, mirroring this project's own precedent of not reaching into shared/inherited code without full regression coverage for it.)

All of this is backed by 458 passing tests (2 skipped, zero regressions), a clean ruff check and mypy --strict run, and two rounds of adversarial CodeRabbit review on the new backends.

Installation

tracehub-mcp is on PyPI. Pick whichever of these your workflow already uses — they're equivalent.

Option 1: uvx (no install step)

uvx tracehub-mcp --backend jaeger --url http://localhost:16686

This is what the Quick Start config above uses — uv fetches the package and runs the tracehub-mcp entry point in one shot, nothing left behind on disk between runs.

Option 2: pip / pipx

pipx install tracehub-mcp
# or: pip install tracehub-mcp

tracehub-mcp --backend jaeger --url http://localhost:16686

Option 3: Clone and run from source

git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync

uv run tracehub-mcp --backend jaeger --url http://localhost:16686

Use this if you're developing locally, want to pin to a specific commit, or want the dev tooling installed (uv sync --group dev).

Option 4: Docker

docker run --rm -p 8000:8000 \
  -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://host.docker.internal:16686 \
  ghcr.io/mcpsmiths/tracehub-mcp:latest

Runs HTTP transport by default (the image's CMD); clients connect to http://localhost:8000/mcp. This is also the form to use for MCP clients whose config takes a command/args pair pointing at docker directly (Cursor, Windsurf) instead of a local binary.

Prerequisites: Python 3.11+, plus uv for Options 1 and 3; Docker for Option 4.

Configuration

Configuration comes from environment variables, CLI flags, or both. Precedence: CLI arguments > environment variables > defaults.

# .env (see .env.example)
BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686

# Equivalent via CLI flags
tracehub-mcp --backend jaeger --url http://localhost:16686

All Configuration Options

Every backend-related CLI flag has a matching env var (--backend/BACKEND_TYPE, --url/BACKEND_URL, --api-key/BACKEND_API_KEY, --app-key/BACKEND_APP_KEY, --tempo-instance-id/BACKEND_TEMPO_INSTANCE_ID, --sentry-org/BACKEND_SENTRY_ORG, --sentry-project/BACKEND_SENTRY_PROJECT, --aws-region/BACKEND_AWS_REGION, --environments/BACKEND_ENVIRONMENTS). --disable-tools <name1,name2,...> / --enabled-tools <name1,name2,...> (CLI-only, no env var) remove/allowlist tools for reduced-trust deployments - --enabled-tools is applied first, --disable-tools on top of whatever it kept. Run tracehub-mcp --help for the full list.

Known third-party egress dependency: the underlying FastMCP framework checks PyPI (https://pypi.org/pypi/fastmcp/json) for a newer FastMCP release once every 12 hours when it prints its startup banner - this is FastMCP's own behavior, not tracehub-mcp's, and unrelated to the OTel self-instrumentation above. It fails silently if there's no network access. For network-restricted/air-gapped deployments, disable it with FASTMCP_CHECK_FOR_UPDATES=off, or suppress the banner entirely with FASTMCP_SHOW_SERVER_BANNER=false.

Backend-Specific Setup

BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686

BACKEND_TYPE=tempo
BACKEND_URL=http://localhost:3200

BACKEND_TYPE=tempo
BACKEND_URL=https://tempo-prod-XX-prod-XX-XXXX.grafana.net
BACKEND_TEMPO_INSTANCE_ID=your_stack_instance_id
BACKEND_API_KEY=your_cloud_access_policy_token

BACKEND_TYPE=traceloop
BACKEND_URL=https://api.traceloop.com
BACKEND_API_KEY=your_api_key_here

BACKEND_TYPE=datadog
# US site (default): https://api.datadoghq.com
# EU site:            https://api.datadoghq.eu
BACKEND_URL=https://api.datadoghq.com
BACKEND_API_KEY=your_api_key_here
BACKEND_APP_KEY=your_application_key_here

BACKEND_TYPE=sentry
# SaaS (may be region-specific, e.g. https://us.sentry.io):
BACKEND_URL=https://sentry.io
BACKEND_API_KEY=your_auth_token_here
BACKEND_SENTRY_ORG=your-org-slug
# Optional: narrow queries to one project
BACKEND_SENTRY_PROJECT=your-project-slug

BACKEND_TYPE=xray
# Decorative only - never dereferenced for a live request. Keep it
# consistent with BACKEND_AWS_REGION by convention.
BACKEND_URL=https://xray.us-east-1.amazonaws.com
BACKEND_AWS_REGION=us-east-1

BACKEND_TYPE=newrelic
# US datacenter (default): https://api.newrelic.com/graphql
# EU datacenter:            https://api.eu.newrelic.com/graphql
BACKEND_URL=https://api.newrelic.com/graphql
BACKEND_API_KEY=your_user_api_key_here
BACKEND_NEWRELIC_ACCOUNT_ID=1234567

BACKEND_TYPE=honeycomb
# US instance (default): https://api.honeycomb.io
# EU instance:            https://api.eu1.honeycomb.io
BACKEND_URL=https://api.honeycomb.io
BACKEND_API_KEY=your_configuration_key_here
BACKEND_HONEYCOMB_DATASET=your-dataset-slug

Transport Modes

# stdio (default) — local use, Claude Desktop, single process
uvx tracehub-mcp
tracehub-mcp                      # pipx/pip install
uv run tracehub-mcp               # from-source install

# HTTP — remote access, multiple clients, network deployment, sample applications
uvx tracehub-mcp --transport http --host 0.0.0.0 --port 8000
tracehub-mcp --transport http --host 0.0.0.0 --port 8000              # pipx/pip install
uv run tracehub-mcp --transport http --host 0.0.0.0 --port 8000       # from-source install

With HTTP transport, clients connect to http://<host>:<port>/mcp (streamable-HTTP, for compatibility across MCP clients).

Fly.io: the one-click fly mcp launch command only supports stdio-transport servers — it can't deploy this server's --transport http mode. Running tracehub-mcp on Fly.io with HTTP transport needs the manual fly.toml + fly deploy path instead.

Diagnostics

Validate a config before wiring it into a real MCP client:

tracehub-mcp doctor --backend jaeger --url http://localhost:16686

[OK] Configuration loaded and validated
[OK] Backend constructed: jaeger @ http://localhost:16686/
[OK] Health check: healthy
[OK] Connectivity probe (list_services): 2 service(s)

doctor accepts the same --backend/--url/etc. flags as the main command and exits non-zero if any step fails - unlike normal server startup, which lazily initializes the backend and deliberately keeps running even if the initial health check fails.

To see the fully-resolved configuration (env vars + CLI overrides merged) without starting the server:

tracehub-mcp --print-config --backend jaeger --url http://localhost:16686

Secrets (--api-key/--app-key) are reported as api_key_set/app_key_set booleans, never their actual value.

Security Considerations

Trace and span data returned by this server — attribute values, error messages, operation names — comes from whatever application your observability backend is instrumenting, not from tracehub-mcp itself. That makes it fundamentally the same category of untrusted external content as a webpage or a file, even though it's a trusted server (this one) handing it back to your MCP client.

• Treat backend data as untrusted input. An LLM client consuming trace/span data from tracehub-mcp should apply the same caution it would to any other external tool output — a span attribute or error message is application data to reason about, not an instruction to follow, no matter how it's phrased.
• This is a known MCP risk category, not a tracehub-mcp-specific one. OWASP's GenAI Security Project covers it in their Practical Guide for Secure MCP Server Development, and Anthropic's own engineering guidance, How We Contain Claude, states plainly that tool output is an attack surface even when the tool itself is trusted.
• Practical implication: if you're querying traces from an application that processes untrusted user input (e.g. a customer-facing chatbot), be aware that adversarial content a user fed into that application could end up in a span attribute this server returns — and from there, in your LLM client's context.

Privacy Policy

tracehub-mcp is self-hosted software with no maintainer-operated service or account — it collects no data of its own. See PRIVACY.md for the full policy: what data the software touches, where network calls go, and how secrets and self-instrumentation are handled.

MCP Client Setup

Every example below uses uvx tracehub-mcp (no install step). Swap in tracehub-mcp (pip/pipx install) or uv run tracehub-mcp (from-source, --directory /absolute/path/to/tracehub-mcp) if you installed it a different way — see Installation.

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "datadog",
        "BACKEND_URL": "https://api.datadoghq.com",
        "BACKEND_API_KEY": "your_api_key_here",
        "BACKEND_APP_KEY": "your_application_key_here"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "sentry",
        "BACKEND_URL": "https://sentry.io",
        "BACKEND_API_KEY": "your_auth_token_here",
        "BACKEND_SENTRY_ORG": "your-org-slug"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "/absolute/path/to/tracehub-mcp/start_locally.sh"
    }
  }
}

claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp

claude mcp list
claude "Show me traces with errors from the last hour"

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

{
  "servers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

gemini "Analyze token usage for gpt-4 requests today"

Tools Reference

tracehub-mcp exposes 19 MCP tools:

Backend Support Matrix

_{* Jaeger requires the service_name parameter for span search.
† Datadog, AWS X-Ray, New Relic, and Honeycomb have no trace-level search API (X-Ray's BatchGetTraces fetches by exact ID); traces are reconstructed by searching (spans, X-Ray trace summaries, NRQL Span events, or Honeycomb breakdown groups) and hydrating, with every result re-verified against the exact ID requested.
‡ Sentry and New Relic both have a native trace-lookup endpoint (unlike Datadog and Honeycomb), so get_trace calls it directly; search_traces still discovers candidate trace IDs via a span/NRQL search first, since both backends' search surface is itself span-centric.
§ AWS X-Ray natively filters only service name, duration, and error/fault/throttle-derived status — OTel attributes (including all gen_ai.* fields) land in unindexed segment metadata by default, not indexed annotations, so they're always applied client-side rather than pushed down to FilterExpression.
¶ New Relic: search_spans reads gen_ai.* attributes directly off NRQL's well-documented flat row shape; search_traces/get_trace additionally rely on an unverified assumption about the shape of the attributes field inside NerdGraph's distributedTracing.trace response, mitigated by overlaying the original search row's attributes back onto the hydrated result. Honeycomb: gen_ai.* attributes are extracted via the same breakdown-column technique used for every other field (module docstring point 2), an equally unverified extraction method.
♦ Honeycomb's entire "Search"/"Get trace by ID" row of checkmarks above depends on the Query Data API, which the research behind this backend found is documented as Enterprise-plan exclusive - a Free/Pro account may not be able to exercise any of this at all. See the Honeycomb setup section above.}

Key Tool Details

search_traces

{
  "service_name": "my-app",
  "start_time": "2024-01-01T00:00:00Z",
  "end_time": "2024-01-01T23:59:59Z",
  "gen_ai_system": "openai",
  "gen_ai_request_model": "gpt-4",
  "min_duration_ms": 1000,
  "has_error": false,
  "limit": 50
}

Parameters: service_name, operation_name, start_time/end_time (ISO 8601), min_duration_ms/max_duration_ms, gen_ai_system, gen_ai_request_model, gen_ai_response_model, has_error, tags, filters (see Generic Filter System), limit (1-1000, default 100). Returns trace summaries with token counts.

get_trace

{ "trace_id": "abc123def456" }

Returns the full trace tree: all spans with attributes, parsed OpenTelemetry gen_ai.* data for LLM spans, per-span token usage, error information, and each span's raw events (e.g. gen_ai.evaluation.result, or any other instrumentation-emitted event — not filtered to a fixed set of names).

triage_trace

{ "trace_id": "abc123def456", "detail_level": "summary" }

Instead of returning raw trace data for an agent to re-derive a diagnosis from every time, this synthesizes one directly: a critical path (the "Last Finishing Child" chain actually responsible for the trace's total latency), the top spans ranked by self-time (latency contribution net of children), and — when the trace contains an error — the deepest error span in the trace's error chain as the likely root cause (confidence: "high", or "medium" when multiple equally-deep error chains make blame ambiguous). Falls back to the highest self-time span as a pure-latency diagnosis (confidence: "low") when no error is present. Deterministic — no LLM call — and works against any configured backend, since it operates on get_trace's already-fetched span data. detail_level: "full" additionally attaches the diagnosed root cause's raw error detail (message/type/stacktrace) when the verdict is error-driven.

correlate_trace

{ "trace_id": "abc123def456" }

Tries to find the corresponding trace in a second, independently-configured backend (e.g. a Datadog trace and its downstream Sentry error, joined). Tries a direct trace_id match in the secondary backend first (confidence: "high"); if that fails, falls back to a time-window + service-name-overlap heuristic search (confidence: "low") - queried once per service name in the primary trace, so this works against backends like Jaeger that require service_name on search_traces. Each match also reports root_cause_consistent: null when neither trace has an error to compare, otherwise true/false for whether both sides agree on the error and which service it traces back to (reusing the same error-chain logic triage_trace uses). This is a best-effort correlation, not a guaranteed join - the result always includes a fixed limitations list (clock skew, sampling mismatches, partial trace visibility, and the fact that trace-id continuity across a vendor boundary isn't guaranteed even under normal W3C Trace Context propagation). Requires a secondary backend configured via SECONDARY_BACKEND_TYPE/SECONDARY_BACKEND_URL (see configuration table); raises a clear error otherwise.

get_llm_usage

{
  "start_time": "2024-01-01T00:00:00Z",
  "end_time": "2024-01-01T23:59:59Z",
  "service_name": "my-app",
  "gen_ai_system": "openai",
  "limit": 1000
}

Returns aggregated prompt/completion/total tokens, broken down by model and by service, plus request counts.

list_services — no parameters. Returns the list of instrumented service names.

find_errors

{
  "start_time": "2024-01-15T14:00:00Z",
  "service_name": "my-app",
  "limit": 50
}

Returns error messages, error types, truncated stack traces, and LLM-specific error info.

list_llm_models / get_llm_model_stats / get_llm_expensive_traces / get_llm_slow_traces / list_llm_tools_tool / search_spans_tool / list_sessions / get_session_stats / compare_time_windows / investigate_cost_spike / investigate_error_spike / get_prompt_version_stats are documented in detail, with worked examples, in CLAUDE.md — this README covers the shape every tool shares; CLAUDE.md is the fuller reference for exact parameters and response fields on the LLM-analysis tools.

Generic Filter System

search_traces and search_spans_tool both accept a filters list in addition to (or instead of) their simple named parameters, for advanced queries. Each filter is:

{
  "field": "gen_ai.usage.total_tokens",
  "operator": "gt",
  "value": 5000,
  "value_type": "number"
}

• field — dotted attribute name, e.g. gen_ai.usage.prompt_tokens, traceloop.span.kind, service.name
• operator — see table below
• value — single value (most operators) or values — list (for in, not_in, between)
• value_type — "string", "number", or "boolean"

Multiple filters combine with AND logic. Legacy simple parameters (service_name, gen_ai_request_model, etc.) still work and are converted to filters internally — mix and match freely.

The server uses a hybrid filtering strategy: filters are pushed to the backend's native query language when supported (TraceQL for Tempo, span-search syntax for Datadog, Discover syntax for Sentry), and applied client-side afterward for anything the backend can't express natively.

Example — expensive OpenAI traces:

{
  "filters": [
    { "field": "gen_ai.system", "operator": "equals", "value": "openai", "value_type": "string" },
    { "field": "gen_ai.usage.total_tokens", "operator": "gt", "value": 5000, "value_type": "number" }
  ]
}

For the full semantic-convention attribute list (gen_ai.* vs legacy llm.*, token-naming variants across providers, finish-reason values, and the token-calculation fallback chain), see CLAUDE.md.

Example Queries

Find Expensive OpenAI Operations

Ask: "Show me OpenAI traces from the last hour that took longer than 5 seconds"

Tool call: search_traces

{
  "service_name": "my-app",
  "gen_ai_system": "openai",
  "min_duration_ms": 5000,
  "start_time": "2024-01-15T10:00:00Z",
  "limit": 20
}

Response:

{
  "traces": [
    {
      "trace_id": "abc123...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "status": "OK",
      "duration_ms": 8250,
      "span_count": 3,
      "llm_span_count": 1,
      "total_tokens": 4523,
      "has_errors": false
    }
  ],
  "count": 1
}

Analyze Token Usage by Model

Ask: "How many tokens did we use for each model today?"

Tool call: get_llm_usage

{
  "start_time": "2024-01-15T00:00:00Z",
  "end_time": "2024-01-15T23:59:59Z",
  "service_name": "my-app"
}

Response:

{
  "period": { "start_time": "2024-01-15T00:00:00Z", "end_time": "2024-01-15T23:59:59Z" },
  "filters": { "service_name": "my-app" },
  "summary": {
    "total_requests": 487,
    "total_prompt_tokens": 82140,
    "total_completion_tokens": 43290,
    "total_tokens": 125430
  },
  "by_model": {
    "gpt-4": { "requests": 156, "prompt_tokens": 58300, "completion_tokens": 26900, "total_tokens": 85200 },
    "gpt-3.5-turbo": { "requests": 331, "prompt_tokens": 23840, "completion_tokens": 16390, "total_tokens": 40230 }
  },
  "by_service": {
    "my-app": { "requests": 487, "prompt_tokens": 82140, "completion_tokens": 43290, "total_tokens": 125430 }
  }
}

Find Traces with Errors

Ask: "Show me all errors from the last hour"

Tool call: find_errors

{
  "start_time": "2024-01-15T14:00:00Z",
  "service_name": "my-app",
  "limit": 10
}

Response:

{
  "count": 1,
  "error_traces": [
    {
      "trace_id": "def456...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "start_time": "2024-01-15T14:23:15Z",
      "duration_ms": 1200,
      "status": "ERROR",
      "span_count": 2,
      "llm_span_count": 1,
      "total_tokens": 310,
      "has_errors": true,
      "error_spans": [
        {
          "span_id": "span789...",
          "operation_name": "chat.completions",
          "service_name": "my-app",
          "status": "ERROR",
          "error_message": "RateLimitError: Too many requests",
          "error_type": "openai.error.RateLimitError",
          "is_llm_error": true,
          "llm_provider": "openai",
          "llm_model": "gpt-4"
        }
      ]
    }
  ]
}

Compare Model Performance

Ask: "What's the performance difference between GPT-4 and Claude?"

Tool call 1: get_llm_model_stats for gpt-4

{ "model_name": "gpt-4", "start_time": "2024-01-15T00:00:00Z" }

Tool call 2: get_llm_model_stats for claude-3-opus

{ "model_name": "claude-3-opus-20240229", "start_time": "2024-01-15T00:00:00Z" }

Investigate High Token Usage

Ask: "Which requests used the most tokens today?"

Tool call: get_llm_expensive_traces

{ "limit": 10, "start_time": "2024-01-15T00:00:00Z", "min_tokens": 5000 }

Common Workflows

Cost Optimization

1. get_llm_expensive_traces — find the highest-token requests
2. get_llm_usage — see which models are costing the most
3. get_trace on a specific trace_id — inspect the exact prompt/response

Performance Debugging

1. get_llm_slow_traces — identify latency outliers
2. find_errors — check for failure patterns
3. get_llm_model_stats — check finish-reason distribution for truncation

Model Adoption Tracking

1. list_llm_models — see every model actually being called
2. get_llm_model_stats per model — compare performance
3. Scan list_llm_models results for unexpected models/services (shadow AI)

Development

git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync --group dev   # pulls in pytest, mypy, ruff, etc. for local iteration

# Tests (458 passed, 2 skipped at time of writing)
uv run pytest

# With coverage
uv run pytest --cov=opentelemetry_mcp --cov-report=html

# Format, lint, type-check
uv run ruff format .
uv run ruff check .
uv run mypy src/

CI (.github/workflows/ci.yml) runs Ruff and mypy (strict) on every push, plus the full pytest suite.

Troubleshooting

Backend connection issues:

curl http://localhost:16686/api/services   # Jaeger
curl http://localhost:3200/api/search/tags  # Tempo

Authentication errors: confirm your key is set —

export BACKEND_API_KEY=your_key_here
# or: tracehub-mcp --api-key your_key_here

No traces found:

• Check the time range (use recent timestamps)
• Verify service names with list_services
• Try searching without filters first

Token usage shows zero:

• Confirm your traces have OpenTelemetry gen_ai.* (or legacy llm.*) instrumentation
• Inspect raw span attributes with get_trace

Datadog/Sentry-specific issues: see the troubleshooting notes under each backend in Configuration.

Roadmap

Both ideas that were previously listed here as deferred are now shipped:

• Agent-native triage — tools that flag a likely root cause rather than just returning raw trace data: see triage_trace in the Tools Reference above.
• Cross-backend correlation — querying a second, independently-configured backend and correlating results against it: see correlate_trace above. This ships as a best-effort, heuristic correlation (direct trace_id match, or a time-window/service-overlap fallback) rather than a guaranteed schema-level join, per the research behind it — trace-id continuity across a vendor boundary is not guaranteed even under normal W3C Trace Context propagation.

New Relic and Honeycomb have since shipped as the 7th and 8th backends, both built entirely from research/documentation without a live account to verify against — see their own setup sections for exactly what's unverified before relying on either in production; Honeycomb in particular carries a real go/no-go risk (its programmatic Query Data API is documented as Enterprise-plan exclusive). There is no further backend currently under research. (Grafana Cloud was never on this list — it already ships today via Tempo's Basic Auth path; see Grafana Tempo above.)

Carried over from upstream's older roadmap, re-prioritized behind the above rather than dropped: a dedicated model-vs-model comparison tool (today's get_llm_model_stats and compare_time_windows cover per-model stats and time-window diffing separately, but not a single tool that diffs two specific models directly), broader prompt-pattern analysis across templates rather than just version-over-version for one prompt (get_prompt_version_stats already covers the latter), MCP resources for common queries, and SigNoz/ClickHouse backend support. Cost calculation with built-in pricing tables and a query-result caching layer are shipped, not pending: every usage-reporting tool (e.g. investigate_cost_spike in the Tools Reference) returns a cost_usd/cost_usd_is_partial pair backed by a vendored litellm pricing table (src/opentelemetry_mcp/pricing/), and query results are cached with in-flight request coalescing via the QUERY_CACHE_TTL_SECONDS env var (see the configuration table above).

Everything else documented elsewhere in this README is shipped.

Contributing

Contributions are welcome. Before opening a PR, make sure:

1. All tests pass: uv run pytest
2. Code is formatted: uv run ruff format .
3. No linting errors: uv run ruff check .
4. Type checking passes: uv run mypy src/

License

Apache License 2.0 — see LICENSE. This project is a fork of traceloop/opentelemetry-mcp-server; full attribution and the fork relationship are documented in NOTICE.

Support

• Issues & feature requests: github.com/mcpsmiths/tracehub-mcp/issues
• Changelog: CHANGELOG.md
• Community: Traceloop Community Slack (upstream's community channel; general OpenTelemetry/GenAI-tracing questions welcome)
• Related: Model Context Protocol · OpenTelemetry GenAI semantic conventions · traceloop/opentelemetry-mcp-server (upstream)