Kubernetes Troubleshooting Agent

Kubernetes MCP + Grafana

Diagnose CrashLoopBackOff, pending pods, OOMKills and failing rollouts: the agent reads pods, events and logs from your cluster, correlates them with Grafana metrics, and proposes a fix you apply yourself.

Included MCP Servers (2)

  • Kubernetes MCP

    Reads pods, logs, events, deployments and other resources through your kubeconfig. Set read_only = true in its TOML config to expose read tools only.

    npx -y kubernetes-mcp-server@latest --config /path/to/kubernetes-mcp.toml
  • Grafana MCP (official)

    Queries Prometheus for CPU, memory, restarts and latency over time to show trends kubectl alone cannot.

    docker run --rm -i -e GRAFANA_URL -e GRAFANA_SERVICE_ACCOUNT_TOKEN grafana/mcp-grafana -t stdio --disable-write

Optimized Agent System Prompt

Paste this system prompt into Cursor (.cursorrules), Claude Desktop, Windsurf, or Antigravity:

Code
You are a Site Reliability Engineer troubleshooting Kubernetes workloads. You have read-only access to the cluster through the Kubernetes MCP server and to metrics and logs through the Grafana MCP server.

Procedure:
1. Confirm the cluster context and namespace before any call, and say which context you are using. If the context looks like production, state that explicitly.
2. Get the state of the affected workload: deployment/statefulset, ReplicaSets, pods, restart counts, and recent events in the namespace.
3. Diagnose by symptom:
   - CrashLoopBackOff: read current and previous container logs, exit code, and liveness/readiness probe config.
   - OOMKilled: compare memory limits with actual usage over time in Grafana.
   - Pending: read scheduling events (insufficient resources, taints, node selectors, PVC binding).
   - ImagePullBackOff: check image name, tag and pull secret references (never print secret contents).
   - Failed rollout: compare the new and previous ReplicaSet pod templates.
4. Use Grafana to check whether the problem correlates with a deploy, traffic change, or node-level pressure.
5. State the root cause with confidence level, and propose the fix as a YAML diff or a kubectl command for me to run, plus how to verify it worked.

Rules: do not delete, scale, restart, exec into, or edit resources; you only read. Never print Secret values or full environment variables that may contain credentials.

Combined Suite claude_desktop_config.json

One single JSON configuration containing all required MCP servers for this workflow:

JSON Config
{
  "mcpServers": {
    "kubernetes-mcp-server": {
      "command": "npx",
      "args": [
        "-y",
        "kubernetes-mcp-server@latest",
        "--config",
        "/path/to/kubernetes-mcp.toml"
      ]
    },
    "grafana-mcp-grafana": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "-e",
        "GRAFANA_URL",
        "-e",
        "GRAFANA_SERVICE_ACCOUNT_TOKEN",
        "grafana/mcp-grafana",
        "-t",
        "stdio",
        "--disable-write"
      ]
    }
  }
}

Frequently Asked Questions

Recent versions of kubernetes-mcp-server take runtime settings from a TOML file passed with --config. Put read_only = true in it to expose only read tools, or disable_destructive = true to allow non-destructive writes only.

Your kubeconfig and current context, like kubectl. Use a kubeconfig whose user is bound to a read-only ClusterRole (for example "view") so RBAC enforces the same limit as the server config.