# MCP Compliance **Category:** 💻 Developer Tools **Repository:** https://github.com/YawLabs/mcp-compliance **Views:** 0 **Installs:** 0 **Upvotes:** 0 **Directory Page:** https://allmcps.com/mcp/mcp-compliance ## Description CLI tool and MCP server that tests MCP servers for spec compliance ## Claude Desktop Quick Installation Heuristic fallback — verify the package name and runner against the repository README before running it. Uses `npx` (confidence: low): ```json "mcpServers": { "mcp-compliance": { "command": "npx", "args": ["-y","mcp-compliance"] } } ``` ## Documentation & README # @yawlabs/mcp-compliance [![Add to Yaw MCP](https://yaw.sh/yaw-mcp-button.svg)](https://yaw.sh/mcp/install?name=mcp-compliance&command=npx&args=-y%2C%40yawlabs%2Fmcp-compliance&description=Test%20any%20MCP%20server%20against%20the%20spec%20%282025-11-25%20or%202026-07-28%29%20with%20letter-grade%20scoring&source=https%3A%2F%2Fgithub.com%2FYawLabs%2Fmcp-compliance) One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below. [![npm version](https://img.shields.io/npm/v/@yawlabs/mcp-compliance)](https://www.npmjs.com/package/@yawlabs/mcp-compliance) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![GitHub stars](https://img.shields.io/github/stars/YawLabs/mcp-compliance)](https://github.com/YawLabs/mcp-compliance/stargazers) **Test any MCP server for spec compliance.** Two test suites — 88 tests for MCP [2025-11-25](https://modelcontextprotocol.io/specification/2025-11-25) and 103 tests for MCP [2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28) — covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. The tool probes the server and grades the newest spec revision it speaks, or you pin one with `--spec-version`. Works against **HTTP endpoints** (`https://my-server.com/mcp`) and **stdio servers** (`npx @modelcontextprotocol/server-filesystem /tmp`) alike. CLI, MCP server, and programmatic API. Built and maintained by [Yaw Labs](https://yaw.sh). ## Why this tool? MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete. This tool solves that: - **Two spec revisions, one tool** — 88 tests for MCP 2025-11-25 (the `initialize` + session era) and 103 tests for MCP 2026-07-28 (stateless, per-request `_meta`, `server/discover`, caching hints, MRTR). Both cover the same 8 categories: transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. HTTP-specific tests (CORS, TLS, header validation, rate limiting) are gated out on stdio: the 2025-11-25 suite runs 85 tests on HTTP and 55 on stdio, the 2026-07-28 suite 99 and 75. - **Auto-detected revision** — one `server/discover` probe tells the tool which era the server speaks; dual-era servers are graded against 2026-07-28 with a warning. Pin either revision with `--spec-version`. See [Spec version](#spec-version). - **Capability-driven** — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim. - **Graded scoring** — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance. - **CI-ready** — `--strict` mode exits with code 1 on required test failures. Drop it into any pipeline. - **Spec-referenced** — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why. - **Three interfaces** — CLI for humans, MCP server for AI assistants, programmatic API for integration. - **Published methodology** — the [testing methodology](https://github.com/YawLabs/mcp-compliance/blob/HEAD/COMPLIANCE_RUBRIC.md) and [rule catalog](https://github.com/YawLabs/mcp-compliance/blob/HEAD/mcp-compliance-rules.json) are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules. ## Quick start **Remote HTTP server:** ```bash npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp ``` **Local stdio server** (the vast majority of MCP servers on npm): ```bash # Pass the command directly, Inspector-style npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp # Or a local build npx @yawlabs/mcp-compliance@latest test node ./dist/server.js # With env vars npx @yawlabs/mcp-compliance@latest test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github ``` **Install globally:** ```bash npm install -g @yawlabs/mcp-compliance mcp-compliance test https://my-server.com/mcp ``` That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score. ## CLI usage ### HTTP targets ```bash # Terminal output with colors and grade mcp-compliance test https://my-server.com/mcp # JSON / SARIF for scripting + GitHub Code Scanning mcp-compliance test https://my-server.com/mcp --format json mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif # Strict mode for CI — exits 1 on required-test failure mcp-compliance test https://my-server.com/mcp --strict # Auth (shorthand or full header) mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123" mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc" # Focus the run mcp-compliance test https://my-server.com/mcp --only transport,lifecycle mcp-compliance test https://my-server.com/mcp --skip prompts,resources mcp-compliance test https://my-server.com/mcp --verbose ``` ### stdio targets Pass the command and its args as positional arguments (MCP Inspector-style). Use `--` to disambiguate when the target needs flags that collide with ours. ```bash # npm-distributed stdio servers mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp mcp-compliance test uvx mcp-server-git # Local build mcp-compliance test node ./dist/server.js # With env vars (repeatable -E, or --env-file) mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server mcp-compliance test --env-file .env -- node ./server.js # Set working directory mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js # Target uses a flag that collides with ours — use `--` to separate mcp-compliance test --verbose -- node ./server.js --verbose ``` On Windows, `npx` and other `.cmd` shims are handled automatically by spawning through the shell. ### Spec version The tool ships one test catalog per MCP specification revision it supports — **2025-11-25** (88 tests: `initialize` handshake, sessions, `ping`) and **2026-07-28** (103 tests: stateless, per-request `_meta`, `server/discover`, caching hints, MRTR). A run grades exactly one revision, chosen by `--spec-version`: ```bash # auto (default): probe the server, grade the newest revision it speaks mcp-compliance test https://my-server.com/mcp # pin a revision — no probe, the named catalog runs regardless of what the server answers mcp-compliance test https://my-server.com/mcp --spec-version 2026-07-28 mcp-compliance test https://my-server.com/mcp --spec-version 2025-11-25 # --list never connects: under auto (the default) it prints every catalog in its own # section; --spec-version previews just that one mcp-compliance test --list --spec-version 2026-07-28 --transport stdio ``` How `auto` decides, following the spec's own rules for dual-era clients: the tool sends one conformant 2026-07-28 `server/discover` (on HTTP it doubles as the preflight connectivity check, so detection costs no extra round-trip; on stdio it is the first exchange). A `DiscoverResult`, or a JSON-RPC error with a modern code (`-32020`, `-32021`, `-32022`), selects **2026-07-28**. Anything else — `-32601`, `-32000`, a 400 "not initialized", a 404, an HTML page, or no reply within the timeout — selects **2025-11-25**; a 401/403 with no modern error body also lands on 2025-11-25, the note says the era could not be determined, and the report's **first** warning (in every output format) says the grade below is not meaningful. With no `Authorization` header configured, a 401, or a 403 carrying a `WWW-Authenticate: Bearer` challenge, reads as authentication required (note `authentication required -- pass --auth`; the warning says to re-run with `--auth`). With one configured, a 401, or a 403 whose Bearer challenge carries an `error` parameter, reads as the credential rejected (note `credential rejected -- check --auth`; the warning names the reason from that error, e.g. `invalid_token` is an invalid or expired token and `insufficient_scope` a missing scope or permission). Any other 403 (the SDK's Host validation behind a tunnel hostname, Origin validation, a gateway) is worded neutrally with or without a header: the note says `forbidden -- Host/Origin validation, a gateway, or missing credentials` (or `... or token permissions` with a header), and the warning quotes the body's JSON-RPC error message when there is one, points at Host/Origin validation and any gateway first, and without a header suggests `--auth` only if the server does require a credential. The fallback is deliberately not keyed to a single error code; a legacy server that ignores unknown methods is still classified correctly, just after the timeout. The report header names the reason (`auto-detected from server/discover: supportedVersions [2026-07-28]`, `... JSON-RPC error -32601, legacy`, `... no response, legacy`) and the JSON `warnings` carry the same note (`Spec version auto-detected as (server/discover -> ). Pin with --spec-version to override.`). An unreachable server is graded as 2025-11-25 so every test that needs the server fails (the 2026-07-28 post-hoc scans fail too when nothing was received, rather than passing over an empty recording). Two budgets bound the probe. On **stdio** it is bounded by `--startup-timeout`, and in terminal mode a dim status line appears on stderr after ~2 s naming the wait and the `--spec-version 2025-11-25` skip, so a legacy server that stays silent on unknown methods does not look like a hang. If the child exits on the probe (a legacy server whose dispatcher throws on unknown pre-`initialize` methods), the report warns with the exit code and stderr tail, spawns a fresh instance and grades it as 2025-11-25. If the fresh instance exits at startup as well, before answering `initialize`, the warning says the server exits at startup regardless of the probe and points at the command, its arguments and its environment instead of suggesting a pin. `benchmark` re-spawns a probe-killed child the same way, bounds its probe by its own `--startup-timeout`, prints the same status line, and carries the warning in its output. On **HTTP** the probe is the preflight and is bounded by `--preflight-timeout`; a preflight that times out is re-probed once within `--startup-timeout` before the run defaults to 2025-11-25, so a slow cold start is graded in its real era instead of as unreachable. A server that answers both eras (the default for servers built on the official SDK 2.0) is graded as **2026-07-28**, and the report carries a warning naming the other era and how to pin it — keyed on `supportedVersions` listing 2025-11-25 *or* on the informational `lifecycle-dual-era` probe being served an `InitializeResult` (SDK 2.0 servers advertise only modern versions yet still serve `initialize`; on stdio that probe goes to a fresh process, since a dual-era server pins its era per process). To grade its legacy side too, run again with `--spec-version 2025-11-25`. A server that serves `initialize` while *rejecting* the conformant `server/discover` is not dual-era but legacy-only: `lifecycle-dual-era` says `legacy-only: ...` and the warning tells you to re-run with `--spec-version 2025-11-25` (or `auto`). The report's `specVersion` is always the **resolved** revision, never `auto`. A pinned run runs the named catalog whatever the server answers, with two honest side notes. On HTTP the preflight is still a modern `server/discover`, so a pinned run whose server plainly answered in the *other* era gets a warning (`Server answered the 2026-07-28 server/discover probe with a DiscoverResult ...; this run is pinned to 2025-11-25. Re-run with --spec-version 2026-07-28 (or auto) to grade it`). The warning needs an era signal -- a `DiscoverResult`, a JSON-RPC body or an HTTP 4xx; a 5xx or an HTML page says nothing about the era and draws none -- and a dual-era server whose `supportedVersions` lists 2025-11-25, pinned to 2025-11-25, is told `Server is dual-era (...); this run grades its 2025-11-25 side.` instead of being sent to the other pin. And a legacy-only server pinned to 2026-07-28 fails `lifecycle-discover` *and* the six `_meta` / standard-header rejection tests as **not evaluable**: a server that rejects the conformant `server/discover` proves nothing by also rejecting a malformed variant, so those required tests are failed rather than credited. Two things to know when you rely on `auto` in CI: - **The grade can move without a config change.** The day your server upgrades to an SDK that speaks 2026-07-28, `auto` switches suites: different test ids, different required set, a new baseline. Pin `--spec-version` (or `"specVersion"` in the config file) if you want the suite to change only when you say so. - **`diff` refuses to compare reports from different revisions** (ids are only comparable within one catalog) and tells you which `--spec-version` to pin. SARIF uploads are tracked per revision (`automationDetails.id` is `mcp-compliance//`), so a switch opens a fresh set of alerts rather than closing the old ones silently. `--only` / `--skip` values are matched against the catalog that was resolved; a value that matches nothing in it (say `lifecycle-init` on a 2026-07-28 run) is reported as a warning instead of silently producing an empty run, and an empty run prints `No tests ran -- check --only/--skip` rather than `All tests passed`. In the 2026-07-28 suite a filtered run still measures the server: the `tools/list` / `resources/list` / `prompts/list` results that a filtered-out feature test would have cached are fetched once on demand by whichever test needs them, so `--only security`, `--only schema` or `--only lifecycle` exercise the real checks instead of skip-passing. When such a list call fails and its `-list` test is filtered out, the checks that needed the list fail with the recorded reason (`tools/list failed (JSON-RPC error -32603 (boom)); no tools list to validate`) rather than grading A over a broken list. An `--only` value that matches only tests gated off the target transport (`transport-post` on a stdio server) is warned about too, and `--list` prints the same warnings under each catalog it lists. ### Options | Option | Applies to | Description | |--------|-----------|-------------| | `--spec-version ` | both | MCP spec revision to grade: `auto` (probe the server, newest wins), `2025-11-25`, or `2026-07-28` (default: `auto`, or `specVersion` in config). See [Spec version](#spec-version) | | `--format ` | both | Output format: `terminal`, `json`, `sarif`, `github`, `markdown`, or `html` (default: `terminal`) | | `--config ` | both | Load defaults from a config file (default: `mcp-compliance.config.json` in cwd) | | `--output ` | both | Write a local SVG badge to the given path after the run | | `--list` | both | Print test IDs that would run given current filters, then exit (no connection; with `--spec-version auto`, the default, every catalog is listed in its own section, and `--spec-version ` previews one) | | `--transport ` | both | Filter by `http` or `stdio` (only used with `--list` when no target is provided) | | `--strict` | both | Exit with code 1 on any required test failure (for CI) | | `--min-grade ` | both | Exit with code 1 if grade is below this threshold (`A`–`F`) | | `-H, --header ` | HTTP | Add header to all requests, format `"Key: Value"` (repeatable) | | `--auth ` | HTTP | Shorthand for `-H "Authorization: "` | | `-E, --env ` | stdio | Set env var for stdio command, format `"KEY=VALUE"` (repeatable) | | `--env-file ` | stdio | Load env vars from a file (one `KEY=VALUE` per line) | | `--cwd ` | stdio | Working directory for the stdio command | | `--watch` | stdio | Re-run the suite whenever a file under the current directory changes (500 ms debounce; ignores `node_modules`, `.git`, `dist`, `coverage`, build caches and log/swap/temp files) until Ctrl+C. stdio targets only, and only with `--format` `terminal`, `markdown` or `html`; `--strict` and `--min-grade` do not apply | | `--timeout ` | both | Per-request timeout in milliseconds after the initial exchange (default: `15000`) | | `--startup-timeout ` | both | Budget for the server's first reply: the stdio era probe under `auto`, the 2025-11-25 `initialize` handshake on either transport, and on HTTP the second era probe sent when the preflight times out (default: `max(--timeout, 60000)`; covers cold `npx` cache fetches before a stdio server starts) | | `--preflight-timeout ` | HTTP | Deadline for the preflight `server/discover` request, which under `auto` is also the era probe; a timeout here re-probes once within `--startup-timeout` before the run defaults to 2025-11-25 (default: `min(--timeout, 10000)`) | | `--retries ` | both | Number of retries for failed tests (default: `0`) | | `--only ` | both | Only run tests matching these categories or test IDs (comma-separated) | | `--skip ` | both | Skip tests matching these categories or test IDs (comma-separated) | | `--concurrency ` | both | Max parallel-safe tests in flight (default: `1`; raising reduces wall time but can perturb timing-sensitive servers) | | `--verbose` | both | Print each test result as it runs, as PASS, FAIL or SKIP (also forwards stdio stderr) | | `--no-color` | both | Disable colored output (the `NO_COLOR` environment variable does the same) | ### CI integration **GitHub Action** (drop into any `.github/workflows/*.yml`): ```yaml - uses: YawLabs/mcp-compliance@v0 with: target: 'node ./dist/server.js' # or a URL like https://my-server.com/mcp format: github # ::error per required failure, ::warning per optional failure and per report warning, ::notice summary with the spec version strict: 'true' # exit non-zero if any required test fails min-grade: 'A' # also exit if grade slips spec-version: auto # or pin 2025-11-25 / 2026-07-28; the resolved value is the `spec-version` output ``` **Manual CLI invocation:** ```bash # GitHub Actions: ::error per required failure, ::warning per optional failure and per # report warning, and a ::notice summary naming the grade and the spec version graded mcp-compliance test https://my-server.com/mcp --format github --strict # Slack/Linear/PR comment: drop the body straight into a comment mcp-compliance test https://my-server.com/mcp --format markdown > report.md # HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages) mcp-compliance test https://my-server.com/mcp --format html > report.html # Block release if grade slips below B mcp-compliance test https://my-server.com/mcp --min-grade B # Preview which tests will run before connecting (handy for --only/--skip authoring) mcp-compliance test --list --transport stdio --skip security --spec-version 2026-07-28 # Diff two runs — exit 1 if anything that was passing (or skipped) is now failing; # checks that start or stop skipping are listed as "Newly skipped" / "No longer skipped" and never fail the diff. # A baseline from an earlier version works too: its skips are read from their wording, and both # reports are scored from their results as this version scores them (skips left out). # Both reports must grade the same spec revision; pin --spec-version on both runs # so a server upgrade cannot flip the suite under the baseline. mcp-compliance test https://my-server.com/mcp --format json --spec-version 2026-07-28 > current.json mcp-compliance diff baseline.json current.json # Watch mode for stdio dev loop — re-runs on file changes in cwd mcp-compliance test --watch -- node ./dist/server.js # Latency benchmark (--spec-version and --startup-timeout work as for test) mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4 ``` **Docker:** ```bash docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcp ``` ### Scaffold a config ```bash mcp-compliance init ``` Interactive prompts walk you through transport (http/stdio), command/url, env vars, timeout, and strict mode — then write a `mcp-compliance.config.json` you can commit. ### Config file Check in a `mcp-compliance.config.json` so CI and your dev loop can run `mcp-compliance test` with no arguments. Supported locations (searched in order): `mcp-compliance.config.json`, `.mcp-compliancerc.json`, `.mcp-compliancerc`, and the `"mcp-compliance"` field of `package.json`. Pass `--config ` to load an explicit file. **HTTP:** ```json { "target": { "type": "http", "url": "https://my-server.com/mcp", "headers": { "Authorization": "Bearer tok123" } }, "timeout": 20000, "specVersion": "2026-07-28", "strict": true } ``` **stdio:** ```json { "target": { "type": "stdio", "command": "node", "args": ["./dist/server.js"], "env": { "LOG_LEVEL": "error" } }, "skip": ["security"], "strict": true } ``` Precedence: CLI flags > config file > defaults. Any field can be overridden on the command line. `specVersion` accepts `auto`, `2025-11-25` or `2026-07-28` and is new in 0.19 — a config that sets it is rejected as an unknown key by older tool versions (0.18.x included), so pin `@yawlabs/mcp-compliance@^0.19` wherever that config is consumed. ### Local SVG badge Write a local SVG reflecting the real grade and commit it to your repo: ```bash mcp-compliance test https://my-server.com/mcp --output badge.svg mcp-compliance test node ./dist/server.js --output badge.svg ``` Then embed it in your README: ```markdown ![MCP Compliance](./badge.svg) ``` ## What the 88 tests check (2025-11-25) The 2025-11-25 catalog grades the `initialize` + session era. It is what `--spec-version auto` selects for every server that does not answer a modern `server/discover`.
Transport (16 tests) HTTP-only (13): - **transport-post** — Server accepts HTTP POST requests (required) - **transport-content-type** — Responds with application/json or text/event-stream (required) - **transport-notification-202** — Notifications return exactly 202 Accepted - **transport-content-type-reject** — Rejects non-JSON request Content-Type - **transport-session-id** — Enforces MCP-Session-Id after initialization - **transport-session-invalid** — Returns 404 for unknown session ID - **transport-get** — GET returns SSE stream or 405 - **transport-delete** — DELETE accepted or returns 405 - **transport-batch-reject** — Rejects JSON-RPC batch requests (required) - **transport-content-type-init** — Initialize response has valid content type - **transport-get-stream** — GET with session returns SSE or 405 - **transport-concurrent** — Handles concurrent requests - **transport-sse-event-field** — SSE responses include required event: message field stdio-only (3): - **stdio-framing** — Newline-delimited JSON framing (required) - **stdio-unicode** — UTF-8 unicode roundtrip preserves non-ASCII payloads - **stdio-unknown-method-recovers** — Returns -32601 for unknown methods and keeps serving
Lifecycle (21 tests) - **lifecycle-init** — Initialize handshake succeeds (required) - **lifecycle-proto-version** — Returns valid YYYY-MM-DD protocol version (required) - **lifecycle-server-info** — Includes serverInfo with name - **lifecycle-capabilities** — Returns capabilities object (required) - **lifecycle-jsonrpc** — Response is valid JSON-RPC 2.0 (required) - **lifecycle-ping** — Responds to ping method (required) - **lifecycle-instructions** — Instructions field is valid string if present - **lifecycle-id-match** — Response ID matches request ID (required) - **lifecycle-string-id** — Supports string request IDs (JSON-RPC 2.0) - **lifecycle-version-negotiate** — Handles unknown protocol version gracefully - **lifecycle-reinit-reject** — Rejects second initialize request - **lifecycle-logging** — logging/setLevel accepted (required if logging capability declared) - **lifecycle-completions** — completion/complete accepted (required if completions capability declared) - **lifecycle-cancellation** — Handles cancellation notifications - **lifecycle-progress** — Handles progress notifications gracefully - **lifecycle-list-changed** — Accepts listChanged notifications for declared capabilities - **lifecycle-progress-token** — Supports progress tokens in requests via SSE - **lifecycle-sampling-capability** — Advisory check for server-side use of the client sampling capability - **lifecycle-roots-capability** — Advisory check for server-side use of the client roots capability - **lifecycle-elicitation-capability** — Advisory check for the 2025-11-25 client elicitation capability - **lifecycle-meta-tolerance** — Server ignores unknown `_meta` fields on incoming requests
Tools (4 tests) - **tools-list** — tools/list returns valid array (required if tools capability declared) - **tools-call** — tools/call responds with correct format - **tools-pagination** — tools/list supports cursor-based pagination - **tools-content-types** — Tool content items have valid types
Resources (5 tests) - **resources-list** — resources/list returns valid array (required if resources capability declared) - **resources-read** — resources/read returns content items - **resources-templates** — resources/templates/list works or returns method-not-found - **resources-pagination** — resources/list supports cursor-based pagination - **resources-subscribe** — Resource subscribe/unsubscribe (required if subscribe capability declared)
Prompts (3 tests) - **prompts-list** — prompts/list returns valid array (required if prompts capability declared) - **prompts-get** — prompts/get returns valid messages - **prompts-pagination** — prompts/list supports cursor-based pagination
Error Handling (10 tests) - **error-unknown-method** — Returns JSON-RPC error for unknown method (required) - **error-method-code** — Uses correct -32601 error code - **error-invalid-jsonrpc** — Handles malformed JSON-RPC (required) - **error-invalid-json** — Handles invalid JSON body - **error-missing-params** — Returns error for tools/call without name - **error-parse-code** — Returns -32700 for invalid JSON - **error-invalid-request-code** — Returns -32600 for invalid request - **tools-call-unknown** — Returns error for nonexistent tool name - **error-capability-gated** — Rejects methods for undeclared capabilities - **error-invalid-cursor** — Handles invalid pagination cursor gracefully
Schema Validation (6 tests) - **tools-schema** — All tools have valid name and inputSchema (required if tools capability declared) - **tools-annotations** — Tool annotations are valid if present - **tools-title-field** — Tools include title field (2025-11-25) - **tools-output-schema** — Tools with outputSchema are valid (2025-11-25) - **prompts-schema** — Prompts have valid name field (required if prompts capability declared) - **resources-schema** — Resources have valid uri and name (required if resources capability declared)
Security (23 tests) - **security-auth-required** — Rejects unauthenticated requests - **security-www-authenticate** — 401 responses include WWW-Authenticate header - **security-auth-malformed** — Rejects malformed auth credentials - **security-tls-required** — Enforces HTTPS/TLS - **security-session-entropy** — Session IDs are high-entropy - **security-session-not-auth** — Session ID does not bypass auth - **security-oauth-metadata** — Protected Resource Metadata endpoint exists (RFC 9728) - **security-token-in-uri** — Rejects auth tokens in query string - **security-cors-headers** — CORS headers are restrictive - **security-origin-validation** — Validates Origin header for DNS rebinding protection - **security-command-injection** — Resists command injection in tool params - **security-sql-injection** — Resists SQL injection in tool params - **security-path-traversal** — Resists path traversal in tool params - **security-ssrf-internal** — Resists SSRF to internal networks - **security-oversized-input** — Handles oversized inputs gracefully - **security-extra-params** — Rejects or ignores extra tool params - **security-tool-schema-defined** — All tools define inputSchema - **security-tool-rug-pull** — Tool definitions are stable across calls - **security-tool-description-poisoning** — Tool descriptions free of injection patterns - **security-tool-cross-reference** — Tools do not reference other tools by name - **security-error-no-stacktrace** — Error responses do not leak stack traces - **security-error-no-internal-ip** — Error responses do not leak internal IPs - **security-rate-limiting** — Rate limiting is enforced
## What the 103 tests check (2026-07-28) The 2026-07-28 catalog grades the stateless, per-request `_meta` era: `server/discover` instead of `initialize`, no sessions, caching hints on every cacheable result, `resultType` on every result, MRTR `input_required` results instead of server-initiated requests, and `-32602` for missing resources. An id shared with the 2025-11-25 list covers the same feature, but its criteria and required flag may differ between the catalogs (`stdio-framing` and `error-invalid-jsonrpc` are optional here, `error-method-code` is required), so ids are comparable only within one catalog; a check whose verdict flipped carries a new id, and `diff` refuses to compare reports across revisions. Post-hoc tests scan a recording of every message the server sent during the run (and fail, rather than pass vacuously, when nothing was received). The `_meta` and standard-header rejection tests are attributable: a rejection is credited only when the conformant `server/discover` was served, so a legacy-only server pinned to this catalog fails them as "not evaluable". The setup `server/discover` is resent once after a 429, and every rule that reads it judges the answer it settles on. The checks that credit any rejection — `transport-batch-reject`, `transport-content-type-reject`, `lifecycle-jsonrpc`, `lifecycle-subscriptions-listen`, and six error checks (`error-unknown-method`, `error-invalid-jsonrpc`, `error-invalid-json`, `error-missing-params`, `error-capability-gated`, `error-invalid-cursor`), plus the bodiless 4xx that `error-parse-code` and `error-invalid-request-code` accept — also ask whose answer it is: a gateway's 401 with a `-32001` body, a 403 that reads as an auth refusal (without `--auth` any Bearer challenge; with `--auth`, one carrying an `error` parameter), a 429 still a 429 after one resend, or any other 403 that a conformant twin sent next to the probe (a `server/discover`; for `error-invalid-cursor`, its list method without the cursor) could not get past either fails as "not evaluable", while a 5xx without the check's own code fails as the server failing on the request. `lifecycle-jsonrpc`'s probe is the conformant `server/discover` itself, so no twin is sent for it: any 403 on it, and a 5xx without a server's own refusal code, is "not evaluable". `transport-batch-reject` and `transport-content-type-reject` also fail any rejection as "not evaluable" when the setup `server/discover` was itself rejected. `tools-call-unknown` is not read this way yet: a gateway's `-32001` on its `tools/call` still passes it. The four injection tests probe a single (tool, argument) target from the safest annotation tier that has a string argument — `readOnlyHint: true`, then `destructiveHint: false`, then unannotated tools (the spec defaults `destructiveHint` to true), then `destructiveHint: true`, with passed-over and live-probed tools named in warnings — fill the other required arguments with schema-honouring placeholders, and report how many payloads were rejected, returned without evidence of execution, or never reached the tool (all never-reached is inconclusive, with a warning, and flagged as a skip). On stdio a check whose own request kills the server (an injection payload, the 1 MB argument, unknown tool arguments, `stdio-unicode`'s CJK/emoji probe, one of `lifecycle-progress-token`'s calls) has it restarted for the checks after it, every time that happens, `--retries` included — the security checks and `stdio-unicode` fail as died; `security-tool-rug-pull` then compares two lists from the new process. `stdio-unicode` and `lifecycle-progress-token` also charge a child that answers and exits right after (a plain `server/discover` follows the answer), and `stdio-unicode` fails one that answers and then stops answering. `security-rate-limiting` bursts 50 `tools/call`s at a read-only, no-argument tool when there is one and passes a quiet burst with a warning either way.
Transport (20 tests) HTTP-only (15): - **transport-post** — HTTP POST accepted (required) - **transport-content-type** — Responds with JSON or SSE (required) - **transport-content-type-reject** — Rejects non-JSON request Content-Type - **transport-batch-reject** — Rejects JSON-RPC batch requests (required) - **transport-notification-202** — Notification returns 202 Accepted - **transport-concurrent** — Handles concurrent requests - **transport-get-removed** — GET returns 405 - **transport-delete-removed** — DELETE returns 405 - **transport-session-ignored** — Ignores Mcp-Session-Id - **transport-header-version-required** — Rejects missing MCP-Protocol-Version header (required) - **transport-header-version-mismatch** — Rejects header/_meta version mismatch (required) - **transport-header-method-required** — Rejects missing Mcp-Method header (required) - **transport-header-method-mismatch** — Rejects Mcp-Method/body mismatch (required) - **transport-header-name-mismatch** — Rejects Mcp-Name/body mismatch - **transport-header-case-insensitive** — Header names are case-insensitive Both transports (1, post-hoc): - **transport-no-server-requests** — No server-initiated requests on any stream (required) stdio-only (4): - **stdio-framing** — Newline-delimited JSON framing - **stdio-unicode** — UTF-8 unicode roundtrip - **stdio-unknown-method-recovers** — Recovers after unknown method - **stdio-cancellation** — Ignores cancellation of unknown request
Lifecycle (22 tests) - **lifecycle-discover** — server/discover returns DiscoverResult (required) - **lifecycle-discover-versions** — supportedVersions are well-formed (required) - **lifecycle-discover-caching** — server/discover carries caching hints (required) - **lifecycle-jsonrpc** — Response is valid JSON-RPC 2.0 (required) - **lifecycle-id-match** — Response id matches request id (required) - **lifecycle-string-id** — Supports string request ids - **lifecycle-capabilities** — Returns capabilities object (required) - **lifecycle-server-info** — Includes serverInfo in _meta - **lifecycle-instructions** — Instructions field is valid - **lifecycle-meta-required** — Rejects request without _meta (required) - **lifecycle-meta-protocol-version-required** — Rejects _meta without protocolVersion (required) - **lifecycle-meta-client-capabilities-required** — Rejects _meta without clientCapabilities (required) - **lifecycle-meta-client-info-optional** — Serves _meta without clientInfo (required) - **lifecycle-version-unsupported** — Rejects unsupported protocol version (required) - **lifecycle-removed-methods** — Removed legacy methods are rejected - **lifecycle-dual-era** — Legacy initialize probe (informational) - **lifecycle-capability-handlers-match** — Capability declarations match handlers - **lifecycle-subscriptions-listen** — subscriptions/listen acknowledges first - **lifecycle-log-level-gating** — No log notifications without logLevel (required) - **lifecycle-meta-tolerance** — Tolerates unknown _meta keys - **lifecycle-completions** — completion/complete accepted (required if completions capability declared) - **lifecycle-progress-token** — Progress notifications echo the token
Tools (6 tests) - **tools-list** — tools/list returns valid response (required if tools capability declared) - **tools-list-caching** — tools/list carries caching hints (required if tools capability declared) - **tools-list-deterministic-order** — tools/list order is deterministic - **tools-call** — tools/call responds correctly (required if tools capability declared) - **tools-content-types** — Tool content items have valid types (required if tools capability declared) - **tools-pagination** — tools/list supports pagination
Resources (8 tests) - **resources-list** — resources/list returns valid response (required if resources capability declared) - **resources-list-caching** — resources/list carries caching hints (required if resources capability declared) - **resources-read** — resources/read returns content (required if resources capability declared) - **resources-read-caching** — resources/read carries caching hints (required if resources capability declared) - **resources-not-found** — Nonexistent resource returns -32602 (required if resources capability declared) - **resources-templates** — resources/templates/list returns valid response - **resources-templates-caching** — resources/templates/list carries caching hints - **resources-pagination** — resources/list supports pagination
Prompts (4 tests) - **prompts-list** — prompts/list returns valid response (required if prompts capability declared) - **prompts-list-caching** — prompts/list carries caching hints (required if prompts capability declared) - **prompts-get** — prompts/get returns valid messages (required if prompts capability declared) - **prompts-pagination** — prompts/list supports pagination
Error Handling (12 tests) - **error-unknown-method** — Unknown method returns JSON-RPC error (required) - **error-method-code** — Unknown method uses -32601 (required) - **error-invalid-jsonrpc** — Handles malformed JSON-RPC - **error-invalid-json** — Handles invalid JSON body - **error-parse-code** — Returns -32700 for invalid JSON - **error-invalid-request-code** — Returns -32600 for invalid request - **error-missing-params** — tools/call without name returns error - **tools-call-unknown** — Unknown tool name returns error - **error-capability-gated** — Rejects methods for undeclared capabilities - **error-invalid-cursor** — Handles invalid pagination cursor - **error-id-echo** — Error responses echo the request id - **error-retired-codes** — No retired error codes
Schema Validation (10 tests) - **tools-schema** — All tools have name and inputSchema - **tools-annotations** — Tool annotations are valid - **tools-title-field** — Tools include title field - **tools-output-schema** — Tools with outputSchema are valid - **prompts-schema** — Prompts have name field - **resources-schema** — Resources have uri and name - **schema-result-type** — Every result carries resultType (required) - **schema-no-input-required-on-lists** — input_required only on MRTR methods (required) - **schema-input-required-shape** — InputRequiredResult is well-formed - **schema-wire-valid** — Messages validate against the 2026-07-28 schema
Security (21 tests) - **security-auth-required** — Rejects unauthenticated requests - **security-www-authenticate** — 401 responses include WWW-Authenticate - **security-auth-malformed** — Rejects malformed auth credentials - **security-tls-required** — Enforces HTTPS/TLS - **security-oauth-metadata** — Protected Resource Metadata endpoint exists - **security-token-in-uri** — Rejects auth tokens in query string - **security-cors-headers** — CORS headers are restrictive - **security-origin-validation** — Validates Origin header - **security-command-injection** — Resists command injection in tool params - **security-sql-injection** — Resists SQL injection in tool params - **security-path-traversal** — Resists path traversal in tool params - **security-ssrf-internal** — Resists SSRF to internal networks - **security-oversized-input** — Handles oversized inputs gracefully - **security-extra-params** — Rejects or ignores extra tool params - **security-tool-schema-defined** — All tools define inputSchema - **security-tool-rug-pull** — Tool definitions are stable across calls - **security-tool-description-poisoning** — Tool descriptions free of injection patterns - **security-tool-cross-reference** — Tools do not reference other tools by name - **security-error-no-stacktrace** — Error responses do not leak stack traces - **security-error-no-internal-ip** — Error responses do not leak internal IPs - **security-rate-limiting** — Rate limiting is enforced
## Grading | Grade | Score | |-------|--------| | A | 90-100 | | B | 75-89 | | C | 60-74 | | D | 40-59 | | F | 0-39 | Required tests are worth 70% of the score, optional tests 30%, counted over the tests that measured something. A test that measured nothing (a skip) is left out of the score entirely, neither a pass nor a failure; when every required or every optional test skipped, the other pool carries the whole score, and a run in which every test skipped scores 0 (F) like a run in which none ran. The terminal, markdown, HTML and GitHub reports, the SARIF invocation properties (`note`) and the MCP `mcp_compliance_test` tool then say "No test measured anything"; the JSON report shows it in its counts (`summary.skipped` equals `summary.total`). Skips still count toward the pass totals (`summary.passed`, the category counts); every report format counts or lists them (the terminal lists them under `SKIPPED CHECKS`), the terminal category bars and HTML category cards score what was measured, `--verbose` prints skips as `SKIP`, and `diff` lists checks that start or stop skipping. Reports written by 0.19.0 and earlier scored a skip as a pass, so their scores run higher for the same results. By how much depends on the run: on the reference servers no rounded score moved, but the drop grows with a pool's failures times its skips, enough to cost a server whose required checks all pass a whole grade, and a run where most checks measured nothing can drop several. `diff` scores both reports from their results the way this version does, so a baseline written by 0.19.0 or earlier still compares like with like, with a note naming the score that file recorded. See the [full scoring algorithm](https://github.com/YawLabs/mcp-compliance/blob/HEAD/COMPLIANCE_RUBRIC.md#2-scoring-algorithm) in the methodology doc. ## CI integration ```yaml # GitHub Actions example - name: MCP Compliance Check run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --strict ``` ```yaml # With JSON output for parsing - name: MCP Compliance Check run: | npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --format json > compliance.json cat compliance.json | jq '.grade' ``` ```yaml # With retries for flaky network conditions - name: MCP Compliance Check run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000 ``` ```yaml # SARIF output for GitHub Code Scanning - name: MCP Compliance Check run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif - name: Upload SARIF uses: github/codeql-action/upload-sarif@v3 with: sarif_file: compliance.sarif ``` ## MCP server (for Claude Code, Cursor, etc.) This package also exposes an MCP server with 2 tools that can be used from Claude Code, Cursor, or any MCP client. ### Setup **Claude Code (one-liner):** ```bash claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance@latest mcp ``` **Or create `.mcp.json` in your project root:** macOS / Linux / WSL: ```json { "mcpServers": { "mcp-compliance": { "command": "npx", "args": ["-y", "@yawlabs/mcp-compliance@latest", "mcp"] } } } ``` Windows: ```json { "mcpServers": { "mcp-compliance": { "command": "cmd", "args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance@latest", "mcp"] } } } ``` > **Tip:** This file is safe to commit — it contains no secrets. Restart your MCP client and approve the server when prompted. ### Tools - **mcp_compliance_test** — Run the full compliance suite against an HTTP MCP endpoint (URL). Supports auth, custom headers, timeout, retries, category/test filtering, and `specVersion` (`auto` | `2025-11-25` | `2026-07-28`, default `auto`). Returns grade, score, the resolved `specVersion`, and detailed results (each test marked PASS, FAIL or SKIP; the Tests line counts the skips). stdio targets are CLI-only. - **mcp_compliance_explain** — Explain what a specific test ID checks and why it matters, with a link to the spec section. Ids are per catalog: pass `specVersion` to read one catalog, or omit it to search both (a shared id is explained once per suite, with a note saying whether the required flag / category also differ, and an id that only exists in the other suite is pointed out). All tools have [MCP tool annotations](https://modelcontextprotocol.io/specification/2025-11-25/server/tools#annotations) (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) so MCP clients can skip confirmation dialogs for safe operations. ## Programmatic usage ```typescript import { runComplianceSuite } from '@yawlabs/mcp-compliance'; const report = await runComplianceSuite('https://my-server.com/mcp'); console.log(`Grade: ${report.grade} (${report.score}%)`); // With options const report2 = await runComplianceSuite('https://my-server.com/mcp', { headers: { 'Authorization': 'Bearer tok123' }, timeout: 30000, retries: 1, only: ['transport', 'lifecycle'], specVersion: '2026-07-28', // 'auto' (default) | '2025-11-25' | '2026-07-28' }); console.log(report2.specVersion); // always the resolved revision, never 'auto' // Per-revision catalogs and helpers import { getTestDefinitions, specBaseFor, SUPPORTED_SPEC_VERSIONS } from '@yawlabs/mcp-compliance'; getTestDefinitions('2026-07-28'); // 103 TestDefinitions; TEST_DEFINITIONS stays the 2025-11-25 list specBaseFor(report2.specVersion); // base URL for that revision's spec links // Live progress for streaming UIs (e.g. server-sent-events to a browser) await runComplianceSuite('https://my-server.com/mcp', { onTestComplete: (result) => { // result has the full TestResult: id, name, category, required, // passed, skipped (true when the check measured nothing), details, // durationMs, specRef. Push it to your client. (onProgress carries no // skip flag: a skip arrives there as passed.) sendToClient(result); }, // Pre-test status lines: the stdio era probe still waiting after ~2 s, // or an HTTP preflight timeout being re-probed within startupTimeout. // The CLI prints these dimmed on stderr in terminal mode only. onStatus: (message) => sendToClient({ status: message }), }); ``` ## Report schema The JSON output of the test suite is a stable, versioned contract. Every report includes a `schemaVersion` field at the top level. The full JSON Schema lives at [`schemas/report.v1.json`](https://github.com/YawLabs/mcp-compliance/blob/HEAD/schemas/report.v1.json) and is shipped with the npm package. ```jsonc { "schemaVersion": "1", // bumped on breaking changes to the report shape "specVersion": "2026-07-28", // MCP spec revision the run graded — the RESOLVED value ("2025-11-25" or "2026-07-28"), never "auto" "toolVersion": "0.19.0", // mcp-compliance version that produced the report "url": "...", "timestamp": "...", "grade": "A", "score": 92.5, "summary": { "total": 99, "passed": 97, "failed": 2, "required": 24, "requiredPassed": 24, "skipped": 4 }, "tests": [ // ids are only meaningful together with specVersion { "id": "security-auth-malformed", "passed": true, "skipped": true, "details": "Skipped: needs a valid credential to compare against (pass --auth)", ... }, ... ], // ... } ``` A test that ran but measured nothing (no `--auth`, no tools declared, a check that does not apply on the transport, or a refusal an earlier check could not attribute) is recorded as `passed: true` with `skipped: true`; `summary.skipped` and each `categories[*].skipped` count them. Skips count in `passed` but are left out of `score` (reports written by 0.19.0 and earlier scored them as passes), so read `skipped` next to `passed` before treating a pass total as evidence. SARIF output keeps its `results` to failures and names the skips in the run's invocation properties (`testsSkipped`, `skippedTests`), so Code Scanning opens no alert for them. Consumer guidance: - Pin against `schemaVersion`. Reject reports with an unknown version rather than guessing at the shape. - Check `report.specVersion` before interpreting `tests`. Since 0.19 **one tool version produces reports for either revision**, so a dashboard that keyed on `toolVersion` alone will see `2025-11-25` and `2026-07-28` reports from the same version; test ids, the required set and `serverInfo.protocolVersion` all belong to the revision named there. Treat an unknown `specVersion` the way you treat an unknown `schemaVersion`: reject, do not guess. - The schema validates with any Draft 2020-12 validator (e.g. `ajv`). It is strict (`additionalProperties: false`), so validate a report against the `report.v1.json` shipped with the tool version that wrote it: 0.19 added optional `skipped` fields (on a test, in `summary` and in `categories`) to v1, and an older copy of the schema rejects them. - Renames, removals, or type changes bump `schemaVersion`; the report shape is otherwise stable within a major version. - Two runs against the same server produce equivalent grade, score, and per-test pass/fail (modulo timings/timestamps) — provided they resolve to the same `specVersion`. Under `auto` that is a property of the server, so pin `--spec-version` where determinism across deploys matters. ## Methodology & docs The testing methodology is published openly so the grading is auditable: - **[Testing methodology](https://github.com/YawLabs/mcp-compliance/blob/HEAD/COMPLIANCE_RUBRIC.md)** — test execution model per spec revision, scoring algorithm, all 88 + 103 test rules with pass/fail criteria (CC BY 4.0) - **[Machine-readable rule catalog](https://github.com/YawLabs/mcp-compliance/blob/HEAD/mcp-compliance-rules.json)** — every rule tagged with the spec revision it applies to, for programmatic consumption (kept in lock-step with the code by `src/tests/catalog-parity.test.ts`) - **[Why `mcp-compliance`](https://github.com/YawLabs/mcp-compliance/blob/HEAD/docs/WHY.md)** — the problem, existing alternatives, what this tool does differently - **[Fixing common failures](https://github.com/YawLabs/mcp-compliance/blob/HEAD/docs/FIXES.md)** — recipes for the most frequent test failures with code snippets, for both revisions - **[Spec version policy](https://github.com/YawLabs/mcp-compliance/blob/HEAD/docs/SPEC_VERSION_MIGRATION.md)** — one tool version, several spec catalogs: how `auto` decides, how to pin, what changes when your server upgrades - **[Performance deep-dive](https://github.com/YawLabs/mcp-compliance/blob/HEAD/docs/PERFORMANCE.md)** — why the suite is sequential and what parallel execution would cost - **[Spec PR drafts](https://github.com/YawLabs/mcp-compliance/blob/HEAD/docs/spec-prs/)** — our proposed MCP spec clarifications for ambiguous cases we've hit The methodology is not an authoritative conformance standard — it's one tool's choices, published so they can be inspected, adopted, or forked. The official MCP specification ([2025-11-25](https://modelcontextprotocol.io/specification/2025-11-25), [2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28)) defines what servers must do; this document describes how `@yawlabs/mcp-compliance` verifies it. ## Requirements - Node.js 20+ ## Contributing ```bash git clone https://github.com/YawLabs/mcp-compliance.git cd mcp-compliance npm install npm run build npm test ``` **Development commands:** | Command | Description | |---------|-------------| | `npm run build` | Compile with tsup | | `npm run dev` | Watch mode | | `npm test` | Run tests (vitest) | | `npm run lint` | Check with Biome | | `npm run lint:fix` | Auto-fix with Biome | | `npm run typecheck` | TypeScript type checking | | `npm run test:ci` | Build + test (CI-safe) | ## Links - [MCP Specification 2025-11-25](https://modelcontextprotocol.io/specification/2025-11-25) - [MCP Specification 2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28) - [Testing methodology](https://github.com/YawLabs/mcp-compliance/blob/HEAD/COMPLIANCE_RUBRIC.md) - [Yaw Labs](https://yaw.sh) ## License MIT — see [LICENSE](https://github.com/YawLabs/mcp-compliance/blob/HEAD/LICENSE). [![Follow @YawLabs on X](https://img.shields.io/badge/follow-%40YawLabs-000000?logo=x&logoColor=white)](https://x.com/YawLabs)