@yawlabs/mcp-compliance

One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.

Test any MCP server for spec compliance. Two test suites — 88 tests for MCP 2025-11-25 and 103 tests for MCP 2026-07-28 — covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. The tool probes the server and grades the newest spec revision it speaks, or you pin one with --spec-version. Works against HTTP endpoints (https://my-server.com/mcp) and stdio servers (npx @modelcontextprotocol/server-filesystem /tmp) alike. CLI, MCP server, and programmatic API.

Built and maintained by Yaw Labs.

Why this tool?

MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.

This tool solves that:

• Two spec revisions, one tool — 88 tests for MCP 2025-11-25 (the initialize + session era) and 103 tests for MCP 2026-07-28 (stateless, per-request _meta, server/discover, caching hints, MRTR). Both cover the same 8 categories: transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. HTTP-specific tests (CORS, TLS, header validation, rate limiting) are gated out on stdio: the 2025-11-25 suite runs 85 tests on HTTP and 55 on stdio, the 2026-07-28 suite 99 and 75.
• Auto-detected revision — one server/discover probe tells the tool which era the server speaks; dual-era servers are graded against 2026-07-28 with a warning. Pin either revision with --spec-version. See Spec version.
• Capability-driven — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
• Graded scoring — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
• CI-ready — --strict mode exits with code 1 on required test failures. Drop it into any pipeline.
• Spec-referenced — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
• Three interfaces — CLI for humans, MCP server for AI assistants, programmatic API for integration.
• Published methodology — the testing methodology and rule catalog are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.

Quick start

Remote HTTP server:

npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp

Local stdio server (the vast majority of MCP servers on npm):

# Pass the command directly, Inspector-style
npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp

# Or a local build
npx @yawlabs/mcp-compliance@latest test node ./dist/server.js

# With env vars
npx @yawlabs/mcp-compliance@latest test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github

Install globally:

npm install -g @yawlabs/mcp-compliance
mcp-compliance test https://my-server.com/mcp

That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.

CLI usage

HTTP targets

# Terminal output with colors and grade
mcp-compliance test https://my-server.com/mcp

# JSON / SARIF for scripting + GitHub Code Scanning
mcp-compliance test https://my-server.com/mcp --format json
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif

# Strict mode for CI — exits 1 on required-test failure
mcp-compliance test https://my-server.com/mcp --strict

# Auth (shorthand or full header)
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"

# Focus the run
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
mcp-compliance test https://my-server.com/mcp --verbose

stdio targets

Pass the command and its args as positional arguments (MCP Inspector-style). Use -- to disambiguate when the target needs flags that collide with ours.

# npm-distributed stdio servers
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-compliance test uvx mcp-server-git

# Local build
mcp-compliance test node ./dist/server.js

# With env vars (repeatable -E, or --env-file)
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
mcp-compliance test --env-file .env -- node ./server.js

# Set working directory
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js

# Target uses a flag that collides with ours — use `--` to separate
mcp-compliance test --verbose -- node ./server.js --verbose

On Windows, npx and other .cmd shims are handled automatically by spawning through the shell.

Spec version

The tool ships one test catalog per MCP specification revision it supports — 2025-11-25 (88 tests: initialize handshake, sessions, ping) and 2026-07-28 (103 tests: stateless, per-request _meta, server/discover, caching hints, MRTR). A run grades exactly one revision, chosen by --spec-version:

# auto (default): probe the server, grade the newest revision it speaks
mcp-compliance test https://my-server.com/mcp

# pin a revision — no probe, the named catalog runs regardless of what the server answers
mcp-compliance test https://my-server.com/mcp --spec-version 2026-07-28
mcp-compliance test https://my-server.com/mcp --spec-version 2025-11-25

# --list never connects: under auto (the default) it prints every catalog in its own
# section; --spec-version <date> previews just that one
mcp-compliance test --list --spec-version 2026-07-28 --transport stdio

How auto decides, following the spec's own rules for dual-era clients: the tool sends one conformant 2026-07-28 server/discover (on HTTP it doubles as the preflight connectivity check, so detection costs no extra round-trip; on stdio it is the first exchange). A DiscoverResult, or a JSON-RPC error with a modern code (-32020, -32021, -32022), selects 2026-07-28. Anything else — -32601, -32000, a 400 "not initialized", a 404, an HTML page, or no reply within the timeout — selects 2025-11-25; a 401/403 with no modern error body also lands on 2025-11-25, the note says the era could not be determined, and the report's first warning (in every output format) says the grade below is not meaningful. With no Authorization header configured, a 401, or a 403 carrying a WWW-Authenticate: Bearer challenge, reads as authentication required (note authentication required -- pass --auth; the warning says to re-run with --auth). With one configured, a 401, or a 403 whose Bearer challenge carries an error parameter, reads as the credential rejected (note credential rejected -- check --auth; the warning names the reason from that error, e.g. invalid_token is an invalid or expired token and insufficient_scope a missing scope or permission). Any other 403 (the SDK's Host validation behind a tunnel hostname, Origin validation, a gateway) is worded neutrally with or without a header: the note says forbidden -- Host/Origin validation, a gateway, or missing credentials (or ... or token permissions with a header), and the warning quotes the body's JSON-RPC error message when there is one, points at Host/Origin validation and any gateway first, and without a header suggests --auth only if the server does require a credential. The fallback is deliberately not keyed to a single error code; a legacy server that ignores unknown methods is still classified correctly, just after the timeout. The report header names the reason (auto-detected from server/discover: supportedVersions [2026-07-28], ... JSON-RPC error -32601, legacy, ... no response, legacy) and the JSON warnings carry the same note (Spec version auto-detected as <v> (server/discover -> <reason>). Pin with --spec-version to override.). An unreachable server is graded as 2025-11-25 so every test that needs the server fails (the 2026-07-28 post-hoc scans fail too when nothing was received, rather than passing over an empty recording).