Log inSubmit server
Crawl Readiness logo

Crawl Readiness

Audits AI crawler access, validates site signals, checks usage permissions, and generates llms.txt, robots.txt, and JSON-LD.

InstallRepositoryWebsite
Full README Compare alternatives Embed widget Claim this listing

Install Crawl Readiness

Pick your client, then add it in one click or paste the config.

STDIONode.js
~/Library/Application Support/Claude/claude_desktop_config.json
claude_desktop_config.json
{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "YOUR_VALUE_HERE",
        "CRAWL_READINESS_API_BASE": "YOUR_VALUE_HERE"
      }
    }
  }
}

Settings → Developer → Edit Config, merge this into mcpServers, then fully quit and reopen Claude Desktop.

Values are written into the snippet above so you can paste it as is. They never leave your browser.

Automated check passed— started and listed 9 tools correctly (9d ago).

Overview

crawl-readiness MCP server connects AI clients to the Crawl Readiness API for live website audits covering 50+ AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot. Its tools inspect robots.txt, JSON-LD, content parity, and published usage signals, then return scores, permissions, findings, and prioritized fixes. With an API key, it can generate llms.txt, AI-aware robots.txt, and schema markup, while also reading eligible Answer Analysis trends. Reach for it when you need to assess how AI systems can access or use a site and turn the findings into project-ready files.

Use cases

  • Audit a website against major AI crawlers
  • Validate robots.txt and JSON-LD before deployment
  • Check whether a page permits training or agent use
  • Generate llms.txt and AI-aware robots.txt
  • Compare human-visible content with crawler responses

Key features

  • 50+ AI crawler readiness scoring
  • Robots.txt and JSON-LD validation
  • Content parity analysis
  • Signed permission receipts
  • llms.txt, robots.txt, and schema generation
  • Answer Analysis trend reporting

Tools9

What Crawl Readiness lets your agent do. Expand a tool for its parameters.

Verified live~2.3k tokens of context

Security & health

What it can reach and how reliably it runs — from our own checks, not the maintainer’s claims.

ActiveLast checked 2d ago

Recent health check succeeded.

9/9 checks healthy over the last 44d

What it can do

From the 9 tools this server returned when we ran it; 9 declare their own safety hints.

Handles credentials · 5Makes network requests · 9Read-only tools · 9
Network at startup3 connections
Files written at startupNone
Install scriptsNone
npm provenanceNot published
Package age45 days
Maintainers1

Observed 2026-11-02 in an isolated sandbox. A description of reach, not a security audit.

Supply-chain signal

No high-severity advisories surfaced by our automated scan.

Critical 0High 0Medium 0Low 0

Scanned 14d ago via OSV.dev · crawl-readiness-mcp (npm)

Adoption & maintenance

Factual signals from GitHub, npm, and our automated checks — not a rating.

npm downloads1.5kPackage downloads in the last 30 days.
Last commit13d agoMost recent push to the default branch.
Availability100%Our rolling endpoint + install checks that succeeded.
Install checkPassedOur sandbox started it and listed its tools.
Tools exposed9Callable tools this server registers over MCP.

How Crawl Readiness works

What crawl-readiness does

crawl-readiness evaluates whether a website publishes signals that match its intended AI access policy. It can score a site from 0 to 100 against more than 50 AI crawlers, inspect crawler-specific robots.txt rules, validate JSON-LD, and compare the content delivered to people with the content seen by AI crawlers.

The permission check answers whether a page may be used for search, answer generation, training, or agent action. It considers robots.txt, Content-Signal, Content-Usage, an RSL license, the TDM reservation, ai.txt, and page directives. Results distinguish denied, allowed-under-terms, allowed, and unknown outcomes. A signed Ed25519 receipt includes the signals read and hashes of the files examined.

How it works

The server runs as an MCP process and exposes nine tools to compatible AI clients. Audit and permission checks can be used without an account. Generator features require an API key, and the Answer Analysis trend tool requires a key on a plan that includes those analyses.

Each tool sends HTTPS requests to the Crawl Readiness service. Permission checks also use machinereadiness.com. The service receives the target website address, generator inputs when applicable, and the configured API key. The local server does not store data, execute other programs, or read local files. If a generated file should be added to a project, the AI client performs that write after receiving user approval.

Setup and configuration

Claude Desktop can use the downloadable MCP bundle from the latest GitHub release, or the server can be registered manually with npx -y crawl-readiness-mcp. Cursor, Cline, Windsurf, and Continue use the same stdio command in their MCP configuration. Claude Code can load the repository as a plugin with claude --plugin-dir ./crawl-readiness-mcp.

The optional CRAWL_READINESS_API_KEY enables the three generator tools, removes the daily limit on permission checks, and permits access to eligible Answer Analyses. CRAWL_READINESS_API_BASE can replace the default API endpoint, which is useful for testing. Restart the MCP client after changing environment settings.

Tools and capabilities

  • check_ai_readiness scores a site, reports crawler-by-crawler access, and ranks fixes.
  • validate_schema checks JSON-LD blocks across more than 20 schema types.
  • validate_robots reports robots.txt problems line by line and summarizes AI-bot coverage.
  • check_content_parity compares human and crawler responses, including status codes and word overlap.
  • check_permission evaluates published usage permissions and returns a signed receipt.
  • generate_llms_txt crawls a site and returns formatted llms.txt content plus a robots.txt snippet.
  • generate_robots_txt produces robots.txt with an applied AI access policy.
  • generate_schema returns JSON-LD for detected Organization, WebSite, or Article types.
  • get_monitor_trend reads Answer Analysis results, including brand share, provider breakdowns, competitors, trends, and sample answers.

Limitations and notes

The free, no-signup checks cover audits and permission evaluation, while generation and Answer Analysis access depend on the API key and applicable account plan. The MCP process does not write files itself; the connected AI client must save generated content to the project. Permission results can be unknown when a source cannot be read, rather than treating that source as permission to proceed.

Read the full README View source on GitHub

Reviews

No reviews yet — be the first to share how it worked for you.

Frequently Asked Questions about Crawl Readiness

crawl-readiness is an MCP server that connects AI clients to the Crawl Readiness and Machine Readiness services. Its main tools audit websites against 50+ AI crawlers, validate robots.txt and JSON-LD, compare crawler content, check permission signals, generate llms.txt and policy files, and read eligible Answer Analysis trends.

Install it by adding `npx -y crawl-readiness-mcp` to your MCP client configuration. Claude Desktop also supports the downloadable `.mcpb` bundle, and Claude Code can load the repository with `claude --plugin-dir ./crawl-readiness-mcp`.

Yes, crawl-readiness works with Cursor through its global Model Context Protocol settings. Add the server with the `npx -y crawl-readiness-mcp` command, then restart or reload the client so its tools appear.

No, the audit tools and permission check work without an API key. Add `CRAWL_READINESS_API_KEY` to use the three generator tools, lift the permission check's daily limit, and read Answer Analysis data on an eligible plan.

Yes, `generate_llms_txt` crawls a site and returns formatted llms.txt content along with a companion robots.txt snippet. An API key is required, and the connected AI client can save the returned content into your project with your approval.

Related MCP servers

All in Developer Tools All alternatives
Add a badge to your READMEShow it’s listed — live SVG, Markdown or HTML.

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:
Live Dynamic SVG PreviewCrawl Readiness AllMCPs Directory Badge
Markdown (GitHub README)
[![AllMCPs](https://allmcps.com/api/badge/crawl-readiness?style=directory)](https://allmcps.com/mcp/crawl-readiness)
HTML Embed
<a href="https://allmcps.com/mcp/crawl-readiness"><img src="https://allmcps.com/api/badge/crawl-readiness?style=directory" alt="Crawl Readiness on AllMCPs" /></a>