# crawl-readiness [Health: Active]

**Category:** 💻 Developer Tools  
**Repository:** https://github.com/Grubbomatic/crawl-readiness-mcp  
**GitHub Stars:** 0  
**npm Downloads (last month):** 1518  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/crawl-readiness

## Description
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.

## Tools
Capabilities this server exposes over MCP:

- **check_ai_readiness** — Check whether AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and 50+ others) can access a website, and what access policy it states. Returns a 0-100 AI readiness score, per-crawler access status, detected AI-specific files (llms.txt, agents.json), structured data presence, meta signals, and a prioritized fix list. The score counts the crawlers that feed AI search and answers: each entry in `robots.results` carries its `purpose` and whether it is `scored`, and training-only crawlers are listed but never counted, so blocking them does not lower the score. Fixes marked `extra: true` (llms.txt, agents.json, Content-Signal, a markdown copy, agent-ready forms) are informational 'Going further' items and do not affect the score; present them as optional. Use this as the first step whenever a user asks whether AI can read a site.
- **validate_schema** — Validate all JSON-LD structured data on a URL. Extracts every <script type="application/ld+json"> block, runs each through a rules engine covering 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, Recipe, Event, etc.), and reports required-field errors, recommended-field warnings, and type-specific gotchas.
- **validate_robots** — Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.
- **check_content_parity** — Compare what human browsers see vs what AI crawlers see. Fetches a page four times in parallel — as Chrome, GPTBot, ClaudeBot, and PerplexityBot — and reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.
- **generate_llms_txt** — Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.
- **generate_robots_txt** — Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in `generated.robotsTxt` — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (every AI crawler allowed), 'recommended' (blocks only the training-only crawlers; everything that feeds AI search and answers stays allowed), 'search-only' (allows only the crawlers that feed AI search and answers, the ones the readiness score counts, and blocks the rest), 'block-all'. A file from 'recommended' or 'search-only' loses no points for crawler access. Write `generated.robotsTxt` to the site's /robots.txt. Requires an API key.
- **generate_schema** — Generate JSON-LD structured data for a website. Detects the site's name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article schemas plus starter templates for BreadcrumbList and FAQPage. Returns each schema as a <script type="application/ld+json"> block ready to paste into the site's <head>. Requires an API key.
- **get_monitor_trend** — Read the user's Answer Analysis: for the questions a brand's customers ask, how often live AI search (Perplexity, Google AI Overviews) named the brand, which competitors were named, and the change run over run. Call with NO argument to list the user's analyses with each brand's latest share of answers naming it and the direction; pass a brand name or project id to get that brand's full trend, per-provider breakdown, competitor comparison, average position, and short example answers. Reads data existing analyses have already collected — it does not trigger new runs, and it measures rather than promises. Read-only. Requires an API key on a plan that includes Answer Analyses (Monitor, Agency or the audit).
- **check_permission** — Ask whether a page may be used for a purpose, from every machine-readable signal the site publishes: robots.txt per crawler, Cloudflare Content-Signal, the IETF Content-Usage directive and header, an RSL licence, a TDM reservation (header or /.well-known/tdmrep.json), ai.txt, and noai/noindex/nosnippet on the page itself. Purposes: 'search' (index the page and show links and excerpts, no AI summary), 'answer' (input to a generated answer, including retrieval for RAG), 'train' (train or fine-tune a model), 'agent' (fetch or act on the page for a user, live). Returns a `verdict`: 'denied' (a signal reserves this use; any reservation wins, a block on any one crawler that does the job included, and a robots.txt answering a server error counts as a full disallow), 'allowed-under-terms' (permitted on the conditions listed in `terms`, such as payment or a licence server), 'unknown' (a source that could carry a reservation could not be read: robots.txt answered 401 or 403, the page refused the request, or a licence could not be fetched; unknown outranks allowed) or 'allowed' (no signal reserves the use, which is not a licence). With observe: true it also requests the page once with a representative crawler's user-agent string, from Crawl Readiness's servers, and reports the status in `observed`: that shows how the server treats the string, not what the real crawler gets. Every source read is listed in `signals` with what was found and what it decided. The `receipt` is a dated record of all of this, with the SHA-256 of every file read, signed with Ed25519; /api/permission-key publishes the current and retired public keys, so the receipt can be verified later. Use it before fetching pages for RAG, a dataset or an agent action, and whenever the user asks whether they may use, quote, index, train on or act on a page. It is a reading of the signals the site published on that date, not legal advice. 25 calls a day per network without an API key. Documented at https://machinereadiness.com/permission.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "crawl-readiness": {
    "command": "npx",
    "args": ["-y","crawl-readiness-mcp"],
    "env": {
      "CRAWL_READINESS_API_KEY": "",
      "CRAWL_READINESS_API_BASE": ""
    }
  }
}
```

**Requires environment variables:** `CRAWL_READINESS_API_KEY`, `CRAWL_READINESS_API_BASE` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation

## What crawl-readiness does

crawl-readiness evaluates whether a website publishes signals that match its intended AI access policy. It can score a site from 0 to 100 against more than 50 AI crawlers, inspect crawler-specific robots.txt rules, validate JSON-LD, and compare the content delivered to people with the content seen by AI crawlers.

The permission check answers whether a page may be used for search, answer generation, training, or agent action. It considers robots.txt, Content-Signal, Content-Usage, an RSL license, the TDM reservation, ai.txt, and page directives. Results distinguish denied, allowed-under-terms, allowed, and unknown outcomes. A signed Ed25519 receipt includes the signals read and hashes of the files examined.

## How it works

The server runs as an MCP process and exposes nine tools to compatible AI clients. Audit and permission checks can be used without an account. Generator features require an API key, and the Answer Analysis trend tool requires a key on a plan that includes those analyses.

Each tool sends HTTPS requests to the Crawl Readiness service. Permission checks also use machinereadiness.com. The service receives the target website address, generator inputs when applicable, and the configured API key. The local server does not store data, execute other programs, or read local files. If a generated file should be added to a project, the AI client performs that write after receiving user approval.

## Setup and configuration

Claude Desktop can use the downloadable MCP bundle from the latest GitHub release, or the server can be registered manually with `npx -y crawl-readiness-mcp`. Cursor, Cline, Windsurf, and Continue use the same stdio command in their MCP configuration. Claude Code can load the repository as a plugin with `claude --plugin-dir ./crawl-readiness-mcp`.

The optional `CRAWL_READINESS_API_KEY` enables the three generator tools, removes the daily limit on permission checks, and permits access to eligible Answer Analyses. `CRAWL_READINESS_API_BASE` can replace the default API endpoint, which is useful for testing. Restart the MCP client after changing environment settings.

## Tools and capabilities

- `check_ai_readiness` scores a site, reports crawler-by-crawler access, and ranks fixes.
- `validate_schema` checks JSON-LD blocks across more than 20 schema types.
- `validate_robots` reports robots.txt problems line by line and summarizes AI-bot coverage.
- `check_content_parity` compares human and crawler responses, including status codes and word overlap.
- `check_permission` evaluates published usage permissions and returns a signed receipt.
- `generate_llms_txt` crawls a site and returns formatted llms.txt content plus a robots.txt snippet.
- `generate_robots_txt` produces robots.txt with an applied AI access policy.
- `generate_schema` returns JSON-LD for detected Organization, WebSite, or Article types.
- `get_monitor_trend` reads Answer Analysis results, including brand share, provider breakdowns, competitors, trends, and sample answers.

## Limitations and notes

The free, no-signup checks cover audits and permission evaluation, while generation and Answer Analysis access depend on the API key and applicable account plan. The MCP process does not write files itself; the connected AI client must save generated content to the project. Permission results can be unknown when a source cannot be read, rather than treating that source as permission to proceed.

_Full upstream README: https://allmcps.com/mcp/crawl-readiness/readme_