AI Crawler Index

What AI Crawler Index MCP server does

AI Crawler Index MCP server exposes a public, read-only catalog of web crawlers and AI user agents through Model Context Protocol. It is intended for developers who need to identify automated clients, understand the operators behind them, and make access-control decisions based on documented crawler behavior.

The index includes crawler categories, operators, robots.txt tokens and stances, verification methods, example User-Agent strings, mirrored IP-range endpoints, and the stated cost of blocking each crawler. A User-Agent classification is treated as a claim rather than proof; IP verification provides a separate check against prefixes published by crawler operators.

How it works

Clients send JSON-RPC tool calls to https://www.pathwren.workers.dev/mcp using streamable HTTP. The service is free to use, read-only, and does not require authentication, signup, or an API key. Calls do not fetch arbitrary URLs or contact third parties on the caller’s behalf.

whoami examines the request that reaches the service. It reports the supplied User-Agent, source address, inferred class, whether the host has seen the caller before, and related crawler-index results. example runs a classification against a canned User-Agent from the service’s own published data, allowing clients to inspect a real response shape without submitting their own value.

Setup and configuration

Configure an MCP client with the service’s streamable HTTP endpoint and call the tools through the normal MCP interface. No environment variables, credentials, consent flow, or local process are required.

The service also exposes HTTP equivalents for User-Agent classification and crawler-IP verification. These endpoints use query parameters and the same underlying implementation as the MCP tools.

Tools and capabilities

• classify_user_agent: Identify a crawler from a raw User-Agent string, including its operator, category, robots stance, verification guidance, and blocking cost.
• lookup_crawler: Retrieve one crawler’s full record by slug, with near matches for unknown slugs.
• list_crawlers: Browse the complete index or filter it by supported values such as category, with a configurable limit.
• generate_robots_txt: Produce a ready-to-use robots.txt block for one stance or for all eight documented stances.
• is_verified_crawler_ip: Check IPv4 or IPv6 addresses against operator-published crawler prefixes.
• whats_changed and changes_since: Inspect source freshness, failures, changelog entries, and incremental record or prefix changes.

Limitations and notes

AI Crawler Index MCP server relies on operator-published information. A matching User-Agent alone does not establish crawler authenticity, and an IP miss is reported as a miss rather than proof of forgery. Crawler IP data is remirrored every six hours, so freshness should be checked before regenerating configuration. The service does not modify robots.txt files or enforce blocking itself.