AI Crawler Index

crawlersrobots-txtuser-agentbot-detectionllms-txt

Read-only MCP server over a public reference index of web crawlers and AI user agents: classify a user-agent, look up a crawler and who runs it, generate robots.txt rules, check an IP, and see what blocking each one costs. Streamable HTTP, no key and no signup; the data is CC0.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.

Add to Cursor Add to VS Code

Manual Client & Custom JSON ConfigExpand JSON ▾

Client Config & Setup

Remote HTTP

Choose your client or environment

Target File:~/Library/Application Support/Claude/claude_desktop_config.json

claude_desktop_config.json

{
  "mcpServers": {
    "ai-crawler-index": {
      "url": "https://www.pathwren.workers.dev/c/allmcps/mcp"
    }
  }
}

💡 Paste the JSON block into your client's configuration file under mcpServers, then restart the application.

Documentation Overview

AI Crawler Index is a 🔎 Search & Data Extraction MCP server listed on AllMCPs. Read-only MCP server over a public reference index of web crawlers and AI user agents: classify a user-agent, look up a crawler and who runs it, generate robots.txt rules, check an IP, and see what blocking each one costs. Streamable HTTP, no key and no signup; the data is CC0.

Read the full README →View source on pathwren.workers.dev →

Related MCP Servers

View all in Search & Data Extraction View all alternatives

Reviews

No reviews yet — be the first to share how this listing worked for you.

Frequently Asked Questions about AI Crawler Index

Add the following block to your claude_desktop_config.json under mcpServers: "mcpServers": { "ai-crawler-index": { "command": "npx", "args": ["-y", "AI Crawler Index"] } }

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:

Live Dynamic SVG Preview

AI Crawler Index AllMCPs Directory Badge

Markdown (GitHub README)

[![AllMCPs](https://allmcps.com/api/badge/ai-crawler-index?style=directory)](https://allmcps.com/mcp/ai-crawler-index)

HTML Embed

<a href="https://allmcps.com/mcp/ai-crawler-index"><img src="https://allmcps.com/api/badge/ai-crawler-index?style=directory" alt="AI Crawler Index on AllMCPs" /></a>

Technical Specs & Signals

CategorySearch & Data Extraction

PricingFreeFree: no key, no account, no rate limit. Data CC0-1.0.

34Quality signal: Emerging · 34/100How this signal is calculated ▾

Server availability13/25

Verified ownership0/20

Documentation & tools20/30

Adoption & activity1/15

Community engagement0/10

A guidance signal from public completeness & health data — not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.

★ Spotlight Slot

Feature Your MCP Server

Get maximum visibility for your server across our directory, search results, and detail pages.

Spotlight Your Server

Own this project?

This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to unlock edit access and the Official badge — proof is checked automatically, then reviewed by our team.

Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Explore more

More in 🔎 Search & Data Extraction →Best MCP servers for Web Search & Scraping →Alternatives to AI Crawler Index →Install in Claude Desktop Install in Cursor Install in VS Code