Robotstxt AI README
The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Robotstxt AI listing page.
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Features
- Visual Toggle UI — Block or allow AI bots with simple on/off switches
- 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
- Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
- Generate robots.txt — Create a complete robots.txt with your chosen rules
- MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
- Check Bot Status — Verify if a specific bot is blocked on any website
MCP Tools
| Tool | Description |
|---|---|
fetch_robots | Fetch and analyze a robots.txt from any URL |
analyze_robots | Analyze pasted robots.txt content for AI bot blocking status |
generate_robots | Generate a robots.txt with specified blocked bots and custom rules |
list_ai_bots | List all known AI bots with user-agents, companies, and descriptions |
check_bot_status | Check if a specific bot is blocked on a given website |
Installation
Web App
Open http://localhost:3000 in your browser.
MCP Server — Claude Desktop
Add to your claude_desktop_config.json:
MCP Server — Cursor
Add to .cursor/mcp.json:
Tracked Bots
The server knows about 25+ bots including:
- AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
- AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
- Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot
Why robotstxt.ai vs Manual Editing
| Feature | robotstxt.ai | Manual Editing |
|---|---|---|
| Visual toggle UI | Yes | No |
| 20+ AI bots database | Yes | Research yourself |
| Analyze existing robots.txt | Yes | No |
| MCP Server | Yes | No |
| Price | Free | Free but tedious |
More Open Source SEO Tools
| Tool | Description |
|---|---|
| awesome-seo-mcp-servers | Curated list of SEO MCP servers and agent skills |
| indexnow-mcp | Instant URL indexing via IndexNow |
| schema-gen | Schema.org JSON-LD markup generator |
| ai-visibility | AI brand visibility tracker |
License
MIT
