Skip to main content
AllMCPs
BrowseBestCategoriesStackCompareToolsGuidesBlog
Log in Submit MCP

Stay in the loop

Get new MCP servers and top picks in your inbox.

AllMCPs

The open directory for discovering and installing Model Context Protocol servers.

AllMCPs on GitHub (opens in a new tab)
Launched onTiny Startupstinystartups.com
Explore
  • Browse servers
  • Best MCP servers
  • Categories
  • MCP clients
  • Agent prompts
  • Stack Builder
  • Compare servers
  • Random discovery New
  • Submit a server
  • Pricing & Boost Boost
Learn
  • Guides hub
  • What is MCP?
  • Install guide
  • Build an MCP server
  • Deploy an MCP server
  • Security guide
  • Troubleshooting
  • MCP for SEO & AEO
  • Protocol versioning
  • Blog & updates
Tools
  • All developer tools
  • Config generator
  • Config validator
  • Config auditor
  • MCP playground
  • Token calculator
  • OpenAPI → MCP
  • Badge generator
For agents
  • REST API docs
  • Trust & traffic Live
  • Remote MCP server SSE ↗ (opens in a new tab)
  • llms.txt ↗ (opens in a new tab)
  • Catalog JSON ↗ (opens in a new tab)
Company
  • About
  • Advertise Sponsor
  • Contact
  • GitHub ↗ (opens in a new tab)
  • Terms
  • Privacy
AllMCPs VerifiedAllMCPs VerifiedFeatured on Nick LaunchesFeatured on Nick LaunchesLaunch Llama NewsletterLaunch Llama NewsletterVerified DR - allmcps.comVerified DR - allmcps.comFeatured on SaaSGrowFeatured on SaaSGrowFeatured on Twelve ToolsFeatured on Twelve ToolsFeatured on Saaspa.geFeatured on Saaspa.geFeatured on Findly.toolsFeatured on Findly.toolsFeatured on Startup FameFeatured on Startup FameFeatured on LaunchKiwiFeatured on LaunchKiwiFeatured on ScrollLaunchFeatured on ScrollLaunchFeatured on DailyPingsFeatured on DailyPingsFazier badgeFazier badgeFeatured on NewTool.siteFeatured on NewTool.siteFeatured on saasfame.comFeatured on saasfame.comDR Checker - Domain RatingDR Checker - Domain RatingListed on Turbo0Listed on Turbo0Launched on LaunchBoard - Product Launch PlatformLaunched on LaunchBoard - Product Launch PlatformList on SimilarlabsList on Similarlabshttps://codetrendy.comhttps://codetrendy.comListed on DevTool.ioFeatured on BuildlistFeatured on BuildlistLaunched on Tiny StartupsFeatured on ShowMeBestAIFeatured on ShowMeBestAIFind us on LaunchZoneFind us on LaunchZoneAllMCPs VerifiedAllMCPs VerifiedFeatured on Nick LaunchesFeatured on Nick LaunchesLaunch Llama NewsletterLaunch Llama NewsletterVerified DR - allmcps.comVerified DR - allmcps.comFeatured on SaaSGrowFeatured on SaaSGrowFeatured on Twelve ToolsFeatured on Twelve ToolsFeatured on Saaspa.geFeatured on Saaspa.geFeatured on Findly.toolsFeatured on Findly.toolsFeatured on Startup FameFeatured on Startup FameFeatured on LaunchKiwiFeatured on LaunchKiwiFeatured on ScrollLaunchFeatured on ScrollLaunchFeatured on DailyPingsFeatured on DailyPingsFazier badgeFazier badgeFeatured on NewTool.siteFeatured on NewTool.siteFeatured on saasfame.comFeatured on saasfame.comDR Checker - Domain RatingDR Checker - Domain RatingListed on Turbo0Listed on Turbo0Launched on LaunchBoard - Product Launch PlatformLaunched on LaunchBoard - Product Launch PlatformList on SimilarlabsList on Similarlabshttps://codetrendy.comhttps://codetrendy.comListed on DevTool.ioFeatured on BuildlistFeatured on BuildlistLaunched on Tiny StartupsFeatured on ShowMeBestAIFeatured on ShowMeBestAIFind us on LaunchZoneFind us on LaunchZone
© 2026 Jackalope Digital LLC. All rights reserved.
  1. Home
  2. Browse
  3. AI Crawler Index
  4. vs Agent Discovery Doctor
Side-by-Side Model Context Protocol Comparison

AI Crawler Index vs Agent Discovery Doctor

In-depth architectural comparison of the AI Crawler Index and Agent Discovery Doctor MCP servers. Compare execution transports, security boundaries, tool capabilities, quality scores, and ready-to-paste client installation snippets for Claude, Cursor, Windsurf, and VS Code.

At a Glance & Executive Verdict

AI Crawler Index
Developer Tools · Local stdio
Quality: 51/100 (Good) | Auth: No auth required
Agent Discovery Doctor
Developer Tools · Local stdio
Quality: 51/100 (Good) | Auth: No auth required
Verdict Summary: Choose AI Crawler Index if you need specialized Developer Tools tools running via a local process. Choose Agent Discovery Doctor if your workspace requires Developer Tools integration with local subprocess execution. Both servers can be configured concurrently in your client's mcpServers manifest.

Which MCP Server Should You Choose?

A

Choose AI Crawler Index when:

  • You need dedicated capabilities in the Developer Tools domain.
  • You prefer local stdio subprocess transport architecture.
  • Your security boundary fits: No auth required (Free / Open Source).
  • Primary tools included: whoami, example, classify_user_agent.
Explore AI Crawler Index Details
A

Choose Agent Discovery Doctor when:

  • You need dedicated capabilities in the Developer Tools domain.
  • You prefer local stdio subprocess transport architecture.
  • Your security boundary fits: No auth required (Free / Open Source).
  • Primary tools included: no_arguments_check_this_hosts_own_discovery_documents, whoami, example.
Explore Agent Discovery Doctor Details

Feature & Specification Comparison

Specification
A
AI Crawler Index
Developer Tools
A
Agent Discovery Doctor
Developer Tools
SummaryIdentify AI crawlers by user-agent or IP, and build the robots.txt that matches your stance.Which discovery documents does a host serve? llms.txt, agent card, owners.json, mcp.json.
Category & ScopeDeveloper ToolsDeveloper Tools
Quality signal

Tools & Capabilities Breakdown

AI Crawler Index Tools (9)

whoami
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back describes YOU, the caller — the user-agent you sent, the address you came from, the class our instrument books you as, and whether this host has seen you before. Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what the crawler index makes of the user-agent you sent, and whether the address you came from is inside a prefix its operator publishes — both run through this server's own classify_user_agent and is_verified_crawler_ip, on the same data a stranger can curl. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.
example
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — classify_user_agent, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — classify_user_agent on the exact user-agent string of the busiest client in this host's own published log (/data/observed-clients.json, 1164 requests in the window) — our data, and no third party is asked anything — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs classify_user_agent with {"user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1. and returns its real answer.

Ready-to-Paste Client Configurations

Paste either (or both) of these JSON server blocks into your client config file (e.g. claude_desktop_config.json or ~/.cursor/mcp.json).

AI Crawler Index Configuration
mcpServers (Claude Desktop / Cursor)
{
  "mcpServers": {
    "ai-crawler-index-2": {
      "command": "npx",
      "args": [
        "-y",
        "ai-crawler-index-2"
      ]
    }
  }
}
Agent Discovery Doctor Configuration
mcpServers (Claude Desktop / Cursor)
{
  "mcpServers": {
    "agent-discovery-doctor": {
      "command": "npx",
      "args": [
        "-y",
        "agent-discovery-doctor"
      ]
    }
  }
}

Frequently Asked Questions

AI Crawler Index is categorized under Developer Tools and uses a local stdio subprocess. In contrast, Agent Discovery Doctor belongs to Developer Tools using local stdio subprocess. Select AI Crawler Index when you need capabilities focused on developer tools and Agent Discovery Doctor when you require tools for developer tools.

More alternatives to AI Crawler IndexMore alternatives to Agent Discovery DoctorDeveloper Tools category hubCanonical compare URL

Related MCP Server Comparisons

Popular comparisons with AI Crawler Index

  • Claude Task Master logoAI Crawler Index vs Claude Task Master
  • MediaWiki MCP Server logoAI Crawler Index vs MediaWiki MCP Server
  • Andrea9293 MCP logoAI Crawler Index vs Andrea9293 MCP
  • UI5 MCP Server logoAI Crawler Index vs UI5 MCP Server

Popular comparisons with Agent Discovery Doctor

51/100 (Good)
51/100 (Good)
Transport ProtocolLocal Subprocess (stdio)Local Subprocess (stdio)
Auth RequirementNo auth requiredNo auth required
Pricing ModelFree / Open SourceFree / Open Source
Required Env VarsNone requiredNone required
Compatible Clients
Claude DesktopCursorWindsurfClineVS Code
Claude DesktopCursorWindsurfClineVS Code
Install path signalnpx · lownpx · low
Engagement & Health 0 views 0 copies 0 upvotes 0 views 0 copies 0 upvotes
Verified / OfficialCommunity ListingCommunity Listing
Open full listingView AI Crawler Index ListingView Agent Discovery Doctor Listing
classify_user_agent
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers
lookup_crawler
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'.
list_crawlers
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.
generate_robots_txt
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.
is_verified_crawler_ip
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers
whats_changed
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'.
changes_since
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'.

Agent Discovery Doctor Tools (8)

no_arguments_check_this_hosts_own_discovery_documents
TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} to https://www.pathwren.workers.dev/mcp/doctor — the answer is the whole agent-discovery catalogue checked against THIS host — every document this server tracks, whether this host serves it, its size and content type, which ones are missing, and the part only this server can add: the named clients it has watched ask for each file, when, and the status that request took. There is nothing to fill in: the input schema is literally empty, `arguments: {}` and no `arguments` key at all both work, and the subject is a file this host already publishes, so the answer does not depend on you at all. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Nothing is fetched to build it — no request leaves this edge, and none is made to you. whoami and example are here too and take nothing either. Every other tool on this server wants a file pasted in; this one wants nothing. The siblings answer one question each under the tool named beside them: /mcp (whoami), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report). Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} returns a present/missing verdict per tracked document with its byte size, the group each belongs to, the score out of the catalogue, and the observed askers per document with the date and status each one got.
whoami
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/doctor and the answer is the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only this server can answer: which of the discovery documents this catalogue tracks (llms.txt, the agent card, mcp.json, apis.json, oauth-protected-resource…) a client like yours has been observed asking THIS host for, when, and what status it took. Takes no arguments. Safe to call. Deterministic. Touches no third party. Nothing is fetched to build it: every fact comes from the headers on your own request or from a file this host already publishes. Only this server answers the last part — its five siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev. Example: arguments={} returns your user-agent, your address, the class we book you as, whether we have seen you here before, and which agent-discovery documents this host has watched a client like yours ask for — the dates, and the status each request took.
example
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/doctor and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — validate_llms_txt, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — validate_llms_txt on a four-line llms.txt held as a constant in this file: it exercises the H1, the blockquote summary and an H2 link section, which is the whole format — and it fetches nothing, from us or from anyone — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs validate_llms_txt with {"text":"# Site\n\n> One line.\n\n## Docs\n\n- [Guide](https://e.com/g): start here."} and returns its real answer.
check_discovery_documents
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} — Probes 22 documents agents and trust indexes ask for — llms.txt, agent card, owners.json, oauth metadata, mcp.json, apis.json, openapi, robots, sitemap — as served, missing, gated or 200-with-HTML soft-404, and says who asks for each missing one. Refuses private, ephemeral and its own hosts. Example: host='example.com'.
explain_document
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} — One catalogue entry: what the document is for, the named clients observed asking this host for it with dates and the status they took, what a 404 costs, and the spec URL. No argument lists all 22. Example: name='owners.json' names the bot that asks for it twice.
validate_llms_txt
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} — Checks pasted llms.txt against the format: one H1, a blockquote summary, H2 sections of `- [name](url): notes`. Errors and warnings with line numbers and fixes, plus the parsed links. Text in, nothing fetched. Example: text='# Site' warns it has no summary and no sections.
llms_txt_from_sitemap
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} — Paste sitemap.xml, or one URL per line, and get a draft llms.txt: URLs grouped into H2 sections by path, titles from slugs, lastmod kept, and a TODO wherever only you can write the sentence. A sitemap index is reported as one. Example: xml='https://e.com/docs/a\nhttps://e.com/blog/b'.
validate_agent_card
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_check_this_hosts_own_discovery_documents","arguments":{}}} — Validates a pasted /.well-known/agent-card.json against the nine fields A2A marks required and each skill's id/name/description/tags, and warns on capabilities declared true that a reader will then try. Example: json='{"name":"a"}' returns the eight missing fields.
MediaWiki MCP Server logo
Agent Discovery Doctor vs MediaWiki MCP Server
  • Claude Task Master logoAgent Discovery Doctor vs Claude Task Master
  • Andrea9293 MCP logoAgent Discovery Doctor vs Andrea9293 MCP
  • Imqueue MCP logoAgent Discovery Doctor vs Imqueue MCP