Scrape Le

What scrape-le MCP server does

scrape-le MCP server exposes robots.txt analysis over the Model Context Protocol. Its analyze_robots_txt tool takes robots.txt content and a path to evaluate. An optional agent value identifies the user-agent being tested, while an optional maxResults value limits returned results.

The response indicates whether the selected rules permit crawling the requested path. It also identifies which user-agent group supplied the answer, reports crawl-delay settings, lists disallowed patterns, and returns any sitemaps found in the content. If a matching named group is not available, the analysis falls back to the generic User-agent: * rules.

How it works

The server communicates over MCP using stdio. An MCP client starts the Node package and sends tool calls through the process. The server parses the supplied robots.txt text locally and returns structured data; it does not read files or make network requests of its own.

Rule evaluation follows RFC 9309 semantics. The analyzer supports grouped user agents, Allow and Disallow directives, longest-match behavior, * wildcards, $ end anchors, crawl delays, and sitemap declarations. Supplying an empty agent value evaluates only the generic user-agent groups.

Setup and configuration

Run the server with npx:

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": ["-y", "scrape-le-mcp"]
    }
  }
}

The -y option avoids an installation prompt when the package starts for the first time. You can also install the package globally and configure the client to run the scrape-le-mcp binary directly:

npm install -g scrape-le-mcp

The server requires no environment variables, API key, or additional configuration. It can be tested by sending a JSON-RPC tools/list request through stdin; a working installation returns analyze_robots_txt in its tool list. Claude Code can register it with claude mcp add scrape-le -- npx -y scrape-le-mcp. VS Code 1.101 and later can use the tool through the Scrape-LE extension's agent mode.

Tools and capabilities

• analyze_robots_txt(content, path, agent?, maxResults?): evaluates whether a path may be crawled.
• Selects a matching named user-agent group or falls back to User-agent: *.
• Applies Allow and Disallow precedence using longest matching patterns.
• Handles wildcards, end-of-path anchors, crawl delays, and sitemap entries.
• Reports the group that produced the decision, along with matching restrictions and other parsed results.

Limitations and notes

The MCP server analyzes content supplied by the caller. It does not retrieve a site’s robots.txt file, inspect a page, launch Chromium, or perform the broader URL scrapeability checks available through the separate extension and CLI.

Results depend on the robots.txt text and the selected agent value. An empty agent causes only generic groups to be considered, so callers testing a specific crawler should pass its user-agent name when appropriate. The analysis describes the rules present in the file; it does not establish that a website permits scraping beyond those rules.