# scrape-le [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/nolindnaidoo/scrape-le  
**GitHub Stars:** 1  
**npm Downloads (last month):** 1541  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/scrape-le

## Description
Analyse robots.txt content and report whether a path may be crawled.

## Tools
Capabilities this server exposes over MCP:

- **analyze_robots_txt** — Given the contents of a robots.txt file and a path, report whether the rules permit crawling it — the group naming `agent` when one does, otherwise the generic (User-agent: *) rules, and `agent` in the answer says which — along with the crawl delay, the disallowed patterns and any sitemaps. Takes the file contents directly and makes no network request of its own — fetch robots.txt with your own HTTP tool and pass what it returned.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "scrape-le": {
    "command": "npx",
    "args": ["-y","scrape-le-mcp"]
  }
}
```

## Documentation

## What scrape-le MCP server does

scrape-le MCP server exposes robots.txt analysis over the Model Context Protocol. Its `analyze_robots_txt` tool takes robots.txt content and a path to evaluate. An optional agent value identifies the user-agent being tested, while an optional `maxResults` value limits returned results.

The response indicates whether the selected rules permit crawling the requested path. It also identifies which user-agent group supplied the answer, reports crawl-delay settings, lists disallowed patterns, and returns any sitemaps found in the content. If a matching named group is not available, the analysis falls back to the generic `User-agent: *` rules.

## How it works

The server communicates over MCP using stdio. An MCP client starts the Node package and sends tool calls through the process. The server parses the supplied robots.txt text locally and returns structured data; it does not read files or make network requests of its own.

Rule evaluation follows RFC 9309 semantics. The analyzer supports grouped user agents, `Allow` and `Disallow` directives, longest-match behavior, `*` wildcards, `$` end anchors, crawl delays, and sitemap declarations. Supplying an empty agent value evaluates only the generic user-agent groups.

## Setup and configuration

Run the server with `npx`:

```json
{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": ["-y", "scrape-le-mcp"]
    }
  }
}
```

The `-y` option avoids an installation prompt when the package starts for the first time. You can also install the package globally and configure the client to run the `scrape-le-mcp` binary directly:

```bash
npm install -g scrape-le-mcp
```

The server requires no environment variables, API key, or additional configuration. It can be tested by sending a JSON-RPC `tools/list` request through stdin; a working installation returns `analyze_robots_txt` in its tool list. Claude Code can register it with `claude mcp add scrape-le -- npx -y scrape-le-mcp`. VS Code 1.101 and later can use the tool through the Scrape-LE extension's agent mode.

## Tools and capabilities

- `analyze_robots_txt(content, path, agent?, maxResults?)`: evaluates whether a path may be crawled.
- Selects a matching named user-agent group or falls back to `User-agent: *`.
- Applies `Allow` and `Disallow` precedence using longest matching patterns.
- Handles wildcards, end-of-path anchors, crawl delays, and sitemap entries.
- Reports the group that produced the decision, along with matching restrictions and other parsed results.

## Limitations and notes

The MCP server analyzes content supplied by the caller. It does not retrieve a site’s robots.txt file, inspect a page, launch Chromium, or perform the broader URL scrapeability checks available through the separate extension and CLI.

Results depend on the robots.txt text and the selected agent value. An empty agent causes only generic groups to be considered, so callers testing a specific crawler should pass its user-agent name when appropriate. The analysis describes the rules present in the file; it does not establish that a website permits scraping beyond those rules.

_Full upstream README: https://allmcps.com/mcp/scrape-le/readme_