Scrape-LE: Zero Hassle Scrapeability Checks

Load a URL in headless Chromium and see what will block your scraper — before you write it
Anti-bot vendors, rate limits, robots.txt rules, login walls, console errors, screenshots

Useful? A star or rating is how other developers find it — ★ GitHub · ★ Open VSX · ★ Marketplace

What it does

Run Scrape-LE: Check URL Scrapeability (Ctrl+Alt+S / Cmd+Alt+S), enter a URL, and the page loads in a real headless Chromium. The report lands in the output channel: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections. Works in VS Code and VS Code–based editors like Cursor and VSCodium (installable from Open VSX).

One-time setup: run Scrape-LE: Setup Browser to install Chromium (~130MB, into Playwright's browser cache).

Install

Use it from an AI agent

The same engine runs as an MCP server, so an agent can call it directly instead of you running a command.

analyze_robots_txt(content, path, maxResults?)

Given robots.txt contents and a path, reports whether the generic (User-agent: *) rules permit crawling it, plus the crawl delay, disallowed patterns and any sitemaps.

The server takes content and returns data — it reads no files and makes no network requests of its own. Published as scrape-le-mcp on npm and as io.github.nolindnaidoo/scrape-le in the MCP registry.

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": ["-y", "scrape-le-mcp"]
    }
  }
}

npm install -g scrape-le-mcp

{
  "mcpServers": {
    "scrape-le": { "command": "scrape-le-mcp" }
  }
}

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | npx -y scrape-le-mcp

The CLI

The same check runs from a terminal or an agent loop: a Rust CLI in crate/ of this repository, sharing one signature corpus with the extension — crate/signatures/ and crate/fixtures/ — so CI fails if the two ever disagree about a URL.

scrape-le https://example.com/search   # JSON on stdout, summary on stderr
scrape-le --input urls.txt             # a batch, streamed as it completes
scrape-le mcp                          # the same check over MCP on stdio

The exit code is the answer: 0 clear · 1 a real no · 2 the question was malformed. ## Detections

Honest limitations: signatures are best-effort fingerprints of public integration patterns — a detected widget means the page can challenge you, not that it will, and a clean result is not proof a site allows scraping. Agent-specific robots.txt groups are ignored (only the * rules are reported). Pages get up to 5 seconds to go network-idle after load, so content rendered later than that can be missed by the page-level detections.

Commands

Settings

Languages

Twelve languages besides English:

German · Spanish · French · Indonesian · Italian · Japanese · Korean · Portuguese (Brazil) · Russian · Ukrainian · Vietnamese · Chinese (Simplified)

Both halves are covered — the manifest (command titles, setting names and descriptions) and everything shown while the extension runs (notifications, the status bar, quick-picks and prompts). The extension follows VS Code's display language, so it matches whatever the editor is already set to; no setting of its own.

Privacy & security