Scrape-LE: Zero Hassle Scrapeability Checks

Load a URL in headless Chromium and see what will block your scraper — before you write it
Anti-bot vendors, rate limits, robots.txt rules, login walls, console errors, screenshots

Useful? A star or rating is how other developers find it — ★ GitHub · ★ Open VSX · ★ Marketplace

What it does

Run Scrape-LE: Check URL Scrapeability (Ctrl+Alt+S / Cmd+Alt+S), enter a URL, and the page loads in a real headless Chromium. The report lands in the output channel: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections. Works in VS Code and VS Code–based editors like Cursor and VSCodium (installable from Open VSX).

One-time setup: run Scrape-LE: Setup Browser to install Chromium (~130MB, into Playwright's browser cache).

Install

Use it from an AI agent

The same engine runs as an MCP server, so an agent can call it directly instead of you running a command.

analyze_robots_txt(content, path, maxResults?)

Given robots.txt contents and a path, reports whether the generic (User-agent: *) rules permit crawling it, plus the crawl delay, disallowed patterns and any sitemaps.

The server takes content and returns data — it reads no files and makes no network requests of its own. Published as scrape-le-mcp on npm and as io.github.nolindnaidoo/scrape-le in the MCP registry.

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": ["-y", "scrape-le-mcp"]
    }
  }
}

npm install -g scrape-le-mcp

{
  "mcpServers": {
    "scrape-le": { "command": "scrape-le-mcp" }
  }
}

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | npx -y scrape-le-mcp

The CLI

The same check runs from a terminal or an agent loop: a Rust CLI in crate/ of this repository, sharing one signature corpus with the extension — crate/signatures/ and crate/fixtures/ — so CI fails if the two ever disagree about a URL.

scrape-le https://example.com/search   # JSON on stdout, summary on stderr
scrape-le --input urls.txt             # a batch, streamed as it completes
scrape-le mcp                          # the same check over MCP on stdio

The exit code is the answer: 0 clear · 1 a real no · 2 the question was malformed. ## Detections

Honest limitations: signatures are best-effort fingerprints of public integration patterns — a detected widget means the page can challenge you, not that it will, and a clean result is not proof a site allows scraping. Agent-specific robots.txt groups are ignored (only the * rules are reported). Pages get up to 5 seconds to go network-idle after load, so content rendered later than that can be missed by the page-level detections.

Commands

Settings

Languages

Twelve languages besides English:

German · Spanish · French · Indonesian · Italian · Japanese · Korean · Portuguese (Brazil) · Russian · Ukrainian · Vietnamese · Chinese (Simplified)

Both halves are covered — the manifest (command titles, setting names and descriptions) and everything shown while the extension runs (notifications, the status bar, quick-picks and prompts). The extension follows VS Code's display language, so it matches whatever the editor is already set to; no setting of its own.

Privacy & security

• Network access is the feature, and it is scoped. A check talks to exactly two things: the URL you enter (loaded in headless Chromium, which fetches that page's own resources like any browser) and that origin's /robots.txt. Nothing is sent anywhere else — no telemetry, no analytics.
• Screenshots stay local, written to the configured path inside your workspace.
• The MCP server makes no network request at all — unlike the extension, deliberately. fetchRobotsTxt builds a URL from an arbitrary origin, which inside an agent loop is an SSRF primitive: the caller supplying the URL is the model, not you. The server analyses robots.txt content you already fetched, and a test asserts no tool accepts a url argument.
• Error notifications redact home directories and credential-shaped fragments.
• Respect the sites you check: a scrapeability report is information, not permission.

Documentation

Performance

Median of 7 runs after warmup, on Apple M5 Pro, 24 GB RAM, Node 24.3.0. Inputs are generated by scripts/benchmark.ts rather than checked in, so the sizes above are exactly what was measured. Reproduce with bun run benchmark.

These are machine-specific and are not asserted in CI — a benchmark that gates a build only tells you how busy the runner was.

Testing

379 test cases across 29 files, plus an integration suite that runs in a real VS Code extension host and an end-to-end test that installs the built .vsix into a clean profile.

Generated from a real run — coverage/coverage-summary.json and coverage/test-results.json — by scripts/coverage-readme.js; CI fails if this section drifts. Reproduce with bun run test:coverage, and the case count is the one vitest prints.

More from the LE family

Sixteen single-purpose tools for the work in front of every model. Each ships a Rust CLI and an MCP server. One page: letools.dev

Get it out

• String-LE — Extract every string in a codebase, with its position, so a person can read them
• Numbers-LE — Extract every hardcoded number in a codebase, so a person can check them
• Units-LE — Extract every quantity with its unit, normalized, and refuse the ambiguous ones by name
• Dates-LE — Extract every date and timestamp, and the exact instant each one resolves to
• IDs-LE — Extract every UUID, ULID, NanoID, ObjectId and Snowflake, and decode the time inside
• IPs-LE — Extract every IP address, CIDR block and MAC, normalized and classified by scope
• URLs-LE — Extract every URL in a codebase, with its protocol and exact position
• Paths-LE — Extract every file path in a codebase, and say whether it still points at anything
• Colors-LE — Extract every color in a codebase, and say which ones are not in your palette

Check it

• Regex-LE — Find every regex in a codebase, and report which can be driven into catastrophic backtracking
• Versions-LE — Find where one dependency is constrained differently across a repository's manifests
• i18n-LE — Identify the i18n library a project uses, then audit its catalogs by that library's rules
• Scrape-LE — Check whether a page is scrapeable before the scraper is written, and say when it cannot tell

Guard it

• Secrets-LE — Find hardcoded credentials in a codebase, and never print one into the report
• EnvSync-LE — Compare the dotenv files in a tree, and say which keys are missing from which
• Unicode-LE — Find the Unicode that hides meaning — bidi controls, invisibles, homoglyphs, mixed scripts

Each stands on its own: no shared crate, no published core. Where two of them agree, it is because the same answer was right twice.

Contact — nolindnaidoo.com · GitHub · LinkedIn

Also by nolindnaidoo

Rust — pixelcoords and pixelactions are one loop: pixelcoords answers where, pixelactions acts there. Their own tools, their own voice — not part of the LE family.

• pixelcoords — Freeze your screen, mark regions, get pixel-exact coordinates and crops pixelcoords.dev · crates.io · docs.rs
• pixelactions — Consume human-verified coordinates, perform the interaction, confirm it landed pixelactions.dev · crates.io · docs.rs

License

MIT © nolindnaidoo