🍮 crawlbrulee mcp

EU-native web scraping for AI agents & developers.

plug crawlbrulee into your agent. the official mcp server for crawlbrulee gives mcp-aware agents — Claude Code, Codex, Cursor, Claude Desktop — native tools to scrape pages, map sites, run background jobs, and check usage. one call turns any url into clean markdown, screenshots, metadata and links.

• everything runs in the EU. the fetch, the render, the cache and your result never leave EU servers. the proxy exit is the one hop you choose: pick an EU exit and nothing leaves at all. gdpr-aligned, with a data processing agreement.
• output made for models. markdown with the page chrome stripped and the links kept, ready for the prompt. full-page screenshots can come back as tiles sized for an image model.
• the hard parts, handled. headless Chrome when a page needs it, rotating proxies with country selection, automatic retries, ad and cookie-banner removal, caching, background jobs and signed webhooks.
• start free. 750 credits, no credit card.

get a free api key → dashboard.crawlbrulee.com

the server:

• npx-runnable — zero install.
• wraps the @crawlbrulee/sdk under the hood; this mcp is just a thin protocol adapter.
• stdio transport for terminal-based agents.
• strict, fully-described tool schemas — agents see what every parameter does without reading docs.

this readme covers the mcp server itself — its tools and how to wire it into a host. for how the api behaves — endpoints, parameters, and error semantics — please see our api docs.

install

# Claude Code
claude mcp add crawlbrulee \
  --env CRAWLBRULEE_API_KEY=cwbl_... \
  -- npx -y @crawlbrulee/mcp

# Cursor — add to ~/.cursor/mcp.json:
{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "cwbl_..." }
    }
  }
}

the same pattern works for Codex, Claude Desktop, and any other host that accepts a stdio mcp launch command — set command: npx, args: ["-y", "@crawlbrulee/mcp"], and forward CRAWLBRULEE_API_KEY via the env block.

configuration

the mcp reads the env var on first tool invocation — not at startup — so a typo in your config surfaces as a clear tool-error message rather than the server failing to come up. see authentication for how the api consumes keys.

tools

scrape

fetch a single url and return the requested content (markdown, cleaned html, raw html, links, images, screenshot, page metadata).

input — only url is required; everything else has sane defaults.

{
  "url": "https://example.com",
  "extract": {
    "markdown": true,
    "links": true,
    "screenshot": { "type": "full_page", "device_mode": "desktop" },
  },
  "require_js": false,
  "proxy": "basic",
  "cleanup": { "ads_and_popups": true, "exclude_selectors": ["nav", "footer"] },
  "cache": { "max_age": 3600 },
  "location": { "locale": "en-US", "country": "US" },
}

output — full scrape result. page metadata (title, OG tags, etc.) is returned under metadata. extracted images are returned as absolute urls — query strings are preserved, and relative srcs are resolved against the page url. screenshots are returned as signed download urls the agent can fetch separately. in rare cases a screenshot can't be captured: when you requested other outputs too, the screenshot field is simply left out while the rest is still returned — but a screenshot-only call that can't deliver errors instead (unsupported_screenshot_output, HTTP 422, when the content type can't be screenshotted) and isn't billed. the result also carries a top-level response_meta.usage block:

{
  "url": "https://example.com",
  "markdown": "...",
  "metadata": { "title": "Example Domain" },
  "response_meta": {
    "usage": {
      "credits": 1,
      "engine": "http", // "http" | "browser" | "screenshot" | "cache"
      "proxy": "basic", // resolved tier actually used: "basic" | "advanced" (never "auto")
      "screenshot_slices": 0, // 1 when the screenshot-split add-on was billed, otherwise 0
    },
  },
}

alongside response_meta.usage, the result surfaces any non-fatal warnings — stable string codes an agent can switch on. an outsized page is truncated rather than refused, and the code names which part was cut:

and if you requested an extract that doesn't apply to the content type (e.g. markdown of a pdf), the field name comes back in an unsupported_fields list — with the rest of the payload still returned.

every input field, its default, and its constraints are documented under the scrape endpoint — with extraction, screenshots, proxies & location, and caching covering the individual blocks.

scrape_async

submit a scrape job to run asynchronously and get back a job_id immediately, instead of holding the connection open. use this for long-running scrapes (heavy js rendering, full-page screenshots of long pages); for a quick one-shot fetch prefer the synchronous scrape tool. then poll scrape_status until the job is done and fetch the page with scrape_result.

takes the same input as scrape plus an optional per-job completion webhook:

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "webhook": {
    // Endpoint that receives one signed `scrape.complete` POST when the job
    // finishes. http/https (HTTPS required in production), max 2048 chars.
    "url": "https://hooks.example.com/cwbl",
    // Opaque correlation object echoed back verbatim in the delivery's
    // `data.metadata`. Serializes to at most 2048 bytes.
    "metadata": { "ref": "order-42" },
  },
}

output — { "job_id": "..." }.

when a webhook is attached, we deliver a single signed scrape.complete POST to your endpoint once the job reaches a terminal state, with your metadata echoed under data.metadata and the job's usage under data.response_meta.usage — so you can react to completion (and track cost) without polling. verify the X-Cwbl-Signature header with the sdk's verifyWebhookSignature (configure the signing secret in the dashboard under account → webhooks).

the job lifecycle is documented under async scrape; the delivery contract and payload shape under webhooks, with the signature scheme in webhook verification.

scrape_status

look up the current lifecycle status of an async job: pending, running, done, or failed (with an error message when failed). once the job is done the response also carries a response_meta.usage block (credits, billed engine, resolved proxy tier, screenshot_slices). a cache hit is represented by engine: "cache". poll until done, then call scrape_result.

{ "job_id": "..." }

scrape_result

fetch the extracted content of a completed async job — the same result shape as the synchronous scrape tool (including metadata and response_meta.usage). errors if the job is still pending/running, so check scrape_status first.

{ "job_id": "..." }

map

build (or fetch a cached) link-map for a website. combines sitemap discovery with homepage link extraction. use this to enumerate a site before scraping selected pages. each link is just { url }.

max_urls (default 5000, max 100000) is a discovery budget, not a trim at the end — discovery stops as soon as that many urls are found, so a smaller value is a faster, cheaper crawl. limit (default 5000, max 10000) only pages the answer.

returned urls are normalized the same way scrape normalizes its returned url, so map-then-scrape stays on one host. results are ordered with the most useful links first.