🍮 crawlbrulee mcp

EU-native web scraping for AI agents & developers.

plug crawlbrulee into your agent. the official mcp server for crawlbrulee gives mcp-aware agents — Claude Code, Codex, Cursor, Claude Desktop — native tools to scrape pages, map sites, run background jobs, and check usage. one call turns any url into clean markdown, screenshots, metadata and links.

everything runs in the EU. the fetch, the render, the cache and your result never leave EU servers. the proxy exit is the one hop you choose: pick an EU exit and nothing leaves at all. gdpr-aligned, with a data processing agreement.
output made for models. markdown with the page chrome stripped and the links kept, ready for the prompt. full-page screenshots can come back as tiles sized for an image model.
the hard parts, handled. headless Chrome when a page needs it, rotating proxies with country selection, automatic retries, ad and cookie-banner removal, caching, background jobs and signed webhooks.
start free. 750 credits, no credit card.

get a free api key → dashboard.crawlbrulee.com

the server:

npx-runnable — zero install.
wraps the @crawlbrulee/sdk under the hood; this mcp is just a thin protocol adapter.
stdio transport for terminal-based agents.
strict, fully-described tool schemas — agents see what every parameter does without reading docs.

this readme covers the mcp server itself — its tools and how to wire it into a host. for how the api behaves — endpoints, parameters, and error semantics — please see our api docs.

install

bash

# Claude Code
claude mcp add crawlbrulee \
  --env CRAWLBRULEE_API_KEY=cwbl_... \
  -- npx -y @crawlbrulee/mcp

# Cursor — add to ~/.cursor/mcp.json:
{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "cwbl_..." }
    }
  }
}

the same pattern works for Codex, Claude Desktop, and any other host that accepts a stdio mcp launch command — set command: npx, args: ["-y", "@crawlbrulee/mcp"], and forward CRAWLBRULEE_API_KEY via the env block.

configuration

env var	required	description
`CRAWLBRULEE_API_KEY`	yes	api key sent as `Authorization: Bearer …`. get one at https://crawlbrulee.com.

the mcp reads the env var on first tool invocation — not at startup — so a typo in your config surfaces as a clear tool-error message rather than the server failing to come up. see authentication for how the api consumes keys.

tools

`scrape`

fetch a single url and return the requested content (markdown, cleaned html, raw html, links, images, screenshot, page metadata).

input — only url is required; everything else has sane defaults.

JSON Config

{
  "url": "https://example.com",
  "extract": {
    "markdown": true,
    "links": true,
    "screenshot": { "type": "full_page", "device_mode": "desktop" },
  },
  "require_js": false,
  "proxy": "basic",
  "cleanup": { "ads_and_popups": true, "exclude_selectors": ["nav", "footer"] },
  "cache": { "max_age": 3600 },
  "location": { "locale": "en-US", "country": "US" },
}

output — full scrape result. page metadata (title, OG tags, etc.) is returned under metadata. extracted images are returned as absolute urls — query strings are preserved, and relative srcs are resolved against the page url. screenshots are returned as signed download urls the agent can fetch separately. in rare cases a screenshot can't be captured: when you requested other outputs too, the screenshot field is simply left out while the rest is still returned — but a screenshot-only call that can't deliver errors instead (unsupported_screenshot_output, HTTP 422, when the content type can't be screenshotted) and isn't billed. the result also carries a top-level response_meta.usage block:

JSON Config

{
  "url": "https://example.com",
  "markdown": "...",
  "metadata": { "title": "Example Domain" },
  "response_meta": {
    "usage": {
      "credits": 1,
      "engine": "http", // "http" | "browser" | "screenshot" | "cache"
      "proxy": "basic", // resolved tier actually used: "basic" | "advanced" (never "auto")
      "screenshot_slices": 0, // 1 when the screenshot-split add-on was billed, otherwise 0
    },
  },
}

alongside response_meta.usage, the result surfaces any non-fatal warnings — stable string codes an agent can switch on. an outsized page is truncated rather than refused, and the code names which part was cut:

code	what it means for the payload
`screenshot_truncated`	the page was taller than the scrolling-capture height cap; the screenshot covers the top of the page.
`links_truncated`	the page had more than 30,000 links; the `links` array is cut at the cap and is incomplete.
`inline_images_truncated`	the page had more than 10,000 inline images; the `images` array is cut at the cap and is incomplete.
`raw_html_truncated`	the page body exceeded 10,000,000 characters; `raw_html` is cut at a tag boundary, never mid-tag.
`metadata_truncated`	the page head exceeded 2,000,000 characters; `metadata` can be missing tags that sat past the cut.

and if you requested an extract that doesn't apply to the content type (e.g. markdown of a pdf), the field name comes back in an unsupported_fields list — with the rest of the payload still returned.

every input field, its default, and its constraints are documented under the scrape endpoint — with extraction, screenshots, proxies & location, and caching covering the individual blocks.

`scrape_async`

submit a scrape job to run asynchronously and get back a job_id immediately, instead of holding the connection open. use this for long-running scrapes (heavy js rendering, full-page screenshots of long pages); for a quick one-shot fetch prefer the synchronous scrape tool. then poll scrape_status until the job is done and fetch the page with scrape_result.

takes the same input as scrape plus an optional per-job completion webhook:

JSON Config

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "webhook": {
    // Endpoint that receives one signed `scrape.complete` POST when the job
    // finishes. http/https (HTTPS required in production), max 2048 chars.
    "url": "https://hooks.example.com/cwbl",
    // Opaque correlation object echoed back verbatim in the delivery's
    // `data.metadata`. Serializes to at most 2048 bytes.
    "metadata": { "ref": "order-42" },
  },
}

output — { "job_id": "..." }.

when a webhook is attached, we deliver a single signed scrape.complete POST to your endpoint once the job reaches a terminal state, with your metadata echoed under data.metadata and the job's usage under data.response_meta.usage — so you can react to completion (and track cost) without polling. verify the X-Cwbl-Signature header with the sdk's verifyWebhookSignature (configure the signing secret in the dashboard under account → webhooks).

the job lifecycle is documented under async scrape; the delivery contract and payload shape under webhooks, with the signature scheme in webhook verification.

`scrape_status`

look up the current lifecycle status of an async job: pending, running, done, or failed (with an error message when failed). once the job is done the response also carries a response_meta.usage block (credits, billed engine, resolved proxy tier, screenshot_slices). a cache hit is represented by engine: "cache". poll until done, then call scrape_result.

JSON Config

{ "job_id": "..." }

`scrape_result`

fetch the extracted content of a completed async job — the same result shape as the synchronous scrape tool (including metadata and response_meta.usage). errors if the job is still pending/running, so check scrape_status first.

JSON Config

{ "job_id": "..." }

`map`

build (or fetch a cached) link-map for a website. combines sitemap discovery with homepage link extraction. use this to enumerate a site before scraping selected pages. each link is just { url }.

max_urls (default 5000, max 100000) is a discovery budget, not a trim at the end — discovery stops as soon as that many urls are found, so a smaller value is a faster, cheaper crawl. limit (default 5000, max 10000) only pages the answer.

returned urls are normalized the same way scrape normalizes its returned url, so map-then-scrape stays on one host. results are ordered with the most useful links first.

the response's response_meta carries pagination, truncation, and a usage block (credits, billed engine, resolved proxy tier). map responses do not include screenshot-slice accounting.

JSON Config

{
  "url": "https://example.com",
  "sitemap_only": false,
  "types": { "internal": true, "external": false, "internal_subdomains": true },
  "max_urls": 5000,
  "page": 1,
  "limit": 1000,
}

a map stopped by your own max_urls returns exactly that many links with response_capped: false — the signal that the site has more is truncation.discovery_cap_reason:

JSON Config

{
  "truncation": {
    "storage_capped": false,
    "response_capped": false,
    "total_before_max_urls": 5000,
    "total_detected_before_storage_cap": 5000,
    "discovery_capped": true, // discovery stopped before reading every sitemap file
    "sitemaps_skipped": 3, // files skipped or only partly read
    "discovery_cap_reason": "max_urls", // retry with a higher max_urls
  },
}

discovery_cap_reason is one of max_urls, time, file_budget, depth, file_size, unread_files, or null when nothing stopped discovery. only max_urls is a limit you can raise from the request. unread_files means a sitemap file the site publishes could not be read at all this time — often temporary, so asking again later can return more. time, file_budget, depth and file_size mean the site itself is big, slow or deep, and a retry will not help.

see the map endpoint for discovery rules and pagination semantics.

`usage`

returns the current billing-cycle snapshot: total / used / available credits, used quota percent, max concurrency, and cycle reset timestamp. takes no arguments. what a call costs, and how credits are counted, is documented under credits & pricing.

`whoami`

returns the organization name, token name, and truncated token preview for the configured api key. useful for confirming which account is in use before credit-consuming operations.

errors

every tool returns an mcp error result (isError: true) when the api call fails. the error text follows a stable format:

Code

[<errorName>] <message> (HTTP <status>)

agents can branch on the errorName code. the set comes from the sdk's ApiErrorName union plus two synthetic codes added by this mcp (missing_api_key, internal_error):

code	meaning
`missing_api_key`	`CRAWLBRULEE_API_KEY` is not set in the mcp host's env.
`invalid_credentials`	server rejected the api key (revoked, wrong env, etc.).
`service_unavailable`	temporary backend failure (HTTP 503). your key is fine — retry with backoff.
`too_many_requests`	rate limit hit — back off and retry.
`usage_allocation_error`	plan credit / concurrency cap exceeded. show `usage` to user.
`validation_error`	input failed server validation.
`invalid_url`	target url was rejected before fetching.
`blocked_url`	target url is on the blocklist.
`antibot_blocked`	origin's anti-bot defenses blocked the fetch.
`too_many_redirects`	origin redirected the fetch in a loop (HTTP 422). the target's doing — don't retry blindly.
`page_too_large`	the page's html was too large to process (HTTP 422). terminal — never retry it.
`scrape_error`	origin returned an error during scraping.
`unsupported_screenshot_output`	screenshot-only request on a content type that can't be screenshotted (HTTP 422). not billed.
`not_found`	async job ID unknown (e.g. bad `job_id` to `scrape_status` / `scrape_result`).
`request_timeout`	network / read timeout. safe to retry.
`client_closed_request`	caller cancelled before completion.
`internal_server_error`	unhandled server-side failure.
`crawlbrulee_error`	sdk error without a typed name.
`internal_error`	bug in this mcp — please open an issue.

the api docs carry the canonical error reference — every error name, what causes it, and how to recover.

development

bash

pnpm install
pnpm typecheck   # tsc --noEmit
pnpm lint        # eslint
pnpm test        # vitest run
pnpm build       # tsup → dist/index.js with shebang
pnpm verify      # all of the above

run the built mcp locally:

bash

CRAWLBRULEE_API_KEY=cwbl_... node ./dist/index.js

it will block waiting for an mcp client on stdio. combine with the MCP Inspector for interactive debugging.

docs

this readme covers the mcp server itself — installing it, wiring it into a host, and the tools it exposes. for how the api behaves — endpoints, parameters, and error semantics — the api docs are canonical. the mcp guide covers host setup in more depth.

part of the crawlbrulee toolkit

one api, many ways to call it:

js/ts sdk — @crawlbrulee/sdk (the sdk this mcp wraps)
python sdk — crawlbrulee on pypi
cli — npx crawlbrulee
mcp server — @crawlbrulee/mcp (this one)
agent skills — for skills-aware coding agents

docs: crawlbrulee.com/docs · dashboard: dashboard.crawlbrulee.com

license

Apache-2.0