CRW Web Scraper

What CRW Web Scraper MCP server does

CRW Web Scraper MCP server connects AI agents to fastCRW’s web data engine. The engine supports five main operations: scraping a single URL, crawling a bounded portion of a site, discovering URLs with a map operation, searching the web, and extracting selected fields from one or multiple URLs.

Scrape results can be returned as Markdown, HTML, links, screenshots, or schema JSON. Search can optionally scrape selected results, while crawl collects pages discovered within the configured scope. Mapping is intended for URL discovery without downloading every page. These operations make the server suitable for research, documentation gathering, website analysis, and agent workflows that need source content.

How it works

The project can run as a local Rust binary or connect to the managed fastCRW API. Local mode avoids an account and can be started with the crw command after installation. Managed mode uses an API key and provides hosted capabilities such as managed proxies, JavaScript rendering, and search.

The MCP package is installed with npx -y crw-mcp@latest install. The installer adds the CRW skill and MCP server to detected AI tools. The command-line setup can perform the same MCP registration step. Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf are detected when already configured.

Deployment affects available capabilities and response formats. The project provides a capabilities endpoint and documents separate response shapes for different deployments, so clients should not assume that local and managed runs expose identical behavior.

Setup and configuration

Install locally with:

curl -fsSL https://fastcrw.com/install | sh

For cloud access, provide CRW_API_KEY while running the installer:

curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh

The key is stored in ~/.config/crw/config.toml rather than copied into each configured AI tool. Add CRW_NO_AGENTS=1 if the installer should skip automatic agent registration. Alternatively, run crw setup to configure the integration interactively. The documented installation targets are macOS and Linux on Intel and ARM systems.

Tools and capabilities

• Scrape one URL into Markdown, HTML, links, screenshots, or schema JSON.
• Crawl a bounded site area and collect its pages.
• Map a site to discover URLs without scraping every page.
• Search the web and optionally scrape chosen results.
• Extract structured fields from one or more URLs.
• Use local execution or the managed API, depending on infrastructure and data-control needs.

Limitations and notes

CRW Web Scraper MCP server does not guarantee identical features or response structures across local and managed deployments. In local or self-hosted mode, the operator chooses renderers, search, authentication, proxies, and capacity. Managed mode handles hosted services and billing instead.

The README documents the MCP installation flow but does not specify individual MCP tool schemas, argument names, crawl limits, authentication headers, or rendering configuration. Consult the project’s API and MCP documentation before building against exact response fields.

The engine and MCP server use AGPL-3.0. The separate Python and TypeScript SDKs use MIT, which does not change the license of the MCP server itself. CRW Web Scraper MCP server is therefore best evaluated separately from the SDKs if licensing requirements apply to an integration.