Crw

What us/crw does

The us/crw MCP server exposes fastCRW’s web retrieval functions to MCP-compatible AI clients. It handles individual-page scraping, bounded site crawling, URL discovery, web search through SearXNG, structured extraction, and local PDF parsing. Results can be returned as markdown, HTML, links, screenshots, or structured JSON where the selected deployment supports that response shape.

The project is implemented as a small Rust binary and can run locally or be used through the managed API. Local operation is intended for environments that need control over data, networking, rendering, authentication, proxies, and capacity. The hosted service provides managed proxies, JavaScript rendering, search, scaling, and billing.

How it works

A scrape request processes one URL and returns the requested representation. Mapping uses a sitemap and/or a short crawl to produce URLs without returning page contents. Crawling is asynchronous: the initial request returns a job identifier, and a separate status tool retrieves the collected pages.

Structured extraction also runs asynchronously. The caller supplies a prompt and/or JSON schema, receives an extraction job ID, and polls until per-URL results are available. Cancellation is requested through a separate tool; the job remains non-terminal until the currently claimed URL finishes settling. PDF parsing accepts local file data as base64 and converts it to markdown, but does not perform OCR.

Setup and configuration

The us/crw MCP server can be installed with the project’s npm MCP package:

npx -y crw-mcp@latest install

The repository also documents a one-command installer for the local crw binary:

curl -fsSL https://fastcrw.com/install | sh

No account is needed for local and free operation. Managed Cloud use requires a CRW_API_KEY; the installer can use that variable while registering the server with detected AI tools. The setup process can detect Cursor and Windsurf, among other listed tools, or can be skipped with CRW_NO_AGENTS=1. macOS and Linux are supported on Intel and ARM systems.

Tools and capabilities

• crw_scrape converts one URL to markdown, HTML, or links.
• crw_crawl starts an asynchronous bounded crawl and returns a job ID.
• crw_check_crawl_status polls a crawl and retrieves its pages.
• crw_map discovers URLs without collecting page content.
• crw_extract creates structured JSON from one or more URLs using a prompt and/or schema; it requires an LLM.
• crw_check_extract_status returns extraction progress and completed per-URL results.
• crw_cancel_extract requests cancellation and returns the canonical job status.
• crw_parse_file parses a base64-encoded local PDF into markdown.

Limitations and notes

Capabilities and response formats can differ between managed and self-hosted deployments. Local operators choose their own renderers, search configuration, authentication, proxies, and capacity. The supplied PDF parser has no OCR support, so scanned PDFs may produce empty markdown with a warning. Extraction depends on an LLM and should be treated as an asynchronous workflow rather than an immediate response.

The engine and MCP server use AGPL-3.0. The repository separately identifies its Python and TypeScript SDKs as MIT-licensed. The hosted deployment offers 1,000 free credits without a credit card, while managed operations include billing beyond the free allocation.

Getting started with this us/crw MCP server

Always refer to the official documentation for the most accurate and up-to-date information.