Crw

What us/crw does

us/crw exposes fastCRW's web retrieval and extraction features through the Model Context Protocol. Agents can turn a single URL into markdown, HTML, or links with crw_scrape, discover URLs with crw_map, and collect pages across a bounded site crawl. The server also supports web search through its SearXNG-backed search capability.

Structured extraction is handled by crw_extract. An agent supplies URLs plus a prompt and/or JSON schema, and the operation produces structured JSON with help from an LLM. For local documents, crw_parse_file accepts a PDF as base64 content and converts it to markdown.

The project is useful when an agent needs current web content rather than only files or application APIs. It fits both self-hosted environments that require control over infrastructure and managed use cases where hosted crawling services are preferable.

How it works

Scraping and mapping are direct operations. Crawling and structured extraction are asynchronous: crw_crawl and crw_extract return job identifiers, which the agent checks with crw_check_crawl_status or crw_check_extract_status. Extraction jobs can be asked to stop with crw_cancel_extract; cancellation remains in progress until the URL currently being handled reaches a settled state.

The server can run locally or connect to the managed API. The local distribution is a Rust binary available for macOS and Linux on Intel and ARM, and the project describes it as a single static binary. A hosted MCP endpoint is available over Streamable HTTP with OAuth.

Setup and configuration

The quick installation command is:

curl -fsSL https://fastcrw.com/install | sh

This installs the local command-line program without requiring an account. Managed use can be configured by supplying CRW_API_KEY during installation. The key is stored in ~/.config/crw/config.toml. Running crw setup configures the MCP connection interactively, while CRW_NO_AGENTS=1 prevents automatic setup for detected agent tools.

The MCP package can also be installed through npm:

npx -y crw-mcp@latest install

The installer detects Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf when they are already configured. Python and Node.js SDKs, a CLI, and a REST API are also available, although agents using MCP generally only need the MCP installation path.

Self-hosting leaves choices such as renderers, search, authentication, proxies, and capacity to the operator. Managed and local deployments can expose different capabilities and response shapes.

Tools and capabilities

• crw_scrape retrieves one URL as markdown, HTML, or links.
• crw_crawl starts a bounded, asynchronous site crawl.
• crw_check_crawl_status checks a crawl job and returns its pages.
• crw_map finds site URLs through a sitemap and/or a short crawl without returning page content.
• crw_extract creates structured JSON from one or more URLs using a prompt and/or JSON schema; it requires an LLM.
• crw_check_extract_status reports extraction progress and returns per-URL results when complete.
• crw_cancel_extract requests cancellation of an extraction job and returns its canonical status.
• crw_parse_file converts a local PDF supplied as base64 into markdown.

Limitations and notes

crw_parse_file does not perform OCR. Scanned PDFs therefore produce empty markdown accompanied by a warning. crw_map returns URLs only, so a separate scrape or crawl is needed to obtain page content.

Crawls and extractions require status polling rather than returning all results immediately. Structured extraction also depends on an LLM. Hosted deployments may include managed proxies, JavaScript rendering, and search; local deployments require the operator to select and configure the relevant infrastructure.

The engine and MCP server use AGPL-3.0. The Python and TypeScript SDKs use MIT, which is separate from the MCP server license.