# us/crw [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/us/crw  
**GitHub Stars:** 1182  
**Views:** 5  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/us-crw

## Description
fastCRW — open-source (AGPL-3.0), self-hostable Rust web crawler & search API for AI agents. Tools: scrape, crawl, map, and SearXNG-backed search. Single 6MB static binary; reproducible 1K-URL benchmarks faster than hosted alternatives. Hosted MCP at fastcrw.com/mcp (Streamable HTTP, OAuth) or self-host.

## Tools
Capabilities this server exposes over MCP:

- **crw_scrape** — Scrape one URL to markdown, HTML, or links.
- **crw_crawl** — Start an async site crawl; returns a job id to poll with crw_check_crawl_status.
- **crw_check_crawl_status** — Poll an async crawl job and retrieve its pages.
- **crw_map** — Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.
- **crw_extract** — Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.
- **crw_check_extract_status** — Poll an extract job; returns status and, when complete, a per-URL results array.
- **crw_cancel_extract** — Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.
- **crw_parse_file** — Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.

## Claude Desktop Quick Installation
No confirmed install details yet, so no config is published. Follow the setup steps in the repository above.

## Documentation

## What us/crw does

us/crw exposes fastCRW's web retrieval and extraction features through the Model Context Protocol. Agents can turn a single URL into markdown, HTML, or links with `crw_scrape`, discover URLs with `crw_map`, and collect pages across a bounded site crawl. The server also supports web search through its SearXNG-backed search capability.

Structured extraction is handled by `crw_extract`. An agent supplies URLs plus a prompt and/or JSON schema, and the operation produces structured JSON with help from an LLM. For local documents, `crw_parse_file` accepts a PDF as base64 content and converts it to markdown.

The project is useful when an agent needs current web content rather than only files or application APIs. It fits both self-hosted environments that require control over infrastructure and managed use cases where hosted crawling services are preferable.

## How it works

Scraping and mapping are direct operations. Crawling and structured extraction are asynchronous: `crw_crawl` and `crw_extract` return job identifiers, which the agent checks with `crw_check_crawl_status` or `crw_check_extract_status`. Extraction jobs can be asked to stop with `crw_cancel_extract`; cancellation remains in progress until the URL currently being handled reaches a settled state.

The server can run locally or connect to the managed API. The local distribution is a Rust binary available for macOS and Linux on Intel and ARM, and the project describes it as a single static binary. A hosted MCP endpoint is available over Streamable HTTP with OAuth.

## Setup and configuration

The quick installation command is:

```bash
curl -fsSL https://fastcrw.com/install | sh
```

This installs the local command-line program without requiring an account. Managed use can be configured by supplying `CRW_API_KEY` during installation. The key is stored in `~/.config/crw/config.toml`. Running `crw setup` configures the MCP connection interactively, while `CRW_NO_AGENTS=1` prevents automatic setup for detected agent tools.

The MCP package can also be installed through npm:

```bash
npx -y crw-mcp@latest install
```

The installer detects Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf when they are already configured. Python and Node.js SDKs, a CLI, and a REST API are also available, although agents using MCP generally only need the MCP installation path.

Self-hosting leaves choices such as renderers, search, authentication, proxies, and capacity to the operator. Managed and local deployments can expose different capabilities and response shapes.

## Tools and capabilities

- `crw_scrape` retrieves one URL as markdown, HTML, or links.
- `crw_crawl` starts a bounded, asynchronous site crawl.
- `crw_check_crawl_status` checks a crawl job and returns its pages.
- `crw_map` finds site URLs through a sitemap and/or a short crawl without returning page content.
- `crw_extract` creates structured JSON from one or more URLs using a prompt and/or JSON schema; it requires an LLM.
- `crw_check_extract_status` reports extraction progress and returns per-URL results when complete.
- `crw_cancel_extract` requests cancellation of an extraction job and returns its canonical status.
- `crw_parse_file` converts a local PDF supplied as base64 into markdown.

## Limitations and notes

`crw_parse_file` does not perform OCR. Scanned PDFs therefore produce empty markdown accompanied by a warning. `crw_map` returns URLs only, so a separate scrape or crawl is needed to obtain page content.

Crawls and extractions require status polling rather than returning all results immediately. Structured extraction also depends on an LLM. Hosted deployments may include managed proxies, JavaScript rendering, and search; local deployments require the operator to select and configure the relevant infrastructure.

The engine and MCP server use AGPL-3.0. The Python and TypeScript SDKs use MIT, which is separate from the MCP server license.

_Full upstream README: https://allmcps.com/mcp/us-crw/readme_