# us/crw [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/us/crw  
**GitHub Stars:** 974  
**Views:** 4  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/us-crw

## Description
fastCRW — open-source (AGPL-3.0), self-hostable Rust web crawler & search API for AI agents. Tools: scrape, crawl, map, and SearXNG-backed search. Single 6MB static binary; reproducible 1K-URL benchmarks faster than hosted alternatives. Hosted MCP at fastcrw.com/mcp (Streamable HTTP, OAuth) or self-host.

## Tools
Capabilities this server exposes over MCP:

- **crw_scrape** — Scrape one URL to markdown, HTML, or links.
- **crw_crawl** — Start an async site crawl; returns a job id to poll with crw_check_crawl_status.
- **crw_check_crawl_status** — Poll an async crawl job and retrieve its pages.
- **crw_map** — Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.
- **crw_extract** — Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.
- **crw_check_extract_status** — Poll an extract job; returns status and, when complete, a per-URL results array.
- **crw_cancel_extract** — Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.
- **crw_parse_file** — Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "crw": {
    "command": "npx",
    "args": ["-y","crw-mcp@latest"],
    "env": {
      "CRW_API_KEY": ""
    }
  }
}
```

**Requires environment variables:** `CRW_API_KEY` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation

## What us/crw does

The us/crw MCP server exposes fastCRW’s web retrieval functions to MCP-compatible AI clients. It handles individual-page scraping, bounded site crawling, URL discovery, web search through SearXNG, structured extraction, and local PDF parsing. Results can be returned as markdown, HTML, links, screenshots, or structured JSON where the selected deployment supports that response shape.

The project is implemented as a small Rust binary and can run locally or be used through the managed API. Local operation is intended for environments that need control over data, networking, rendering, authentication, proxies, and capacity. The hosted service provides managed proxies, JavaScript rendering, search, scaling, and billing.

## How it works

A scrape request processes one URL and returns the requested representation. Mapping uses a sitemap and/or a short crawl to produce URLs without returning page contents. Crawling is asynchronous: the initial request returns a job identifier, and a separate status tool retrieves the collected pages.

Structured extraction also runs asynchronously. The caller supplies a prompt and/or JSON schema, receives an extraction job ID, and polls until per-URL results are available. Cancellation is requested through a separate tool; the job remains non-terminal until the currently claimed URL finishes settling. PDF parsing accepts local file data as base64 and converts it to markdown, but does not perform OCR.

## Setup and configuration

The us/crw MCP server can be installed with the project’s npm MCP package:

```bash
npx -y crw-mcp@latest install
```

The repository also documents a one-command installer for the local `crw` binary:

```bash
curl -fsSL https://fastcrw.com/install | sh
```

No account is needed for local and free operation. Managed Cloud use requires a `CRW_API_KEY`; the installer can use that variable while registering the server with detected AI tools. The setup process can detect Cursor and Windsurf, among other listed tools, or can be skipped with `CRW_NO_AGENTS=1`. macOS and Linux are supported on Intel and ARM systems.

## Tools and capabilities

- `crw_scrape` converts one URL to markdown, HTML, or links.
- `crw_crawl` starts an asynchronous bounded crawl and returns a job ID.
- `crw_check_crawl_status` polls a crawl and retrieves its pages.
- `crw_map` discovers URLs without collecting page content.
- `crw_extract` creates structured JSON from one or more URLs using a prompt and/or schema; it requires an LLM.
- `crw_check_extract_status` returns extraction progress and completed per-URL results.
- `crw_cancel_extract` requests cancellation and returns the canonical job status.
- `crw_parse_file` parses a base64-encoded local PDF into markdown.

## Limitations and notes

Capabilities and response formats can differ between managed and self-hosted deployments. Local operators choose their own renderers, search configuration, authentication, proxies, and capacity. The supplied PDF parser has no OCR support, so scanned PDFs may produce empty markdown with a warning. Extraction depends on an LLM and should be treated as an asynchronous workflow rather than an immediate response.

The engine and MCP server use AGPL-3.0. The repository separately identifies its Python and TypeScript SDKs as MIT-licensed. The hosted deployment offers 1,000 free credits without a credit card, while managed operations include billing beyond the free allocation.

## Getting started with this us/crw MCP server
Always refer to the official documentation for the most accurate and up-to-date information.

_Full upstream README: https://allmcps.com/mcp/us-crw/readme_