fastCRW

The web data API for AI agents — search, scrape, map, and crawl any site into clean markdown or JSON.

Use the managed cloud for zero infra, or self-host the same open-source engine. Start with Python, TypeScript, cURL, or MCP — you never touch Rust.

Beats Firecrawl and Crawl4AI on truth-recall — measured on Firecrawl's own public dataset.

Quickstart · Docs · Pricing

Works with Claude Code · Cursor · Windsurf · Cline · Codex · Gemini CLI

Quickstart — your first scrape in 30 seconds

Get a free API key → fastcrw.com/register — 500 free credits (1 credit ≈ 1 page), no card.

export CRW_API_KEY="crw_live_..."

# cURL — works anywhere, no SDK
curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer $CRW_API_KEY" -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

pip install crw

from crw import CrwClient

crw = CrwClient()  # reads CRW_API_KEY from your env
page = crw.scrape("https://example.com",
                  formats=["markdown"])
print(page["markdown"])

npm install crw-sdk

import { CrwClient } from "crw-sdk";

const crw = new CrwClient();  // reads CRW_API_KEY from your env
const page = await crw.scrape("https://example.com",
                              { formats: ["markdown"] });
console.log(page.markdown);

In both SDKs page is a plain object (markdown, metadata, contentType, …), so page["markdown"] / page.markdown is clean content:

# Example Domain

This domain is for use in documentation examples without needing permission. Avoid use in operations.

[Learn more](https://iana.org/domains/example)

Over cURL you get the same fields wrapped in {"success": true, "data": { … }}.

Prefer no SDK? Every example works over plain HTTP against https://api.fastcrw.com. That first scrape spent 1 of your 500 free credits — see plans → when you need more.

Got one page? Crawl the whole site: crw.crawl("https://docs.example.com") returns every page — then search, map, and extract in Core operations. Full docs: Quickstart → · API reference →

_{Firecrawl's own public 1,000-URL dataset, with all three tools run through the same matcher (diagnose_3way.py): a fairness control, not a looser number. The panel is generated from the run of record, not drawn by hand: scripts/bench-radar.py. Full table and one-command repro ↓}

Why fastCRW?

• Most accurate — the highest truth-recall (how much of the real page content it captures): 63.7% on 819 labeled URLs in Firecrawl's public dataset, vs Firecrawl 56.0% and Crawl4AI 60.0% — and it recovers 34 pages both miss.
• Fast median, tunable latency — the fastest median latency (p50 1914 ms — a statistical tie with Crawl4AI's 1916 ms, ahead of Firecrawl's 2305 ms). Recall mode maximizes accuracy; fast mode trades the recall tail for lower latency. One config toggle — pick accuracy or latency.
• Lighter — one static binary, ~50 MB RAM idle. No Redis, no Node, no Chromium heap in the request path — it runs on a $5 VPS.

Search, map, and crawl run on the same engine — built-in web search (a free self-hostable search backend), so there's no separate search vendor and no per-query search-API bill. See the full benchmark →

Open source (AGPL-3.0), passing OpenSSF Best Practices, and published on PyPI · npm · crates.io · Homebrew · APT — with a benchmark you can rerun yourself, not marketing math.

What you get

• Any URL → LLM-ready output. Clean markdown, HTML, links, or schema-validated JSON — no HTML soup, no boilerplate.
• One API, six operations. search the web and scrape a page, or map / crawl / extract / monitor a whole site — see below.
• Drop-in Firecrawl compatibility. Migrate existing Firecrawl code by changing one base URL.
• Managed or self-hosted, same API. No exit cost — develop against the free open-source binary, ship to the cloud, or the reverse. Nothing changes but the base URL.

Use it in your AI agent (MCP)

fastCRW ships a built-in MCP server, so any MCP host can search/scrape/crawl with no glue code.

# Claude Code — managed
claude mcp add crw \
  -e CRW_API_URL=https://api.fastcrw.com -e CRW_API_KEY=$CRW_API_KEY \
  -- npx -y crw-mcp

# Claude Code — embedded (no server, no key — runs the engine locally, on your machine)
claude mcp add crw -- npx -y crw-mcp

Per-client recipes (Cursor, Windsurf, Cline, Continue.dev, Codex, Gemini CLI): docs.fastcrw.com/mcp-clients/

Agent Skills

Reusable instruction packs that teach coding agents when and how to use each verb. Install all 13 into every detected agent with one command:

npx skills add us/crw            # all skills, every detected agent
npx skills add us/crw@crw-scrape # just one
npx skills add -g us/crw         # global (user-level)

crw (hub) · crw-search · crw-scrape · crw-map · crw-crawl · crw-parse · crw-extract · crw-watch · crw-research · crw-dynamic-search (biggest token-saver) · crw-best-practices · crw-migrate · crw-self-host. Full catalog: skills/.

Core operations

SDK return shapes: scrape / extract → one object · map → list of URLs · crawl → list of result objects · search → list, or a dict grouped by source when sources=[...] is set. Full reference: docs.fastcrw.com/#rest-api.

SDKs & integrations

pip install crw          # Python package: crw
npm install crw-sdk      # Node / TypeScript package: crw-sdk (not crw)

from crw import CrwClient

client = CrwClient()   # reads CRW_API_KEY; set CRW_LOCAL=1 for local embedded mode
client.scrape("https://example.com", formats=["markdown", "links"])
# .search() .map() .crawl() .extract() — one method per operation in the table above

import { CrwClient } from "crw-sdk";

const crw = new CrwClient();  // reads CRW_API_KEY; new CrwClient({ apiUrl }) for self-host
await crw.scrape("https://example.com", { formats: ["markdown", "links"] });
// .search() .map() .crawl() .extract() — same methods, all typed

The TypeScript client is typed and zero-dependency; its cloud path is pure fetch, so it runs on Node 18+, Bun, Deno, and edge runtimes. The Python client is synchronous — wrap long calls like crawl() / extract() in asyncio.to_thread inside async code. Both client SDKs (crw, crw-sdk) are MIT-licensed — installing them imposes nothing on your code; AGPL-3.0 covers only the engine.

LangChain and CrewAI integrations ship in the package:

from crw.integrations.langchain import CrwLoader          # pip install crw[langchain]
from crw.integrations.crewai import CrwScrapeWebsiteTool   # pip install crw[crewai]

All integrations → · SDK examples →

Managed cloud vs self-host

Same binary, same API in both modes — switch anytime by changing the base URL. Most teams run on the managed cloud: it scales with your traffic, rotates proxies, and stays patched, so you ship features instead of operating a scraper.

Air-gapped note: the crw engine itself makes no telemetry calls. The default Docker Compose stack disables the LightPanda renderer's third-party telemetry (LIGHTPANDA_DISABLE_TELEMETRY=true). If you run a fully air-gapped deployment, verify egress for any renderer sidecars you enable and pin their images to a digest. See self-hosting hardening.

Self-host in one command

docker run -p 3000:3000 ghcr.io/us/crw
curl http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

Or install the CLI and scrape straight from your shell:

brew install us/crw/crw                            # macOS & Linux
curl -fsSL https://apt.fastcrw.com/setup.sh | sudo sh   # Debian & Ubuntu
curl -fsSL https://fastcrw.com/install | sh        # any platform, no package manager

crw scrape https://example.com

Prefer Cargo, Docker Compose with a stealth tier, or building from source? All install paths and production hardening: docs.fastcrw.com/installation/ · self-hosting guide →

_{Skip the setup — 500 free credits on the managed cloud, no card.}

Why it's fast (built in Rust)

You don't need Rust to use fastCRW — it's why the numbers below are what they are. The engine is a single static binary: no Redis, no Node runtime, no Python venv, no headless-browser sidecar parked in the request path. Cold start is sub-second and idle RAM sits around ~50 MB, so one process saturates a $5 VPS instead of a multi-container stack. An agent that fires N scrapes per task pays the network floor N times — fastCRW strips process-spawn, JIT-warmup, and browser-navigation overhead out of every one.

Benchmark

The numbers above come from Firecrawl's own public 1,000-URL dataset, run identically across all three tools with the same matcher — a fairness control, not a looser number. Reproducible, not marketing math. Full table, methodology, and the repro harness: BENCHMARKS.md · fastcrw.com/benchmarks.

Migrating from Firecrawl

New projects use native /v1. Existing Firecrawl v2 SDK code works against the /firecrawl/v2/* compatibility layer — often just a base-URL swap (point your existing Firecrawl client at api.fastcrw.com, the address it sends requests to):

from firecrawl import FirecrawlApp
app = FirecrawlApp(api_url="https://api.fastcrw.com", api_key="YOUR_CRW_API_KEY")

Compatibility reduces migration work, not every behavioral difference — check request bodies, response fields, and unsupported features before moving production traffic. Field-by-field diff: COMPATIBILITY-firecrawl.md.

Security

SSRF protection (blocks loopback, private IPs, cloud metadata, non-HTTP schemes), optional constant-time Bearer auth, RFC 9309 robots.txt, token-bucket rate limiting, and resource caps (1 MB body, depth 10, 1,000 pages). Hardening guide →

Contributing

Issues and PRs welcome. make hooks installs the pre-commit hook; make check runs the same checks as CI. Setup, architecture, and crate layout: CONTRIBUTING.md.

Contributors

_{Everyone who has landed a commit, refreshed weekly from the contributor graph.}

License

Open source under AGPL-3.0. Calling the API over the network — managed or self-hosted — imposes nothing on your own code. AGPL only applies if you bundle the engine's code directly inside your own app (not just call its API) or run a modified copy as a public service; for those cases the managed offering at fastcrw.com includes a commercial carve-out, and standalone commercial licenses are available — hello@fastcrw.com.

Links

Docs · API reference · MCP setup · Benchmarks · Pricing · Changelog · Discord · X

Star History

_{It is the sole responsibility of end users to respect websites' policies when scraping. By default, fastCRW respects robots.txt directives.}