Webcrawl MCP

What andyliszewski/webcrawl-mcp MCP server does

The andyliszewski/webcrawl-mcp MCP server gives MCP clients four web research tools: single-page scraping, DuckDuckGo search, same-domain URL mapping, and multi-page crawling. Scraped results are returned as Markdown together with a source indicator that identifies whether the content came from a normal HTTP request, a retry, or a Firecrawl fallback.

The local-first design is intended for public articles, documentation, blogs, and similar pages that do not require browser rendering. It can reduce dependence on external services because ordinary extraction happens on the machine running the server.

How it works

webcrawl_scrape fetches one URL and extracts its main content. Trafilatura handles the primary HTML-to-content conversion. If that output is unavailable or shorter than the quality threshold, the server can use Markdownify on the raw HTML, and then optionally call Firecrawl if the result remains poor.

webcrawl_search queries DuckDuckGo and can optionally scrape returned results. webcrawl_map starts at a page and returns links belonging to the same domain. webcrawl_crawl follows discovered pages breadth-first, up to the requested depth. Crawl requests include rate limiting between requests, and search usage may need spacing because DuckDuckGo can throttle bursts.

For HTTP transport errors such as 403, 429, or 503, a 429 response with a usable Retry-After header receives one bounded retry by default. Firecrawl transport fallback is opt-in.

Setup and configuration

Install the package with Python 3.12 or newer using pip install webcrawl-mcp. The executable is webcrawl-mcp; MCP clients can also run it through uvx with the package name as its argument. The README provides configuration examples for Claude Code and Claude Desktop, and notes that the standard command, argument, and environment structure also applies to clients such as Cursor and Cline.

All configuration variables are optional. USER_AGENT changes the HTTP user agent, REQUEST_TIMEOUT controls request timeouts, and POLITE_MODE controls the bounded 429 retry behavior. Set FIRECRAWL_API_KEY to enable Firecrawl fallback, and use FIRECRAWL_API_URL to change its endpoint. FALLBACK_ON_TRANSPORT_ERROR enables Firecrawl handling for selected transport failures when a key is present.

Tools and capabilities

• Scrape one URL into Markdown with provenance.
• Search DuckDuckGo and optionally retrieve result content.
• Discover same-domain links from a starting page.
• Crawl multiple pages using breadth-first traversal and a maximum depth.

Limitations and notes

JavaScript-rendered pages may produce little content through local extraction. Configure a Firecrawl key for those cases, or use another approach when browser rendering is essential. Without that key, the server remains self-contained and does not use the Firecrawl fallback.

The tool is intended for public web content and includes per-domain rate limiting, but users must still follow each site's terms, robots.txt, applicable law, and reasonable request limits. The project is released under the MIT license.