# andyliszewski/webcrawl-mcp [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/andyliszewski/webcrawl-mcp  
**GitHub Stars:** 16  
**Views:** 3  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/andyliszewski-webcrawl-mcp

## Description
Local-first web scraping, search, and crawling. Static pages extracted locally via trafilatura; optional Firecrawl fallback only when JS rendering is needed. Four tools: scrape, search (DuckDuckGo), map, crawl.

## Tools
Capabilities this server exposes over MCP:

- **webcrawl_scrape** — Fetch a URL and extract main content as markdown.
- **webcrawl_search** — Search the web using DuckDuckGo.
- **webcrawl_map** — Discover URLs on a website.

Fetches the given URL and extracts all same-domain links.
- **webcrawl_crawl** — Crawl multiple pages starting from a URL.

Uses BFS to discover and fetch pages up to max_depth links away.
Respects rate limiting between requests.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `uvx` (confidence: high):

```json
"mcpServers": {
  "webcrawl-mcp": {
    "command": "uvx",
    "args": ["webcrawl-mcp"]
  }
}
```

## Documentation

## What andyliszewski/webcrawl-mcp MCP server does

The andyliszewski/webcrawl-mcp MCP server gives MCP clients four web research tools: single-page scraping, DuckDuckGo search, same-domain URL mapping, and multi-page crawling. Scraped results are returned as Markdown together with a source indicator that identifies whether the content came from a normal HTTP request, a retry, or a Firecrawl fallback.

The local-first design is intended for public articles, documentation, blogs, and similar pages that do not require browser rendering. It can reduce dependence on external services because ordinary extraction happens on the machine running the server.

## How it works

`webcrawl_scrape` fetches one URL and extracts its main content. Trafilatura handles the primary HTML-to-content conversion. If that output is unavailable or shorter than the quality threshold, the server can use Markdownify on the raw HTML, and then optionally call Firecrawl if the result remains poor.

`webcrawl_search` queries DuckDuckGo and can optionally scrape returned results. `webcrawl_map` starts at a page and returns links belonging to the same domain. `webcrawl_crawl` follows discovered pages breadth-first, up to the requested depth. Crawl requests include rate limiting between requests, and search usage may need spacing because DuckDuckGo can throttle bursts.

For HTTP transport errors such as 403, 429, or 503, a 429 response with a usable `Retry-After` header receives one bounded retry by default. Firecrawl transport fallback is opt-in.

## Setup and configuration

Install the package with Python 3.12 or newer using `pip install webcrawl-mcp`. The executable is `webcrawl-mcp`; MCP clients can also run it through `uvx` with the package name as its argument. The README provides configuration examples for Claude Code and Claude Desktop, and notes that the standard command, argument, and environment structure also applies to clients such as Cursor and Cline.

All configuration variables are optional. `USER_AGENT` changes the HTTP user agent, `REQUEST_TIMEOUT` controls request timeouts, and `POLITE_MODE` controls the bounded 429 retry behavior. Set `FIRECRAWL_API_KEY` to enable Firecrawl fallback, and use `FIRECRAWL_API_URL` to change its endpoint. `FALLBACK_ON_TRANSPORT_ERROR` enables Firecrawl handling for selected transport failures when a key is present.

## Tools and capabilities

- Scrape one URL into Markdown with provenance.
- Search DuckDuckGo and optionally retrieve result content.
- Discover same-domain links from a starting page.
- Crawl multiple pages using breadth-first traversal and a maximum depth.

## Limitations and notes

JavaScript-rendered pages may produce little content through local extraction. Configure a Firecrawl key for those cases, or use another approach when browser rendering is essential. Without that key, the server remains self-contained and does not use the Firecrawl fallback.

The tool is intended for public web content and includes per-domain rate limiting, but users must still follow each site's terms, robots.txt, applicable law, and reasonable request limits. The project is released under the MIT license.

_Full upstream README: https://allmcps.com/mcp/andyliszewski-webcrawl-mcp/readme_