WebReaper

What alex-on-ai/WebReaper MCP server does

alex-on-ai/WebReaper MCP server gives MCP-compatible agents access to a self-hosted web scraping engine. Its exposed tools cover six operations: scrape, map, extract, extract_with_prompt, extract_inferred, and crawl. Scraping can return page content as Markdown, while extraction supports structured fields defined by a JSON schema or inferred from a natural-language description.

The underlying WebReaper application can discover links on a site, recursively crawl same-domain pages, and write crawl results as JSON Lines. It also supports output intended for agent workflows, including clean page content rather than raw HTML. The project is distributed under the MIT license and is described as a single-binary application.

How it works

The stdio implementation, WebReaper.Mcp, is intended for local clients that launch the server process. The Streamable HTTP implementation, WebReaper.Mcp.AspNetCore, lets a client connect to a URL instead; the README identifies n8n as a use case for the HTTP variant. The two MCP servers expose the same tool set.

Page loading starts with an HTTP request. When WebReaper detects signs of blocking, it can escalate to a browser and, when enabled, a stealth backend. This escalation can happen per page, while a blocked host keeps the higher working tier for later pages. A page that remains blocked at the highest available tier is omitted from the output and causes a non-zero run result.

LLM-assisted operations accept an OpenAI-compatible endpoint and can also work with providers supported through Microsoft.Extensions.AI, including OpenAI, Anthropic, Ollama, Azure OpenAI, and llamafile. The supplied material does not specify MCP-specific configuration for choosing among those providers.

Setup and configuration

Install options for the underlying application include a Homebrew formula, a shell installation script, Windows release binaries, and the WebReaper .NET package. The README names the MCP packages WebReaper.Mcp and WebReaper.Mcp.AspNetCore, but the supplied excerpt does not provide a complete command for launching either package directly, so the exact MCP installation command should be confirmed from the package documentation.

The HTTP example runs a Chromium-baked container on port 8080 and sets WEBREAPER_MCP_TOKEN. Clients connect to the resulting URL with a bearer token. The stdio server is intended for local process-spawning clients such as Cursor and Claude Desktop.

Tools and capabilities

• scrape retrieves one page, with Markdown as the default style.
• map discovers URLs and can filter or limit results.
• extract uses a supplied JSON schema for structured output.
• extract_with_prompt extracts requested fields using an LLM endpoint.
• extract_inferred infers a schema from a description before extraction.
• crawl recursively processes on-domain pages.
• Browser and stealth loading support JavaScript-rendered or bot-protected pages.

Limitations and notes

alex-on-ai/WebReaper MCP server is self-hosted; the excerpt does not describe a hosted public endpoint. Browser fallback does not require an explicit flag for ordinary automatic escalation, but stealth mode must be selected or enabled for unattended use. The stealth backend is an optional download of approximately 220 MB. LLM-based extraction requires an endpoint and model configuration, while basic HTTP scraping does not require an LLM. The listed compatible clients are limited here to Cursor, Claude Desktop, and the HTTP-oriented n8n use case; broader client compatibility is not established by the supplied material.