Log inSubmit server
PyreCrawl logo

PyreCrawl

Self-hosted MCP web tools for scraping, crawling, searching, extracting, monitoring, and researching online content.

InstallRepositoryWebsite
Full README Compare alternatives Embed widget Claim this listing

Install PyreCrawl

Pick your client, then add it in one click or paste the config.

STDIOPython
~/Library/Application Support/Claude/claude_desktop_config.json
claude_desktop_config.json
{
  "mcpServers": {
    "pyrecrawl": {
      "command": "uvx",
      "args": [
        "pyrecrawl@latest"
      ],
      "env": {
        "PYRECRAWL_CACHE": "YOUR_VALUE_HERE",
        "PYRECRAWL_CACHE_TTL": "YOUR_VALUE_HERE",
        "PYRECRAWL_MONITOR_DIR": "YOUR_VALUE_HERE",
        "PYRECRAWL_NO_TELEMETRY": "YOUR_VALUE_HERE",
        "DO_NOT_TRACK": "YOUR_VALUE_HERE"
      }
    }
  }
}

Settings → Developer → Edit Config, merge this into mcpServers, then fully quit and reopen Claude Desktop.

Values are written into the snippet above so you can paste it as is. They never leave your browser.

Automated check passed— started and listed 13 tools correctly (22d ago).

Overview

PyreCrawl MCP server gives AI agents tools for scraping pages, crawling sites, extracting structured data, searching the web, and processing documents. It runs locally through MCP and uses an automatic fallback from HTTP to a stealth Chromium browser and deeper processing when needed. Use it for web research, content ingestion, academic searches, page monitoring, and browser sessions without external API keys. Installation requires Python tooling and a one-time browser-engine setup.

Use cases

  • Scrape web pages into Markdown for agent analysis
  • Crawl documentation sites with path and depth filters
  • Extract structured product data with CSS schemas
  • Research topics with cited source evidence
  • Monitor pages for content changes

Key features

  • Automatic HTTP, browser, and deep-processing fallback
  • Structured extraction with JsonCss schemas
  • Web and academic search without API keys
  • Parallel batch scraping with cache awareness
  • Persistent browser sessions and page monitoring
  • PDF, DOCX, and PPTX conversion to Markdown

Tools13

What PyreCrawl lets your agent do. Expand a tool for its parameters.

Verified live~3.7k tokens of context

Security & health

What it can reach and how reliably it runs — from our own checks, not the maintainer’s claims.

ActiveLast checked 2d ago

Recent health check succeeded.

5/5 checks healthy over the last 46d

Supply-chain signal

No high-severity advisories surfaced by our automated scan.

Critical 0High 0Medium 0Low 0

Scanned 23h ago via OSV.dev · pyrecrawl@latest (PyPI)

Adoption & maintenance

Factual signals from GitHub, npm, and our automated checks — not a rating.

GitHub stars22Stargazers on the source repository.
Last commit1mo agoMost recent push to the default branch.
Availability100%Our rolling endpoint + install checks that succeeded.
Install checkPassedOur sandbox started it and listed its tools.
Tools exposed13Callable tools this server registers over MCP.
Directory activity6 viewsConfig copies, upvotes, and views on AllMCPs.

How PyreCrawl works

What PyreCrawl MCP server does

PyreCrawl MCP server exposes web access and content-processing operations to MCP-compatible AI agents. Its core tools cover single-page scraping, structured extraction, site mapping, multi-page crawling, parallel URL fetching, web search, academic paper search, document conversion, research collection, change monitoring, persistent browser sessions, cache management, and health checks.

The server is intended for local, self-hosted use and does not require API keys. It belongs in the Search & Data Extraction category for agents that need to gather current web content or turn online sources into material for summarization, retrieval, or analysis.

How it works

Requests can use one of four preference modes: automatic selection, HTTP-only fetching, stealth browser fetching, or deeper processing. In automatic mode, PyreCrawl starts with a fast HTTP request. If the response is blocked, empty, or affected by a Cloudflare challenge, it can escalate to a Chromium-based browser. Pages that need JavaScript rendering or remain blocked can then use deeper processing that produces LLM-ready Markdown, citations, or structured output.

The crawl tool discovers and processes multiple pages with path filters, page limits, and breadth-first depth handling. map_site lists internal URLs without performing a full crawl, while batch_scrape processes multiple URLs in parallel with deduplication and cache awareness. deep_research combines web search and top-result scraping into an evidence pack with numbered citations; it does not perform the final LLM synthesis.

Setup and configuration

Install the package with uvx, uv tool, pipx, or pip in a virtual environment. Run pyrecrawl setup once to install Chromium and the stealth browser engines. The pyrecrawl install command can detect supported agents and write MCP configuration, or target specific agents such as Claude Desktop, Cursor, Claude Code, VS Code, and Codex. A manual stdio configuration is also available for Claude Desktop and Claude Code.

Optional environment variables control local behavior. PYRECRAWL_CACHE enables an in-memory cache or specifies a cache directory, PYRECRAWL_CACHE_TTL sets cache lifetime, PYRECRAWL_MONITOR_DIR selects the directory for saved monitor snapshots, and PYRECRAWL_NO_TELEMETRY disables the anonymous startup ping. The server also honors DO_NOT_TRACK=1 for that telemetry setting.

Tools and capabilities

  • scrape returns a URL as LLM-ready Markdown.
  • extract applies a JsonCss schema to produce structured data.
  • search uses DuckDuckGo HTML without an API key.
  • search_papers searches arXiv and Crossref; returned PDF URLs can be sent to document.
  • document converts PDF, DOCX, and PPTX files to Markdown, with optional document extras.
  • monitor stores snapshots and reports unified diffs for content changes.
  • session preserves browser cookies for login walls, multi-step flows, and screenshots.

Read-only MCP resources expose cache statistics, sessions, and monitors. The included prompts provide playbooks for research, RAG ingestion, and page watching.

Limitations and notes

Browser-dependent behavior requires the one-time engine installation and may take longer than HTTP fetching. The automatic fallback is not a guarantee that every protected or JavaScript-heavy page can be accessed. Academic search is limited to the stated arXiv and Crossref sources, and web search uses DuckDuckGo HTML. Persistent monitoring depends on the configured snapshot directory. PyreCrawl MCP server supplies research evidence and extracted content, but an agent must perform any final synthesis.

Read the full README View source on GitHub

Reviews

No reviews yet — be the first to share how it worked for you.

Frequently Asked Questions about PyreCrawl

PyreCrawl is an MCP server that connects AI agents to websites, documents, DuckDuckGo, arXiv, and Crossref. Its main tools scrape, crawl, extract, map, search, search papers, process documents, collect cited research evidence, monitor changes, and manage persistent browser sessions for local web data extraction.

Install it with uvx, uv tool, pipx, or pip, then run `pyrecrawl setup` to install Chromium and stealth browser engines. Run `pyrecrawl install` to register the server with detected agents, or target supported agents explicitly.

No, PyreCrawl does not require API keys for its web or academic search tools. Web search uses DuckDuckGo HTML, while academic search uses arXiv and Crossref. The server is designed to run locally.

PyreCrawl can return a page as LLM-ready Markdown or extract structured data using a JsonCss schema. It can also enumerate internal URLs, crawl multiple pages, and fetch batches of URLs in parallel.

Yes, PyreCrawl lists Cursor among its supported agents. Use `pyrecrawl install cursor` to target Cursor's MCP configuration, then restart the agent or begin a new session.

Related MCP servers

All in Search & Data Extraction All alternatives
Add a badge to your READMEShow it’s listed — live SVG, Markdown or HTML.

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:
Live Dynamic SVG PreviewPyreCrawl AllMCPs Directory Badge
Markdown (GitHub README)
[![AllMCPs](https://allmcps.com/api/badge/pyrecrawl?style=directory)](https://allmcps.com/mcp/pyrecrawl)
HTML Embed
<a href="https://allmcps.com/mcp/pyrecrawl"><img src="https://allmcps.com/api/badge/pyrecrawl?style=directory" alt="PyreCrawl on AllMCPs" /></a>