# PyreCrawl [Health: Active] **Category:** πŸ”Ž Search & Data Extraction **Repository:** https://github.com/SanggonBoy/PyreCrawl **GitHub Stars:** 22 **Views:** 6 **Installs:** 0 **Upvotes:** 0 **Directory Page:** https://allmcps.com/mcp/pyrecrawl ## Description One tool for AI agents to scrape, crawl, extract, map, and search the web. Self-hosted, no API keys. ## Tools Capabilities this server exposes over MCP: - **scrape** β€” Scrape a single URL β†’ LLM-ready markdown. Use this when the user shares a URL and wants its content (read, analyze, summarize, extract). Auto-escalates through fastβ†’stealthβ†’llm when blocked. Args: url: Target URL (http/https). prefer: "auto" | "fast" | "stealth" | "llm". auto = fast first, escalate to stealth on block/short page. fast = cheap HTTP only (no JS). stealth = real Chromium + Cloudflare solver. llm = full Crawl4AI browser + BM25 fit-markdown. timeout: per-attempt timeout in seconds. include_html: include raw HTML in the response (large; off by default). js: (stealth only) JS expression evaluated against the live page after it settles. The value comes back in ``meta.js_result``. Use for data that lives in DOM *properties* (e.g. an input's ``.value``) rather than in serialized HTML. wait_for: (stealth only) JS predicate expression polled until truthy (bounded by ``timeout``). Use to wait for content that arrives asynchronously after ``network_idle``. Returns: {url, final_url, status, markdown, title, method, elapsed_ms, meta} or {error, url, method} on failure. - **extract** β€” Scrape + structured extraction using a CSS-based JSON schema. Use when the user wants structured data (tables, lists, product info) extracted from a page. Define a CSS schema to target specific elements. The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] } Returns parsed JSON in `data`. - **map_site** β€” Enumerate all internal URLs reachable from `root`. Use when the user wants to map a site's structure or find all pages before crawling. Often paired with `crawl` or `batch_scrape`. Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs. - **crawl** β€” Multi-page crawl: discover URLs on `root`, then scrape each. Use this when the user wants to crawl an entire site section or docs, or needs multiple pages scraped in bulk. For single pages use `scrape`; for research questions use `deep_research`. Args: root: start URL. max_pages: hard cap on pages scraped. css_selector: scope each page's html/markdown to the matched element (non-llm: lxml re-scope of the fetched HTML; llm: native crawl4ai css_selector). prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl). include_paths: regex β€” keep only URLs matching (matched against full URL). exclude_paths: regex β€” drop URLs matching (e.g. `/tag/|/page/\d+`). max_depth: 0 = flat harvest from the root page's links (default); >0 = true BFS up to that link depth, honoring the filters. Returns: {root, pages: [{url, markdown, title, ...}], count, discovered, elapsed_ms} or {error, root} on failure. - **document** β€” Extract text from a PDF/DOCX/PPTX URL β†’ markdown (no browser). Use when the user shares a link to a document (PDF, Word, PowerPoint) and wants its text content. Also useful after `search_papers` to get full text from a paper's pdf_url. Content-type sniffed and routed to pypdf / python-docx / python-pptx. Optional deps β€” install with `pip install 'pyrecrawl[docs]'`. - **search** β€” Web search via DuckDuckGo HTML (no API key required). Use this for targeted searches where you need anti-bot bypass (Cloudflare protection on DDG). For simple searches, the built-in web_search may suffice. For research questions, prefer `deep_research` (search + scrape + citations). Returns [{url, title, snippet}, ...]. The smart ladder bypasses DDG's bot detection if needed. Returns: {query, results: [{url, title, snippet}], count} or {error, query} on failure. - **batch_scrape** β€” Scrape MANY URLs in ONE call (parallel, deduped, cache-aware). Use when the user provides multiple URLs or you have a list of pages to fetch. More efficient than calling `scrape` N times. Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default). Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated β€” other URLs still succeed. Returns: {requested, unique, succeeded, failed, results: [{url, markdown, ...}]} - **deep_research** β€” Search the web, then pull the top sources as EVIDENCE (no LLM synthesis). PRIMARY RESEARCH TOOL β€” use when the user asks to research, investigate, deep-dive, fact-check, or learn about a topic. Returns a ``citations`` list with stable [n] numbers and an ``evidence`` list of per-source markdown β€” the agent does the synthesis from evidence. Multi-pass mode: set iterations=2-3 to auto-run additional searches with refined queries (alternatives, criticism, latest developments) and append deduplicated evidence. Each pass adds up to ``scrape_top`` new sources. Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm". iterations: 1 (default, single pass), 2-3 (multi-pass with refined queries targeting evidence gaps). Each pass searches from a different angle and deduplicates by URL. Returns: {query, iterations_run, queries: [str], hits: [{url, title, snippet}], citations: [{url, title}], evidence: [{url, title, markdown}], scraped, used_engines, elapsed_ms} or {error, query, hint} on failure. - **monitor** β€” Track a URL over time and report meaningful content changes. Use when the user wants to watch a page for updates (price changes, new blog posts, status updates). Ask me to 'set up monitoring for ' for a guided setup playbook. Args: url: target URL. action: "check" | "history" | "forget". prefer: ladder preference, same as ``scrape``. css_selector: scope the diff to one element (so banner / nav changes don't trigger false positives). Snapshots persist under ``PYRECRAWL_MONITOR_DIR`` (default ``~/.pyrecrawl/monitors/``). ``check`` returns ``status`` of ``new`` | ``unchanged`` | ``changed`` | ``error`` and a unified diff when the page changed. Returns: {url, status: "new"|"unchanged"|"changed"|"error", diff?: str, snapshot_chars?: int, elapsed_ms?: int} - **search_papers** β€” Search academic papers via arXiv or Crossref β€” no API keys. Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV". Returns papers with id/url/pdf_url/title/authors/summary/published. Feed pdf_url into the `document` tool to extract full text. Returns: {query, source, papers: [{title, authors, abstract, url, pdf_url?, ...}]} or {error, query} on failure. - **cache** β€” Inspect the response cache: stats, clear, enable, or disable. Use this to check cache hit rates before large batch jobs, or to clear stale cached responses when a site's content has changed. Args: action: "stats" (default) | "clear" | "disable" | "enable". - **health** β€” Verify PyreCrawl is working: engine versions, dependencies, update status. Use this at the start of a session or before a large scraping job to confirm all engines are installed and up to date. - **session** β€” Drive a persistent browser session β€” cookies & JS state kept across calls. Use for login walls and multi-step flows the one-shot ladder can't handle (e.g. user needs to log in first, then scrape a protected page). For simple pages use `scrape`; for research use `deep_research`. One-shot scrape has no session memory; here each action runs against the same live page. Args: session: named session; reuse the same name to keep state. action: "open" (url) β€” navigate, returns url/title/status "click" (selector) β€” click an element "fill" (selector, text) β€” type into an input "type" (key) β€” press a key, e.g. "Enter" "eval" (js) β€” run a JS expression, returns value "wait" (selector?) β€” wait for selector or sleep timeout_ms "content" β€” url/title/visible text of current page "screenshot" (full_page?) β€” returns png_base64 "cookies" β€” list session cookies "close" β€” destroy the session "list" β€” show live sessions Returns {session, action, ...result, elapsed_ms}; errors as {error}. ## Claude Desktop Quick Installation Install path detected from listing signals. Uses `uvx` (confidence: high): ```json "mcpServers": { "pyrecrawl": { "command": "uvx", "args": ["pyrecrawl@latest"], "env": { "PYRECRAWL_CACHE": "", "PYRECRAWL_CACHE_TTL": "", "PYRECRAWL_MONITOR_DIR": "", "PYRECRAWL_NO_TELEMETRY": "", "DO_NOT_TRACK": "" } } } ``` **Requires environment variables:** `PYRECRAWL_CACHE`, `PYRECRAWL_CACHE_TTL`, `PYRECRAWL_MONITOR_DIR`, `PYRECRAWL_NO_TELEMETRY`, `DO_NOT_TRACK` β€” the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for). ## Documentation ## What PyreCrawl MCP server does PyreCrawl MCP server exposes web access and content-processing operations to MCP-compatible AI agents. Its core tools cover single-page scraping, structured extraction, site mapping, multi-page crawling, parallel URL fetching, web search, academic paper search, document conversion, research collection, change monitoring, persistent browser sessions, cache management, and health checks. The server is intended for local, self-hosted use and does not require API keys. It belongs in the Search & Data Extraction category for agents that need to gather current web content or turn online sources into material for summarization, retrieval, or analysis. ## How it works Requests can use one of four preference modes: automatic selection, HTTP-only fetching, stealth browser fetching, or deeper processing. In automatic mode, PyreCrawl starts with a fast HTTP request. If the response is blocked, empty, or affected by a Cloudflare challenge, it can escalate to a Chromium-based browser. Pages that need JavaScript rendering or remain blocked can then use deeper processing that produces LLM-ready Markdown, citations, or structured output. The `crawl` tool discovers and processes multiple pages with path filters, page limits, and breadth-first depth handling. `map_site` lists internal URLs without performing a full crawl, while `batch_scrape` processes multiple URLs in parallel with deduplication and cache awareness. `deep_research` combines web search and top-result scraping into an evidence pack with numbered citations; it does not perform the final LLM synthesis. ## Setup and configuration Install the package with `uvx`, `uv tool`, `pipx`, or `pip` in a virtual environment. Run `pyrecrawl setup` once to install Chromium and the stealth browser engines. The `pyrecrawl install` command can detect supported agents and write MCP configuration, or target specific agents such as Claude Desktop, Cursor, Claude Code, VS Code, and Codex. A manual stdio configuration is also available for Claude Desktop and Claude Code. Optional environment variables control local behavior. `PYRECRAWL_CACHE` enables an in-memory cache or specifies a cache directory, `PYRECRAWL_CACHE_TTL` sets cache lifetime, `PYRECRAWL_MONITOR_DIR` selects the directory for saved monitor snapshots, and `PYRECRAWL_NO_TELEMETRY` disables the anonymous startup ping. The server also honors `DO_NOT_TRACK=1` for that telemetry setting. ## Tools and capabilities - `scrape` returns a URL as LLM-ready Markdown. - `extract` applies a JsonCss schema to produce structured data. - `search` uses DuckDuckGo HTML without an API key. - `search_papers` searches arXiv and Crossref; returned PDF URLs can be sent to `document`. - `document` converts PDF, DOCX, and PPTX files to Markdown, with optional document extras. - `monitor` stores snapshots and reports unified diffs for content changes. - `session` preserves browser cookies for login walls, multi-step flows, and screenshots. Read-only MCP resources expose cache statistics, sessions, and monitors. The included prompts provide playbooks for research, RAG ingestion, and page watching. ## Limitations and notes Browser-dependent behavior requires the one-time engine installation and may take longer than HTTP fetching. The automatic fallback is not a guarantee that every protected or JavaScript-heavy page can be accessed. Academic search is limited to the stated arXiv and Crossref sources, and web search uses DuckDuckGo HTML. Persistent monitoring depends on the configured snapshot directory. PyreCrawl MCP server supplies research evidence and extracted content, but an agent must perform any final synthesis. _Full upstream README: https://allmcps.com/mcp/pyrecrawl/readme_