PyreCrawl vs Arxiv MCP Server
A side-by-side look at the PyreCrawl and Arxiv MCP Server MCP servers: transport, auth, tools, quality signals, and the config to install either one in Claude, Cursor, Windsurf or VS Code.
Verdict: choose PyreCrawl for search & data extraction work as a local process; choose Arxiv MCP Server for search & data extraction as a local process. You can also run both side by side in one mcpServers block.
Which MCP server should you choose?
Choose PyreCrawl when
- You need dedicated capabilities in the Search & Data Extraction domain.
- You prefer a local stdio subprocess transport.
- Its auth model fits you: No auth required.
- You have the keys it needs:
PYRECRAWL_CACHE, PYRECRAWL_CACHE_TTL, PYRECRAWL_MONITOR_DIR, PYRECRAWL_NO_TELEMETRY, DO_NOT_TRACK. - You want tools like
scrape, extract, map_site.
Choose Arxiv MCP Server when
- You need dedicated capabilities in the Search & Data Extraction domain.
- You prefer a local stdio subprocess transport.
- Its auth model fits you: No auth required.
- You want tools like
search_papers, download_paper, list_papers.
Feature & specification comparison
Rows where the two servers differ are marked.
| Spec | ||
|---|---|---|
| Category | Search & Data Extraction | Search & Data Extraction |
| Quality signal | 61/100 · Good | 68/100 · Great |
| Install command | Confirmed | Confirmed |
| Transport | Local subprocess (stdio) | Local subprocess (stdio) |
| Auth | No auth required | No auth required |
| Pricing | Free | Free |
| Required env vars | PYRECRAWL_CACHEPYRECRAWL_CACHE_TTLPYRECRAWL_MONITOR_DIRPYRECRAWL_NO_TELEMETRYDO_NOT_TRACK | None listed |
| Verified owner | Community listing | Community listing |
| GitHub stars | 22 | 3,230 |
| Directory activity | 6 views 0 installs 0 upvotes | 9 views 0 installs 0 upvotes |
| Tools | 13 | 19 |
| Full listing | View PyreCrawl | View Arxiv MCP Server |
Tools & capabilities
PyreCrawl tools 13
scrapeScrape a single URL → LLM-ready markdown. Use this when the user shares a URL and wants its content (read, analyze, summarize, extract). Auto-escalates through fast→stealth→llm when blocked. Args: url: Target URL (http/https). prefer: "auto" | "fast" | "stealth" | "llm". auto = fast first, escalate to stealth on block/short page. fast = cheap HTTP only (no JS). stealth = real Chromium + Cloudflare solver. llm = full Crawl4AI browser + BM25 fit-markdown. timeout: per-attempt timeout in seconds. include_html: include raw HTML in the response (large; off by default). js: (stealth only) JS expression evaluated against the live page after it settles. The value comes back in ``meta.js_result``. Use for data that lives in DOM *properties* (e.g. an input's ``.value``) rather than in serialized HTML. wait_for: (stealth only) JS predicate expression polled until truthy (bounded by ``timeout``). Use to wait for content that arrives asynchronously after ``network_idle``. Returns: {url, final_url, status, markdown, title, method, elapsed_ms, meta} or {error, url, method} on failure.extractScrape + structured extraction using a CSS-based JSON schema. Use when the user wants structured data (tables, lists, product info) extracted from a page. Define a CSS schema to target specific elements. The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] } Returns parsed JSON in `data`.map_siteEnumerate all internal URLs reachable from `root`. Use when the user wants to map a site's structure or find all pages before crawling. Often paired with `crawl` or `batch_scrape`. Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs.crawlMulti-page crawl: discover URLs on `root`, then scrape each. Use this when the user wants to crawl an entire site section or docs, or needs multiple pages scraped in bulk. For single pages use `scrape`; for research questions use `deep_research`. Args: root: start URL. max_pages: hard cap on pages scraped. css_selector: scope each page's html/markdown to the matched element (non-llm: lxml re-scope of the fetched HTML; llm: native crawl4ai css_selector). prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl). include_paths: regex — keep only URLs matching (matched against full URL). exclude_paths: regex — drop URLs matching (e.g. `/tag/|/page/\d+`). max_depth: 0 = flat harvest from the root page's links (default); >0 = true BFS up to that link depth, honoring the filters. Returns: {root, pages: [{url, markdown, title, ...}], count, discovered, elapsed_ms} or {error, root} on failure.documentExtract text from a PDF/DOCX/PPTX URL → markdown (no browser). Use when the user shares a link to a document (PDF, Word, PowerPoint) and wants its text content. Also useful after `search_papers` to get full text from a paper's pdf_url. Content-type sniffed and routed to pypdf / python-docx / python-pptx. Optional deps — install with `pip install 'pyrecrawl[docs]'`.searchWeb search via DuckDuckGo HTML (no API key required). Use this for targeted searches where you need anti-bot bypass (Cloudflare protection on DDG). For simple searches, the built-in web_search may suffice. For research questions, prefer `deep_research` (search + scrape + citations). Returns [{url, title, snippet}, ...]. The smart ladder bypasses DDG's bot detection if needed. Returns: {query, results: [{url, title, snippet}], count} or {error, query} on failure.batch_scrapeScrape MANY URLs in ONE call (parallel, deduped, cache-aware). Use when the user provides multiple URLs or you have a list of pages to fetch. More efficient than calling `scrape` N times. Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default). Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated — other URLs still succeed. Returns: {requested, unique, succeeded, failed, results: [{url, markdown, ...}]}deep_researchSearch the web, then pull the top sources as EVIDENCE (no LLM synthesis). PRIMARY RESEARCH TOOL — use when the user asks to research, investigate, deep-dive, fact-check, or learn about a topic. Returns a ``citations`` list with stable [n] numbers and an ``evidence`` list of per-source markdown — the agent does the synthesis from evidence. Multi-pass mode: set iterations=2-3 to auto-run additional searches with refined queries (alternatives, criticism, latest developments) and append deduplicated evidence. Each pass adds up to ``scrape_top`` new sources. Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm". iterations: 1 (default, single pass), 2-3 (multi-pass with refined queries targeting evidence gaps). Each pass searches from a different angle and deduplicates by URL. Returns: {query, iterations_run, queries: [str], hits: [{url, title, snippet}], citations: [{url, title}], evidence: [{url, title, markdown}], scraped, used_engines, elapsed_ms} or {error, query, hint} on failure.monitorTrack a URL over time and report meaningful content changes. Use when the user wants to watch a page for updates (price changes, new blog posts, status updates). Ask me to 'set up monitoring for <url>' for a guided setup playbook. Args: url: target URL. action: "check" | "history" | "forget". prefer: ladder preference, same as ``scrape``. css_selector: scope the diff to one element (so banner / nav changes don't trigger false positives). Snapshots persist under ``PYRECRAWL_MONITOR_DIR`` (default ``~/.pyrecrawl/monitors/``). ``check`` returns ``status`` of ``new`` | ``unchanged`` | ``changed`` | ``error`` and a unified diff when the page changed. Returns: {url, status: "new"|"unchanged"|"changed"|"error", diff?: str, snapshot_chars?: int, elapsed_ms?: int}search_papersSearch academic papers via arXiv or Crossref — no API keys. Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV". Returns papers with id/url/pdf_url/title/authors/summary/published. Feed pdf_url into the `document` tool to extract full text. Returns: {query, source, papers: [{title, authors, abstract, url, pdf_url?, ...}]} or {error, query} on failure.cacheInspect the response cache: stats, clear, enable, or disable. Use this to check cache hit rates before large batch jobs, or to clear stale cached responses when a site's content has changed. Args: action: "stats" (default) | "clear" | "disable" | "enable".healthVerify PyreCrawl is working: engine versions, dependencies, update status. Use this at the start of a session or before a large scraping job to confirm all engines are installed and up to date.- +1 more on the listing page
Arxiv MCP Server tools 19
search_papersSearch arXiv by query with optional categories, date range, sort, and pagination. Query: prefer quoted phrases; ti:/au:/abs:/cat:; AND/OR/ANDNOT. Unprefixed terms match title+abstract (not authors). Use categories (cs.AI, cs.LG, cs.CL, cs.CV, cs.MA, cs.RO, stat.ML, quant-ph). Catalog/examples: README 'search_papers query guide'. Dates YYYY-MM-DD (date_from/date_to). sort_by relevance|date. max_results default 5 (cap 50). abstract_mode none|snippet|full (default snippet). start default 0; response: total_results, returned, has_more, next_start, abstract_mode. Pass next_start with same abstract_mode. Use get_abstract after compact search — not after abstract_mode=full. arXiv ~3s between requests (server-side). Transient 429/503 are retried with backoff; persistent rate limits return status=rate_limited.download_paperDownload a paper from arXiv and return its text content. Tries the HTML version first for clean extraction; falls back to PDF conversion if HTML is unavailable. Stores the paper locally. Returned text is bounded to roughly 12,000 characters by default so one call cannot return an unbounded paper body. When is_truncated is true, call again with start=next_start (see next_retrieval) to continue, or pass return_full_text=true for the entire remaining paper. Set force=true to re-fetch and overwrite a cached paper (required to replace a newer stored arXiv version with an older one).list_papersList all papers that have been downloaded and stored locally via download_paper. Returns id, title, authors, published, and arxiv_version/versioned_id from local metadata — no live re-fetch. Set compact=true to return arXiv IDs only. Returns an empty list if no papers have been downloaded yet. Workflow: search_papers -> download_paper -> list_papers -> read_paper.read_paperRead the text content of a paper that was previously downloaded via download_paper. Returns the paper in markdown format, bounded to roughly 12,000 characters by default so one call cannot return an unbounded paper body. When is_truncated is true, call again with start=next_start (see next_retrieval) to continue, or pass return_full_text=true for the entire remaining paper. Will fail with a clear error if the paper has not been downloaded yet — call download_paper first. Workflow: search_papers -> download_paper -> read_paper.get_abstractFetch abstract and metadata by arXiv ID without downloading the paper. Use before download_paper to assess relevance. Returns title, authors, abstract, categories, published date, and PDF URL. After compact search, use for one full abstract; skip if search used abstract_mode=full.semantic_searchSemantic similarity search over papers you have already downloaded locally via download_paper. Supports free-text queries (e.g. 'attention mechanisms for long sequences') or finding papers similar to a given paper_id. IMPORTANT: only searches your local downloaded collection — will return empty results if no papers have been downloaded yet. Use search_papers to find papers on arXiv, then download_paper to add them to the local index before using this tool. Requires pro dependencies: uvx --from 'arxiv-mcp-server[pro]' arxiv-mcp-server (or uv tool install 'arxiv-mcp-server[pro]').reindexRebuild the local semantic index for downloaded papers.citation_graphReturn papers citing an arXiv paper and papers that it references using Semantic Scholar's citation graph. Results are bounded (default 50) to stay within the unauthenticated quota. Under load, export SEMANTIC_SCHOLAR_API_KEY for a higher limit; without a key, persistent rate limits return status=rate_limited instead of failing hard.export_citationsExport BibTeX citations for one or more arXiv papers using authoritative arXiv metadata (title, authors, year, primary category), never model-generated fields. Version suffixes (e.g. '2401.12345v2') are preserved and citation keys are deterministic. Returns the rendered BibTeX plus per-paper status/error. BibTeX only; RIS/CSL-JSON are not yet supported.watch_topicSave or update a persistent research topic watch. When checked via check_alerts, returns only papers published since the last check — acting as a standing alert for new work on a topic. New watches seed last_checked to creation time so the first check does not dump historical matches. The topic string uses the same query syntax as search_papers (quoted phrases, field specifiers, boolean operators). Examples: '"diffusion models" AND ti:"video generation"', 'au:"LeCun" AND cs.LG'. Calling watch_topic with the same topic string updates the existing watch rather than creating a duplicate. On update, omit categories to preserve existing filters; pass categories: [] to clear them. Pair with check_alerts to poll for new papers.check_alertsCheck all saved topic watches for newly published papers since the last check. Omitting the topic parameter runs ALL saved watches and returns new papers for each. Passing a topic string checks only that specific watch. Advances each watch's drain cursor after running: when a page is truncated by max_results, has_more=true and check_start advances so later calls return the next papers in the same window (Atom date bounds alone are day-granular and would otherwise re-hit the boundary); last_checked tracks the newest returned paper. When the page is not full, last_checked becomes now and the drain cursor resets. Use watch_topic to register topics before calling this. Returns a clear not-found error if a topic is provided but no matching watch exists. Returns a summary with new paper counts, has_more, and full paper metadata per topic.list_watchesList all saved topic watches without checking for new papers. Returns each watch's topic, categories, last_checked timestamp, and other stored fields. Does not update last_checked — use this to inspect what is saved. Use unwatch_topic to remove a watch, or check_alerts to poll for new papers.- +7 more on the listing page
Ready-to-paste client configs
Paste either block (or both) into your client config, e.g. claude_desktop_config.json or ~/.cursor/mcp.json. We only show a config when the install command is confirmed.
PyreCrawl
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": [
"pyrecrawl@latest"
],
"env": {
"PYRECRAWL_CACHE": "YOUR_PYRECRAWL_CACHE_HERE",
"PYRECRAWL_CACHE_TTL": "YOUR_PYRECRAWL_CACHE_TTL_HERE",
"PYRECRAWL_MONITOR_DIR": "YOUR_PYRECRAWL_MONITOR_DIR_HERE",
"PYRECRAWL_NO_TELEMETRY": "YOUR_PYRECRAWL_NO_TELEMETRY_HERE",
"DO_NOT_TRACK": "YOUR_DO_NOT_TRACK_HERE"
}
}
}
}
Arxiv MCP Server
{
"mcpServers": {
"blazickjp-arxiv-mcp-server": {
"command": "uvx",
"args": [
"arxiv-mcp-server"
]
}
}
}
Frequently asked questions
PyreCrawl is categorized under Search & Data Extraction and uses a local stdio subprocess. Arxiv MCP Server belongs to Search & Data Extraction and uses a local stdio subprocess. Pick PyreCrawl for search & data extraction work and Arxiv MCP Server for search & data extraction.
Copy the mcpServers block from the config section on this page into your client's config (for example claude_desktop_config.json or ~/.cursor/mcp.json), then fully restart the client.
PyreCrawl is listed as free. Auth: no auth required. Arxiv MCP Server is listed as free. Auth: no auth required. If a server needs environment variables such as API keys, define them under the env key in your client's MCP config.
Yes. MCP clients support multiple servers: add pyrecrawl and blazickjp-arxiv-mcp-server as separate keys in the same mcpServers object, and your assistant will call tools from either one.
On AllMCPs, PyreCrawl has a quality signal of 61/100 (Good) with 6 views, 0 installs and 22 GitHub stars. Arxiv MCP Server has 68/100 (Great) with 9 views, 0 installs and 3,230 GitHub stars.