web-crawling MCP servers & tools

Crawl websites into clean Markdown, search pages, and extract structured data with LLMs. Built-in MCP server for web research and RAG pipelines.

fastCRW — open-source (AGPL-3.0), self-hostable Rust web crawler & search API for AI agents. Tools: scrape, crawl, map, and SearXNG-backed search. Single 6MB static binary; reproducible 1K-URL benchmarks faster than hosted alternatives. Hosted MCP at fastcrw.com/mcp (Streamable HTTP, OAuth) or self-host.

Runs and manages its own local models, or uses your existing Ollama or LM Studio if you prefer. Indexes your files and code, crawls the websites you point it at, and answers with citations to the source.

MCP server for advanced web search using Tavily

MCP server for advanced web search using Tavily API.

API to search, extract and structure web data. Web scraping, AI-powered answers with citations, batch processing (10k URLs), and autonomous site crawling.

Unified web layer for AI agents. Search (8 engines), stealth browse, cookie auth, and act on 24 platforms. 5,000 free searches/month via Gemini Grounded Search.

Self-hosted web research for MCP agents: search (SearXNG, with DuckDuckGo fallback), crawl, dense+BM25 rerank, and dedupe into a source-grounded, cited prompt. Local ONNX embeddings by default, or bring an OpenAI-compatible embedding API.

Crawleo MCP Server - Real-Time Web Knowledge for AI Crawleo's Model Context Protocol (MCP) server enables AI assistants like Claude to access real-time web data directly through native tool integration.

Japanese-web-native web fetching optimized for low target-site impact and token efficiency. Outline-first progressive disclosure, ShiftJIS/EUC-JP mojibake handling, PDF/CSV extraction, and polite crawling (robots.txt, rate limits, honest User-Agent). Install via npx -y amenbo.

Four free tools to check if AI engines can find, read and recognise a site. No API key.

Inspect a site's AI-search readiness: AI crawler access, llms.txt, schema markup, meta directives

One tool for AI agents to scrape, crawl, extract, map, and search the web. Self-hosted, no API keys.

Free, open-source, token-efficient local alternative to Firecrawl with native MCP Server for LLMs (>73% token reduction, Playwright stealth anti-bot bypass).

Read-only MCP server over a public reference index of web crawlers and AI user agents: classify a user-agent, look up a crawler and who runs it, generate robots.txt rules, check an IP, and see what blocking each one costs. Streamable HTTP, no key and no signup; the data is CC0.

Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.

搜尋引擎與 AI 爬蟲工具箱：搜尋指令、爬蟲驗證、收錄與 AEO 健檢。台灣繁體中文 MCP 工具。

檢查網站爬蟲設定、驗證爬蟲 IP、IndexNow 提交與 YouTube SEO 檢查。

Run SEO Content Machine tasks from an MCP client: scraping, crawls, Google Maps, Workspace pages.