Markcrawl

What the AIMLPM/markcrawl MCP server does

AIMLPM/markcrawl MCP server is built around a local web-crawling engine that converts public webpages into Markdown intended for LLM pipelines. It can process a single URL or follow links across a website, including documentation, blogs, and subsections. The crawler removes common page noise such as navigation, scripts, and boilerplate so downstream retrieval receives the main page content.

Each crawled page is written as a Markdown file and represented in pages.jsonl. The index records the source URL, title, fetch timestamp, a preformatted citation containing the access date, the tool name, and the extracted text. Optional records can also reference downloaded files, images, or screenshots. The project description additionally identifies page search and LLM-based structured extraction as use cases, while the README separates LLM extraction from the core installation.

How it works

AIMLPM/markcrawl MCP server is intended for local execution. The underlying command accepts a starting URL, an output directory, and crawl controls such as a maximum page count. URL include and exclude rules, dry runs, smart sampling, and interrupted-run resumption are available through the crawler’s documented recipes. A default aggregator-page filter avoids common documentation bundle URLs such as /print.html and /_print/ unless that behavior is disabled.

The standard extraction path uses a Beautiful Soup backend. The project also documents alternatives including trafilatura, an ensemble mode, and ReaderLM-v2. JavaScript-rendered pages are listed among the supported scraping recipes. Binary downloads can be enabled for selected PDF and DOCX links, with content-type and size checks plus a pre-fetch filtering callback. Image downloads and page screenshots are optional outputs.

Setup and configuration

Install the core package from PyPI with pip install markcrawl. A basic run supplies a public starting address and output directory, for example markcrawl --base https://quotes.toscrape.com --out ./demo --max-pages 5 --show-progress. The resulting directory contains Markdown files and the JSONL index.

The local embedder is included by default and does not require an API key. Users can select another embedding model through the embedding_model option or the MARKCRAWL_EMBEDDER environment variable; the README gives text-embedding-3-small as an OpenAI alternative. LLM extraction, Supabase upload, the MCP server, and LangChain tools are installed separately rather than being required by the base crawler.

Limitations and notes

AIMLPM/markcrawl MCP server is a local crawler, not a hosted scraping endpoint. The README states that a hosted API is only under consideration, so deployments that require managed infrastructure should not assume one is available. The documented comparison also warns that sites requiring login authentication or facing aggressive bot protection may need a custom solution.

The project reports benchmark results openly: in the cited benchmark it ranked first for cost but seventh of seven for answer quality and retrieval accuracy. Those figures describe the referenced benchmark, not a guarantee for every site. Its narrower crawl strategy can omit pages that another discovery method finds, although the project is working on crawl selection and benchmark methodology.

The supplied material does not list the MCP server’s individual tool names, client configuration blocks, transport details, or a dedicated command for launching that server. Those details should be verified in the repository before configuring an MCP client.