Scrape, crawl, and map websites to Markdown or JSON via local CLI.
Quick Install
Automated & IDE SetupCopy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.
Manual Client & Custom JSON ConfigExpand JSON ▾
Client Config & Setup
Choose your client or environment
Target File:
💡 Paste the JSON block into your client's configuration file under mcpServers, then restart the application.
Documentation Overview
Crawlberg
Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.
What and Why?
You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.
It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.
Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.
Features
| Feature | Description |
|---|---|
| Structured extraction | Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, response headers |
| Markdown conversion | Clean Markdown with citations, document structure, and fit-content mode |
| Concurrent crawling | Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency |
| 14 language bindings | Rust, Python, Node.js, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly |
| Smart filtering | BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, sitemap discovery |
| Browser rendering | Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass |
| Batch & streaming | Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams |
| SSRF-safe by default | Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or CrawlConfig |
| Auth & rate limiting | HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling |
| MCP server & REST API | Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec |
Supported Platforms
Precompiled binaries for glibc Linux (x86_64/aarch64), macOS, and Windows across every binding. On musl Linux (Alpine) coverage is deliberately partial — the CLI, Docker image, Rust, Ruby, Java, C#, and Elixir work there; the Node, Python, Go, and PHP bindings do not ship musl artifacts. See the platform support matrix for the full table and the Alpine workarounds.
⭐ Star this repo to show your support — it helps others discover Crawlberg.
Quick Start
Language Packages
Python
See Python README for full documentation.
Node.js
See Node.js README for full documentation.
Rust
See Rust README for full documentation.
Go
See Go README for full documentation.
Java
Available on Maven Central as io.xberg.crawlberg:crawlberg. See Java README for the dependency snippet and current version.
C#
See C# README for full documentation.
Ruby
See Ruby README for full documentation.
PHP
See PHP README for full documentation.
Elixir
Add {:crawlberg, "~> 0.3"} to your mix.exs dependencies. See Elixir README for full documentation.
Dart / Flutter
See Dart README for full documentation.
Kotlin (Android)
Available on Maven Central as io.xberg.crawlberg.android:crawlberg-android. See Kotlin README for the dependency snippet and current version.
Swift
Related MCP Servers
Reviews
No reviews yet — be the first to share how this listing worked for you.
Frequently Asked Questions about Crawlberg
AllMCPs Directory Badge
Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.
Badge Style:
Live Dynamic SVG Preview
Markdown (GitHub README)
[](https://allmcps.com/mcp/crawlberg)HTML Embed
<a href="https://allmcps.com/mcp/crawlberg"><img src="https://allmcps.com/api/badge/crawlberg?style=directory" alt="Crawlberg on AllMCPs" /></a>Technical Specs & Signals
CategoryDeveloper Tools
27Quality signal: Emerging · 27/100How this signal is calculated ▾
Server availabilityNot measured
Not scored for repo-hosted servers — we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.
Verified ownership8/20
Documentation & tools11/30
Adoption & activity1/15
Community engagement0/10
A guidance signal from public completeness & health data — not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.
Own this project?
This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to unlock edit access and the Official badge — proof is checked automatically, then reviewed by our team.
Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.
Claim & get free dofollowShare & Embed
Add our SVG badge (dark/light directory styles) or embeddable widget to your site.