Commoncrawl logo

Commoncrawl

Search Common Crawl snapshots, inspect archived URL captures, and retrieve page content from WARC records.

Install Commoncrawl

Pick your client, then add it in one click or paste the config.

STDIONode.js
~/Library/Application Support/Claude/claude_desktop_config.json
claude_desktop_config.json
{
  "mcpServers": {
    "commoncrawl": {
      "command": "npx",
      "args": [
        "-y",
        "@pipeworx/mcp-commoncrawl"
      ]
    }
  }
}

Settings → Developer → Edit Config, merge this into mcpServers, then fully quit and reopen Claude Desktop.

Install not yet sandbox-tested— the server itself is confirmed live (we called its tools/list endpoint). The stdio install command is a separate, ongoing check.

Overview

Commoncrawl provides MCP tools for searching the Common Crawl public web archive and reading archived page bytes. Use its crawl-listing tool to select a monthly collection, search that collection's CDX index, then fetch a matching WARC record by byte range. It is a good fit for checking historical page content, URL crawl history, HTTP metadata, or pages that are no longer available online. Access is keyless through the hosted endpoint, with an optional local stdio bridge.

Use cases

  • Find captures of a URL during a specified month
  • Compare archived versions of a page
  • Verify whether Common Crawl saw a domain
  • Retrieve page content after a site went offline
  • Filter archived captures by status or date

Key features

  • Lists monthly Common Crawl collections
  • Searches CDX indexes by URL, host, or domain
  • Retrieves WARC records by byte range
  • Returns capture and HTTP metadata
  • Supports pagination, filters, and date ranges
  • Provides hosted HTTP and local stdio access

Tools3

What Commoncrawl lets your agent do. Expand a tool for its parameters.

Verified live~1.0k tokens of context

Security & health

What it can reach and how reliably it runs — from our own checks, not the maintainer’s claims.

ActiveLast checked

Recent health check succeeded.

11/11 checks healthy over the last 46d

What it can do

From the 3 tools this server returned when we ran it; 3 declare their own safety hints.

Makes network requests · 3Read-only tools · 3
Network at startup5 connections
Files written at startupNone
Install scriptsNone
npm provenanceSigned build
Package age37 days
Maintainers1

Observed 2026-11-02 in an isolated sandbox. A description of reach, not a security audit.

Supply-chain signal

No high-severity advisories surfaced by our automated scan.

Critical 0High 0Medium 0Low 0

Scanned 7d ago via OSV.dev · @pipeworx/mcp-commoncrawl (npm)

Adoption & maintenance

Factual signals from GitHub, npm, and our automated checks — not a rating.

npm downloads969Package downloads in the last 30 days.
Last commit12d agoMost recent push to the default branch.
Tools exposed3Callable tools this server registers over MCP.
Directory activity2 viewsConfig copies, upvotes, and views on AllMCPs.

How Commoncrawl works

What Commoncrawl does

Commoncrawl exposes the Common Crawl Foundation's public web archive through MCP. It supports historical web research at the level of individual captures: you can identify which monthly crawl covered a date, find captures for a URL or domain, and retrieve the archived response recorded during crawling.

The returned index data includes the capture timestamp, HTTP status, MIME type, detected language, content digest, and the WARC file location needed to read the record. Fetching that record returns WARC headers, captured HTTP response headers, and the page body as it was stored.

Reach for Commoncrawl when a question depends on what a page contained at an earlier point in time, whether a URL was ever crawled, or which URLs were collected under a host or domain. It is separate from a live web search and from link-graph data: the results describe Common Crawl's archived corpus.

How it works

Start with commoncrawl_crawls to list monthly crawl collections. The result includes crawl identifiers such as CC-MAIN-2026-34 and the date range covered by each collection. The optional contains_date argument helps select the snapshot associated with a particular day.

Pass the selected crawl identifier to commoncrawl_index_search. This queries that collection's CDX index for a URL, host, or domain. Matching behavior can be controlled with match_type, and the search supports limits, pagination, filters, and date bounds. Each matching record contains the filename, byte offset, and length required by the next step.

Call commoncrawl_fetch_record with those WARC location values to retrieve the archived record. The operation reads only the requested byte range rather than downloading an entire WARC file. max_body_bytes controls the page-body limit.

The service handles transient upstream 5xx responses with retries. A no-capture response is represented as an empty result with an explanatory note rather than being treated as an indistinguishable transport failure.

Setup and configuration

The hosted streamable MCP endpoint is:

text
https://gateway.pipeworx.io/commoncrawl/mcp

An MCP client can register that URL directly. The endpoint requires no API key for basic use. A full Pipeworx gateway endpoint is also available at https://gateway.pipeworx.io/mcp; both routes access the same gateway, while the pack endpoint lists Commoncrawl tools directly.

For local stdio use, run the package with npx:

config.json
{
  "mcpServers": {
    "commoncrawl": {
      "command": "npx",
      "args": ["-y", "@pipeworx/mcp-commoncrawl"]
    }
  }
}

The bridge forwards requests to the hosted endpoint and needs network access. Anonymous use is capped at 50 calls per day. Adding a PIPEWORX_API_KEY environment variable from a Pipeworx account raises the allowance to 200 calls per day. Set PIPEWORX_META_TOOLS=1 if the local bridge should also list shared Pipeworx tools such as ask_pipeworx; otherwise it lists the Commoncrawl pack tools.

Tools and capabilities

  • commoncrawl_crawls lists monthly collections, their capture windows, and the collection matching an optional date.
  • commoncrawl_index_search searches a collection's CDX index by URL, host, or domain.
  • commoncrawl_fetch_record reads an archived WARC record using its filename, offset, and length.
  • Index results expose capture timestamps, status codes, MIME types, detected languages, content digests, and WARC locations.
  • Record retrieval returns archive metadata, captured response headers, and the stored page body.
  • Queries support result limits, pagination, filters, and date ranges.

Limitations and notes

Commoncrawl depends on network access to the Pipeworx gateway and Common Crawl data services; the local bridge has no offline mode. Anonymous requests have a daily call limit, while authenticated use requires a Pipeworx API key and is metered.

Broad domain searches can be expensive upstream and are more likely to encounter load-related errors. Narrowing a query to a host and applying a filter such as a successful HTTP status can reduce that pressure. The archived body is subject to the max_body_bytes limit, so a fetched result may be intentionally truncated.

Pack endpoints also expose shared Pipeworx meta-tools in addition to the three Commoncrawl tools. Their exact set is reported during MCP initialization and may increase the tool schemas visible to a client.

Written by AllMCPs with AI assistance from the project’s README and listing data. How we build listings

Reviews

No reviews yet — be the first to share how it worked for you.

Frequently Asked Questions about Commoncrawl

Commoncrawl is an MCP server that connects AI clients to Common Crawl's public web archive. Its main tools list crawl collections, search CDX indexes for URL or domain captures, and fetch archived WARC records by byte range. Results include crawl metadata, captured HTTP headers, and the stored page body.

Run `npx -y @pipeworx/mcp-commoncrawl` from an MCP client's stdio configuration, or connect directly to `https://gateway.pipeworx.io/commoncrawl/mcp`. The local bridge forwards requests to the hosted service and requires network access.

Yes, Commoncrawl can be added to Cursor using its hosted MCP URL or the local npx stdio configuration. The hosted configuration uses `https://gateway.pipeworx.io/commoncrawl/mcp`.

Call `commoncrawl_crawls` with `contains_date` to identify the monthly collection covering that date. Use the returned crawl ID as the `crawl` argument for `commoncrawl_index_search`.

Search the selected collection with `commoncrawl_index_search`, then pass its WARC `filename`, `offset`, and `length` to `commoncrawl_fetch_record`. Set `max_body_bytes` when you need to limit the returned page body.

Related MCP servers

Add a badge to your READMEShow it’s listed — live SVG, Markdown or HTML.

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:
Live Dynamic SVG PreviewCommoncrawl AllMCPs Directory Badge
Markdown (GitHub README)
[![AllMCPs](https://allmcps.com/api/badge/commoncrawl?style=directory)](https://allmcps.com/mcp/commoncrawl)
HTML Embed
<a href="https://allmcps.com/mcp/commoncrawl"><img src="https://allmcps.com/api/badge/commoncrawl?style=directory" alt="Commoncrawl on AllMCPs" /></a>