# Commoncrawl [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/pipeworx-io/mcp-commoncrawl  
**GitHub Stars:** 0  
**npm Downloads (last month):** 969  
**Views:** 2  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/commoncrawl

## Description
Common Crawl's public web archive — find every time a URL was crawled since 2008 and read the…

## Tools
Capabilities this server exposes over MCP:

- **commoncrawl_crawls** — List the Common Crawl monthly crawl collections (crawl ids like "CC-MAIN-2026-34") with the date range each one covers. AUTHORITATIVE for "which Common Crawl snapshot covers <date>" — call this first to pick the crawl id that commoncrawl_index_search needs. Newest first. Keyless.
- **commoncrawl_index_search** — Search one Common Crawl collection's CDX index for archived captures of a URL, host, or whole domain. PREFER OVER WEB SEARCH when the question is "what did this page look like in <month>", "did Common Crawl ever see this URL", or "list the URLs crawled under this domain" — it returns the crawl record (timestamp, HTTP status, MIME type, detected language, content digest) plus the WARC filename/offset/length that commoncrawl_fetch_record needs to read the page bytes. Keyless.
- **commoncrawl_fetch_record** — Read one archived page back out of Common Crawl by byte range — pass the filename, offset and length from a commoncrawl_index_search result and get the WARC headers, the captured HTTP response headers, and the page body as it was crawled. AUTHORITATIVE for "what did this page actually say when it was crawled", including pages that have since changed or gone offline. The body is truncated to a byte cap you control. Keyless.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "commoncrawl": {
    "command": "npx",
    "args": ["-y","@pipeworx/mcp-commoncrawl"]
  }
}
```

## Documentation

## What Commoncrawl does

Commoncrawl exposes the Common Crawl Foundation's public web archive through MCP. It supports historical web research at the level of individual captures: you can identify which monthly crawl covered a date, find captures for a URL or domain, and retrieve the archived response recorded during crawling.

The returned index data includes the capture timestamp, HTTP status, MIME type, detected language, content digest, and the WARC file location needed to read the record. Fetching that record returns WARC headers, captured HTTP response headers, and the page body as it was stored.

Reach for Commoncrawl when a question depends on what a page contained at an earlier point in time, whether a URL was ever crawled, or which URLs were collected under a host or domain. It is separate from a live web search and from link-graph data: the results describe Common Crawl's archived corpus.

## How it works

Start with `commoncrawl_crawls` to list monthly crawl collections. The result includes crawl identifiers such as `CC-MAIN-2026-34` and the date range covered by each collection. The optional `contains_date` argument helps select the snapshot associated with a particular day.

Pass the selected crawl identifier to `commoncrawl_index_search`. This queries that collection's CDX index for a URL, host, or domain. Matching behavior can be controlled with `match_type`, and the search supports limits, pagination, filters, and date bounds. Each matching record contains the filename, byte offset, and length required by the next step.

Call `commoncrawl_fetch_record` with those WARC location values to retrieve the archived record. The operation reads only the requested byte range rather than downloading an entire WARC file. `max_body_bytes` controls the page-body limit.

The service handles transient upstream 5xx responses with retries. A no-capture response is represented as an empty result with an explanatory note rather than being treated as an indistinguishable transport failure.

## Setup and configuration

The hosted streamable MCP endpoint is:

```text
https://gateway.pipeworx.io/commoncrawl/mcp
```

An MCP client can register that URL directly. The endpoint requires no API key for basic use. A full Pipeworx gateway endpoint is also available at `https://gateway.pipeworx.io/mcp`; both routes access the same gateway, while the pack endpoint lists Commoncrawl tools directly.

For local stdio use, run the package with `npx`:

```json
{
  "mcpServers": {
    "commoncrawl": {
      "command": "npx",
      "args": ["-y", "@pipeworx/mcp-commoncrawl"]
    }
  }
}
```

The bridge forwards requests to the hosted endpoint and needs network access. Anonymous use is capped at 50 calls per day. Adding a `PIPEWORX_API_KEY` environment variable from a Pipeworx account raises the allowance to 200 calls per day. Set `PIPEWORX_META_TOOLS=1` if the local bridge should also list shared Pipeworx tools such as `ask_pipeworx`; otherwise it lists the Commoncrawl pack tools.

## Tools and capabilities

- `commoncrawl_crawls` lists monthly collections, their capture windows, and the collection matching an optional date.
- `commoncrawl_index_search` searches a collection's CDX index by URL, host, or domain.
- `commoncrawl_fetch_record` reads an archived WARC record using its filename, offset, and length.
- Index results expose capture timestamps, status codes, MIME types, detected languages, content digests, and WARC locations.
- Record retrieval returns archive metadata, captured response headers, and the stored page body.
- Queries support result limits, pagination, filters, and date ranges.

## Limitations and notes

Commoncrawl depends on network access to the Pipeworx gateway and Common Crawl data services; the local bridge has no offline mode. Anonymous requests have a daily call limit, while authenticated use requires a Pipeworx API key and is metered.

Broad domain searches can be expensive upstream and are more likely to encounter load-related errors. Narrowing a query to a host and applying a filter such as a successful HTTP status can reduce that pressure. The archived body is subject to the `max_body_bytes` limit, so a fetched result may be intentionally truncated.

Pack endpoints also expose shared Pipeworx meta-tools in addition to the three Commoncrawl tools. Their exact set is reported during MCP initialization and may increase the tool schemas visible to a client.

_Full upstream README: https://allmcps.com/mcp/commoncrawl/readme_

