Crawlgraph MCP

What the pucilpet/crawlgraph-mcp MCP server does

The pucilpet/crawlgraph-mcp MCP server connects MCP clients to the CrawlGraph backlink-intelligence API. Its data comes from the public Common Crawl webgraph, which the project describes as covering 4.4 billion edges and 120 million domains. The server is intended for competitor backlink research, gap analysis, and outreach prospecting.

The backlinks tool accepts a target domain and returns referring domains, host counts, CrawlGraph authority scores, and the target domain's own authority or rank. The gap_analysis tool finds domains that link to one or more competitors but not to the specified domain. Each gap includes a found_on field identifying which competitors link from that domain.

How it works

The pucilpet/crawlgraph-mcp MCP server sends requests to CrawlGraph using an API key. Backlink lookups are direct API calls. Gap analysis submits an asynchronous job and polls until the result is ready; completion usually takes 5–30 seconds.

gap_outreach_targets combines those operations into a prospecting workflow. It runs a gap analysis, removes platform and CDN noise unless configured otherwise, and classifies results into priority and secondary groups. Priority domains link to all supplied competitors but not to the user's domain. Secondary domains link to at least two competitors. The tool can enrich top priority targets with authority scores. The documentation recommends using two or three competitors for this workflow.

The releases tool lists Common Crawl snapshots available through the API. A returned release ID can be supplied to backlinks to query a particular snapshot.

Setup and configuration

Install the package with:

npx -y crawlgraph-mcp

The server requires a CrawlGraph API key in CRAWLGRAPH_API_KEY. The API documentation provides free keys by email, with a free allowance of 15 backlink calls per month. The gap tools require the $99 lifetime tier. That tier includes 1,000 backlink calls and 50 gap analyses per calendar month, according to the supplied material.

CRAWLGRAPH_BASE_URL is optional and defaults to https://crawlgraph.com. Configure the server in an MCP client by running the package with npx -y and passing the API key through the process environment. The README gives configuration examples for Claude Desktop, Claude Code, Cursor, Windsurf, Cline, and Zed.

Tools and capabilities

• backlinks: Look up referring domains for one target. Results can be limited, sorted by authority or host count, and associated with a selected release.
• gap_analysis: Compare one domain with one to five competitor domains and return missing referring-domain opportunities.
• gap_outreach_targets: Compare two to five competitors, filter noise, group prospects, and score selected targets by authority.
• releases: List queryable Common Crawl snapshots without consuming the monthly quota.

Backlink calls and gap jobs consume separate quotas. Enriching outreach targets also consumes backlink calls for the targets being scored.

Limitations and notes

The data is based on quarterly Common Crawl snapshots, not a live crawl. This makes the server suitable for periodic prospecting and research, but not for tracking backlink changes over a period of days. Gap results identify the competitors associated with each domain but do not include per-domain authority; use backlinks when an individual prospect needs an authority score.

The project is released under the MIT license. The pucilpet/crawlgraph-mcp MCP server does not itself provide outreach writing; an MCP client or agent can use the returned prospect data for that purpose.