# CrawlCheck [Health: Active]

**Category:** 💻 Developer Tools  
**Repository:** https://github.com/emmanuelorta/crawlcheck-core  
**GitHub Stars:** 0  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/crawlcheck

## Description
Verification layer for the agentic web: a signed answer about any site before an agent acts.

## Claude Desktop Quick Installation
No confirmed install details yet, so no config is published. Follow the setup steps in the repository above.

## Documentation

## What CrawlCheck MCP server does

The provided repository contains the open-core readers behind CrawlCheck's site-verification measurements. It examines how a site presents access rules and content to crawlers and answer engines, while leaving scoring and hosted-service operations outside the package. The modules run without a database, API key, or build step.

The repository covers robots.txt policy interpretation, crawler identification and access-log verification, quotable-content analysis, image signals, and probes related to markdown negotiation and agent surfaces. It also includes JavaScript and Python SDK directories for the hosted API and verification of signed answers, evidence bundles, and receipts. The README does not describe an MCP protocol implementation or a command that starts an MCP server.

## How it works

Install the repository as an npm dependency and import individual subpaths. The robots reader parses groups and rules, selects the most specific user-agent group, evaluates path permissions, and reports named groups that shadow wildcard disallows. It recognizes a table of 92 agents.

The agent verifier identifies crawler tokens in user-agent strings and compares logged IP addresses with published operator ranges. Results distinguish verified, spoofed, unverifiable, and non-bot lines. `loadRanges(fetch)` is the package's network operation; it retrieves feeds for supported operators, and the README recommends caching the result for a day.

The quotable reader accepts HTML and a site name, then reports signals such as whether the opening sentence defines the subject, the presence of answer-sized paragraphs, attributable sentence share, and FAQ questions visible in the page. These are readings rather than grades. Thresholds, weights, historical monitoring, fetching as fifteen crawler identities, and corpus calibration remain in the hosted service.

## Setup and configuration

Install the package directly from GitHub because the README says it is not yet on the npm registry:

```bash
npm install github:emmanuelorta/crawlcheck-core
```

The package uses ESM and requires Node 20 or newer. No environment variables, API keys, store, or build step are documented for the core readers. Import modules from paths such as `crawlcheck-core/robots`, `crawlcheck-core/agents`, and `crawlcheck-core/quotable`.

The repository also points to `@crawlcheck/sdk` for Node clients and offline verification, and to the `crawlcheck` Python package for Python 3.9 or newer. Those SDKs are separate from the core reader modules.

## Tools and capabilities

The CrawlCheck MCP server material supports these local capabilities:

- Parse robots.txt groups and determine path or root access.
- Detect named groups that lose wildcard disallow rules through shadowing.
- Identify named crawlers from user-agent strings.
- Verify crawler IP claims against published operator ranges.
- Match IPv4, IPv6, and IPv4-mapped addresses to CIDR ranges.
- Analyze HTML for quotable units and visible FAQ content.

The robots implementation strips `*` from rule paths and treats the remainder as a prefix. It does not honor `$` end anchors and ignores `Crawl-delay`, `Sitemap`, and unknown fields. The agent verifier reports operators without usable feeds as unverifiable rather than forged.

## Limitations and notes

The open core does not include the service's scoring calibration, section weights, thresholds, dataset, billing, monitoring, historical records, or multi-identity fetching. `loadRanges` depends on the availability of the published operator feeds. A user-agent string alone is treated as a claim, not proof of identity.

The supplied material does not name Claude Desktop, Cursor, or another MCP client, and it does not document an MCP transport or server entry point. Therefore, the CrawlCheck MCP server should not be assumed to be directly installable as an MCP server from this repository without additional integration documentation.

_Full upstream README: https://allmcps.com/mcp/crawlcheck/readme_