Crawlie

What Crawlie does

Crawlie turns website audits into MCP tools that an AI agent can call directly. A full crawl reports separate Health, GEO, and Accessibility scores, along with detected issues and data for individual pages. Health represents technical SEO, GEO represents readiness for discovery and citation in AI search, and Accessibility represents WCAG conformance.

The crawler checks common technical problems such as broken links, redirects, missing metadata, 4xx and 5xx responses, and more than 40 SEO and generative-engine rules. Findings include plain-language guidance, so an agent can move from identifying a problem to explaining why it matters and how to address it.

How it works

The local distribution includes the crawlie-mcp executable and communicates with MCP clients using JSON-RPC over stdio. Agents can submit a whole-site crawl, a single URL audit, or an explicit list of URLs. Whole-site crawls respect robots.txt and use sitemap.xml as a seed by default.

Crawlie can save crawl results to local report history. The list_reports tool lets an agent read that history, making it possible to revisit earlier audits without immediately starting another crawl. The same engine is also available through Crawlie Cloud at https://crawlie.app/mcp, using MCP Streamable HTTP and bearer-key authentication. Cloud crawls return a report ID and are metered against the account plan.

Setup and configuration

Install the CLI and MCP server globally with npm:

npm i -g crawlie

After installation, crawlie-mcp is available on the system path. A stdio client can launch it with the command crawlie-mcp. Claude Desktop uses an MCP server entry with that command, while Claude Code can add it with claude mcp add crawlie crawlie-mcp.

Building from source requires Rust. The repository can produce both the crawlie and crawlie-mcp binaries; clients should use the absolute path to the compiled MCP executable when it is not installed on the path.

For the hosted endpoint, configure the URL and an Authorization: Bearer header containing a Crawlie API key. API keys are created in the Crawlie dashboard under Settings and API keys.

Tools and capabilities

• crawl_site crawls and audits a complete site, returning scores, issues, and per-page information.
• audit_url checks one page.
• audit_urls checks a supplied list of pages.
• explain_issue describes the importance of a rule and suggests how to fix it.
• list_rules returns the catalogue of available checks.
• list_reports reads saved crawl history.

Agents can use these tools for site-wide audits, targeted landing-page reviews, GEO investigations, and structured remediation plans. The underlying CLI also supports JSON, CSV, pretty terminal, and HTML report output, plus page limits, depth limits, URL inclusion and exclusion patterns, severity filtering, and CI failure thresholds.

Limitations and notes

Local crawls run on the machine hosting the MCP process and do not require a hosted account. The default crawl limit is 500 pages and the default request concurrency is 16; both can be adjusted through CLI options. Robots, sitemap seeding, and external-link checks can be disabled when needed.

Crawlie also offers a hosted service, but hosted crawls are scoped to a team and consume the applicable plan allowance. The local and hosted deployments expose different hosted-only operations, so integrations should use the tool catalogue returned by the connected server rather than assuming every deployment has identical tools.