Best Alternatives to Crawler Log Triage

Check up to 500 IPs against operator-published crawler ranges; export a firewall allowlist.

Lint a robots.txt, test paths against RFC 9309, see which AI crawlers it really blocks.

Which discovery documents does a host serve? llms.txt, agent card, owners.json, mcp.json.

Markdown

Runtime: Remote (SSE)

Six checks: llms.txt, llms-full.txt, .md twins, Accept: text/markdown, sitemap.md, alt link. Free.

Identify AI crawlers by user-agent or IP, and build the robots.txt that matches your stance.

Identify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.

Lint any MCP endpoint you name: handshake, tool schemas, error conformance. Free, no key.

Draw and visually collaborate with your agents on tldraw's canvas.

Claude Task Master

Runtime: npx

Parses PRDs into structured tasks, expands implementation steps, and manages AI development work across multiple model providers.

MediaWiki MCP Server

Runtime: npx

MCP server enabling AI clients to interact with any MediaWiki wiki through standard tools

PraisonAI

Runtime: npx

AI Agents Framework with Self Reflection and MCP support

Affset

Runtime: npx

Run affset from chat: stats, campaigns, zones, payouts, targeting, and team.