screaming-frog-mcp

MCP server that lets Claude run Screaming Frog SEO Spider headless crawls, export data, and manage crawl storage — without anyone opening the GUI.

Type a URL into Claude. Screaming Frog runs in the background. You get the data back. That's it.

Forked from bzsasson/screaming-frog-mcp v0.1.0 with bug fixes. The original had issues that made it unusable in practice — pipe deadlocks that hung crawls, false GUI detection that blocked everything after the first run, a delete command that could wipe your entire crawl database. All fixed.

What's fixed

Bug	Fix
Pipe deadlock	stdout/stderr redirected to log files instead of PIPE. Crawls no longer hang when SF produces large output.
GUI detection	Uses `psutil` instead of `ps aux`. Works on Mac and Windows. Headless CLI processes no longer get mistaken for the GUI.
Stale crawl cleanup	SF leaves a temp `crawl.seospider` file inside its own app bundle when a crawl gets interrupted. Every crawl after that fails. Now auto-cleaned before each run.
Delete safety	`delete_crawl(".")` used to resolve to the root data directory and wipe everything. Fixed.
Export dir leak	Failed exports left temp directories on disk. Now cleaned up.
Input validation	Stricter character allowlists for CLI arguments and db_id.

Requirements

Screaming Frog SEO Spider with a paid license — headless crawls require a license
Python 3.10+
uv (recommended) or pip

Installation

Mac

bash

uvx --from git+https://github.com/marykovziridze/screaming-frog-mcp screaming-frog-mcp

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

config.json

{
  "mcpServers": {
    "screaming-frog": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/marykovziridze/screaming-frog-mcp", "screaming-frog-mcp"]
    }
  }
}

Windows

Install uv first:

powershell

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Add to C:\Users\[name]\AppData\Roaming\Claude\claude_desktop_config.json:

config.json

{
  "mcpServers": {
    "screaming-frog": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/marykovziridze/screaming-frog-mcp", "screaming-frog-mcp"],
      "env": {
        "SF_CLI_PATH": "C:\\Program Files (x86)\\Screaming Frog SEO Spider\\ScreamingFrogSEOSpiderCli.exe"
      }
    }
  }
}

Restart Claude Desktop after editing the config.

Tools

Tool	What it does
`sf_check`	Verify SF is installed and licensed
`crawl_site`	Start a headless crawl
`crawl_status`	Check crawl progress
`list_crawls`	List saved crawls in SF's database
`export_crawl`	Export crawl data as CSV
`read_crawl_data`	Read and filter exported CSV data
`delete_crawl`	Delete a saved crawl
`storage_summary`	Show disk usage of crawl storage

Configuration

Variable	Default	Notes
`SF_CLI_PATH`	Mac: auto-detected	Set manually on Windows or custom installs

Known limitations

Windows stale crawl path — auto-cleanup works on Mac. On Windows, if crawls fail after an interruption, check for a crawl.seospider file in your SF install directory and delete it manually.
No crawl progress percentage — SF's headless CLI doesn't report progress mid-crawl. You know when it starts and when it finishes.
Large sites — tested on sites up to ~160 pages. Not stress-tested on 10k+ page sites.

License

MIT — see LICENSE

Credits

Original MCP server by Boaz Sasson.