SnoopScan
Turn any public page into clean markdown, structured JSON, or a schema you
define — including the pages that block everything else. REST API, MCP server,
and SDKs.
This repository is SnoopScan's open core. It runs on its own, and the hosted
API at snoopscan.com adds the parts that are not here.
See Open source vs hosted API.
Quick start
The examples below call the hosted API and read your key from
SNOOPSCAN_API_KEY. The free plan includes 1,500 credits a month.
curl
curl -X POST https://api.snoopscan.com/v1/scrape \
-H "Authorization: Bearer $SNOOPSCAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/", "formats": ["markdown", "links"]}'
Python
from snoopscan import SnoopScan
snoop = SnoopScan(api_key="...")
print(snoop.scrape("https://example.com/").markdown)
JavaScript / TypeScript
import { SnoopScan } from 'snoopscan';
const snoop = new SnoopScan({ apiKey: '...' });
console.log((await snoop.scrape('https://example.com/')).markdown);
Features
- Six fetch tiers on the hosted API, from a plain HTTP request up to a stealth
browser on a residential exit. The engine starts at the cheapest tier a
domain has historically needed and moves up only when it has to.
- Four layers of block detection, including a statistical comparison against
the domain's own baseline. A 200 that carries a challenge page, a consent
wall or a generated decoy counts as a failure.
- Extraction is routed by page type. Each page is classified first. Articles
go through the heuristic path. Forums, listings, products and tables go
through the structured path. Confidence is scored from 0 to 1 and reported.
- Shopify stores, WordPress blogs and Substacks are read from their own public
JSON, so you get the content in one request instead of a crawl.
- Failed requests cost nothing. Every response reports the tier that
succeeded and what it consumed.
- Every endpoint is also an MCP tool.
Open source vs hosted API
This repository contains:
- the REST API
- extraction
- block detection
- the crawl frontier
- the MCP server
- the first two fetch tiers (plain HTTP and browser-grade TLS)
- storage
- both SDKs
These parts are not in this repository and run only on the hosted API at
snoopscan.com:
- the proxy layer
- the browser and stealth tiers
- the anti-bot knowledge base
- the lead-gen pipeline
A self-hosted copy fetches with the first two tiers. A page that needs a real
browser or a residential exit comes back as blocked. It is reported as
blocked, never as a false success. The hosted API uses all six tiers, and its
free plan includes 1,500 credits a month.
MCP server
Every endpoint is also an MCP tool, so an agent can call them directly with no
glue code.
claude mcp add --transport http snoopscan https://api.snoopscan.com/mcp \
--header "Authorization: Bearer $SNOOPSCAN_API_KEY"
Apps that sign in instead of taking a key (the Claude app, ChatGPT) connect to
https://api.snoopscan.com/mcp-oauth and sign in with a SnoopScan account.
Any MCP client that speaks streamable HTTP connects the same way, including
Cursor, VS Code, Windsurf, Zed and Codex.
Tools
scrape, fetchMore, crawl, crawlStatus, crawlPages, map, search,
extract, checkChanges, listProducts, listPosts, domain, company,
findContacts, hiring, people, findLeads and leadsStatus.
The company, contact, people, hiring and lead tools need the hosted API. This
repository answers them with a plain "not available on this deployment".
Every tool has a title and read-only / destructive labels. None of them
submits, posts or buys anything.
Limits
The tools are designed to keep an agent's context small:
- Every content tool takes a
maxChars budget.
- Truncation is always visible and returns a continuation token.
- Crawl tools never inline page bodies.
- Guardrails cap pages per session, concurrent crawls, crawl size and
bandwidth.
- Every refusal explains the limit, so an agent can adapt instead of retrying
blindly.
executeJavascript is not exposed over MCP at any level.
API endpoints
POST /v1/scrape one URL -> markdown, html, links, screenshot, json
(+ `actions`: click, type, scroll, then read the result)
POST /v1/crawl a site, with a frontier, robots and politeness
POST /v1/map every URL on a site, ordered by relevance
POST /v1/search the web, with the results scraped
POST /v1/extract your JSON schema — or a named template — filled from the page
GET /v1/templates the named templates (product, article, jobPosting, …) and their fields
POST /v1/batch many URLs, one job
POST /v1/parse a PDF, DOCX or XLSX into text
POST /v1/monitor watch URLs on a schedule, webhook on change
POST /v1/products a Shopify or WooCommerce catalogue, from its own API
POST /v1/posts WordPress, Substack, Squarespace or Discourse posts
POST /v1/company enrich one company from its own site
POST /v1/domain what a domain runs, and who it belongs to
POST /v1/places Maps listings, details and enrichment
Every page includes metadata.platform, which says what built it. Every
response reports the tier that succeeded, the tiers attempted, and the
extraction path.
Extract data with a schema
curl -X POST https://api.snoopscan.com/v1/extract \
-H "Authorization: Bearer $SNOOPSCAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com/product/1"],
"schema": {"type": "object",
"properties": {"name": {"type": "string"},
"price": {"type": "string"}}}
}'
Monitor a page for changes
This checks the page every 60 minutes and sends a webhook when it changes.
curl -X POST https://api.snoopscan.com/v1/monitor \
-H "Authorization: Bearer $SNOOPSCAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/pricing"],
"everyMinutes": 60,
"webhookUrl": "https://your.app/hook"}'
Development
Set up and run the engine locally:
uv venv --python 3.12 && uv pip install -e ".[dev]"
uv pip install -e sdk/python --python .venv/bin/python
cp .env.example .env && createdb scraping_engine && .venv/bin/alembic upgrade head
.venv/bin/python tools/create_key.py "local-dev" --rpm 600
.venv/bin/uvicorn engine.api.app:app --reload --port 8099
The JS/TS SDK is in sdk/js and builds on its own:
cd sdk/js && npm install && npm run build && npm test
Run the tests, lint and type checks:
.venv/bin/pytest engine/tests -q
.venv/bin/ruff check engine tools && .venv/bin/mypy --strict engine/core
.venv/bin/python tools/smoke.py # live, not part of CI
The engine is built to the specs in docs/spec/. Spec 01 (API surface) and
spec 02 (data model) are the contract. Everything else is implemented behind
them.
Dependency licences
Every dependency must be MIT, Apache-2.0, BSD, ISC or MPL-2.0. The check is
blocking: a build that pulls in any other licence fails instead of warning. It
also runs weekly, because a transitive dependency can change its licence
without you noticing.
.venv/bin/pip-licenses --format=json > licences.json \
&& .venv/bin/python tools/check_licences.py licences.json
Two results of this rule are already in the code. psycopg2 (LGPL) is not
used, and Alembic migrates through asyncpg instead. tld, which comes in as a
transitive dependency, has a recorded allowlist entry that names which branch
of its tri-licence we use.
Contributing
Contributions must follow three rules:
- Clean-room implementation. Do not use code from any AGPL, GPL or SSPL
project. The API mirrors the option names common in this category, since
interface compatibility is legitimate, but the implementation is
independent.
- The licence check is blocking, not advisory.
- No personal or identifying data in the repo. Configuration comes from
environment variables only. Every fixture uses
example.com or .invalid.
Licence
The engine is licensed under AGPL-3.0. If you run a modified version as a
network service, section 13 requires you to offer its source to your users.
The running instance does this itself at GET /v1/source, so it does not
depend on a document that a fork might forget to update.
Both SDKs, Python and JS/TS, are MIT. An AGPL client library would push the
copyleft into every application that imports it, which is not what a client
library is for.
Some modules are proprietary and are not covered by the AGPL: the proxy layer,
the browser and stealth tiers, the anti-bot knowledge base, and the lead-gen
pipeline. They are listed in LICENSE-PROPRIETARY, which is generated from
engine/licensing.py. CI enforces the split: tools/check_split.py fails the
build if any public module depends on one of them.