ScrapingBot README

ScrapingBot

One API key for public web data: scrape any page (plain or JavaScript-rendered), pull fields out with AI, and get public TikTok, Instagram, Google and Amazon data as JSON. A hosted MCP server gives AI agents the same data as tools.

100 free credits when you sign up. No credit card. Get your API key

Docs · Pricing · MCP server · Examples · Agent Skills

Quickstart

export SCRAPINGBOT_API_KEY="YOUR_API_KEY"

curl "https://scrapingbot.io/api/v1/scrape?url=https://example.com" \
  -H "x-api-key: $SCRAPINGBOT_API_KEY"

{
  "success": true,
  "url": "https://example.com",
  "html": "<!doctype html><html lang=en><head>…",
  "status": 200,
  "duration": "0.79",
  "credits_used": 1,
  "job_id": "…"
}

That call cost 1 credit. Failed requests are refunded automatically.

Base URL: https://scrapingbot.io/api/v1 Auth: x-api-key: YOUR_API_KEY header (or Authorization: Bearer YOUR_API_KEY). Keep the key on the server.

Endpoints

The data APIs (TikTok, Instagram, Google, Amazon) are all POST with a JSON body naming the endpoint and its params:

curl -X POST "https://scrapingbot.io/api/v1/tiktok" \
  -H "x-api-key: $SCRAPINGBOT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"endpoint": "/user/info", "params": {"unique_id": "tiktok"}}'

Full parameters and response shapes: docs.

How charging works

• Data APIs charge only for a successful 200. Any error is refunded.
• Website API charges when the page answers (2xx/3xx, or a real 400/404 from the site, reported as fault: "user"). Timeouts, other 4xx such as 403 and 429, 5xx and errors on our side are refunded (credits_used: 0).
• Requests rejected before they run (missing parameter, bad key, concurrency limit) cost nothing.

Limits

Limits are on requests in flight at once, not per minute. The free plan allows 1 concurrent request; paid plans allow 10 to 200. Going over returns 429 right away (free); retry with a short, growing delay. Timeouts: 45 s for scraping and Instagram, 30 s for TikTok and Amazon, 15 s for Google.

MCP server for AI agents

Hosted at https://scrapingbot.io/api/mcp (Streamable HTTP, stateless). Nothing to install. Authenticate with your API key in the x-api-key header, or Authorization: Bearer.

Claude Code

claude mcp add --transport http scrapingbot https://scrapingbot.io/api/mcp \
  --header "x-api-key: YOUR_API_KEY"

Add --scope user to use it in every project. Type /mcp in a session to see the tools.

Claude Desktop (and Claude on the web)

Open Settings → Connectors → Add custom connector, name it ScrapingBot, and paste:

https://scrapingbot.io/api/mcp?api_key=YOUR_API_KEY

Then turn it on from the tools menu in a chat. Connectors take a URL only, so the key goes in the URL: keep that URL private, like a password.

Cursor

~/.cursor/mcp.json (or .cursor/mcp.json in a project):

{
  "mcpServers": {
    "scrapingbot": {
      "url": "https://scrapingbot.io/api/mcp",
      "headers": { "x-api-key": "YOUR_API_KEY" }
    }
  }
}

VS Code

.vscode/mcp.json in your project, then pick the ScrapingBot tools in Copilot's agent mode:

{
  "servers": {
    "scrapingbot": {
      "type": "http",
      "url": "https://scrapingbot.io/api/mcp",
      "headers": { "x-api-key": "YOUR_API_KEY" }
    }
  }
}

Any other MCP client

URL https://scrapingbot.io/api/mcp, header x-api-key: YOUR_API_KEY (or Authorization: Bearer YOUR_API_KEY). If the client only takes a URL, append ?api_key=YOUR_API_KEY. When calling it by hand, send Accept: application/json, text/event-stream.

Tools

Tool calls are ordinary API requests: same credits, same concurrency slots, same refunds.

Agent Skills

skills/ holds Agent Skills that teach an agent to call the REST API directly with SCRAPINGBOT_API_KEY: which endpoint for which task, the parameters, the response fields worth reading, error handling and costs.

Claude Code: copy a folder into ~/.claude/skills/ (or .claude/skills/ in a project). Set SCRAPINGBOT_API_KEY in the environment the agent runs in.

Examples

Small, runnable scripts in examples/, each in curl, Python (requests) and Node (built-in fetch, Node 18+):

Python needs pip install requests; the curl scripts that build JSON use jq. Each language folder has a tiny shared client (scrapingbot.py, scrapingbot.mjs) that retries refunded failures (408, 429, 5xx) with backoff.

export SCRAPINGBOT_API_KEY="YOUR_API_KEY"
sh examples/curl/scrape.sh https://example.com
python examples/python/google_search.py "best espresso machine"
node examples/node/tiktok_user.mjs tiktok

Pricing

Free: 100 credits on sign-up, 1 concurrent request. Paid plans start at $49.99/month for 275,000 credits and 10 concurrent requests. See pricing.

Responsible use

ScrapingBot returns publicly available data. Respect each site's terms and applicable law, and handle any personal data you collect accordingly.

License

The examples and skills in this repository are MIT licensed. See LICENSE. Use of the ScrapingBot API is subject to the terms at scrapingbot.io.