Scrapewright

Give it a store URL, get structured rows. A model writes the parser once; replays are free.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.

Add to Cursor Add to VS Code

Manual Client & Custom JSON ConfigExpand JSON ▾

Client Config & Setup

Choose your client or environment

Target File:~/Library/Application Support/Claude/claude_desktop_config.json

claude_desktop_config.json

{
  "mcpServers": {
    "scrapewright": {
      "command": "npx",
      "args": [
        "-y",
        "scrapewright"
      ]
    }
  }
}

💡 Paste the JSON block into your client's configuration file under mcpServers, then restart the application.

Documentation Overview

scrapewright

Give it a URL. It writes the scraper.

Most e-commerce catalog scraping splits into two worlds: sites on a known platform (Shopify, WooCommerce) that expose a clean JSON feed, and everything else — bespoke HTML where you hand-write a parser per site and re-write it every time the markup shifts. scrapewright collapses both into one call:

Detect the platform behind a URL.
For known platforms, extract deterministically from their public catalog API — free, stable, no LLM.
For custom HTML, synthesize a reusable extractor once with an LLM, cache it, and replay it deterministically forever after.

The LLM is a compiler, not a runtime. It runs once per site to produce a recipe of CSS selectors; every page after that is parsed by plain BeautifulSoup at zero marginal cost. That is the whole cost-control story — no per-page model calls, no token bill that scales with your crawl.

Code

                    ┌─────────────┐
   store URL  ───▶  │   detect    │
                    └──────┬──────┘
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                   ▼
    shopify            woocommerce         generic HTML
   products.json      wc/store/products    (page mode)
        │                  │                   │
        │  deterministic   │                   ▼
        │  (free)          │            cached recipe? ──yes──▶ replay (free)
        └────────┬─────────┘                   │ no
                 ▼                              ▼
             Product{}  ◀───── selectors ── JSON-LD? ──yes──▶ Product{} (free)
                 ▲                              │ no
                 │                              ▼
                 └──────── replay ◀── LLM synthesizes recipe ONCE ──▶ cache

Everything normalizes to one Product shape, so downstream code never knows or cares which path a record came from.

Install

Terminal

pip install scrapewright               # deterministic paths (Shopify, Woo, JSON-LD)
pip install "scrapewright[llm]"        # + LLM recipe synthesis for custom HTML
pip install "scrapewright[llm,js,excel,mcp]"   # + JS rendering, XLSX, MCP server
playwright install chromium                    # only needed for --js

Use it

server.ts

from scrapewright import Scrapewright

sw = Scrapewright()

# Catalog mode — a whole Shopify/WooCommerce store, deterministically
for product in sw.scrape_catalog("https://shop.example.com", max_items=200):
    print(product.brand, product.title, product.price, product.currency)

# Page mode — one custom-HTML product page.
# First call: tries JSON-LD (free); if absent, the LLM writes a recipe once.
# Every later call on that domain: replayed from the cached recipe, no LLM.
item = sw.scrape_page("https://boutique.example.com/products/wool-coat")
print(item.model_dump(exclude={"raw"}))

# Crawl mode — walk a WHOLE custom store from one listing/category URL.
# The frontier discovers product pages (deterministic, free); the first page
# pays the single synthesis cost, every other page replays the recipe.
for product in sw.crawl("https://boutique.example.com/collection", max_items=100):
    print(product.title, product.price)

CLI

bash

scrapewright detect https://shop.example.com          # platform + strategy
scrapewright run    https://shop.example.com --max 50 # scrape a catalog → JSONL
scrapewright crawl  https://boutique.example.com/collection -o products.xlsx
scrapewright run    https://shop.example.com -o products.csv   # Excel-ready CSV
scrapewright add    https://boutique.example.com/products/coat  # learn a site
scrapewright run    https://boutique.example.com/products/coat --no-llm
scrapewright list                                     # cached recipe domains

-o writes .csv (Excel-ready, UTF-8 BOM), .xlsx (pip install scrapewright[excel]), or .jsonl; without it, products stream to stdout as JSONL.

Know what you are dealing with

detect answers the routing question before a job starts:

Code

$ scrapewright detect https://some-store.com
https://some-store.com
  platform: bigcommerce
  catalog:  -
  strategy: crawl
  note:     BigCommerce (Stencil) markup

Twelve platforms are recognized: Shopify and WooCommerce publish a free JSON catalog, so those route to catalog — deterministic, no LLM, no browser. Magento, BigCommerce, Salesforce Commerce Cloud, Squarespace, Wix, Webflow, PrestaShop, Shopware, Ecwid and OpenCart are recognized by fingerprint and route to crawl, where the recipe path handles them like any custom site — the point of naming them is knowing what you face, not writing twelve parsers. Wix and Ecwid render client-side, so detection says crawl+js up front.

A site behind an anti-bot wall reports strategy: blocked with the HTTP status, rather than pretending it found nothing.

Bring your own schema

Products are just the built-in default. Declare the fields you want and the same compile-once/replay-free loop works on any structured page — job posts, listings, registry records:

bash

scrapewright run https://jobs.example.com/p/123 -f title -f company -f salary:number -f tags:list --schema-name job

server.ts

from scrapewright import Scrapewright, Schema

job = Schema.from_names(["title", "company", "salary:number", "tags:list"], name="job")
record = Scrapewright().extract("https://jobs.example.com/p/123", job)
print(record.data)   # {'title': ..., 'company': ..., 'salary': ..., 'tags': [...]}

Field kinds are text (default), number, url, and list. Recipes are cached per site and per schema, so one domain can be compiled against several field sets without them overwriting each other.

Use it from an AI agent (MCP)

scrapewright ships an MCP server, so an agent can call it as a tool instead of reading raw HTML itself:

Terminal

pip install "scrapewright[mcp,llm]"
scrapewright mcp

Point any MCP client at that command and the agent gains five tools: detect_site, scrape_catalog, extract_page, crawl_site, and list_learned_sites.

Drop this into your client's config — Claude Desktop, Cursor, or anything else that speaks MCP:

config.json

{
  "mcpServers": {
    "scrapewright": {
      "command": "uvx",
      "args": ["--from", "scrapewright[mcp,llm]", "scrapewright", "mcp"],
      "env": { "ANTHROPIC_API_KEY": "sk-ant-..." }
    }
  }
}

The key is only needed for sites on no known platform, where a recipe has to be written once. Shopify and WooCommerce stores work without it.

The economics are the point. An agent that reads pages itself pays model tokens per page, forever. These tools pay once per site — an agent crawling 500 pages spends one synthesis, not five hundred, and platform stores (Shopify, WooCommerce) cost nothing at all.

Run it as a service

The same core behind an HTTP API, with keys, quotas, metering and background jobs:

Terminal

pip install "scrapewright[service,llm]"
scrapewright keys create --label alice --plan free
scrapewright serve --port 8000

Terminal

curl -X POST localhost:8000/v1/extract   -H "X-API-Key: sw_..." -H "Content-Type: application/json"   -d '{"url": "https://shop.example.com/products/coat"}'

Endpoint	Purpose
`POST /v1/detect`	platform + strategy (cheap)
`POST /v1/extract`	one page -> structured record
`POST /v1/crawl`	a whole site -> job id (crawls outlive a request)
`GET /v1/jobs/{id}`	poll a crawl
`GET /v1/usage`	what this key has consumed, against its plan

Prepaid credits, no subscription

One action costs real money: compiling a new site, a single LLM pass over a page, measured at $0.02 on a small product page and $0.15 on a heavy rendered one. Everything after that is BeautifulSoup — the ten-thousandth record from a compiled site is free to serve. So credits are priced off that one action, and everything else is denominated relative to it:

Action	Credits
1 record delivered	1
1 browser render	5
1 new site compiled	300
page fetches, `detect`	free

Code

$ scrapewright plans
pack         credits   price   $/credit   margin
starter       10,000     $10    0.00100    80.0%
growth        50,000     $40    0.00080    75.0%
scale        250,000    $150    0.00060    66.7%

Free: 1,000 credits a month, resetting.

Margin is measured on compiling a site, because that is the only step that costs anything; a test fails if a price edit drops any pack below 60%. A free account can cost us at most $0.20 a month, even if every free credit goes to the most expensive action there is.

Credits are a ledger, not a counter — every grant and every charge is a row, so a disputed bill can be reconstructed line by line, and a replayed payment webhook cannot double-credit (grants take an idempotency key). Running out returns 402 with the balance and what to do about it; a crawl is capped by the credits on hand, so a job stops at what the caller can pay for instead of overdrawing.

bash

scrapewright credits grant <key_id> --pack starter --idempotency <payment_id>
scrapewright credits balance <key_id>

Taking payment

Stripe is wired in and turned on by environment, not by a code change:

Terminal

pip install "scrapewright[service,stripe]"
export STRIPE_SECRET_KEY=sk_test_...      # absent -> nothing is for sale
export STRIPE_WEBHOOK_SECRET=whsec_...    # absent -> webhooks are refused
scrapewright serve

Read the full README →View source on GitHub →

Related MCP Servers

View all in Developer Tools View all alternatives

Reviews

No reviews yet — be the first to share how this listing worked for you.

Frequently Asked Questions about Scrapewright

Add the following block to your claude_desktop_config.json under mcpServers: "mcpServers": { "scrapewright": { "command": "npx", "args": ["-y", "scrapewright"] } }

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:

Live Dynamic SVG Preview

Markdown (GitHub README)

[![AllMCPs](https://allmcps.com/api/badge/scrapewright?style=directory)](https://allmcps.com/mcp/scrapewright)

HTML Embed

<a href="https://allmcps.com/mcp/scrapewright"><img src="https://allmcps.com/api/badge/scrapewright?style=directory" alt="Scrapewright on AllMCPs" /></a>

Technical Specs & Signals

CategoryDeveloper Tools

27Quality signal: Emerging · 27/100How this signal is calculated ▾

Server availabilityNot measured

Not scored for repo-hosted servers — we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.

Verified ownership8/20

Documentation & tools11/30

Adoption & activity1/15

Community engagement0/10

A guidance signal from public completeness & health data — not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.

★ Spotlight Slot

Feature Your MCP Server

Get maximum visibility for your server across our directory, search results, and detail pages.

Spotlight Your Server

Own this project?

This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to unlock edit access and the Official badge — proof is checked automatically, then reviewed by our team.

Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Explore more

More in 💻 Developer Tools →Best MCP servers for Developers →Alternatives to Scrapewright →Install in Claude Desktop Install in Cursor Install in VS Code

Scrapewright

Give it a store URL, get structured rows. A model writes the parser once; replays are free.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.

Add to Cursor Add to VS Code

Manual Client & Custom JSON ConfigExpand JSON ▾

Client Config & Setup

Choose your client or environment

Target File:~/Library/Application Support/Claude/claude_desktop_config.json

claude_desktop_config.json

{
  "mcpServers": {
    "scrapewright": {
      "command": "npx",
      "args": [
        "-y",
        "scrapewright"
      ]
    }
  }
}

💡 Paste the JSON block into your client's configuration file under mcpServers, then restart the application.

Documentation Overview

scrapewright

Give it a URL. It writes the scraper.

Detect the platform behind a URL.
For known platforms, extract deterministically from their public catalog API — free, stable, no LLM.
For custom HTML, synthesize a reusable extractor once with an LLM, cache it, and replay it deterministically forever after.

Code

                    ┌─────────────┐
   store URL  ───▶  │   detect    │
                    └──────┬──────┘
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                   ▼
    shopify            woocommerce         generic HTML
   products.json      wc/store/products    (page mode)
        │                  │                   │
        │  deterministic   │                   ▼
        │  (free)          │            cached recipe? ──yes──▶ replay (free)
        └────────┬─────────┘                   │ no
                 ▼                              ▼
             Product{}  ◀───── selectors ── JSON-LD? ──yes──▶ Product{} (free)
                 ▲                              │ no
                 │                              ▼
                 └──────── replay ◀── LLM synthesizes recipe ONCE ──▶ cache

Everything normalizes to one Product shape, so downstream code never knows or cares which path a record came from.

Install

Terminal

pip install scrapewright               # deterministic paths (Shopify, Woo, JSON-LD)
pip install "scrapewright[llm]"        # + LLM recipe synthesis for custom HTML
pip install "scrapewright[llm,js,excel,mcp]"   # + JS rendering, XLSX, MCP server
playwright install chromium                    # only needed for --js

Use it

server.ts

from scrapewright import Scrapewright

sw = Scrapewright()

# Catalog mode — a whole Shopify/WooCommerce store, deterministically
for product in sw.scrape_catalog("https://shop.example.com", max_items=200):
    print(product.brand, product.title, product.price, product.currency)

# Page mode — one custom-HTML product page.
# First call: tries JSON-LD (free); if absent, the LLM writes a recipe once.
# Every later call on that domain: replayed from the cached recipe, no LLM.
item = sw.scrape_page("https://boutique.example.com/products/wool-coat")
print(item.model_dump(exclude={"raw"}))

# Crawl mode — walk a WHOLE custom store from one listing/category URL.
# The frontier discovers product pages (deterministic, free); the first page
# pays the single synthesis cost, every other page replays the recipe.
for product in sw.crawl("https://boutique.example.com/collection", max_items=100):
    print(product.title, product.price)

CLI

bash

scrapewright detect https://shop.example.com          # platform + strategy
scrapewright run    https://shop.example.com --max 50 # scrape a catalog → JSONL
scrapewright crawl  https://boutique.example.com/collection -o products.xlsx
scrapewright run    https://shop.example.com -o products.csv   # Excel-ready CSV
scrapewright add    https://boutique.example.com/products/coat  # learn a site
scrapewright run    https://boutique.example.com/products/coat --no-llm
scrapewright list                                     # cached recipe domains

-o writes .csv (Excel-ready, UTF-8 BOM), .xlsx (pip install scrapewright[excel]), or .jsonl; without it, products stream to stdout as JSONL.

Know what you are dealing with

detect answers the routing question before a job starts:

Code

$ scrapewright detect https://some-store.com
https://some-store.com
  platform: bigcommerce
  catalog:  -
  strategy: crawl
  note:     BigCommerce (Stencil) markup

A site behind an anti-bot wall reports strategy: blocked with the HTTP status, rather than pretending it found nothing.

Bring your own schema

Products are just the built-in default. Declare the fields you want and the same compile-once/replay-free loop works on any structured page — job posts, listings, registry records:

bash

scrapewright run https://jobs.example.com/p/123 -f title -f company -f salary:number -f tags:list --schema-name job

server.ts

from scrapewright import Scrapewright, Schema

job = Schema.from_names(["title", "company", "salary:number", "tags:list"], name="job")
record = Scrapewright().extract("https://jobs.example.com/p/123", job)
print(record.data)   # {'title': ..., 'company': ..., 'salary': ..., 'tags': [...]}

Use it from an AI agent (MCP)

scrapewright ships an MCP server, so an agent can call it as a tool instead of reading raw HTML itself:

Terminal

pip install "scrapewright[mcp,llm]"
scrapewright mcp

Point any MCP client at that command and the agent gains five tools: detect_site, scrape_catalog, extract_page, crawl_site, and list_learned_sites.

Drop this into your client's config — Claude Desktop, Cursor, or anything else that speaks MCP:

config.json

{
  "mcpServers": {
    "scrapewright": {
      "command": "uvx",
      "args": ["--from", "scrapewright[mcp,llm]", "scrapewright", "mcp"],
      "env": { "ANTHROPIC_API_KEY": "sk-ant-..." }
    }
  }
}

The key is only needed for sites on no known platform, where a recipe has to be written once. Shopify and WooCommerce stores work without it.

Run it as a service

The same core behind an HTTP API, with keys, quotas, metering and background jobs:

Terminal

pip install "scrapewright[service,llm]"
scrapewright keys create --label alice --plan free
scrapewright serve --port 8000

Terminal

curl -X POST localhost:8000/v1/extract   -H "X-API-Key: sw_..." -H "Content-Type: application/json"   -d '{"url": "https://shop.example.com/products/coat"}'

Endpoint	Purpose
`POST /v1/detect`	platform + strategy (cheap)
`POST /v1/extract`	one page -> structured record
`POST /v1/crawl`	a whole site -> job id (crawls outlive a request)
`GET /v1/jobs/{id}`	poll a crawl
`GET /v1/usage`	what this key has consumed, against its plan

Prepaid credits, no subscription

Action	Credits
1 record delivered	1
1 browser render	5
1 new site compiled	300
page fetches, `detect`	free

Code

$ scrapewright plans
pack         credits   price   $/credit   margin
starter       10,000     $10    0.00100    80.0%
growth        50,000     $40    0.00080    75.0%
scale        250,000    $150    0.00060    66.7%

Free: 1,000 credits a month, resetting.

bash

scrapewright credits grant <key_id> --pack starter --idempotency <payment_id>
scrapewright credits balance <key_id>

Taking payment

Stripe is wired in and turned on by environment, not by a code change:

Terminal

pip install "scrapewright[service,stripe]"
export STRIPE_SECRET_KEY=sk_test_...      # absent -> nothing is for sale
export STRIPE_WEBHOOK_SECRET=whsec_...    # absent -> webhooks are refused
scrapewright serve

Read the full README →View source on GitHub →

Related MCP Servers

View all in Developer Tools View all alternatives

Reviews

No reviews yet — be the first to share how this listing worked for you.

Frequently Asked Questions about Scrapewright

Add the following block to your claude_desktop_config.json under mcpServers: "mcpServers": { "scrapewright": { "command": "npx", "args": ["-y", "scrapewright"] } }

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:

Live Dynamic SVG Preview

Markdown (GitHub README)

[![AllMCPs](https://allmcps.com/api/badge/scrapewright?style=directory)](https://allmcps.com/mcp/scrapewright)

HTML Embed

<a href="https://allmcps.com/mcp/scrapewright"><img src="https://allmcps.com/api/badge/scrapewright?style=directory" alt="Scrapewright on AllMCPs" /></a>

Technical Specs & Signals

CategoryDeveloper Tools

27Quality signal: Emerging · 27/100How this signal is calculated ▾

Server availabilityNot measured

Not scored for repo-hosted servers — we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.

Verified ownership8/20

Documentation & tools11/30

Adoption & activity1/15

Community engagement0/10

★ Spotlight Slot

Feature Your MCP Server

Get maximum visibility for your server across our directory, search results, and detail pages.

Spotlight Your Server

Own this project?

Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Explore more

More in 💻 Developer Tools →Best MCP servers for Developers →Alternatives to Scrapewright →Install in Claude Desktop Install in Cursor Install in VS Code