Sluicer

Web scrapers that fail loudly when a site's layout changes,
instead of quietly returning empty values.
No model, no API key, no bill.

Documentation · Getting started · In your agent · Scoreboards · FAQ · Changelog

A real site, as the Wayback Machine kept it. The extractor learnt in January 2016 still fits in February. After the 2024 redesign it stops with exit code 3. heal then proposes new places for the listing and five of its ten fields, each move resting on one of five learnt values: moves to check, not a repair.

Most scrapers break without a sound. The site changes its markup, and the scraper keeps running and returns nulls, or the wrong column, for weeks before anyone notices.

Sluicer works the other way round. Show it a value on a few pages of a site (a price, a title, a date) and it learns where that value lives. On every page it reads after that, it checks the page against what it learnt. If the layout has changed, the run fails with exit code 3 and names the check that broke. sluicer heal then proposes where each field went, when the new page still shows values the extractor was learnt from.

It also reads everything a page already declares about itself: JSON-LD, microdata, RDFa, OpenGraph and four more vocabularies, merged into one record per thing, with every value pointing to the exact place on the page it came from. No model reads any page, so the same page always gives the same answer.

Quick start

pip install sluicer   # or: uv tool install sluicer, pipx install sluicer

# Learn where a book's title and price sit, from two pages of one template.
sluicer compile https://books.toscrape.com/catalogue/page-1.html \
  https://books.toscrape.com/catalogue/page-2.html \
  --want title="A Light in the Attic" --want price=51.77 -o books.json

# Replay it on another page of that template: 20 rows of title and price, exit 0.
sluicer run books.json https://books.toscrape.com/catalogue/page-3.html

# A page it was not learnt for: exit 3, naming the check that failed.
sluicer run books.json https://quotes.toscrape.com/

books.toscrape.com and quotes.toscrape.com are public sandboxes made for trying scrapers on. The last command prints FAILED https://quotes.toscrape.com/: expected the listing at html>body>…>ol.row, got not found, its path shortened here, and exits 3.

After a redesign, sluicer heal looks on the new page for the values the extractor was learnt from, and proposes a new place for each field it finds them in. In a clone of this repository, examples/shop/ holds a made-up shop before and after a redesign that renamed every class:

sluicer compile examples/shop/before-1.html examples/shop/before-2.html \
  --want title="A Light in the Attic" --want price=51.77 -o shop.json
sluicer run shop.json examples/shop/after.html   # exit 3: the listing is not found
sluicer heal shop.json examples/shop/after.html -o shop-healed.json

container: html>body>div.page>ol.row -> html>body>main.content>div.page>section.grid
member: li.product -> div.card
moved: title -> h2.name>a (5 of 5 learnt values found there; the next best place had 0)
moved: price -> div.cost (5 of 5 learnt values found there; the next best place had 0)
Wrote shop-healed.json.

Each move says how many of the field's learnt values were found in its new place, and how many the next best place held. heal proposes; it does not repair. It can only move a field whose old values the new page still shows, so run it on a page you learnt from, or one listing the same items. On the drift benchmark's 21 real redesigns, 18 of the new pages shared no item with the old ones, and heal was fully right on none and partly right on 2. When a field or the listing is lost, it exits 3 and writes nothing unless given --force.

Exit codes follow grep: 0 found -- a record or a summary answer, a <title> alone included -- 1 found nothing, 2 could not read the page, and 3 when a page broke its extractor's checks, a heal lost a field or left a move undecided, or an audit found a documented rule broken. diff exits 1 when something changed.

The checks are about structure, not truth: a run fails when a field is no longer where it was learnt, no longer reads the way it did, or no longer has its shape. A change that keeps all three, such as a different number in the price's place, passes. On SWDE, the checks flagged 18% of the extractors' wrong answers; the other 82% passed.

Reading what a page declares needs no example at all. The product page read here is examples/brake-pads.html, in a clone of this repository; without one, fetch it first with mkdir -p examples && curl -o examples/brake-pads.html https://raw.githubusercontent.com/Gi0tto/sluicer/main/examples/brake-pads.html. The library is imported from the Python you ran pip install sluicer in:

>>> import sluicer
>>> page = open("examples/brake-pads.html", "rb").read()
>>> result = sluicer.extract(page, url="https://example.com/p/bp-2210")
>>> price = result.summary["price"]
>>> price.value, price.source, price.key
('41.90', 'jsonld', 'Product.offers.price')
>>> price.where
'/html/head/script[1]#/offers/price'
>>> result.normalised
{'price': '41.90', 'currency': 'EUR', 'gtin': '4006381333931'}
>>> [(answer.value, answer.source) for answer in result.conflicts[0].answers]
[('41.90', 'jsonld'), ('39.90', 'opengraph')]

The page states one price in its JSON-LD and another in its OpenGraph tags; Sluicer reports the conflict instead of picking one in silence. sluicer inspect page.html shows the same reading laid out for a person.

Install

pip install sluicer

That is the whole install for every command but one kind of page: it reads HTML you have, fetches and crawls over plain HTTP, audits, learns extractors and turns a page into markdown. For the sluicer command in an environment of its own, uv tool install sluicer or pipx install sluicer; to run it once, installing nothing, uvx sluicer --version; in a uv project, uv add sluicer.

A page a script draws, which plain HTTP brings back as an empty shell, needs a browser: add the browser extra, then let Sluicer download Playwright's Chromium once.

pip install "sluicer[browser]"
sluicer install browser

sluicer doctor says what is installed, what each missing piece is for, and the command that adds it for the way you installed Sluicer (pip, uv tool, pipx or uvx); a command that needs a missing extra names the same command.

What you can give it