The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Crawl Census listing page.
Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.
Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:
HTTP 402 Payment Required. That is a price,
not a refusal. Treating it as a block walks away from content the operator wants to sell you.
Retrying around it takes something they are charging for.No dependencies. No key required.
The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches
rather than after it fails. Listed in the
official MCP registry as
io.github.taylorsmithgg/crawl-census.
| Tool | Answers |
|---|---|
crawl_preflight | will these domains serve my agent, refuse it, or charge it? |
agent_profile | what does this census publish about my crawler, and how do I correct it? |
census_facts | the headline findings as dated records with denominators and citation lines |
site_report | the stored audit for one domain |
scan_site | measure a domain now |
census_stats | corpus-level totals |
No authentication for read tools. Streamable HTTP.
Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.
One call per 1,000 domains instead of one per host:
For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:
The delta feed is https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> and each
response carries next_since, so a long-running crawler stays current on a few hundred bytes
an hour instead of re-downloading the list.
That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and
still need preflight or politeFetch.
Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls carrying one domain each; the same host requested three times at once cost three, because the cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour, so a crawler hit its ceiling at 240 hosts when one call covers twenty-five.
politeFetch now shares work automatically: lookups issued in the same tick leave as one
batched call, and concurrent lookups for the same host await a single request.
| pattern | before | now |
|---|---|---|
| 20 hosts, concurrent | 20 calls | 1 call of 20 |
| 1 host, 3 URLs, concurrent | 3 calls | 1 call |
| 60 hosts, concurrent | 60 calls | 3 calls (25 / 25 / 10) |
partition then fetch | 2 calls | 1 call |
batchSize defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key.
batchWaitMs widens the coalescing window for concurrency that arrives in waves rather than
all at once; the default of zero flushes on the next tick.
A pay verdict carries the amount when the origin named one:
Two things worth knowing. Most origins answering HTTP 402 name no amount at all, so price
is usually null and the arrangement has to be made out of band. And pricing is per crawler:
across the measured corpus, 78 of 213 charging origins charge some agents and serve others
free, so ask with your own token rather than assuming a domain on the list will charge you.
partition splits a work queue into crawl, pay, skip, unknown and undecidable.
The last two look alike and are not. unknown means the census has not measured that domain
yet: submit it and the next pass gets a real verdict. undecidable means the site's robots.txt
disallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste,
and a loop that resubmits its unknowns each pass would resubmit those forever. The server marks
the difference with a measurable boolean; read that, never the reason prose.
Submission is a separate call on purpose. A library that quietly POSTs during what reads as a lookup is a bad citizen, and you should choose when your queue positions are spent.
A deny list is the smaller half. Measured against the live census, a crawler that skips only robots disallows still spends around 2,900 requests a pass on domains that permit it in robots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is larger than its deny list. Those fetches return nothing and cost a round trip each.
| agent | deny list | also skippable | total |
|---|---|---|---|
| GPTBot | 3,542 | 2,944 | 6,486 |
| ClaudeBot | 3,169 | 3,194 | 6,363 |
| PerplexityBot | 1,128 | 3,658 | 4,786 |
The three sets are kept apart internally, so a change moves the one it belongs to. why()
follows the same precedence as preflight: a disallow outranks a price, because a price is not
permission.
syncBlocklist / BlocklistSync download the list once, then apply only what changed.
The list is served with the exact position in the change feed it was built at, in an
x-cursor header and a # cursor: comment. The clients read it and resume from there, so
there is no gap between the snapshot and the first poll, and no reliance on your clock being
in step with the server's. Polling by second cannot express a position inside a second, and a
crawl batch writes dozens of events into one, so a second-granularity resume can drop the
remainder of it: measured live, resuming after the first of three same-second changes
recovered both siblings by cursor and neither by second.
refresh() applies only robots transitions to the list, because that is what the list is made
of. Edge refusals, new prices and llms.txt changes come back in other for you to act on
separately — an earlier version deleted those domains from the deny list, so a crawler resumed
fetching exactly what had just started refusing it.
The authoritative definition of each verdict — what it means, what it obliges a crawler to do,
and whether asking again could change it — is published as data at
/api/v1/verdicts. The list below is a summary; if
the two ever disagree, the endpoint is right and this file is stale.
politeFetch skips disallow, refuse and pay by default, which is the endpoint's derived
do_not_fetch set. The copy here is deliberate — a crawl loop should not need a network call to
decide — and a test compares the two so it cannot drift unnoticed.
| Verdict | Meaning | Default behaviour |
|---|---|---|
allow | robots.txt permits this agent, and a live request carrying its user agent was served | fetch |
disallow | robots.txt forbids this agent at the site root | skip |
refuse | robots.txt permits it; the edge refused it anyway. The allowance is not real | skip |
pay | the origin answered HTTP 402. It will serve this agent on commercial terms | skip |
unknown | not measured recently enough to answer | fetch |
onPay: "fetch" (on_pay="fetch") overrides the paywall default. It is an explicit opt-in and
is recorded on the result as paidRouteOverridden so it shows up in your logs.
If the census is unreachable every verdict becomes unknown and your crawl proceeds as it
normally would. A third-party outage must never stop your pipeline. There is a live test for
exactly this.
If a figure is wrong, the correction channel is in that response and on your operator page. Registry facts are corrected without argument; disputed measurements are published alongside the dispute with the underlying scan records, rather than quietly amended.
25 domains per preflight call anonymously, 200 with a Pro key, 1,000 with a Data key. Pass
apiKey. Details at https://crawlcensus.com/for-crawlers.
node test.mjs runs against the live census on purpose. The value of this client is whether
its verdicts match reality, and a mocked test would assert only that the mock agrees with itself.
MIT. Data is CC BY 4.0, attribute as "Source: Crawl Census (crawlcensus.com)".