CleanPage

A paid MCP tool that turns any URL into clean, structured, agent-optimized content. Agents pay $0.02 USDC per successful call via x402 on Base — no accounts, no API keys, no subscriptions.

clean_page(url) → { title, clean_markdown, plain_text, structured_facts,
                     metadata, links, images, quality_score, extraction_notes }

Built for Cloudflare Workers. Full TypeScript. No paid infra required beyond a domain and a Base wallet address.

How agents use this

1. Discover the service for free — no payment needed for any of this:

   • GET /.well-known/agent.json — machine-readable capability card
   • GET /.well-known/mcp.json — MCP connection + pricing manifest
   • GET /llms.txt — plain-text summary for LLM crawlers
   • MCP tools health, list_capabilities, pricing (all free)

2. Connect an MCP client to https://<your-domain>/mcp over Streamable HTTP:

   config.json

   Copy

   {
     "mcpServers": {
       "cleanpage": { "url": "https://<your-domain>/mcp", "transport": "streamable-http" }
     }
   }
   

3. Call clean_page with no payment attached first:

   Code

   Copy

   POST /mcp
   { "jsonrpc": "2.0", "id": 1, "method": "tools/call",
     "params": { "name": "clean_page", "arguments": { "url": "https://example.com/article" } } }
   

   The response is HTTP 402 with an accepts array describing exactly what payment is required (amount, network, asset, payee address).

4. Sign and retry. Construct a USDC transferWithAuthorization (EIP-3009) payment payload for the requested amount, base64-encode it, and retry the same request with an X-PAYMENT header. Most x402 client libraries (e.g. x402-fetch, x402-axios) do steps 3–4 for you automatically.

5. Get the result. On success, the response carries the full JSON result plus an X-PAYMENT-RESPONSE header with the settlement transaction hash. Failed extractions (paywalls, bot-blocks, empty pages) are never charged — settlement only happens after a successful extraction.

Full protocol details: see /llms.txt and /.well-known/mcp.json once deployed, or the "For agents" section of the landing page.

Project structure

cleanpage/
├── src/
│   ├── index.ts                # Hono app: routes, x402 payment gate, static asset fallback
│   ├── types.ts                # Shared types (Env, CleanPageResult, etc.)
│   ├── config.ts                # Reads env vars into a typed config object
│   ├── mcp/
│   │   ├── server.ts            # JSON-RPC 2.0 / MCP method dispatcher (stateless)
│   │   └── tools.ts             # Tool schemas + descriptions (clean_page, health, ...)
│   ├── extraction/
│   │   ├── extractor.ts         # Core pipeline: fetch → Readability → markdown/links/metadata
│   │   └── enrichment.ts        # Optional cheap LLM pass for structured_facts
│   ├── payments/
│   │   └── x402.ts              # x402 payment requirements, facilitator verify/settle
│   └── middleware/
│       └── limits.ts            # KV-backed rate limiting + call/revenue analytics
├── public/                       # Static assets served at the Worker's root
│   ├── index.html                # Landing page (live demo, pricing, docs, for-agents)
│   ├── robots.txt                 # Explicitly welcomes agent/LLM crawlers
│   ├── llms.txt                   # Plain-text agent-facing summary
│   └── .well-known/
│       ├── agent.json             # Agent discovery/capability card
│       └── mcp.json               # MCP connection + pricing manifest
├── wrangler.toml                  # Cloudflare Workers config (bindings, vars)
├── package.json
├── tsconfig.json
└── .env.example                   # Documents every config value

Architecture notes

MCP transport. The MCP Streamable HTTP transport is implemented directly in src/mcp/server.ts as a stateless JSON-RPC 2.0 dispatcher (initialize, tools/list, tools/call, ping), rather than via the official SDK's Node-oriented HTTP transport class. This is a deliberate choice for correctness on Cloudflare Workers: the SDK's transport targets Node's http.IncomingMessage/ServerResponse, which Workers doesn't have. The dispatcher here follows the same wire protocol (JSON-RPC over a single POST endpoint, one server-scoped session per request) so any standard MCP client can connect to it. If you later run this on a Node-based host, you can swap in @modelcontextprotocol/sdk's StreamableHTTPServerTransport directly.

Payment gating. POST /mcp peeks at the parsed JSON-RPC body: only tools/call requests naming clean_page are payment-gated (src/index.ts); health, list_capabilities, pricing, and all non-tool-call methods (initialize, tools/list, etc.) pass straight through, rate-limited but free. This keeps the discovery surface fully open to agents while gating only the one paid capability.

Settlement timing. Payment is verified before extraction runs, but settled (broadcast on-chain) only after extraction succeeds. If extraction fails, the agent's payment authorization is simply discarded — nothing is charged.

No private key required. Because USDC on Base supports EIP-3009 meta-transactions, the resource server never holds a private key. PAYMENT_ADDRESS is a public wallet address; the facilitator submits the signed authorization on-chain and sponsors gas.

Extraction engine. @mozilla/readability (the engine behind Firefox Reader Mode) does the core content extraction, running against a DOM built with linkedom (a fast, dependency-light DOM implementation that works in Workers — jsdom does not). A small hand-written HTML→Markdown walker converts the extracted fragment to markdown, since no existing markdown-conversion library is both Workers-compatible and DOM-based. Heuristics before and after extraction catch common failure modes: paywall markers, bot-check pages, and JS-only empty shells all produce typed errors instead of low-quality output.

Optional LLM enrichment. structured_facts is populated only when the caller passes options.extract_facts: true, and only if ENABLE_LLM_ENRICHMENT=true and ANTHROPIC_API_KEY is set. It's off the critical path — if the enrichment call fails or is disabled, clean_page still returns a complete result with an empty structured_facts array and a note explaining why. There's no additional charge for using it.

Rate limiting & analytics. Both are implemented with a single KV namespace: a fixed 1-minute-window counter per client IP for free-tier calls, and a running JSON counter object for total calls / successes / failures / revenue, exposed read-only at GET /analytics.

Required services / infrastructure

No database, no queue, no separate backend — everything runs in the one Worker.

Deployment (step by step)

# 1. Install dependencies
npm install

# 2. Create the KV namespace used for rate limiting + analytics
npx wrangler kv namespace create CLEANPAGE_KV
npx wrangler kv namespace create CLEANPAGE_KV --preview
# Copy the two returned IDs into wrangler.toml under [[kv_namespaces]]

# 3. Edit wrangler.toml [vars]:
#    - PAYMENT_ADDRESS   → your Base wallet address
#    - PUBLIC_BASE_URL   → your real domain, e.g. https://cleanpage.yourdomain.com
#    - X402_NETWORK      → "base-sepolia" while testing, "base" for mainnet
#    Also update the same URLs/addresses in:
#    - public/.well-known/agent.json
#    - public/.well-known/mcp.json
#    - public/llms.txt
#    - public/index.html (og:url, structured data, MCP config example)

# 4. (Optional) enable LLM fact extraction
npx wrangler secret put ANTHROPIC_API_KEY

# 5. Type-check
npm run typecheck

# 6. Deploy
npm run deploy

# 7. Point your domain at the Worker (Cloudflare dashboard → Workers →
#    your worker → Triggers → Custom Domains), or use the workers.dev
#    subdomain Wrangler prints after deploy.

Local development

npm run dev
# Worker runs at http://localhost:8787
# Try: curl -X POST http://localhost:8787/mcp -H "Content-Type: application/json" \
#        -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

Going from testnet to mainnet

1. Switch X402_NETWORK from base-sepolia to base in wrangler.toml.
2. Switch X402_FACILITATOR_URL to a mainnet-capable facilitator (confirm your chosen facilitator supports Base mainnet settlement — verify current facilitator options, as this ecosystem is still young and options change).
3. Re-deploy.

Environment variables

See .env.example for the full annotated list. Summary: