Agent Scraper MCP

What aparajithn/agent-scraper-mcp MCP server does

The aparajithn/agent-scraper-mcp MCP server provides six tools for retrieving and inspecting public web pages. scrape_url converts a page into readable text, Markdown, or HTML. scrape_structured accepts a map of field names to CSS selectors, which is useful for extracting values such as product names, prices, authors, or review text.

screenshot_url captures either the visible viewport or the complete scrollable page and returns a base64-encoded PNG. extract_links returns link text and URLs, with an optional regular-expression filter. extract_meta collects the page title, description, canonical URL, favicon, Open Graph tags, and Twitter Card tags. search_google accepts a query and returns result titles, URLs, and snippets.

How it works

The hosted MCP endpoint is available at https://agent-scraper-mcp.onrender.com/mcp. An MCP client can connect to that URL and call the tools using the protocol. The project also exposes REST routes under the same host, including /api/v1/scrape_url, /api/v1/scrape_structured, /api/v1/screenshot_url, /api/v1/extract_links, /api/v1/extract_meta, and /api/v1/search_google.

Content extraction uses readability-style processing, while HTML parsing and metadata handling are provided by the project’s Python web stack. Screenshots use Playwright and Chromium. Structured extraction is controlled by selectors supplied with each request, so the agent or calling application must know the relevant page structure.

Setup and configuration

For MCP clients, add the hosted URL to the client’s MCP settings. The README gives a Cline-compatible configuration using the url field. Local development requires Python 3.11, installation of the project with its development dependencies, and Chromium installation through Playwright. The documented local server command is uvicorn src.main:app --reload --port 8080.

Docker is also documented: build the repository image and run it with port 8080 exposed. The deployment configuration lists PUBLIC_HOST and X402_WALLET_ADDRESS; the latter identifies the wallet used for HTTP 402 payments.

Tools and capabilities

The aparajithn/agent-scraper-mcp MCP server supports:

• Readable page extraction in text, Markdown, or HTML
• CSS-selector-based structured extraction
• Viewport and full-page PNG screenshots
• Link collection with optional regex filtering
• Title, description, canonical, favicon, Open Graph, and Twitter metadata
• Google result retrieval with configurable result count

Limitations and notes

The hosted service provides 50 requests per IP per day in its free tier, with all tools included and no credit card requirement. After that allowance, scraping tools are priced at $0.005 per request and screenshots at $0.01 per request. Payment uses HTTP 402 and requires an X-Payment header containing payment proof.

The README does not document authentication credentials or client-specific behavior beyond the hosted MCP configuration and Cline example. Screenshot responses contain base64 image data, and full-page captures may require more browser resources than viewport captures.