# aparajithn/agent-scraper-mcp [Health: Active]

**Category:** 📂 Browser Automation  
**Repository:** https://github.com/aparajithn/agent-scraper-mcp  
**GitHub Stars:** 8  
**Views:** 4  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/aparajithn-agent-scraper-mcp

## Description
Web scraping MCP server for AI agents. 6 tools: clean content extraction, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction, metadata extraction (OG/Twitter cards), and Google search. Free tier with x402 micropayments.

## Tools
Capabilities this server exposes over MCP:

- **title** — Page title
- **description** — Meta description
- **canonical** — Canonical URL
- **favicon** — Favicon URL
- **og** — Open Graph tags
- **twitter** — Twitter Card tags

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `uvx` (confidence: high):

```json
"mcpServers": {
  "agent-scraper-mcp": {
    "command": "uvx",
    "args": ["x402-fetch"],
    "env": {
      "PUBLIC_HOST": "",
      "X402_WALLET_ADDRESS": ""
    }
  }
}
```

**Requires environment variables:** `PUBLIC_HOST`, `X402_WALLET_ADDRESS` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation

## What aparajithn/agent-scraper-mcp MCP server does

The aparajithn/agent-scraper-mcp MCP server provides six tools for retrieving and inspecting public web pages. `scrape_url` converts a page into readable text, Markdown, or HTML. `scrape_structured` accepts a map of field names to CSS selectors, which is useful for extracting values such as product names, prices, authors, or review text.

`screenshot_url` captures either the visible viewport or the complete scrollable page and returns a base64-encoded PNG. `extract_links` returns link text and URLs, with an optional regular-expression filter. `extract_meta` collects the page title, description, canonical URL, favicon, Open Graph tags, and Twitter Card tags. `search_google` accepts a query and returns result titles, URLs, and snippets.

## How it works

The hosted MCP endpoint is available at `https://agent-scraper-mcp.onrender.com/mcp`. An MCP client can connect to that URL and call the tools using the protocol. The project also exposes REST routes under the same host, including `/api/v1/scrape_url`, `/api/v1/scrape_structured`, `/api/v1/screenshot_url`, `/api/v1/extract_links`, `/api/v1/extract_meta`, and `/api/v1/search_google`.

Content extraction uses readability-style processing, while HTML parsing and metadata handling are provided by the project’s Python web stack. Screenshots use Playwright and Chromium. Structured extraction is controlled by selectors supplied with each request, so the agent or calling application must know the relevant page structure.

## Setup and configuration

For MCP clients, add the hosted URL to the client’s MCP settings. The README gives a Cline-compatible configuration using the `url` field. Local development requires Python 3.11, installation of the project with its development dependencies, and Chromium installation through Playwright. The documented local server command is `uvicorn src.main:app --reload --port 8080`.

Docker is also documented: build the repository image and run it with port 8080 exposed. The deployment configuration lists `PUBLIC_HOST` and `X402_WALLET_ADDRESS`; the latter identifies the wallet used for HTTP 402 payments.

## Tools and capabilities

The aparajithn/agent-scraper-mcp MCP server supports:

- Readable page extraction in text, Markdown, or HTML
- CSS-selector-based structured extraction
- Viewport and full-page PNG screenshots
- Link collection with optional regex filtering
- Title, description, canonical, favicon, Open Graph, and Twitter metadata
- Google result retrieval with configurable result count

## Limitations and notes

The hosted service provides 50 requests per IP per day in its free tier, with all tools included and no credit card requirement. After that allowance, scraping tools are priced at $0.005 per request and screenshots at $0.01 per request. Payment uses HTTP 402 and requires an `X-Payment` header containing payment proof.

The README does not document authentication credentials or client-specific behavior beyond the hosted MCP configuration and Cline example. Screenshot responses contain base64 image data, and full-page captures may require more browser resources than viewport captures.

_Full upstream README: https://allmcps.com/mcp/aparajithn-agent-scraper-mcp/readme_