AI First Scraper MCP

What yubinkim444/ai-first-scraper-mcp MCP server does

The yubinkim444/ai-first-scraper-mcp MCP server connects an MCP-compatible agent to web retrieval and content cleanup services. It returns pages as Markdown rather than exposing the agent to full HTML documents containing advertisements and other page noise. The server supports individual pages, batches of pages, and search-driven retrieval.

The project delegates processing to the ai-first-scraper and ai-first-search APIs. Public service instances are used by default, while deployments can point the MCP server at their own backend endpoints.

How it works

An MCP client starts the server locally over standard input and output. When the agent invokes a tool, the server sends the request to the configured scraper or search backend and returns Markdown content through MCP.

For direct retrieval, a URL can refer to an HTML page or a PDF. Batch retrieval accepts as many as 25 URLs and runs those requests in parallel. Search combines web discovery with page conversion, returning the top-k results as Markdown so the agent can work with page content directly.

This design fits research, summarization, and extraction workflows where the agent needs current web pages but should not parse raw HTML itself. It does not describe a browser-control workflow or a local crawler; its documented interface is the three MCP tools backed by the companion APIs.

Setup and configuration

The yubinkim444/ai-first-scraper-mcp MCP server is available from PyPI and can be launched with uvx without a separate package installation:

{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "uvx",
      "args": ["ai-first-scraper-mcp"]
    }
  }
}

A conventional pip installation is also documented. After installation, the ai-first-scraper-mcp executable starts the server. The README provides configuration locations for Claude Desktop on macOS and Windows, Cursor, and Cline in VS Code; clients must be restarted after adding the entry.

Three optional environment variables select alternate backends or change the request timeout:

• SCRAPER_URL sets the scraper service URL.
• SEARCH_URL sets the search service URL.
• AFS_TIMEOUT sets the timeout value.

If these values are omitted, the server uses the listed public Render-hosted scraper and search instances.

Tools and capabilities

The yubinkim444/ai-first-scraper-mcp MCP server provides:

• fetch_page: retrieves one HTML or PDF URL and converts it to clean Markdown.
• fetch_pages_batch: retrieves up to 25 URLs concurrently and returns Markdown for each.
• search_web: performs a web search and returns the top-k result pages after Markdown conversion.

These outputs are intended for direct agent reasoning, including tasks such as finding articles, summarizing multiple sources, or extracting information from selected pages.

Limitations and notes

The project depends on the availability and behavior of its scraper and search backends. The README documents public default endpoints and optional self-hosting, but it does not specify uptime guarantees, search-provider coverage, authentication options, rate limits, or Markdown conversion rules.

Batch requests are capped at 25 URLs. Search returns a configurable top-k result set, but the provided material does not define a default value or expose additional search parameters. Python 3.10 or newer is indicated by the project metadata. The server communicates over MCP stdio, so it requires an MCP client configuration rather than operating as a standalone hosted endpoint.