# bch1212/agentfetch-mcp [Health: Active] **Category:** 📂 Browser Automation **Repository:** https://github.com/bch1212/agentfetch-mcp **GitHub Stars:** 3 **Views:** 4 **Installs:** 0 **Upvotes:** 0 **Directory Page:** https://allmcps.com/mcp/bch1212-agentfetch-mcp ## Description Token-budgeted web fetch for AI agents. Auto-routes between Trafilatura, Jina Reader, FireCrawl, and pypdf based on URL pattern. estimatetokens before fetchurl, 6h Redis cache, server-side maxtokens truncation. Open source MCP server (MIT) plus hosted REST API at agentfetch.dev — 500 free fetches/mo, no card. ## Tools Capabilities this server exposes over MCP: - **fetch_url** — Fetch any URL and return clean, LLM-ready Markdown with token count, metadata, and 6h caching. WHEN TO USE: - You have a specific URL whose content you need. - You want to cap response size to stay inside your context window. - You want repeat fetches to be cheap (cache hits ≈ $0.0001). - The URL might be JS-rendered, a PDF, or behind a paywall — this tool auto-routes to the right fetcher (Trafilatura → Jina → FireCrawl → PDF). WHEN NOT TO USE: - You don't know which URL to fetch — use search_and_fetch instead. - You have many URLs to fetch — use fetch_multiple instead. Args: url: The URL to fetch. max_tokens: Hard cap on response size. Default unlimited. Pass this if you're tight on context budget — cheaper than over-fetching. format: "markdown" (default — recommended), "text", or "json". use_cache: True returns a cached copy if one exists (≤6h old). Pass False only when freshness matters (live news, prices). Returns: { "url": str, "success": bool, "markdown": str, "metadata": {title, author, published_date, domain, word_count, token_count, reading_time_seconds, content_type, language}, "cache": {hit, cached_at, expires_at}, "fetch_info": {fetcher_used, fetch_time_ms, cost_credits}, "error": str | None } - **estimate_tokens** — Estimate token count of a URL's content WITHOUT fetching the body. WHEN TO USE: - You're considering fetching a URL but unsure if it fits your remaining context window. This call is ~10x cheaper than a full fetch. - You want to triage a list of candidate URLs before deciding which to actually retrieve. IMPORTANT: Many servers omit Content-Length on dynamic / chunked responses. When that happens, this tool returns confident=false and estimated_tokens=null. In that case, call fetch_url with a max_tokens cap instead of trusting the estimate. Args: url: The URL to estimate. Returns: { "url": str, "success": bool, "estimated_tokens": int | null, "byte_size": int | null, "content_type": str, "confident": bool, "note": str } - **fetch_multiple** — Fetch up to 20 URLs concurrently. Each result is the same shape as fetch_url. WHEN TO USE: - You have a list of URLs (search results, links from a doc, sitemap) and want them retrieved in parallel rather than one at a time. Args: urls: 1–20 URLs. Larger batches: split into multiple calls. max_tokens_each: Per-result cap. Apply this to keep total response inside your context budget — total ≈ len(urls) * max_tokens_each. use_cache: True for cache-aware fetching (default). Returns: {"count": int, "results": [, ...]} - **search_and_fetch** — Web search + fetch top results in one call. WHEN TO USE: - You have a research question, not specific URLs. E.g. "what's the latest on X", "find docs for Y library", "recent news about Z". - You'd otherwise have to call a search tool, parse results, then call fetch — this collapses that into one round-trip. Args: query: Search query (2–500 chars). num_results: Top N to fetch (1–10, default 3). max_tokens_each: Per-result cap (default 2000). Returns: {"query": str, "count": int, "results": [, ...]} ## Claude Desktop Quick Installation Install path detected from listing signals. Uses `uvx` (confidence: high): ```json "mcpServers": { "agentfetch-mcp": { "command": "uvx", "args": ["agentfetch-mcp"], "env": { "JINA_API_KEY": "", "FIRECRAWL_API_KEY": "", "REDIS_URL": "", "CACHE_TTL_SECONDS": "" } } } ``` **Requires environment variables:** `JINA_API_KEY`, `FIRECRAWL_API_KEY`, `REDIS_URL`, `CACHE_TTL_SECONDS` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for). ## Documentation ## What bch1212/agentfetch-mcp MCP server does The bch1212/agentfetch-mcp MCP server gives an MCP-compatible agent a small set of web retrieval tools. It can fetch a known URL, estimate its likely token size without downloading the body, retrieve several URLs in parallel, or search the web and fetch the leading results in one operation. A normal fetch returns Markdown, success status, the requested URL, metadata, cache details, and information about the fetcher used. Metadata can include the title, author, publication date, domain, word count, token count, reading time, content type, and language. The response can also include an error when retrieval fails. ## How it works URL patterns determine which retrieval method is attempted. Trafilatura handles many conventional pages locally, Jina Reader handles other HTML pages, FireCrawl is used for JavaScript-heavy sites such as Twitter/X, LinkedIn, and Notion, and pypdf extracts PDF content. The routing order described by the project is Trafilatura, Jina, FireCrawl, then PDF handling where appropriate. Before making an expensive or large retrieval, `estimate_tokens` checks the response size using available information such as `Content-Length`. It can return no estimate and mark the result as uncertain when a site uses dynamic or chunked responses. In that case, use `fetch_url` with `max_tokens` instead. `fetch_url` supports Markdown, plain text, and JSON output formats. Its `max_tokens` value caps the returned content, and `use_cache` controls whether a recent cached response may be used. `fetch_multiple` accepts between 1 and 20 URLs and applies a per-result token limit. `search_and_fetch` accepts a query, fetches 1 to 10 top results by default, and applies a per-result cap. ## Setup and configuration Install the Python package with: ```bash pip install agentfetch-mcp ``` Run the stdio server with: ```bash python -m agentfetch.mcp.server ``` The project documents Python 3.11 or newer. For Claude Desktop, add a server entry whose command is `python` and whose arguments are `-m agentfetch.mcp.server`. Claude Code can also register the same command through its MCP configuration. Configuration is controlled through environment variables: - `JINA_API_KEY`: Recommended for Jina Reader. Without it, only Trafilatura is available. - `FIRECRAWL_API_KEY`: Optional; used for JavaScript-heavy domains. - `REDIS_URL`: Optional Redis connection URL. Without Redis, results are not cached. - `CACHE_TTL_SECONDS`: Optional cache lifetime, defaulting to 21,600 seconds. ## Tools and capabilities - `fetch_url`: Retrieve one URL with a token cap, selected format, metadata, and cache information. - `estimate_tokens`: Estimate size before downloading the body when the response provides enough information. - `fetch_multiple`: Retrieve up to 20 URLs concurrently. - `search_and_fetch`: Search for a query and fetch the top results in one call. - Six-hour Redis caching by default when Redis is configured. - Automatic handling for standard HTML, JavaScript-heavy pages, and PDFs. ## Limitations and notes The bch1212/agentfetch-mcp MCP server does not guarantee a token estimate for dynamic or chunked responses. A missing estimate should be treated as uncertainty, not as evidence that the page is small. JavaScript-heavy retrieval depends on FireCrawl configuration, while Jina-backed retrieval depends on `JINA_API_KEY`; the local Trafilatura path remains available without those keys for supported pages. Redis is optional, so deployments without `REDIS_URL` run without caching. The hosted AgentFetch service is separate from this open-source repository and has its own pricing and account model. The repository is licensed under MIT. _Full upstream README: https://allmcps.com/mcp/bch1212-agentfetch-mcp/readme_