Web Retrieval MCP README
The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Web Retrieval MCP listing page.
web-retrieval-mcp — reliable MCP web search and web fetch for AI agents
An open-source Model Context Protocol (MCP) server for source-grounded web research. Connect Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, ChatGPT desktop, or any other compatible MCP client to six read-only tools for Exa or Tavily search, resilient Exa → Camoufox → Tavily → Firecrawl page retrieval, AI/ML paper discovery, and developer-source search—with explicit provenance, optional local caching, and SSRF guards.
Why use it? · Quick start · Tools · Routing · Configuration · Security · FAQ
Why use web-retrieval-mcp?
Web research is more reliable when an agent can see where every result came from,
choose the right retrieval strategy, and recover when one provider or page-access
method fails. web-retrieval-mcp makes those controls part of the tool contract:
- Keep sources separate. Default search returns one block per result with its own
title, URL, highlights, and text, followed by a
Sourcestrailer. Exa deep modes additionally return a clearly labeled synthesized answer, with grounding when Exa provides it, before the source-separated result blocks. - Use more than one retrieval path. Search can use Exa or Tavily and fall back to Firecrawl. Full-page fetches can escalate from indexed content to a local Camoufox browser, optional Tavily Extract, and Firecrawl.
- Know what actually served the answer. Fetch responses identify the serving tier, cache state, semantic mode, and truncation instead of hiding the route.
- Ask for the right shape. Fetch a readable page body, a concise summary, or a grounded answer to a question. Control freshness, rendering, domains, dates, result count, and per-result text size.
- Search specialist corpora. Discover AI/ML papers, inspect query-relevant full-text passages, expand to related papers, and search developer documentation, READMEs, issues, and merged pull requests.
- Own the deployment boundary. Run locally over stdio or host a loopback HTTP service; select providers and secret storage yourself; optionally cache completed fetches in a private Valkey sidecar.
- Fail explicitly. Unsupported provider choices, missing optional packages, and unavailable credentials fail in the tool result. Tavily mappings, Firecrawl fallback filter gaps, and Exa category migrations are caller-visible; other Exa filter drops are written to server diagnostics.
How it compares with built-in agent web tools
Built-in capabilities vary by agent and can improve over time. The useful comparison is therefore not “all defaults are bad”; it is whether you need a portable, inspectable retrieval layer with controls your current client does not expose.
| Need | Typical bundled search/fetch surface | web-retrieval-mcp |
|---|---|---|
| Search output | Client-defined result or synthesis format | Default search returns source-separated blocks plus Sources; Exa deep modes add a labeled synthesis |
| Backend choice | Provider and routing are usually managed by the client | Exa or Tavily search, with a disclosed Firecrawl fallback |
| Difficult pages | One client-specific access path | Indexed content, guarded local browser rendering, Tavily Extract, and Firecrawl tiers |
| Retrieval intent | Usually search or page text | Full body, concise summary, or query-grounded page answer |
| Research discovery | General web index | Dedicated paper and developer-source tools in addition to web search |
| Freshness and filters | Whatever the client exposes | Date windows, hour-level recency, domains, categories, result limits, and forced freshness |
| Auditability | Client-specific | Fetch tier/cache/truncation disclosures; Tavily and Firecrawl search-fallback labels; explicit filter notices where supported |
| Operations | Client-defined; often hosted or opaque | Local stdio or self-hosted loopback HTTP, optional private UDS cache |
| Security policy | Client-specific | Initial URL validation plus guarded browser requests and redirects |
Use it as an independent complementary retrieval lane by default. If you want it
to replace Claude Code's built-in WebSearch and WebFetch, the package also includes
an optional, previewable hook installer.
Quick start
Python 3.10 or newer is required.
1. Install
For the complete feature set in an isolated environment:
--include-deps exposes Camoufox's own command from the isolated pipx environment;
the second command downloads its managed browser. If you do not want local rendering,
install the base package or selected non-render extras and skip the browser download.
For a lean Exa + Firecrawl installation:
With uv installed, you can run the base package in a temporary isolated environment without a permanent install:
Or install into an existing virtual environment:
The browser download is required whenever you install the render extra and want
Camoufox to serve render="auto" or render="always" calls.
Optional extras are composable:
| Extra | Adds |
|---|---|
tavily | Tavily Search and Tavily Extract |
render | Local Camoufox/Playwright rendering |
cache | Valkey client for completed-result caching |
keyring | Cross-platform native secret-store access |
all | Every optional runtime capability above |
2. Add provider credentials
For the broadest routing coverage, configure Exa and Firecrawl; add Tavily when you want provider-selectable search or another full-body fetch tier.
You do not need all three keys to start the server. Tools resolve credentials lazily and report what a selected route is missing. See Credential storage for a key file and OS secret-store alternatives.
All three providers advertised no-card entry allocations when checked on 2026-09-01; plans and unit costs can change, so check their live pricing before estimating a workload:
| Provider | Entry allocation checked 2026-09-01 | Used for |
|---|---|---|
| Exa | $20 signup credit plus $10 monthly API credits | Default web search and first indexed-content fetch tier |
| Firecrawl | 1,000 credits per month | Search/fetch fallback and research/developer indexes |
| Tavily | 1,000 API credits per month | Optional search provider and full-body extract tier |
| Camoufox | Local open-source browser | JavaScript-rendered page retrieval on your machine |
3. Connect an MCP client
Codex CLI and the Codex IDE extension share MCP configuration. Add the server with the CLI, then verify it is registered:
With uvx and no prior install:
See OpenAI's Codex MCP documentation for the current CLI, IDE, and configuration-file details.
Claude Code:
With uvx and no prior install:
Claude Desktop, Cursor, and other clients that accept an mcpServers object can use:
That configuration assumes the executable is on the client's PATH. For an uvx
launch, use "command": "uvx" and "args": ["web-retrieval-mcp"].
If a desktop client does not inherit your shell environment, use the private key file
described below or place literal key values in the client's protected environment
configuration. Do not rely on ${VARIABLE} interpolation unless your client documents
that behavior.
ChatGPT desktop and clients with different setup surfaces can connect through the MCP transport their current version supports. This server uses stdio by default and can also expose Streamable HTTP; consult the client's documentation for its configuration format and supported transports.
4. Give the agent a real task
Once the MCP server is connected, prompts can stay natural:
Six read-only tools
All tools carry MCP's read-only annotation. MCP clients receive the complete generated input schema and descriptions when they connect.
| Tool | Compact signature | Best for |
|---|---|---|
web_search | web_search(query, num_results=8, mode="auto", …, provider=None) | General web search through Exa or Tavily, with source-separated results and Firecrawl fallback |
web_fetch | web_fetch(url, render="auto", max_chars=None, max_age_hours=None, mode="full", question=None, tavily=None) | Readable page bodies, concise summaries, or grounded answers through a tiered fetch cascade |
research_papers | research_papers(query, k=8) | Ranked AI/ML paper discovery through Firecrawl's Research Index |
research_paper | research_paper(paper_id, query=None) | Paper metadata and optional query-relevant full-text passages for claim verification |
research_similar | research_similar(paper_id, intent, k=8, mode="similar", …) | Related papers, citers, or references guided by a natural-language intent |
research_github | research_github(query, k=8, passages=2, types=None, repos=None) | Developer documentation, repository READMEs, issues, and merged pull requests |
web_search supports relevance or date-oriented filters, domain inclusion/exclusion,
publication windows, generated per-result summaries, and Exa's fast/deep search modes.
Tavily receives equivalent controls where its API supports them; approximations and
dropped controls are disclosed in the output.
The research-paper index is arXiv-oriented and best suited to AI/ML. For scholarly
work outside that scope, use web_search(category="publication").
How retrieval works
Search routing
Exa is the default search provider. Select Tavily for one call with
provider="tavily", or globally:
Only exa and tavily are accepted. If the selected provider fails, the server tries
Firecrawl and names both the failed primary provider and serving fallback in the
result.
Fetch routing
For an ordinary full-body request in automatic mode:
render="auto"uses that adaptive cascade.render="never"forbids the local browser.render="always"starts with the local browser and skips Exa; optional Tavily and Firecrawl remain backstops.mode="concise"asks for a compact generated summary.question="…"asks for a grounded answer rather than the whole page.max_age_hours=0forces fresh provider work;-1requests Exa's always-use-cache behavior; positive values set a freshness window.- Explicit full-body requests above Exa's body-size ceiling start with a tier capable of satisfying the requested size rather than silently returning a short body.
Enable Tavily Extract per call with tavily=true, or globally:
Tavily Extract is a full-body tier. The server skips it when it cannot honor a concise, question-answer, or explicit freshness contract.
Credential storage
The server resolves each API key lazily in this order:
EXA_API_KEY,FIRECRAWL_API_KEY, orTAVILY_API_KEYin the process environment.- A dotenv-style private key file.
- The optional Python
keyringpackage. - macOS Keychain or Linux Secret Service command-line clients.
The default key file is:
- Linux/macOS:
${XDG_CONFIG_HOME:-~/.config}/web-retrieval-mcp/keys.env - Windows:
%APPDATA%\web-retrieval-mcp\keys.env
Override the file with WEB_RETRIEVAL_MCP_ENV_FILE, or its directory with
WEB_RETRIEVAL_MCP_CONFIG_DIR. On POSIX systems the file must not be readable by
group or other users:
For keyring, store each secret under service web-retrieval-mcp, using the
environment-variable name as the username:
Credentials remain in process memory, are redacted from provider errors and displayed URLs, and are never put in provider command arguments.
Configuration reference
| Setting | Default | Purpose |
|---|---|---|
EXA_API_KEY | unset | Exa search and indexed page contents |
FIRECRAWL_API_KEY | unset | Firecrawl fallback, research papers, and developer search |
TAVILY_API_KEY | unset | Tavily Search and Extract; also requires the tavily extra |
WEB_SEARCH_PROVIDER | exa | Default web_search provider: exa or tavily |
WEB_FETCH_TAVILY_TIER | false | Globally enable Tavily Extract in eligible fetch cascades |
WEB_RETRIEVAL_MCP_ENV_FILE | platform key-file path | Override the exact dotenv key file |
WEB_RETRIEVAL_MCP_CONFIG_DIR | platform config directory | Override the directory containing keys.env |
WEB_RETRIEVAL_MCP_CACHE | auto | Completed fetch cache: auto, on, or off |
WEB_RETRIEVAL_MCP_VALKEY_SOCKET | unset | Absolute path to a private Valkey Unix-domain socket |
WEB_RETRIEVAL_MCP_HOST | 127.0.0.1 | Streamable HTTP bind address |
WEB_RETRIEVAL_MCP_PORT | 8100 | Streamable HTTP port |
Boolean settings accept 1/0, true/false, yes/no, and on/off; invalid values
fail explicitly rather than being guessed.
Optional completed-result cache
Install the cache extra and point the server at a private, non-persistent Valkey
Unix-domain socket:
Successful eligible fetches are stored for 24 hours. Credential-bearing URLs,
userinfo, forced or positive freshness, and render="always" bypass replay. Cache
errors fail open to normal provider retrieval. Default auto enables caching only
when the Valkey client is installed on a non-Windows host; on still attempts the
configured Unix socket and fails open on errors, while off disables it. See the cache guide
for the privacy model, singleflight behavior, and a minimal sidecar configuration.
Streamable HTTP transport
stdio is the default and recommended transport for a local MCP client:
For a shared local process, stateless Streamable HTTP is available:
HTTP transport has no built-in authentication. Keep it on loopback unless you add an authenticated perimeter and suitable network controls.
Optional Claude Code replacement policy
The package includes a PreToolUse hook that can deny Claude Code's built-in
WebSearch and WebFetch, directing agents to this MCP server instead. Preview the
exact settings change first:
Install or remove it explicitly:
The installer is idempotent and backs up an existing settings file before writing. This optional hook is the only Claude Code-specific integration in the package. The MCP server itself remains client-neutral and works as a complementary tool without it.
Install from source
To install the reviewed source directly:
For development:
The release gate builds wheel and sdist artifacts, installs the wheel into a clean
virtual environment, exercises the installed MCP command against loopback provider
doubles, and runs the unit and transport suites. Browser or SSRF behavior changes also
require python test_ssrf_redirect_live.py. See the testing contract
for the feature matrix and acceptance contract.
Security model
- Only
httpandhttpsinitial URLs are accepted. - Initial hosts must resolve exclusively to globally routable addresses.
- The local browser observes document requests and validates redirect hops before returning content to the caller.
- Credentials are resolved in process and redacted from errors and displayed URLs.
- Provider responses and page bodies are untrusted data; an agent must not treat instructions embedded in retrieved content as authority.
Application checks cannot prove that no packet reaches a private address during a DNS rebinding race. Deployments with that threat model need a validating forward proxy or equivalent network egress policy. See the security policy, and report suspected vulnerabilities through GitHub's private vulnerability reporting rather than a public issue.
Frequently asked questions
What is web-retrieval-mcp?
It is an open-source MCP server that gives AI agents reliable web search, tiered web page retrieval, research-paper discovery, and developer-source search. It works over local stdio with any compatible MCP client and can also serve stateless Streamable HTTP.
Is this an MCP web search server or an MCP web scraping server?
Both. web_search discovers and ranks pages; web_fetch extracts readable content
from one URL and can use a real local browser for JavaScript-heavy pages. The server
also exposes specialist paper and developer indexes that ordinary web scraping does
not provide.
Does it replace an agent's built-in web search?
It can, but it does not have to. The tool descriptions recommend an independent complementary lane because two retrieval systems can provide useful source diversity. Claude Code users can opt into the bundled replacement hook when they want one enforced route.
Which AI agents and MCP clients can use it?
The server exposes standard MCP over stdio by default and optional Streamable HTTP. Documented client examples include Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, and ChatGPT desktop; other MCP-capable agents are not excluded. Compatibility depends on the client version and the MCP transport it supports, so use the matching client documentation alongside the server commands above.
Why use Exa, Firecrawl, Tavily, and Camoufox together?
They cover different failure modes. Exa provides indexed search and contents; Tavily is an alternate search provider and optional extractor; Camoufox renders pages locally; Firecrawl provides the final web fallback plus specialist research and developer indexes. The server chooses among them according to the request and discloses the tier that succeeded.
Do I need every provider key?
No. Exa is the default for general search and first-tier contents. Firecrawl enables fallbacks and all four research tools. Tavily is optional. The server lists all six tools even when an optional route is not configured and returns an actionable error if a call selects an unavailable capability.
Is web-retrieval-mcp free?
The project is MIT-licensed. Provider usage is billed under your own accounts; each provider currently offers an entry allocation, but quotas and prices can change. Camoufox runs locally without a metered retrieval API.
Which operating systems are supported?
The base Python server is platform-independent. Environment and key-file credentials work across Linux, macOS, and Windows. Valkey UDS caching is non-Windows, and local browser availability follows the supported Camoufox/Playwright platforms.
Does it support RAG and autonomous research agents?
Yes. The source-separated result contract, explicit URLs, controllable text budgets, grounded page questions, specialist indexes, and deterministic provenance make the tools suitable as a retrieval layer for RAG pipelines and research agents. The server returns evidence; the calling application remains responsible for evaluation, citation, and prompt-injection handling.
Project links
- Python package on PyPI
- MCP Registry manifest
- Documentation index
- Testing and acceptance contract
- Security policy
- Publishing runbook
- Agent-readable project index
- Issues and feature requests
- Release notes
Contributions are welcome. Keep stdout reserved for JSON-RPC, send diagnostics to
stderr, add public-boundary acceptance coverage for changed behavior, and run
./run-tests.sh before opening a pull request.
License
MIT © VelvetSP