Rust MCP server for denoised web search β fetch, clean, and rerank web content for AI agents.
Copy the AI prompt to install this server into Claude Code, Cursor, or another agent β or use 1-click editor setup below.
π‘ Paste into ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows)
WebShift is a Rust library and MCP server that shifts noisy web pages into clean, right-sized text for LLM consumption.
Raw HTML is mostly junk: scripts, ads, navigation menus, cookie banners, tracking pixels. Feeding it directly to an LLM floods the context window with tens of thousands of useless tokens and leaves no room for reasoning. WebShift strips all that noise, sterilizes the text, and enforces strict size budgets so the model receives only the content that matters.
Depending on the features you enable, WebShift can be four things:
| Use case | Crate | Feature flags | What it does |
|---|---|---|---|
| HTML denoiser | webshift | default-features = false | clean() β pure Rust HTML-to-text pipeline. Strips noise elements, sterilizes Unicode/BiDi, collapses whitespace. Zero network, zero config. Drop into any Rust project that processes web content for LLMs. |
| HTML text rewriter | webshift | features = ["text-map"] | extract_text_nodes() + replace_text_nodes() β extract individual text nodes from HTML, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure intact. Tags, attributes, and links are never touched. |
| Web content client | webshift | default or features = ["llm"] | fetch() + query() β streaming HTTP fetcher with size caps, 8 search backends, BM25 reranking, optional LLM query expansion and summarization. Full pipeline from search query to structured results. |
| MCP server | webshift-mcp | all features | Native binary (mcp-webshift) that exposes webshift_query, webshift_fetch, and webshift_onboarding over MCP stdio. Single static binary, zero runtime dependencies. |
text-map does preserve the DOM structure for text rewriting use cases.)For a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see Under the Hood. For the full configuration reference (TOML, env vars, CLI args) see Configuration. For ready-to-use examples see Use Cases.
The binary is called mcp-webshift.
The easiest option is SearXNG β free, self-hosted, no API key:
No Docker? Use a cloud backend β see Search backends.
That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.
For client-specific setup see docs/integrations/.
| Tool | Description |
|---|---|
webshift_query | Full search pipeline: search + fetch + clean + rerank + (optional) summarize |
webshift_fetch | Single page fetch and clean |
webshift_onboarding | Returns a JSON guide for the agent (budgets, backends, tips) |
webshift_query parameters| Parameter | Type | Default | Description |
|---|---|---|---|
queries | string or list | required | Search query or list of queries |
num_results_per_query | integer | 5 | Results per query |
lang | string | none | Language filter (e.g. "en") |
backend | string | config default | Override search backend |
Resolution order (highest priority first):
--default-backend, --brave-api-key, etc.WEBSHIFT_* prefixwebshift.toml (current dir, then ~/webshift.toml)For every setting with all three config methods (TOML, env vars, CLI args)
and plain-language descriptions, see the full Configuration Reference.
Ready-to-use config examples are in Use Cases and examples/.
| Backend | Auth | Notes |
|---|---|---|
| SearXNG | none | Self-hosted, free. Default: http://localhost:8080 |
| Brave | API key | Free tier. brave.com/search/api |
| Tavily | API key | AI-oriented. tavily.com |
| Exa | API key | Neural search. exa.ai |
| SerpAPI | API key | Multi-engine proxy (Google, Bing, DDG...). serpapi.com |
| API key + CX | Custom Search. Free: 100 req/day. programmablesearchengine.google.com | |
| Bing | API key | Web Search API. Free: 1,000 req/month. Microsoft Azure |
| HTTP | configurable | Generic REST backend β no code required, TOML-only config |
All opt-in β disabled by default, no data leaves your machine unless enabled.
| Feature | What it does |
|---|---|
| Query expansion | Single query -> N complementary search variants |
| Summarization | Markdown report with inline [1] [2] citations |
| LLM reranking | Tier-2 reranking on top of deterministic BM25 |
Cross-language normalization (bonus): when BM25 reranking surfaces pages in foreign languages (e.g. Chinese, Japanese, Arabic), the LLM summarizer still produces the final report in the prompt language. The agent receives clean, readable output regardless of the language mix in the source pages.
Works with any OpenAI-compatible API (OpenAI, Ollama, vLLM, LM Studio, etc.):
Always active β the core value proposition:
| Protection | Description |
|---|---|
max_download_mb | Streaming cap β never buffers full response |
max_result_length | Hard cap on characters per cleaned page |
max_query_budget | Total character budget across all sources |
max_total_results | Hard cap on results per call |
| Binary filter | .pdf, .zip, .exe, etc. filtered before any network request |
| Unicode sterilization | BiDi control chars, zero-width chars removed |
Extract text nodes, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure, attributes, and links intact.
Requires features = ["text-map"]. See Use Cases #11 for a full translation example.
| Feature | Default | Enables |
|---|---|---|
backends | on | All search backends + query pipeline |
llm | off | LLM client, expander, summarizer, LLM reranking |
text-map | off | extract_text_nodes() + replace_text_nodes() β DOM round-trip for content rewriting |
| Platform | Guide |
|---|---|
| Claude Desktop, Claude Code, Cursor, Windsurf, VS Code | IDE Integration |
| Zed β native extension with auto-download and Configure Server modal | Zed Extension |
| Gemini CLI, Claude CLI, custom agents | Agent Integration |
WebShift is in beta. Core functionality is stable and the server is used daily, but the API surface may still change before 1.0.
Feedback is very welcome. If something doesn't work as expected, behaves oddly, or you have a use case that isn't covered:
Bug reports, configuration questions, and feature requests all help shape the roadmap.
Contributions are welcome! Please see CONTRIBUTING.md for detailed guidelines on:
MIT License β see LICENSE for details.
Need help? Check the documentation or open an issue on GitHub.
Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.
[](https://allmcps.com/mcp/webshift)<a href="https://allmcps.com/mcp/webshift"><img src="https://allmcps.com/api/badge/webshift?style=directory" alt="Webshift on AllMCPs" /></a>