Skip to main content
AllMCPs
BrowseBestCategoriesStackCompareToolsGuidesBlog Log in Submit MCP

Stay in the loop

Get new MCP servers and top picks in your inbox.

AllMCPs

The open directory for discovering and installing Model Context Protocol servers.

Explore

  • Browse servers
  • Best MCP servers
  • Categories
  • MCP clients
  • Agent prompts
  • Stack Builder
  • Compare servers
  • Tags index
  • Submit a server
  • Pricing

Learn

  • Guides hub
  • What is MCP?
  • Install guide
  • Troubleshooting
  • Security
  • Blog
  • Blog RSS

Tools

  • All tools
  • Config generator
  • Config validator
  • MCP playground
  • OpenAPI β†’ MCP
  • Badge generator

For agents

  • API docs
  • Trust & traffic
  • llms.txt β†— (opens in a new tab)
  • Catalog JSON β†— (opens in a new tab)
  • Remote MCP β†— (opens in a new tab)

Company

  • About
  • Contact
  • X (@AllMCPs) β†— (opens in a new tab)
  • GitHub β†— (opens in a new tab)
  • Terms
  • Privacy
AllMCPs VerifiedAllMCPs VerifiedFeatured on Nick LaunchesFeatured on Nick LaunchesLaunch Llama NewsletterLaunch Llama NewsletterVerified DR - allmcps.comVerified DR - allmcps.comFeatured on SaaSGrowFeatured on SaaSGrowFeatured on Twelve ToolsFeatured on Twelve ToolsFeatured on Saaspa.geFeatured on Saaspa.geFeatured on Findly.toolsFeatured on Findly.toolsFeatured on Startup FameFeatured on Startup FameFeatured on LaunchKiwiFeatured on LaunchKiwiFeatured on ScrollLaunchFeatured on ScrollLaunchFeatured on DailyPingsFeatured on DailyPingsFazier badgeFazier badgeFeatured on NewTool.siteFeatured on NewTool.siteFeatured on saasfame.comFeatured on saasfame.comDR Checker - Domain RatingDR Checker - Domain RatingListed on Turbo0Listed on Turbo0Launched on LaunchBoard - Product Launch PlatformLaunched on LaunchBoard - Product Launch PlatformList on SimilarlabsList on Similarlabshttps://codetrendy.comhttps://codetrendy.comListed on DevTool.ioFeatured on BuildlistFeatured on BuildlistAllMCPs VerifiedAllMCPs VerifiedFeatured on Nick LaunchesFeatured on Nick LaunchesLaunch Llama NewsletterLaunch Llama NewsletterVerified DR - allmcps.comVerified DR - allmcps.comFeatured on SaaSGrowFeatured on SaaSGrowFeatured on Twelve ToolsFeatured on Twelve ToolsFeatured on Saaspa.geFeatured on Saaspa.geFeatured on Findly.toolsFeatured on Findly.toolsFeatured on Startup FameFeatured on Startup FameFeatured on LaunchKiwiFeatured on LaunchKiwiFeatured on ScrollLaunchFeatured on ScrollLaunchFeatured on DailyPingsFeatured on DailyPingsFazier badgeFazier badgeFeatured on NewTool.siteFeatured on NewTool.siteFeatured on saasfame.comFeatured on saasfame.comDR Checker - Domain RatingDR Checker - Domain RatingListed on Turbo0Listed on Turbo0Launched on LaunchBoard - Product Launch PlatformLaunched on LaunchBoard - Product Launch PlatformList on SimilarlabsList on Similarlabshttps://codetrendy.comhttps://codetrendy.comListed on DevTool.ioFeatured on BuildlistFeatured on Buildlist
Β© 2026 Jackalope Digital LLC. All rights reserved.
  1. Home
  2. πŸ’» Developer Tools
  3. Webshift
W
Health: Not checked yetWe have not completed a health check for this listing yet.Last checked 8/11/2026, 12:26:29 AM

Webshift

Enrichment pendingWe haven’t run our AI enrichment pass on this listing yet, so the overview, use cases, and FAQ below may be sparse or missing. We work through the catalog over time β€” check back soon.
View Repository

Rust MCP server for denoised web search β€” fetch, clean, and rerank web content for AI agents.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent β€” or use 1-click editor setup below.

Add to CursorAdd to VS Code
Manual Client & Custom JSON ConfigExpand JSON β–Ύ

Install Config Generator

Choose your client
claude_desktop_config.json
{
  "mcpServers": {
    "webshift": {
      "command": "npx",
      "args": [
        "-y",
        "webshift"
      ]
    }
  }
}

πŸ’‘ Paste into ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows)

Install Directory Badge Claim listing AlternativesπŸ’» More in Developer Tools

Documentation Overview

WebShift

Crates.io docs.rs Latest Release Beta License


What is WebShift

WebShift is a Rust library and MCP server that shifts noisy web pages into clean, right-sized text for LLM consumption.

Raw HTML is mostly junk: scripts, ads, navigation menus, cookie banners, tracking pixels. Feeding it directly to an LLM floods the context window with tens of thousands of useless tokens and leaves no room for reasoning. WebShift strips all that noise, sterilizes the text, and enforces strict size budgets so the model receives only the content that matters.

What you get

Depending on the features you enable, WebShift can be four things:

Use caseCrateFeature flagsWhat it does
HTML denoiserwebshiftdefault-features = falseclean() β€” pure Rust HTML-to-text pipeline. Strips noise elements, sterilizes Unicode/BiDi, collapses whitespace. Zero network, zero config. Drop into any Rust project that processes web content for LLMs.
HTML text rewriterwebshiftfeatures = ["text-map"]extract_text_nodes() + replace_text_nodes() β€” extract individual text nodes from HTML, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure intact. Tags, attributes, and links are never touched.
Web content clientwebshiftdefault or features = ["llm"]fetch() + query() β€” streaming HTTP fetcher with size caps, 8 search backends, BM25 reranking, optional LLM query expansion and summarization. Full pipeline from search query to structured results.
MCP serverwebshift-mcpall featuresNative binary (mcp-webshift) that exposes webshift_query, webshift_fetch, and webshift_onboarding over MCP stdio. Single static binary, zero runtime dependencies.

When to use WebShift

  • You're building an AI agent that needs web search and you want clean, budget-controlled text β€” not raw HTML.
  • You're processing web pages in a Rust pipeline and need a reliable HTML-to-text cleaner that strips noise without losing real content.
  • You need an LLM to translate, rewrite, or simplify text inside HTML without corrupting the markup β€” text-map gives you a safe round-trip.
  • You want an MCP web search server that works as a single binary β€” no Python, no pip, no venv, no Docker (unless you want it).
  • You need hard guarantees on output size: per-page caps, total budget caps, streaming download limits.

When NOT to use WebShift

  • You need a headless browser that renders JavaScript-heavy SPAs. WebShift parses static HTML β€” it doesn't execute JS.
  • You need to render or screenshot a page preserving its visual layout. WebShift processes HTML structure but does not render CSS or compute layout. (Note: text-map does preserve the DOM structure for text rewriting use cases.)
  • You're building a web scraper that needs to navigate across pages, fill forms, or handle authentication flows.

How it works

Code
Question

  |

  +- (optional) LLM query expansion -> multiple search variants

  |

  +- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP)

  |

  +- Deduplicate + filter binary URLs

  |

  +- Streaming fetch with per-page size cap

  |

  +- HTML cleaning -> plain text (noise elements, scripts, nav removed)

  |

  +- Unicode/BiDi sterilization

  |

  +- BM25 deterministic reranking

  |   +- (optional) LLM-assisted tier-2 reranking

  |

  +- Budget-aware truncation across all sources

  |

  +- (optional) LLM Markdown summary with inline citations

  |

  +- Structured JSON output

For a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see Under the Hood. For the full configuration reference (TOML, env vars, CLI args) see Configuration. For ready-to-use examples see Use Cases.


Installation

Binary (MCP server)

bash
cargo install webshift-mcp

The binary is called mcp-webshift.

From source

bash
cargo install --path crates/webshift-mcp

As a library

toml
# Full pipeline (search + fetch + clean + rerank)

webshift = "0.2"



# Cleaner + fetcher only (no search backends)

webshift = { version = "0.2", default-features = false }



# Text-map only (extract/replace text nodes in HTML)

webshift = { version = "0.2", default-features = false, features = ["text-map"] }



# Everything including LLM features

webshift = { version = "0.2", features = ["llm"] }

Quick start

1. Set up a search backend

The easiest option is SearXNG β€” free, self-hosted, no API key:

Terminal
docker run -d -p 8080:8080 searxng/searxng

No Docker? Use a cloud backend β€” see Search backends.

2. Configure your MCP client

config.json
{

  "mcpServers": {

    "webshift": {

      "command": "mcp-webshift",

      "args": ["--default-backend", "searxng"]

    }

  }

}

That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.

For client-specific setup see docs/integrations/.


MCP tools

ToolDescription
webshift_queryFull search pipeline: search + fetch + clean + rerank + (optional) summarize
webshift_fetchSingle page fetch and clean
webshift_onboardingReturns a JSON guide for the agent (budgets, backends, tips)

webshift_query parameters

ParameterTypeDefaultDescription
queriesstring or listrequiredSearch query or list of queries
num_results_per_queryinteger5Results per query
langstringnoneLanguage filter (e.g. "en")
backendstringconfig defaultOverride search backend

Configuration

Resolution order (highest priority first):

  1. CLI args β€” --default-backend, --brave-api-key, etc.
  2. Environment variables β€” WEBSHIFT_* prefix
  3. Config file β€” webshift.toml (current dir, then ~/webshift.toml)
  4. Built-in defaults

Config file

toml
[server]

max_query_budget    = 32000   # total char budget across all sources

max_result_length   = 8000    # per-page char cap

max_total_results   = 20      # hard cap on results per call

max_download_mb     = 1       # streaming cap per page (MB)

search_timeout      = 8       # seconds

results_per_query   = 5

oversampling_factor = 2

adaptive_budget     = "auto"  # "auto" | "on" | "off" β€” budget allocation mode



[backends]

default = "searxng"



[backends.searxng]

url = "http://localhost:8080"



[backends.brave]

api_key = "BSA-..."



[backends.tavily]

api_key = "tvly-..."



[backends.exa]

api_key = "..."



[backends.serpapi]

api_key = "..."

engine  = "google"    # google | bing | duckduckgo | yandex



[backends.google]

api_key = "..."

cx      = "..."       # Custom Search Engine ID



[backends.bing]

api_key = "..."

market  = "en-US"



[backends.http]

url           = "https://my-search.example.com/api/search"

query_param   = "q"

count_param   = "limit"

results_path  = "data.items"     # dot-path to results array in JSON response

title_field   = "title"

url_field     = "link"

snippet_field = "description"



[backends.http.headers]

"Authorization" = "Bearer my-token"



[llm]

enabled               = false

base_url              = "http://localhost:11434/v1"   # OpenAI-compatible

api_key               = ""

model                 = "gemma3:27b"

timeout               = 60

expansion_enabled     = true

summarization_enabled = true

llm_rerank_enabled    = false

For every setting with all three config methods (TOML, env vars, CLI args) and plain-language descriptions, see the full Configuration Reference. Ready-to-use config examples are in Use Cases and examples/.

Key environment variables

bash
WEBSHIFT_DEFAULT_BACKEND=searxng

WEBSHIFT_SEARXNG_URL=http://localhost:8080

WEBSHIFT_BRAVE_API_KEY=BSA-xxx

WEBSHIFT_GOOGLE_API_KEY=xxx

WEBSHIFT_GOOGLE_CX=xxx

WEBSHIFT_BING_API_KEY=xxx

WEBSHIFT_LLM_ENABLED=true

WEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1

WEBSHIFT_LLM_MODEL=gemma3:27b

Search backends

BackendAuthNotes
SearXNGnoneSelf-hosted, free. Default: http://localhost:8080
BraveAPI keyFree tier. brave.com/search/api
TavilyAPI keyAI-oriented. tavily.com
ExaAPI keyNeural search. exa.ai
SerpAPIAPI keyMulti-engine proxy (Google, Bing, DDG...). serpapi.com
GoogleAPI key + CXCustom Search. Free: 100 req/day. programmablesearchengine.google.com
BingAPI keyWeb Search API. Free: 1,000 req/month. Microsoft Azure
HTTPconfigurableGeneric REST backend β€” no code required, TOML-only config

LLM features (optional)

All opt-in β€” disabled by default, no data leaves your machine unless enabled.

FeatureWhat it does
Query expansionSingle query -> N complementary search variants
SummarizationMarkdown report with inline [1] [2] citations
LLM rerankingTier-2 reranking on top of deterministic BM25

Cross-language normalization (bonus): when BM25 reranking surfaces pages in foreign languages (e.g. Chinese, Japanese, Arabic), the LLM summarizer still produces the final report in the prompt language. The agent receives clean, readable output regardless of the language mix in the source pages.

Works with any OpenAI-compatible API (OpenAI, Ollama, vLLM, LM Studio, etc.):

toml
[llm]

enabled  = true

base_url = "http://localhost:11434/v1"

model    = "gemma3:27b"

Anti-flooding protections

Always active β€” the core value proposition:

ProtectionDescription
max_download_mbStreaming cap β€” never buffers full response
max_result_lengthHard cap on characters per cleaned page
max_query_budgetTotal character budget across all sources
max_total_resultsHard cap on results per call
Binary filter.pdf, .zip, .exe, etc. filtered before any network request
Unicode sterilizationBiDi control chars, zero-width chars removed

Library usage

server.ts
use webshift::{Config, clean, fetch, query};



// Clean raw HTML β€” cap output at 8000 chars

let result = clean("<html><body><p>Hello world</p></body></html>", 8000);

println!("{}", result.text);



// Pass 0 to disable the per-page cap entirely (no truncation)

let full = clean("<html><body><p>Hello world</p></body></html>", 0);

assert!(!full.truncated);



// Fetch and clean a single page

let config = Config::default();

let page = fetch("https://example.com", &config).await?;



// Full search pipeline

let results = query(&["rust async programming"], &config).await?;

for source in &results.sources {

    println!("[{}] {} β€” {} chars", source.id, source.title, source.content.len());

}



// Backend partial failures (CAPTCHA, rate-limits, engine outages) are surfaced

// in `warnings` rather than as Err. Empty sources + non-empty warnings means

// "all backends blocked" β€” distinct from a legitimate "no matches found".

if results.sources.is_empty() && !results.warnings.is_empty() {

    eprintln!("all backends failed: {:?}", results.warnings);

}

Text-map: rewrite HTML content without breaking markup

Extract text nodes, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure, attributes, and links intact.

rust
use webshift::{extract_text_nodes, replace_text_nodes, TextReplacement};



let html = r#"<p>Hello <a href="https://example.com">world</a></p>"#;

let map = extract_text_nodes(html);

// map.nodes = [(0, "Hello"), (1, "world")]



let replacements = vec![

    TextReplacement { id: 0, text: "Ciao".into() },

    TextReplacement { id: 1, text: "mondo".into() },

];

let result = replace_text_nodes(html, &replacements).unwrap();

// β†’ <p>Ciao <a href="https://example.com">mondo</a></p>

// href untouched, tags intact, only text changed.

Requires features = ["text-map"]. See Use Cases #11 for a full translation example.

Feature flags

FeatureDefaultEnables
backendsonAll search backends + query pipeline
llmoffLLM client, expander, summarizer, LLM reranking
text-mapoffextract_text_nodes() + replace_text_nodes() β€” DOM round-trip for content rewriting

Integrations

PlatformGuide
Claude Desktop, Claude Code, Cursor, Windsurf, VS CodeIDE Integration
Zed β€” native extension with auto-download and Configure Server modalZed Extension
Gemini CLI, Claude CLI, custom agentsAgent Integration

Beta Status

WebShift is in beta. Core functionality is stable and the server is used daily, but the API surface may still change before 1.0.

Feedback is very welcome. If something doesn't work as expected, behaves oddly, or you have a use case that isn't covered:

Open an issue on GitHub

Bug reports, configuration questions, and feature requests all help shape the roadmap.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for detailed guidelines on:

  • Development setup and workflow
  • Code style and conventions
  • Testing requirements
  • Documentation standards
  • Pull request process

License

MIT License β€” see LICENSE for details.

Links

  • GitHub Repository β€” Source code and issues
  • Docs.rs β€” API documentation
  • MCP Registry β€” WebShift on Model Context Protocol Registry
  • MCP Protocol β€” Model Context Protocol specification

Need help? Check the documentation or open an issue on GitHub.

Related MCP Servers

View all in Developer Tools View all alternatives
  • O
    Oxylabs Oxylabs Mcp

    Fetch and process content from specified URLs using the Oxylabs Web Scraper API.

    πŸ’» Developer Tools0 views
    Compare vs Oxylabs Oxylabs Mcp β†’
  • P
    Pinkpixel Dev Web Scout Mcp

    Search the web and extract clean, readable text from webpages. Process multiple URLs at once to sp…

    πŸ’» Developer Tools0 views
    Compare vs Pinkpixel Dev Web Scout Mcp β†’
  • S
    Sofya

    Web search, fetch, extract, and research for AI agents. Markdown output + AI-synthesized answers.

    πŸ’» Developer Tools0 views
    Compare vs Sofya β†’
  • BoostedTravel logoBoostedTravel

    Flight search & booking for AI agents. 400+ airlines, $20-50 cheaper than OTAs.

    πŸ’» Developer Tools0 views
    Compare vs BoostedTravel β†’

Frequently Asked Questions about Webshift

Add the following block to your claude_desktop_config.json under mcpServers: "mcpServers": { "webshift": { "command": "npx", "args": ["-y", "webshift"] } }

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:
Live Dynamic SVG PreviewWebshift AllMCPs Directory Badge
Markdown (GitHub README)
[![AllMCPs](https://allmcps.com/api/badge/webshift?style=directory)](https://allmcps.com/mcp/webshift)
HTML Embed
<a href="https://allmcps.com/mcp/webshift"><img src="https://allmcps.com/api/badge/webshift?style=directory" alt="Webshift on AllMCPs" /></a>

Technical Specs & Signals

CategoryπŸ’»Developer Tools
More technical detailsExpand β–Ύ
TransportSTDIO
RuntimeNode.js
0/4 checks healthy over the last 6h
Views0
Unique ViewsTotal visits recorded for this listing page on AllMCPs.
Installs0
Installs & Copy ActionsTotal times users copied install commands or configuration snippets for this server.
27Quality signal: Emerging Β· 27/100How this signal is calculated β–Ύ
Server availabilityNot measured

Not scored for repo-hosted servers β€” we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.

Verified ownership8/20
Documentation & tools11/30
Adoption & activity1/15
Community engagement0/10

A guidance signal from public completeness & health data β€” not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.

β˜… Spotlight Slot

Feature Your MCP Server

Get maximum visibility for your server across our directory, search results, and detail pages.

Spotlight Your Server

Own this project?

This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to get the verified badge and attach your website.

Free dofollow backlink: after claiming, verify your product site and place a dofollow AllMCPs badge β€” we recheck it stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Explore more

More in πŸ’» Developer Tools β†’Best MCP servers for Developers β†’Alternatives to Webshift β†’Install in Claude DesktopInstall in CursorInstall in VS Code