Webshift

Rust MCP server for denoised web search — fetch, clean, and rerank web content for AI agents.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.

One-click editor setup isn’t available for this listing yet — we don’t have a confirmed install command, and we’d rather show nothing than point your editor at the wrong package or host. Follow the project’s own setup instructions, linked above.

Manual Client & Custom JSON ConfigExpand JSON ▾

No confirmed setup config for this listing yet. We only publish a config block when the install details come from the project itself — its README, its docs, or a verified owner. We haven’t found those for webshift, and we’d rather show nothing than a guess you’d paste into your client. Follow the project’s own setup instructions for the current steps.

Capabilities & Tool Schemas (3)Self-reported

Inspect callable tools, capabilities, and parameters exposed to AI agents by Webshift.

webshift_query

Full search pipeline: search + fetch + clean + rerank + (optional) summarize

webshift_fetch

Single page fetch and clean

webshift_onboarding

Returns a JSON guide for the agent (budgets, backends, tips)

Documentation Overview

WebShift

What is WebShift

WebShift is a Rust library and MCP server that shifts noisy web pages into clean, right-sized text for LLM consumption.

Raw HTML is mostly junk: scripts, ads, navigation menus, cookie banners, tracking pixels. Feeding it directly to an LLM floods the context window with tens of thousands of useless tokens and leaves no room for reasoning. WebShift strips all that noise, sterilizes the text, and enforces strict size budgets so the model receives only the content that matters.

What you get

Depending on the features you enable, WebShift can be four things:

Use case	Crate	Feature flags	What it does
HTML denoiser	`webshift`	`default-features = false`	`clean()` — pure Rust HTML-to-text pipeline. Strips noise elements, sterilizes Unicode/BiDi, collapses whitespace. Zero network, zero config. Drop into any Rust project that processes web content for LLMs.
HTML text rewriter	`webshift`	`features = ["text-map"]`	`extract_text_nodes()` + `replace_text_nodes()` — extract individual text nodes from HTML, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure intact. Tags, attributes, and links are never touched.
Web content client	`webshift`	`default` or `features = ["llm"]`	`fetch()` + `query()` — streaming HTTP fetcher with size caps, 8 search backends, BM25 reranking, optional LLM query expansion and summarization. Full pipeline from search query to structured results.
MCP server	`webshift-mcp`	all features	Native binary (`mcp-webshift`) that exposes `webshift_query`, `webshift_fetch`, and `webshift_onboarding` over MCP stdio. Single static binary, zero runtime dependencies.

When to use WebShift

You're building an AI agent that needs web search and you want clean, budget-controlled text — not raw HTML.
You're processing web pages in a Rust pipeline and need a reliable HTML-to-text cleaner that strips noise without losing real content.
You need an LLM to translate, rewrite, or simplify text inside HTML without corrupting the markup — text-map gives you a safe round-trip.
You want an MCP web search server that works as a single binary — no Python, no pip, no venv, no Docker (unless you want it).
You need hard guarantees on output size: per-page caps, total budget caps, streaming download limits.

When NOT to use WebShift

You need a headless browser that renders JavaScript-heavy SPAs. WebShift parses static HTML — it doesn't execute JS.
You need to render or screenshot a page preserving its visual layout. WebShift processes HTML structure but does not render CSS or compute layout. (Note: text-map does preserve the DOM structure for text rewriting use cases.)
You're building a web scraper that needs to navigate across pages, fill forms, or handle authentication flows.

How it works

Code

Question

  |

  +- (optional) LLM query expansion -> multiple search variants

  |

  +- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP)

  |

  +- Deduplicate + filter binary URLs

  |

  +- Streaming fetch with per-page size cap

  |

  +- HTML cleaning -> plain text (noise elements, scripts, nav removed)

  |

  +- Unicode/BiDi sterilization

  |

  +- BM25 deterministic reranking

  |   +- (optional) LLM-assisted tier-2 reranking

  |

  +- Budget-aware truncation across all sources

  |

  +- (optional) LLM Markdown summary with inline citations

  |

  +- Structured JSON output

For a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see Under the Hood. For the full configuration reference (TOML, env vars, CLI args) see Configuration. For ready-to-use examples see Use Cases.

Installation

Binary (MCP server)

bash

cargo install webshift-mcp

The binary is called mcp-webshift.

From source

bash

cargo install --path crates/webshift-mcp

As a library

toml

# Full pipeline (search + fetch + clean + rerank)

webshift = "0.2"



# Cleaner + fetcher only (no search backends)

webshift = { version = "0.2", default-features = false }



# Text-map only (extract/replace text nodes in HTML)

webshift = { version = "0.2", default-features = false, features = ["text-map"] }



# Everything including LLM features

webshift = { version = "0.2", features = ["llm"] }

Quick start

1. Set up a search backend

The easiest option is SearXNG — free, self-hosted, no API key:

Terminal

docker run -d -p 8080:8080 searxng/searxng

No Docker? Use a cloud backend — see Search backends.

2. Configure your MCP client

config.json

{

  "mcpServers": {

    "webshift": {

      "command": "mcp-webshift",

      "args": ["--default-backend", "searxng"]

    }

  }

}

That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.

For client-specific setup see docs/integrations/.

MCP tools

Tool	Description
`webshift_query`	Full search pipeline: search + fetch + clean + rerank + (optional) summarize
`webshift_fetch`	Single page fetch and clean
`webshift_onboarding`	Returns a JSON guide for the agent (budgets, backends, tips)

`webshift_query` parameters

Parameter	Type	Default	Description
`queries`	string or list	required	Search query or list of queries
`num_results_per_query`	integer	5	Results per query
`lang`	string	none	Language filter (e.g. `"en"`)
`backend`	string	config default	Override search backend

Configuration

Resolution order (highest priority first):

CLI args — --default-backend, --brave-api-key, etc.
Environment variables — WEBSHIFT_* prefix
Config file — webshift.toml (current dir, then ~/webshift.toml)
Built-in defaults

Config file

toml

[server]

max_query_budget    = 32000   # total char budget across all sources

max_result_length   = 8000    # per-page char cap

max_total_results   = 20      # hard cap on results per call

max_download_mb     = 1       # streaming cap per page (MB)

search_timeout      = 8       # seconds

results_per_query   = 5

oversampling_factor = 2

adaptive_budget     = "auto"  # "auto" | "on" | "off" — budget allocation mode



[backends]

default = "searxng"



[backends.searxng]

url = "http://localhost:8080"



[backends.brave]

api_key = "BSA-..."



[backends.tavily]

api_key = "tvly-..."



[backends.exa]

api_key = "..."



[backends.serpapi]

api_key = "..."

engine  = "google"    # google | bing | duckduckgo | yandex



[backends.google]

api_key = "..."

cx      = "..."       # Custom Search Engine ID



[backends.bing]

api_key = "..."

market  = "en-US"



[backends.http]

url           = "https://my-search.example.com/api/search"

query_param   = "q"

count_param   = "limit"

results_path  = "data.items"     # dot-path to results array in JSON response

title_field   = "title"

url_field     = "link"

snippet_field = "description"



[backends.http.headers]

"Authorization" = "Bearer my-token"



[llm]

enabled               = false

base_url              = "http://localhost:11434/v1"   # OpenAI-compatible

api_key               = ""

model                 = "gemma3:27b"

timeout               = 60

expansion_enabled     = true

summarization_enabled = true

llm_rerank_enabled    = false

For every setting with all three config methods (TOML, env vars, CLI args) and plain-language descriptions, see the full Configuration Reference. Ready-to-use config examples are in Use Cases and examples/.

Key environment variables

bash

WEBSHIFT_DEFAULT_BACKEND=searxng

WEBSHIFT_SEARXNG_URL=http://localhost:8080

WEBSHIFT_BRAVE_API_KEY=BSA-xxx

WEBSHIFT_GOOGLE_API_KEY=xxx

WEBSHIFT_GOOGLE_CX=xxx

WEBSHIFT_BING_API_KEY=xxx

WEBSHIFT_LLM_ENABLED=true

WEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1

WEBSHIFT_LLM_MODEL=gemma3:27b

Search backends

Backend	Auth	Notes
SearXNG	none	Self-hosted, free. Default: `http://localhost:8080`
Brave	API key	Free tier. brave.com/search/api
Tavily	API key	AI-oriented. tavily.com
Exa	API key	Neural search. exa.ai

Read the full README →View source on GitHub →

Related MCP Servers

View all in Search & Data Extraction View all alternatives

Adoption & maintenance

Factual signals from GitHub, npm, and our automated checks — not a rating.

GitHub stars

Stargazers on the source repository.

npm downloads

Package downloads in the last 30 days.

Last commit

4mo ago

Most recent push to the default branch.

Tools exposed

Callable tools this server registers over MCP.

Reviews

No reviews yet — be the first to share how this listing worked for you.

Frequently Asked Questions about Webshift

We don't have a confirmed install command for webshift yet, so we don't publish a generated one — a guessed package name would point at the wrong package or none at all. Follow the project's own README or setup instructions (https://github.com/annibale-x/webshift) for the current steps.

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:

Live Dynamic SVG Preview

Markdown (GitHub README)

[![AllMCPs](https://allmcps.com/api/badge/webshift?style=directory)](https://allmcps.com/mcp/webshift)

HTML Embed

<a href="https://allmcps.com/mcp/webshift"><img src="https://allmcps.com/api/badge/webshift?style=directory" alt="Webshift on AllMCPs" /></a>

Technical Specs & Signals

CategorySearch & Data Extraction

6/10 checks healthy over the last 45d

45Quality signal: Fair · 45/100How this signal is calculated ▾

Server availabilityNot measured

Not scored for repo-hosted servers — we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.

Verified ownership10/20

Documentation & tools20/30

Adoption & activity4/15

Community engagement0/10

A guidance signal from public completeness & health data — not a user rating. New listings start lower and rise as they add docs, get verified, and grow adoption. Signals we can't observe for a listing are skipped, not counted against it.

★ Featured

AllMCPs Server

The official MCP server for AllMCPs.com - submit and manage tools directly from your AI. The open directory for MCP servers. Connect Claude, Cursor, Windsurf, and AI agents to databases, tools, files, and APIs. Explore 10,000+ servers. AllMCPs is the premier, open directory for discovering, evaluating, and installing Model Context Protocol (MCP) servers to equip AI agents and LLMs with real-world superpowers.

Explore Server →

Own this project?

This directory is pre-filled from public sources. Claim via GitHub README, site badge, or DNS TXT to unlock edit access and the Official badge — proof is checked automatically, then reviewed by our team.

Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Explore more

More in 🔎 Search & Data Extraction →Best MCP servers for Web Search & Scraping →Alternatives to Webshift →Install in Claude Desktop Install in Cursor Install in VS Code Setup guides for all 13 MCP clients

Webshift

Rust MCP server for denoised web search — fetch, clean, and rerank web content for AI agents.

Quick Install

Automated & IDE Setup

Copy the AI prompt to install this server into Claude Code, Cursor, or another agent — or use 1-click editor setup below.

Manual Client & Custom JSON ConfigExpand JSON ▾

Capabilities & Tool Schemas (3)Self-reported

Inspect callable tools, capabilities, and parameters exposed to AI agents by Webshift.

webshift_query

Full search pipeline: search + fetch + clean + rerank + (optional) summarize

webshift_fetch

Single page fetch and clean

webshift_onboarding

Returns a JSON guide for the agent (budgets, backends, tips)

Documentation Overview

WebShift

What is WebShift

WebShift is a Rust library and MCP server that shifts noisy web pages into clean, right-sized text for LLM consumption.

What you get

Depending on the features you enable, WebShift can be four things:

Use case	Crate	Feature flags	What it does
HTML denoiser	`webshift`	`default-features = false`	`clean()` — pure Rust HTML-to-text pipeline. Strips noise elements, sterilizes Unicode/BiDi, collapses whitespace. Zero network, zero config. Drop into any Rust project that processes web content for LLMs.
HTML text rewriter	`webshift`	`features = ["text-map"]`	`extract_text_nodes()` + `replace_text_nodes()` — extract individual text nodes from HTML, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure intact. Tags, attributes, and links are never touched.
Web content client	`webshift`	`default` or `features = ["llm"]`	`fetch()` + `query()` — streaming HTTP fetcher with size caps, 8 search backends, BM25 reranking, optional LLM query expansion and summarization. Full pipeline from search query to structured results.
MCP server	`webshift-mcp`	all features	Native binary (`mcp-webshift`) that exposes `webshift_query`, `webshift_fetch`, and `webshift_onboarding` over MCP stdio. Single static binary, zero runtime dependencies.

When to use WebShift

You're building an AI agent that needs web search and you want clean, budget-controlled text — not raw HTML.
You're processing web pages in a Rust pipeline and need a reliable HTML-to-text cleaner that strips noise without losing real content.
You need an LLM to translate, rewrite, or simplify text inside HTML without corrupting the markup — text-map gives you a safe round-trip.
You want an MCP web search server that works as a single binary — no Python, no pip, no venv, no Docker (unless you want it).
You need hard guarantees on output size: per-page caps, total budget caps, streaming download limits.

When NOT to use WebShift

You need a headless browser that renders JavaScript-heavy SPAs. WebShift parses static HTML — it doesn't execute JS.
You need to render or screenshot a page preserving its visual layout. WebShift processes HTML structure but does not render CSS or compute layout. (Note: text-map does preserve the DOM structure for text rewriting use cases.)
You're building a web scraper that needs to navigate across pages, fill forms, or handle authentication flows.

How it works

Code

Question

  |

  +- (optional) LLM query expansion -> multiple search variants

  |

  +- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP)

  |

  +- Deduplicate + filter binary URLs

  |

  +- Streaming fetch with per-page size cap

  |

  +- HTML cleaning -> plain text (noise elements, scripts, nav removed)

  |

  +- Unicode/BiDi sterilization

  |

  +- BM25 deterministic reranking

  |   +- (optional) LLM-assisted tier-2 reranking

  |

  +- Budget-aware truncation across all sources

  |

  +- (optional) LLM Markdown summary with inline citations

  |

  +- Structured JSON output

Installation

Binary (MCP server)

bash

cargo install webshift-mcp

The binary is called mcp-webshift.

From source

bash

cargo install --path crates/webshift-mcp

As a library

toml

# Full pipeline (search + fetch + clean + rerank)

webshift = "0.2"



# Cleaner + fetcher only (no search backends)

webshift = { version = "0.2", default-features = false }



# Text-map only (extract/replace text nodes in HTML)

webshift = { version = "0.2", default-features = false, features = ["text-map"] }



# Everything including LLM features

webshift = { version = "0.2", features = ["llm"] }

Quick start

1. Set up a search backend

The easiest option is SearXNG — free, self-hosted, no API key:

Terminal

docker run -d -p 8080:8080 searxng/searxng

No Docker? Use a cloud backend — see Search backends.

2. Configure your MCP client

config.json

{

  "mcpServers": {

    "webshift": {

      "command": "mcp-webshift",

      "args": ["--default-backend", "searxng"]

    }

  }

}

That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.

For client-specific setup see docs/integrations/.

MCP tools

Tool	Description
`webshift_query`	Full search pipeline: search + fetch + clean + rerank + (optional) summarize
`webshift_fetch`	Single page fetch and clean
`webshift_onboarding`	Returns a JSON guide for the agent (budgets, backends, tips)

`webshift_query` parameters

Parameter	Type	Default	Description
`queries`	string or list	required	Search query or list of queries
`num_results_per_query`	integer	5	Results per query
`lang`	string	none	Language filter (e.g. `"en"`)
`backend`	string	config default	Override search backend

Configuration

Resolution order (highest priority first):

CLI args — --default-backend, --brave-api-key, etc.
Environment variables — WEBSHIFT_* prefix
Config file — webshift.toml (current dir, then ~/webshift.toml)
Built-in defaults

Config file

toml

[server]

max_query_budget    = 32000   # total char budget across all sources

max_result_length   = 8000    # per-page char cap

max_total_results   = 20      # hard cap on results per call

max_download_mb     = 1       # streaming cap per page (MB)

search_timeout      = 8       # seconds

results_per_query   = 5

oversampling_factor = 2

adaptive_budget     = "auto"  # "auto" | "on" | "off" — budget allocation mode



[backends]

default = "searxng"



[backends.searxng]

url = "http://localhost:8080"



[backends.brave]

api_key = "BSA-..."



[backends.tavily]

api_key = "tvly-..."



[backends.exa]

api_key = "..."



[backends.serpapi]

api_key = "..."

engine  = "google"    # google | bing | duckduckgo | yandex



[backends.google]

api_key = "..."

cx      = "..."       # Custom Search Engine ID



[backends.bing]

api_key = "..."

market  = "en-US"



[backends.http]

url           = "https://my-search.example.com/api/search"

query_param   = "q"

count_param   = "limit"

results_path  = "data.items"     # dot-path to results array in JSON response

title_field   = "title"

url_field     = "link"

snippet_field = "description"



[backends.http.headers]

"Authorization" = "Bearer my-token"



[llm]

enabled               = false

base_url              = "http://localhost:11434/v1"   # OpenAI-compatible

api_key               = ""

model                 = "gemma3:27b"

timeout               = 60

expansion_enabled     = true

summarization_enabled = true

llm_rerank_enabled    = false

Key environment variables

bash

WEBSHIFT_DEFAULT_BACKEND=searxng

WEBSHIFT_SEARXNG_URL=http://localhost:8080

WEBSHIFT_BRAVE_API_KEY=BSA-xxx

WEBSHIFT_GOOGLE_API_KEY=xxx

WEBSHIFT_GOOGLE_CX=xxx

WEBSHIFT_BING_API_KEY=xxx

WEBSHIFT_LLM_ENABLED=true

WEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1

WEBSHIFT_LLM_MODEL=gemma3:27b

Search backends

Backend	Auth	Notes
SearXNG	none	Self-hosted, free. Default: `http://localhost:8080`
Brave	API key	Free tier. brave.com/search/api
Tavily	API key	AI-oriented. tavily.com
Exa	API key	Neural search. exa.ai

Read the full README →View source on GitHub →

Related MCP Servers

View all in Search & Data Extraction View all alternatives

Adoption & maintenance

Factual signals from GitHub, npm, and our automated checks — not a rating.

GitHub stars

Stargazers on the source repository.

npm downloads

Package downloads in the last 30 days.

Last commit

4mo ago

Most recent push to the default branch.

Tools exposed

Callable tools this server registers over MCP.

Reviews

No reviews yet — be the first to share how this listing worked for you.

Frequently Asked Questions about Webshift

AllMCPs Directory Badge

Full Badge Customizer

Showcase your server listing on GitHub or your project documentation. Embed this dynamic SVG badge to highlight official listing status and live engagement.

Badge Style:

Live Dynamic SVG Preview

Markdown (GitHub README)

[![AllMCPs](https://allmcps.com/api/badge/webshift?style=directory)](https://allmcps.com/mcp/webshift)

HTML Embed

<a href="https://allmcps.com/mcp/webshift"><img src="https://allmcps.com/api/badge/webshift?style=directory" alt="Webshift on AllMCPs" /></a>

Technical Specs & Signals

CategorySearch & Data Extraction

6/10 checks healthy over the last 45d

45Quality signal: Fair · 45/100How this signal is calculated ▾

Server availabilityNot measured

Not scored for repo-hosted servers — we can't reach the running server, only its GitHub page. Hosted MCP endpoints are health-checked live.

Verified ownership10/20

Documentation & tools20/30

Adoption & activity4/15

Community engagement0/10

★ Featured

AllMCPs Server

Explore Server →

Own this project?

Free dofollow backlink: add your website and place the AllMCPs badge on it — no claim needed. We detect it automatically and keep it verified as long as the badge stays live.

Claim & get free dofollow

Share & Embed

Add our SVG badge (dark/light directory styles) or embeddable widget to your site.

Webshift

Quick Install

Documentation Overview

WebShift

What is WebShift

What you get

When to use WebShift

When NOT to use WebShift

How it works

Installation

Binary (MCP server)

From source

As a library

Quick start

1. Set up a search backend

2. Configure your MCP client

MCP tools

webshift_query parameters

Configuration

Config file

Key environment variables

Search backends

Related MCP Servers

Adoption & maintenance

Reviews

Frequently Asked Questions about Webshift

AllMCPs Directory Badge

Technical Specs & Signals

AllMCPs Server

Own this project?

Share & Embed

Explore more

Webshift

Quick Install

Documentation Overview

WebShift

What is WebShift

What you get

When to use WebShift

When NOT to use WebShift

How it works

Installation

Binary (MCP server)

From source

As a library

Quick start

1. Set up a search backend

2. Configure your MCP client

MCP tools

webshift_query parameters

Configuration

Config file

Key environment variables

Search backends

Related MCP Servers

Adoption & maintenance

Reviews

Frequently Asked Questions about Webshift

AllMCPs Directory Badge

Technical Specs & Signals

AllMCPs Server

Own this project?

Share & Embed

Explore more

`webshift_query` parameters

`webshift_query` parameters