"; my-websearch search "query" --engines exa
```
If the key is missing, the exa engine fails fast with an error message that includes these instructions instead of silently returning nothing.
**Common configurations:**
```bash
# Enable proxy for restricted regions
USE_PROXY=true PROXY_URL=http://127.0.0.1:7890 npx my-websearch@latest
# Only if a target website has a broken certificate chain
FETCH_WEB_INSECURE_TLS=true npx my-websearch@latest
# Request first, then fallback to Playwright if available
SEARCH_MODE=auto npx my-websearch@latest
# Force request-only Bing search
SEARCH_MODE=request npx my-websearch@latest
# Full configuration
DEFAULT_SEARCH_ENGINE=bing ENABLE_CORS=true USE_PROXY=true PROXY_URL=http://127.0.0.1:7890 PORT=8080 npx my-websearch@latest
```
**Proxy guidance for mainland China:**
`duckduckgo`, `exa`, `brave`, and `startpage` are overseas engines and **cannot be reached without a proxy from mainland China** — they will time out or return errors. Domestic engines (`bing`, `baidu`, `csdn`, `juejin`, `sogou`) work without a proxy.
Use `PROXY_ENGINES` to keep domestic engines on a fast direct connection while routing only the overseas engines through the proxy (avoiding the redirects/timeouts that a global proxy causes for Chinese engines):
```bash
USE_PROXY=true PROXY_URL=http://127.0.0.1:7890 PROXY_ENGINES=duckduckgo,exa,brave,startpage npx my-websearch@latest
```
If a search includes overseas engines but the proxy is off, those engines will fail fast instead of hanging until timeout: my-websearch probes direct connectivity to `duckduckgo`/`brave`/`startpage` (3s timeout, one retry, result cached for 5 minutes) — unreachable engines immediately return a "proxy required, or use domestic engines" error, while reachable engines (e.g. overseas users) work normally. `exa` is excluded from probing because `api.exa.ai` is directly reachable from mainland China. When the proxy is on, engines in `PROXY_ENGINES` are never probed — they go straight through the proxy.
Browser-enhanced Bing fallback and the `startpage` engine work out of the box: `playwright-core` is bundled as an optional dependency (auto-installed with the package, no browser download — system browsers are auto-discovered, e.g. Edge on Windows). If that install fails (network/platform), browser-based features degrade gracefully and all other engines stay unaffected.
Optional advanced setups (full Playwright with its own browser, custom module paths, remote/CDP browsers):
1. Full local Playwright install:
```bash
npm install playwright
npx playwright install chromium
SEARCH_MODE=auto npx my-websearch@latest
```
2. Reuse an existing browser binary with a slim client:
```bash
npm install playwright-core
PLAYWRIGHT_PACKAGE=playwright-core PLAYWRIGHT_EXECUTABLE_PATH=/path/to/chromium SEARCH_MODE=auto npx my-websearch@latest
```
3. Reuse a Playwright package that already exists elsewhere on the machine:
```bash
PLAYWRIGHT_MODULE_PATH=/absolute/path/to/node_modules/playwright SEARCH_MODE=playwright npx my-websearch@latest
```
4. Connect to an existing remote browser:
```bash
npm install playwright-core
PLAYWRIGHT_PACKAGE=playwright-core PLAYWRIGHT_WS_ENDPOINT=ws://127.0.0.1:3000/ SEARCH_MODE=auto npx my-websearch@latest
```
5. Reuse a local Chrome/Chromium session over CDP:
```bash
npm install playwright-core
# Start Chrome/Chromium with a debugging port first
chrome --remote-debugging-port=9222 --user-data-dir=/tmp/my-websearch-chrome
# Then connect through CDP
PLAYWRIGHT_PACKAGE=playwright-core PLAYWRIGHT_CDP_ENDPOINT=http://127.0.0.1:9222 SEARCH_MODE=auto npx my-websearch@latest
```
This is the most practical setup when you want to reuse your own logged-in or previously verified browser session.
Windows PowerShell example:
```powershell
npm install playwright-core
& "$env:LOCALAPPDATA\Google\Chrome\Application\chrome.exe" `
--remote-debugging-port=9222 `
--user-data-dir="$env:TEMP\my-websearch-chrome"
$env:PLAYWRIGHT_PACKAGE="playwright-core"
$env:PLAYWRIGHT_CDP_ENDPOINT="http://127.0.0.1:9222"
$env:SEARCH_MODE="auto"
npx my-websearch@latest
```
Mode behavior:
- `request`: only uses request-based Bing scraping
- `auto`: tries request first, and only falls back to Playwright when request fails and a manually accessible Playwright client + browser are available
- `playwright`: forces Playwright and errors if the configured Playwright client or browser target is unavailable
Notes:
- `PLAYWRIGHT_MODULE_PATH` takes precedence over `PLAYWRIGHT_PACKAGE`
- `PLAYWRIGHT_WS_ENDPOINT` takes precedence over `PLAYWRIGHT_CDP_ENDPOINT`
- Remote endpoints ignore `PLAYWRIGHT_EXECUTABLE_PATH` and local proxy launch flags
- When Playwright is available, blocked CSDN/Zhihu article fetches and generic web fetches can also retry with browser-acquired cookies
- Without Playwright, `fetchWebContent` stays on the request-only path. Public pages can still work, but pages that require browser cookies or browser-rendered HTML may fail.
### Local Installation
1. Clone or download this repository
2. Install dependencies:
```bash
npm install
```
This installs the core MCP server only. Browser fallback remains optional until you install or connect a Playwright client yourself.
3. Build the server:
```bash
npm run build
```
4. Add the server to your MCP configuration:
**Cherry Studio:**
```json
{
"mcpServers": {
"web-search": {
"name": "Web Search MCP",
"type": "streamableHttp",
"description": "Multi-engine web search with article fetching",
"isActive": true,
"baseUrl": "http://localhost:3211/mcp"
}
}
}
```
**VSCode (Claude Dev Extension):**
```json
{
"mcpServers": {
"web-search": {
"transport": {
"type": "streamableHttp",
"url": "http://localhost:3211/mcp"
}
},
"web-search-sse": {
"transport": {
"type": "sse",
"url": "http://localhost:3211/sse"
}
}
}
}
```
**Claude Desktop:**
```json
{
"mcpServers": {
"web-search": {
"type": "http",
"url": "http://localhost:3211/mcp"
},
"web-search-sse": {
"type": "sse",
"url": "http://localhost:3211/sse"
}
}
}
```
**NPX Command Line Configuration:**
```json
{
"mcpServers": {
"web-search": {
"args": [
"my-websearch@latest"
],
"command": "npx",
"env": {
"MODE": "stdio",
"DEFAULT_SEARCH_ENGINE": "bing",
"ALLOWED_SEARCH_ENGINES": "bing,duckduckgo,exa"
}
}
}
}
```
Windows NPX configuration:
```json
{
"mcpServers": {
"web-search": {
"command": "cmd",
"args": [
"/c",
"npx",
"-y",
"my-websearch@latest"
],
"env": {
"MODE": "stdio",
"DEFAULT_SEARCH_ENGINE": "bing",
"SYSTEMROOT": "C:/Windows"
}
}
}
}
```
Proxy and TLS notes:
- my-websearch now disables Axios environment-proxy auto-detection internally and only uses the explicit `USE_PROXY` + `PROXY_URL` path.
- When `USE_PROXY=true`, all Axios-based network requests follow the configured `PROXY_URL` path instead of mixing direct requests with environment-proxy behavior.
- If `PROXY_URL` points to a local rule-based proxy client, that client can still decide which destinations go `DIRECT` and which ones are proxied.
- If `PROXY_URL` points to a fixed upstream proxy or overseas egress, region-sensitive sites such as Baidu, CSDN, Juejin, or GitHub may behave differently than before.
- If your host machine already sets `HTTP_PROXY` or `HTTPS_PROXY`, they will no longer override the server's internal request behavior.
- Prefer configuring `NODE_EXTRA_CA_CERTS` on Windows when a site has a missing intermediate CA.
- Use `FETCH_WEB_INSECURE_TLS=true` only as a last resort for `fetchWebContent`, since it weakens TLS verification.
**Local STDIO Configuration for Cherry Studio (Windows):**
```json
{
"mcpServers": {
"my-websearch-local": {
"command": "node",
"args": ["C:/path/to/your/project/build/index.js"],
"env": {
"MODE": "stdio",
"DEFAULT_SEARCH_ENGINE": "bing",
"ALLOWED_SEARCH_ENGINES": "bing,duckduckgo,exa"
}
}
}
}
```
## Usage Guide
The server provides seven tools: `search`, `resolveLibraryId`, `queryDocs`, `fetchCsdnArticle`, `fetchGithubReadme`, `fetchJuejinArticle`, and `fetchWebContent`.
For the local daemon HTTP API (`serve`, `status`, `GET /health`, `POST /search`, `POST /fetch-*`), see [docs/http-api.md](https://github.com/wtznicy/my-websearch/blob/HEAD/docs/http-api.md).
### search Tool Usage
```typescript
{
"query": string, // Search query
"limit": number, // Optional: Number of results to return (default: 10)
"engines": string[], // Optional: Engines to use (bing,baidu,csdn,duckduckgo,exa,brave,juejin,startpage,sogou) default runtime-configured engine. Note: duckduckgo/exa/brave/startpage need a proxy from mainland China (see PROXY_ENGINES)
"searchMode": string // Optional: request, auto, or playwright (currently only affects Bing)
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "search",
arguments: {
query: "search content",
limit: 3, // Optional parameter
engines: ["bing", "csdn", "duckduckgo", "exa", "brave", "juejin", "sogou"] // Optional parameter, supports multi-engine combined search
}
})
```
Response example:
```json
[
{
"title": "Example Search Result",
"url": "https://example.com",
"description": "Description text of the search result...",
"source": "Source",
"engine": "Engine used"
}
]
```
### fetchCsdnArticle Tool Usage
Used to fetch complete content of CSDN blog articles.
```typescript
{
"url": string // URL from CSDN search results using the search tool
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "fetchCsdnArticle",
arguments: {
url: "https://blog.csdn.net/xxx/article/details/xxx"
}
})
```
Response example:
```json
[
{
"content": "Example search result"
}
]
```
### fetchGithubReadme Tool Usage
Used to fetch README content from GitHub or Gitee repositories (Gitee uses the official API, reachable without a proxy).
```typescript
{
"url": string // GitHub/Gitee repository URL (supports HTTPS, SSH formats)
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "fetchGithubReadme",
arguments: {
url: "https://gitee.com/wtznicy/my-websearch"
}
})
```
Supported URL formats:
- GitHub HTTPS: `https://github.com/owner/repo`
- GitHub HTTPS with .git: `https://github.com/owner/repo.git`
- GitHub SSH: `git@github.com:owner/repo.git`
- URLs with parameters: `https://github.com/owner/repo?tab=readme`
- Gitee HTTPS: `https://gitee.com/owner/repo`
- Gitee SSH: `git@gitee.com:owner/repo.git`
Response example:
```json
[
{
"content": "\n\n# MyWebSearch MCP Server..."
}
]
```
### fetchWebContent Tool Usage
Fetch content directly from public HTTP(S) links, including Markdown files (`.md`) and ordinary web pages.
```typescript
{
"url": string, // Public HTTP(S) URL
"maxChars": number // Optional: max returned content length (1000-200000, default 30000)
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "fetchWebContent",
arguments: {
url: "https://gitee.com/wtznicy/my-websearch/raw/main/README.md",
maxChars: 12000
}
})
```
Response example:
```json
{
"url": "https://gitee.com/wtznicy/my-websearch/raw/main/README.md",
"finalUrl": "https://gitee.com/wtznicy/my-websearch/raw/main/README.md",
"contentType": "text/plain; charset=utf-8",
"title": "",
"truncated": false,
"content": "# MyWebSearch MCP Server ..."
}
```
### fetchJuejinArticle Tool Usage
Used to fetch complete content of Juejin articles.
```typescript
{
"url": string // Juejin article URL from search results
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "fetchJuejinArticle",
arguments: {
url: "https://juejin.cn/post/7520959840199360563"
}
})
```
Supported URL format:
- `https://juejin.cn/post/{article_id}`
Response example:
```json
[
{
"content": "🚀 开源 AI 联网搜索工具：MyWebSearch MCP 全新升级，支持多引擎 + 流式响应..."
}
]
```
## Usage Limitations
Since this tool works by scraping multi-engine search results, please note the following important limitations:
1. **Rate Limiting**:
- Too many searches in a short time may cause the used engines to temporarily block requests
- Recommendations:
- Maintain reasonable search frequency
- Use the limit parameter judiciously
- Add delays between searches when necessary
- **Brave is the strictest**: it throttles consecutive automated requests aggressively — a burst of searches triggers HTTP 429 for minutes (even from residential proxy IPs), and the block window outlasts short cooldowns. Use brave at low frequency; prefer `duckduckgo` / `startpage` as the daily overseas engines (they are stable and of similar quality). A 429 on brave fails fast and `minResults` cascade automatically falls back to other engines.
2. **Result Accuracy**:
- Depends on the HTML structure of corresponding engines, may fail when engines update
- Some results may lack metadata like descriptions
- Complex search operators may not work as expected
3. **Legal Terms**:
- This tool is for personal use only
- Please comply with the terms of service of corresponding engines
- Implement appropriate rate limiting based on your actual use case
4. **Search Engine Configuration**:
- Default search engine can be set via the `DEFAULT_SEARCH_ENGINE` environment variable
- Supported engines: bing, duckduckgo, exa, brave, baidu, csdn, juejin, startpage, sogou
- Overseas engines (duckduckgo, exa, brave, startpage) require a proxy from mainland China (see `PROXY_ENGINES`); domestic engines (bing, baidu, csdn, juejin, sogou) work direct
- The default engine is used when searching specific websites
5. **Proxy Configuration**:
- HTTP proxy can be configured when certain search engines are unavailable in specific regions
- Enable proxy with environment variable `USE_PROXY=true`
- Configure proxy server address with `PROXY_URL`
- With `USE_PROXY=true`, `PROXY_ENGINES` (comma-separated whitelist) limits which engines route through the proxy; empty = all engines proxied. Overseas engines (`duckduckgo`, `exa`, `brave`, `startpage`) require a proxy from mainland China, while domestic engines stay direct — recommended: `PROXY_ENGINES=duckduckgo,exa,brave,startpage`
- For Clash fake-ip / TUN setups, configure synthetic DNS ranges with `FAKE_IP_CIDRS` (for example `198.18.0.0/15`)
## Contributing
Welcome to submit issue reports and feature improvement suggestions!
### resolveLibraryId Tool Usage
Resolves a library/package name into a Context7-compatible library ID, with reputation and quality metadata. Powered by the [Context7](https://context7.com) documentation index — official, version-specific library docs without needing a separate MCP server.
```typescript
{
"libraryName": string, // e.g. "Next.js", "express", "prisma"
"query": string, // The user's question, used to rank matches (e.g. "how to implement authentication")
"limit": number // Optional: max matches (default 5, max 10)
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "resolveLibraryId",
arguments: {
libraryName: "Next.js",
query: "how to set up middleware with auth"
}
})
```
### queryDocs Tool Usage
Retrieves up-to-date, version-specific documentation snippets and code examples for a library. Use `resolveLibraryId` first if you don't know the library ID.
```typescript
{
"libraryId": string, // Context7-compatible ID, e.g. "/vercel/next.js", "/packages/express" (optional version: "/vercel/next.js@v15.1.8")
"query": string, // The question or task to get relevant documentation for
"limit": number // Optional: max code snippets (default 5, max 10)
}
```
Usage example:
```typescript
use_mcp_tool({
server_name: "web-search",
tool_name: "queryDocs",
arguments: {
libraryId: "/vercel/next.js",
query: "how to set up middleware with authentication"
}
})
```
> **Note:** Both Context7 tools call the public REST API directly (no API key required at low rate limits). Set `CONTEXT7_API_KEY` for higher rate limits.
## Author & Acknowledgements
**Author: wtznicy**
This project is a modified fork of **Open-WebSearch** (originally by Aas-ee) — thanks to the original author for the great work.
Thanks also to these open-source projects:
- **context7** (Upstash): powers the `resolveLibraryId` / `queryDocs` library-docs lookup
- **fetch** (official MCP servers): reference for the `fetchWebContent` web-fetching design