websearch-skill

Open-source multi-engine web search and content extraction for AI agents. Internal layers communicate through versioned JSON Schema contracts.

Status: early. First public release 2026-06-22; current version in CHANGELOG.md. Available now: keyless search, Markdown extraction, five agent commands, self-hosted SearXNG, and a NordVPN egress proxy that two env vars enable and lock. These are covered by the test suite. There is no MCP server; use the CLI directly or install the skill (see No MCP below). Hard anti-bot tiers and local reranking remain on the roadmap. Pin a version and test it in a sandbox before using it with sensitive data.

What it is

A self-hosted web search and page reader for AI agents, with no API keys or paid search service. It queries multiple engines, combines and deduplicates the results, then fetches pages as clean Markdown. The CLI runs locally and sends queries directly to the configured engines.

Core commands:

• init reads the settings files, starts local SearXNG (and Tor, when that layer is on), and reports available capabilities.
• web_search finds pages: ranked, deduplicated results across engines, each with a reusable handle.
• web_fetch reads a page: clean Markdown, fenced as untrusted, paginated so a long page never overflows context.
• web_open pages back through a document you already fetched, from cache, without hitting the network again.

Additional keyless commands:

• arxiv searches arXiv papers and returns structured metadata (authors, abstract, categories, abstract and PDF links).
• github searches GitHub repositories and returns typed fields you can sort on (stars, language, topics).
• tor starts a local Tor and routes everything through it, which is what makes .onion reachable and --onion search the Tor indexes. Off by default; see Tor below.

One runtime, one call

The whole toolkit is one Python package with one CLI, websearch. There is no Node service, no npm runtime, no daemon of its own, and nothing to compose; the only requirement is uv, which fetches a compatible Python on its own. The npx skills add line under Install copies skill folders into agents once; nothing from Node runs afterwards.

An agent brings everything online with one call:

websearch init

init reads the settings files, starts the local SearXNG when it is not already running, starts Tor when that layer is on, runs the self-test, and reports capabilities and next actions. There is nothing to probe first and no order to follow: the same call is correct on a fresh machine and on one where everything is already up.

SearXNG is an optional extra engine catalog that the toolkit operates for you: init (or websearch searxng up) clones it, installs it into its own virtualenv, and runs it as a detached local Python process. Docker is never required; docker/searxng/ exists only as an alternative for hosts that prefer a container.

Engines, out of the box, no keys

The default engine is the keyless ddgs metasearch library, which spans Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. Each query uses the providers that respond. The agent-facing web-search command requires no account, service, or engine flags. The lower-level search command can restrict providers with --ddgs-backends google,brave,mojeek. Provider availability varies by network; websearch doctor reports it per provider.

For broader and more reliable search you can run your own SearXNG (~280 engines, your own server, still no keys), and the router fuses it with ddgs and de-correlates the engines they share. websearch searxng up sets one up and starts it on any machine with git and Python; on a Docker host, docker/searxng/ is the curated stack, including a probe that enables every engine that answers on your connection. See Self-hosting SearXNG below. Public SearXNG instances are deliberately not a default: most disable the JSON API and rate-limit automated clients, so depending on them would break on a fresh install.

The project is MIT-licensed. Optional integrations include self-hosted SearXNG, keyed engines (Brave, Exa, Tavily), and a paid egress adapter.

The scorecard below compares result quality, latency, cost, privacy, and retrieval limits. The software layer is free and self-hosted, with multi-engine search, deduplication, and Markdown extraction. Protected sites may require the planned paid egress adapter; residential proxies and CAPTCHA solving have no reliable free equivalent.

Layer status

Contracts are frozen as JSON Schema 2020-12: envelope@1.0.0, search@1.2.0, fetch@1.3.0, extract@1.0.0, format@1.0.0, store@1.0.0, agent-io@1.2.0, arxiv@1.1.0, github@1.1.0, doctor@2.1.0, searxng@1.2.0, init@1.1.0, tor@1.0.0, proxy@1.0.0. Every response is wrapped in one Envelope { contract_version, ok, data, error, meta }.

Layer 1: Search

Search normalizes a request, sends it to each configured EngineAdapter, canonicalizes and deduplicates URLs, and combines the rankings with weighted Reciprocal Rank Fusion (RRF, k=60). The keyless default is ddgs, a metasearch library for Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek, Wikipedia, and Grokipedia. On the lower-level search command, --ddgs-backends selects a provider subset and --engines selects the ddgs and SearXNG adapters. The agent-facing web-search command uses the defaults. Set WEBSEARCH_SEARXNG_URL to add a self-hosted SearXNG instance. Keyed engines such as Brave, Exa, and Tavily are planned.

ddgs is a keyless Python library that runs in-process. SearXNG is a separate self-hosted server with a larger engine catalog. Search can use either adapter or combine both. SearXNG provides search results only; page fetching and extraction remain in Layer 2A.

De-correlation prevents duplicate upstream data from distorting the ranking. SearXNG and ddgs both use crawlers such as Google and Bing, so a naive union can count the same source twice. Results are grouped by correlation_group; each group contributes one RRF term at its best rank, and the consensus bonus uses only the number of distinct groups. SearXNG and ddgs agreeing counts as one independent vote; SearXNG and a neural index agreeing counts as two. The response includes a warning when de-correlation is applied.

Each result records which engines returned it and its rank within each engine.

Layer 2A: Fetch + Extract

Fetching and extraction use separate interfaces and can be replaced independently.

Fetch starts with httpx and retries with curl_cffi browser impersonation only when it detects an anti-bot challenge. It does not retry 404s, rate limits, authentication failures, or legal and geographic blocks because the same network address is unlikely to change the result. Detection checks response headers, then known body markers for Cloudflare, DataDome, PerimeterX, Akamai, and Imperva. Imperva body markers are checked on every status because its blocks can return HTTP 200. Browser and stealth tiers (Crawl4AI, nodriver) are present in the contract but are not part of the base install.

Extract defaults to Trafilatura, a heuristic extractor. Per the May 2026 WCXB benchmark, heuristic extractors beat neural extractors on both quality and cost (neural runs tens to hundreds of times more expensive), and Trafilatura lands at about 0.79 F1 on CPU in roughly 100ms; article pages saturate around 0.93 F1. The adapter parses the raw HTML once with lxml to recover the raw schema.org JSON-LD blocks and og:type (Trafilatura folds JSON-LD into metadata and never exposes the raw blocks), runs Trafilatura for the Markdown body and plain text plus metadata, then computes:

• a heuristic quality_score (0..1) from runtime signals (text density, word-count saturation, paragraph count, inverse link density, JSON-LD presence, clean title) with hard vetoes for soft-404s and shells; below about 0.80 a page is a fallback candidate.
• a cheap page_type resolved from JSON-LD @type, then og:type, then URL shape.