searchts

The missing layer between AI and the web. A Python CLI and library that lets an AI agent read and search the internet, fronted by a fully open-source "unlocker" that gets through common bot-walls with no paid proxy and no API key.

▶ Watch the full 1-minute demo

Why searchts?

• Reads pages behind common bot walls
• Reads complete ChatGPT / Claude / Gemini / Grok / Poe / DeepSeek / Perplexity / Copilot shared conversations
• Works with Claude, Codex, and MCP agents
• Extracts clean Markdown, ready to feed a model
• Searches the web without API keys
• Downloads a page's assets (images, fonts, palette)
• Transcribes videos, subtitles-first

Why it's free

AI agents constantly need to read web pages, but the naive way they fetch is trivially blocked by modern anti-bot systems (Cloudflare, PerimeterX, DataDome). Paid unlocker services solve this, but the thing they really charge for is a large pool of clean residential IP addresses. searchts runs on your own machine, from your own connection, at personal volume, so it sidesteps that cost and gets through most of those walls for free.

The unlocker

searchts reads any URL through an escalating ladder and stops at the first tier that returns real content:

1. curl_cffi: a fetch that impersonates a real Chrome's TLS/JA3 and HTTP2 fingerprint. Beats user-agent and fingerprint filters. Fast, local, private.
2. Jina Reader: a JavaScript-rendering relay (r.jina.ai), for pages that only fill in content after running JS. Default on — the target URL is sent to Jina's servers on this rung. Opt out with SEARCHTS_NO_JINA=1 or config jina: false (local curl + stealth only).
3. stealth browser: an undetected headless Chromium (patchright), launched lazily only when the cheaper tiers fail, for live JS / Cloudflare managed challenges.

If no tier comes back with real content, an optional human-in-the-loop step opens a real browser so you can clear the page once and continue. That covers interactive CAPTCHAs and soft walls alike: a login page served as HTTP 200 is not a challenge, but it is still a page only a human gets past. Block detection is phrase-based (not vendor-name based), so legitimate pages that merely embed a bot-sensor script are not falsely rejected. Content is extracted to clean Markdown with trafilatura.

Walls (F12 playbook, not a bypass): fail loud on login/challenge/thin. Do not cut a release that claims Reddit/LinkedIn now read (N7). Order: stealth already retries page.content after a navigation race (P3.11) → next is a persistent Chromium profile so clearance can survive across reads (F1, not shipped) → then --human / device session for extras only (F7, never silent, never inside read_url). Never paid residential as default (N1). Never a keyed commercial unlocker as default (N3).

AI-chat share links

Share links from AI chat apps are a special kind of hard: the conversation never appears in the page HTML as extractable text, so generic readers (and most AI agents' built-in fetch) return an empty shell or a fragment cut off mid-chat. searchts read recognizes these URLs and decodes each provider's own data channel instead, returning the complete conversation as role-labeled Markdown — keyless, no login:

The first five need no browser. The last three are JavaScript shells with nothing in the initial HTML, so those reuse the stealth tier: wait for the conversation to render, auto-scroll until the page height stops changing (list virtualization will otherwise truncate a long chat), then expand the collapsed sections before reading. The benchmark currently covers the five that read without a browser and passes all five; the three that need one are not in it yet.

ChatGPT issues two shapes: /share/<uuid> for a whole conversation, and the newer /s/<prefix>_<id> short links for a single shared turn (t_ thread, m_ message, dr_ deep research, cd_ Codex). Both are read.

Each provider is a drop-in plugin module (searchts/share_extractors/); if a provider changes its format, extraction falls back to the normal unlocker ladder instead of failing.

Install

Keep it (global isolated CLI, MCP extra included):

pipx install "searchts[mcp]"

Try it without installing (one-shot, copy-paste):

uvx --from "searchts[mcp]" searchts <verb>

venv / packaging only (not the recommended path for the CLI):

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install "searchts[mcp]"
# optional stealth-browser extra, still venv-only:
# pip install "searchts[browser]" && patchright install chromium

Quickstart

searchts read https://en.wikipedia.org/wiki/Ada_Lovelace   # fetch a page as clean Markdown
searchts search "open source vector db"    # multi-provider web search (keyless by default)
searchts transcribe https://youtu.be/...   # transcript of a YouTube/TikTok/Instagram/Reddit video
searchts grab https://example.com          # download a page's assets + extract palette/fonts
searchts get https://example.com/logo.png  # download one asset (image/PDF/font/file)
searchts doctor                            # see what is configured and working

read flags: --json, --backend <tier>, --human (hand off a CAPTCHA or login wall to a real browser), --scrub (redact injection). search flags: -n <count>, --json, --provider <name>. Content goes to stdout (pipeable); status to stderr. grab flags: --out <dir>, --kinds <images,icons,css,fonts,svg>, --read (also save page.md), --max <n>, --json.

Use it from your AI agent

Add searchts to your agent in one line - as an MCP server, or as a Claude Code slash command:

Two ways, both one command:

# 1) MCP: always-on read_url + web_search + fetch_asset + grab_site + get_status
# Try / no install / Claude cannot see PATH:
claude mcp add searchts -- uvx --from "searchts[mcp]" searchts mcp serve
# Keep (after pipx install "searchts[mcp]"):
#   claude mcp add searchts -- searchts mcp serve
# Desktop / Cursor JSON: `searchts mcp install`  (or uvx the same serve command)
# First read: Wikipedia — example.com is thinner than _MIN_CHARS and looks like a failed install.

# 2) Slash command: type /searchts <url-or-query> in Claude Code
searchts skill install        # writes ~/.claude/commands/searchts.md

See the MCP server reference for all five tools (read_url, web_search, fetch_asset, grab_site, get_status), their inputs and outputs, and when to use each.

Features

• Escalating open-source unlocker: curl_cffi, then Jina Reader, then a stealth browser.
• Multi-provider search with rank fusion: DuckDuckGo (keyless default), plus SearXNG, Exa, Brave, and Tavily when configured; results merged with reciprocal rank fusion and de-duplicated.
• Video transcription: yt-dlp audio plus Whisper for YouTube, TikTok, Instagram, and Reddit videos.
• Asset + design grabber: searchts grab <url> downloads a page's images/icons/css/fonts and extracts a color palette plus the fonts in use; searchts get <url> pulls a single asset. Both go through the same escalating unlock ladder, so they work on fingerprint-gated CDNs, not just open ones.
• Prompt-injection scrubbing: strips invisible/bidi characters, flags injection indicators, optional redaction, so untrusted page content is safer to feed a model.
• Per-domain backend memory: remembers which tier worked per domain and tries it first (SEARCHTS_NO_MEMORY=1 to disable).
• Jina opt-out: the JS-render relay is on by default; SEARCHTS_NO_JINA=1 (or jina: false in ~/.searchts config) skips it so URLs never hit r.jina.ai.
• Surfaces: a CLI, an MCP server (read_url, web_search, fetch_asset, grab_site, get_status), and a Python library.

Use as a library