Ketch Cli README

ketch

A stateless CLI for web search, code search, library docs, and scraping — one binary, no daemon, no API server to run.

Why ketch

Most research tooling for agents means wiring up several provider SDKs, each with its own auth and response shape. ketch collapses that into one binary with three research surfaces:

• ketch search — web search, no API key required (Brave, DuckDuckGo, SearXNG, Exa, Firecrawl, Keenable, Tavily, TinyFish, Parallel, SerpBase, Serply, or You.com)
• ketch code — grep real OSS source across public repos (Grep, Sourcegraph, or GitHub Code Search)
• ketch docs — curated, version-aware library documentation (Context7)

Plus ketch scrape and ketch crawl to turn HTML pages and text-based PDFs into clean markdown.

It's built for two audiences at once:

• Humans, who want a fast terminal tool for the same job curl | pandoc or a browser tab would otherwise do.
• AI agents, who want structured, predictable output (--json everywhere), documented exit codes for control flow, and a single ketch config call to discover what backends are active — no environment probing, no per-provider glue code.

An operator configures the backend once (ketch config set backend searxng); every agent invocation afterward just calls ketch search or ketch scrape without knowing or caring which provider is behind it.

Install

# macOS / Linux, any of x86_64 or arm64
curl -fsSL https://ketch.run/install | sh

# Homebrew
brew install ketch

# npm
npm install -g ketch-cli

# go install
go install github.com/1broseidon/ketch@latest

# Or download a prebuilt binary (linux/darwin/windows, amd64/arm64)
# from https://github.com/1broseidon/ketch/releases

The install script picks the right build for your OS and architecture, verifies it against the release's checksums.txt, and drops the binary in /usr/local/bin when that's writable, otherwise ~/.local/bin. Read it first if you'd rather not pipe to a shell: install.sh. Pin a version or change the target directory with sh -s -- --version v0.17.0 --bin-dir ~/bin.

Quickstart

$ ketch scrape https://go.dev/doc/effective_go
---
url: https://go.dev/doc/effective_go
title: Effective Go - The Go Programming Language
words: 16582
---
1. [Documentation](https://go.dev/doc/)
2. [Effective Go](https://go.dev/doc/effective_go)
...
## Introduction

Go is an open-source programming language that focuses on simplicity, reliability, and efficiency...

Search real OSS code with zero configuration:

$ ketch code "http.NewRequestWithContext" --lang go --limit 2
---
query: http.NewRequestWithContext
lang: go
backend: grepapp
result_count: 2
---
harness/harness  registry/app/remote/clients/registry/client.go  (line 207)
  req, err := http.NewRequestWithContext(ctx, http.MethodGet, buildPingURL(c.url), nil)
  https://github.com/harness/harness/blob/main/registry/app/remote/clients/registry/client.go
...

Web search works with zero configuration too — the default backend (auto) falls back through the keyless providers, so there is no key to set first:

ketch search "golang error handling"
ketch search "golang error handling" --scrape   # fetch + extract full content per result
ketch search "golang error handling" --multi    # federate across every usable backend, rank-fused
ketch search "golang error handling" --random  # pick one random backend, fallback to rest on failure

auto tries providers in a fixed order and returns the first that answers, reporting which one served in the backend: field. It prefers whatever you have actually configured — your own SearXNG or Degoog instance first, then any provider you have set a key for — and only then the keyless hosted providers. So setting a key is still how you get a specific provider and higher limits, and you do not have to also set backend for it to take effect:

ketch config set brave_api_key <key>   # auto now prefers Brave
ketch search "golang error handling" -b ddg   # or pick a provider explicitly

--multi queries several backends at once and fuses their rankings with Reciprocal Rank Fusion (a page several engines rank highly floats to the top), deduplicating by URL and tagging each result with the engines that returned it. --random shuffles the backend list, tries one, and falls back to the rest — ideal when you want one provider's results without wasting rate limits on all of them. Both support bare (all usable backends) or =brave,exa explicit lists, and both are mutually exclusive with --backend and each other. See the command reference.

Every command takes --json for structured output:

ketch scrape https://example.com --json
# {"url":"https://example.com","title":"Example Domain","markdown":"..."}

Pipe any HTML through ketch's extraction pipeline without a fetch:

curl -L https://chain.sh/ketch | ketch extract
cat page.html | ketch extract --select article --max-chars 4000

Extraction

Extraction reads the page's own structure: the content landmark it declares (main, article), a document assembled from uniform sections, or the smallest element holding its prose. Site furniture is removed by what it is — navigation, hidden and collapsed controls, link rails, tables of contents — and readability is the fallback for a page that declares no structure. The extract_mode config key sets what pruning may drop: clean (the default) also drops blocks by name and phrase — related-post rails, comment threads, share bars, "was this helpful?" boxes — for the leanest markdown, and complete keeps everything the structure does not condemn, trading a little precision for the last fraction of recall. Pages served in a legacy encoding are decoded before extraction.

PDF extraction

ketch scrape detects PDFs from the response MIME type or %PDF- signature and extracts their text with a built-in pure-Go parser:

ketch scrape https://example.com/report.pdf

Scanned/image-only PDFs need OCR and return a precondition error with an OCR-converter hint in the built-in path. Operators can configure an external PDF-to-Markdown converter that writes Markdown to stdout (capped at 10 MiB); its shlex-parsed command must contain exactly one {input} placeholder:

ketch config set external_pdf_to_md_converter_command 'pdftotext "{input}" -'
ketch config set external_pdf_to_md_converter_timeout_sec 300

When configured, the external converter is authoritative: failures are returned rather than silently falling back to the built-in parser. PDF binary output is never emitted: --raw and --select reject PDFs as validation errors. With --force-browser, PDF markdown still uses text extraction and never opens Chromium's PDF viewer.

Tags — bookmarks for agent workflows

Save useful sources while you research a project, then find them again in a later session. Tags group documentation, code references, search results and write-ups under the same project or topic. One source can belong to several tags.

ketch search "guacamole ldap authentication" --scrape --tag remote-access
ketch code "guacamole ldap" --tag remote-access
ketch docs "apache guacamole" --tag remote-access
ketch tag add remote-access https://example.com/read-this-later  # no network

ketch tag show remote-access
ketch tag show remote-access --limit 10 --json
ketch tag show remote-access --limit 0       # all entries
ketch tag remove remote-access https://example.com/read-this-later

--tag works on search, code, docs, scrape and crawl, including --no-cache. A bookmark keeps the source URL, a bounded title and description, and the time it was tagged. Descriptions come from search results, snippets or fetched content; full bodies belong to the separate page cache.

tag show returns the newest 50 entries by default, ordered by tagging time and then URL for ties. Use --limit N to change that, or --limit 0 for all. Text output says showing N of M when limited; --minimal keeps its TSV output and puts that notice on stderr. JSON and MCP return entries and cached totals for the whole tag, shown for the returned pages, and cache_status. The MCP tag tool accepts the same limit on operation: show.

Read the index, choose a source, and scrape its URL. cached: false usually means there is no fresh local body, not that the link is broken. If the page cache is locked or unreadable, bookmarks still work: cache_status: unavailable means the cached: false flags are unverified. Fetches can proceed without the page cache. A bookmark does not guarantee an upstream page still exists.

Bookmarks survive cache expiry and cache clear. Clear removes page bodies and deletes the cache file, returning the space; expired pages are also swept automatically, so the file stays near the size of what is fresh. tag add accepts absolute http(s) URLs without fetching them (anything else is bad input, exit 2), preserves existing metadata on a cold re-add, and fills missing titles and descriptions on a later fetch. Removing a bookmark uses the displayed URL, regardless of cookies, User-Agent or URL rewrite settings. Nothing expires bookmarks; use tag remove to tidy up — with URLs, it reports the ones that were not under the tag as missing.

The independent tags.db is opened only for short index operations, so an idle MCP server or a background crawl does not monopolize it. Its default location uses the operating system's configuration directory:

Set KETCH_TAGS_PATH to override the complete filename. Isolated labs must set it as well as their page-cache override; XDG_CACHE_HOME alone no longer isolates bookmarks. This replaces the unreleased in-cache tag layout; existing experimental tag buckets are left untouched and are not automatically imported.

A failed explicit tag write returns exit 5 / MCP [precondition]. If saving a bookmark fails during research, the research result is retained: the CLI emits a stderr warning (warning.code: tag_write_failed with --json), and MCP adds warnings to the result. Research success alone does not confirm bookmark persistence.

Commands

After configured URL rewrites, scrape and crawl fetch only absolute http:// and https:// URLs; other final schemes such as file:// are rejected. Private or internal HTTP(S) addresses remain fetchable when reachable from the machine running ketch.

Every command supports -h/--help for its full flag list; --json is the only flag global to every command. Full flag reference lives at ketch.run.

Backends

Why it works well for agents

• Stateless, single binary. No daemon, no server to keep alive — call, get a result, exit.
• Documented exit codes, not just stderr text, for scripted control flow: 2 bad input, 3 not found, 4 upstream/network failure, 5 missing precondition (e.g. no API key), 6 cancelled (SIGINT/SIGTERM).
• Automatic JS-rendering fallback. ketch scrape and ketch crawl detect JS-shell pages (React/Vue/Svelte SPAs, streaming hydration frameworks) and transparently re-fetch via headless Chrome when needed — same output shape either way.
• Smart input detection on scrape. Single URL, multiple positional args, a JSON array, a file of URLs, or stdin — no --batch flag required.
• Page cache. Fetches are cached (bbolt, default TTL 72h); repeat scrapes return cached pages, and repeat crawls can reuse entries with known link metadata. Crawl entries keep resolved outgoing HTTP(S) links—including external links and query strings—locally for the same TTL; this metadata does not require or store full HTML, and is not sent elsewhere. A deeper crawl may fetch a still-cached page when its links are unknown; that can recur after a normal scrape refreshes the entry. If backfill fails, the cached page is still returned with an incomplete-link diagnostic (CLI stderr; MCP pages plus errors). Background crawl status counts the returned page and error but does not retain individual error details. --no-cache bypasses the cache.
• One discovery call. ketch config returns the full effective configuration and active backends as JSON, so an agent can inspect capabilities without parsing --help text.

Configuration

ketch reads defaults from ~/.config/ketch/config.json. Flags always override config values.

ketch config init                          # write a default config file
ketch config set backend searxng           # set a default backend
ketch config set searxng_url http://my-searxng:8080
ketch config set browser chrome            # enable browser fallback for JS-rendered pages
ketch config                               # print effective config + available backends
ketch config path                          # print the config file path

Environment variables

Every scalar config key can also be set through the environment as KETCH_ + the upper-snake key name — handy for containers and CI where writing a config file is awkward:

KETCH_BRAVE_API_KEY=<key> ketch search "query"     # keys/tokens without touching disk
KETCH_BACKEND=ddg KETCH_LIMIT=10 ketch search "query"
KETCH_CONFIG=/etc/ketch/config.json ketch config    # point at an alternate config file

Precedence is CLI flag > KETCH_* env > config file > built-in default. Notes:

• Per-provider key vars (KETCH_BRAVE_API_KEY, KETCH_EXA_API_KEY, KETCH_FIRECRAWL_API_KEY, KETCH_KEENABLE_API_KEY) accept a comma-separated list and replace the provider's whole key pool (there are no *_API_KEYS plural vars).
• KETCH_GITHUB_TOKEN beats the config file, which beats an ambient $GITHUB_TOKEN/$GH_TOKEN.
• url_rewrites and spa_markers are file-only (their JSON/regex values don't survive env quoting).
• ketch config (show) reports an env_overrides section, so you can always see which effective values came from the environment; ketch config set writes only file values and never persists env-derived ones.
• Invalid env values (e.g. KETCH_LIMIT=abc) fail loudly on commands that use config, naming the offending variable; ketch version and ketch config set/path still work.
• A config file that isn't valid JSON or has a wrong type for a key (e.g. "limit": "5") fails the same way (exit 2), naming the key and the type it needs; ketch config set refuses to overwrite it, so fix it by hand. None of the file is applied until it decodes.
• Secret KETCH_* vars are stripped from the environment of spawned subprocesses (headless browser, external PDF converter).

Other configurable keys include per-backend API keys (brave_api_key, brave_api_keys for multi-key rotation, exa_api_key, firecrawl_api_key, keenable_api_key, parallel_api_key, tavily_api_key, tinyfish_api_key, serpbase_api_key, serply_api_key, youcom_api_key, context7_api_key, github_token), firecrawl_url / sourcegraph_url / degoog_url (self-hosted overrides), http_headers (per-origin headers, such as a Cloudflare Access service token, for self-hosted SearXNG, Firecrawl, Degoog, and Sourcegraph instances), cache_ttl, url_rewrites (regex rewrite rules applied before fetch), spa_markers (extra JS-shell detection tokens), cookie_file (see below), user_agent (User-Agent override for HTTP and browser fetches; setting one scopes cached pages to it, so entries cached under the default stay valid), extract_mode (clean by default, or complete; see extraction — a non-default mode scopes cached pages the same way), and the optional external PDF converter command/timeout. Multiple keys per provider are picked randomly per request to spread rate limits. See the config reference for the full list.

Cookies (BYO cookies.txt)

Some pages only render their real content once a session or consent cookie is present — a logged-in dashboard, or a consent-banner wall (e.g. Nvidia NGC docs) that shows only the banner to an anonymous headless browser. ketch can attach your own cookies to every fetch, on both the HTTP and browser paths.

Supply a jar in the Netscape cookies.txt format — the same format exported by browser "cookies.txt" extensions and consumed by curl and yt-dlp:

ketch scrape https://catalog.ngc.nvidia.com/... --cookie-file ~/cookies.txt
ketch search "query" --scrape --cookie-file ~/cookies.txt
ketch crawl https://internal.example.com --cookie-file ~/cookies.txt
ketch config set cookie_file ~/cookies.txt          # persist as a default

• The --cookie-file flag overrides the cookie_file config value; an explicit empty flag (--cookie-file "") disables cookies for that run.
• Only cookies whose Domain, HostOnly, Path, and Secure scope matches the request URL are sent; scope is re-evaluated on every redirect, and expired entries are rechecked before every request.
• Cookie values are never printed — not in output frontmatter, --json, errors, or ketch doctor. ketch doctor reports configured (N cookies, M expired) and nothing more. Keep the jar at chmod 600; ketch warns on stderr if it is group/world-readable.
• A configured jar uses a jar-specific page-cache namespace, including before redirects, so authenticated content cannot collide with anonymous cached copies. Authenticated content is still cached locally: on POSIX, ketch protects the cache directory/database as 0700/0600; use --no-cache if it must not be stored.

Responsibility: respecting a site's Terms of Service and using only your own session cookies is entirely the operator's responsibility.

Agent integration

Point an agent's system prompt at ketch instead of teaching it individual search/scrape APIs:

Use `ketch` for external research — web pages, OSS code, library docs.
- `ketch search "query"` / `ketch search "query" --scrape` for web results with optional full content (add `--multi` to federate across backends and rank-fuse)
- `ketch scrape <url> [url...]` for clean markdown from one or more URLs
- `ketch extract` for already-fetched/piped HTML (`curl ... | ketch extract`) — no fetch, no cache, no browser
- `ketch code "query" --lang go` for real OSS code with repo/line context; `--repo owner/name` searches one repository
- `ketch docs "query" --library /org/repo` for version-aware library docs
- All commands support `--json`. `ketch config` reports active backends.

The operator configures backends once; the agent's prompt never needs to mention which provider is behind ketch search.

For a fuller agent playbook — surface routing, token budgets, error-code control flow, a deep-research recipe, and guided backend setup — install the bundled skill from skills/ketch/ (works with any agent that loads SKILL.md-style skills, e.g. Claude Code).

MCP server

For agents that speak MCP instead of shelling out, ketch mcp serve runs the same surfaces — search, code, docs, scrape, crawl, plus tag — as MCP tools over stdio, using the same config and backends as the CLI. To register it with Claude Code:

claude mcp add ketch -- ketch mcp serve

# or, with no install step at all:
claude mcp add ketch -- npx -y ketch-cli mcp serve

The npm package carries the binary in a per-platform dependency, so npx runs it without a postinstall download — which keeps the server's cold start quick when a client relaunches it.

Tool errors carry the exit-code taxonomy as stable message prefixes: [validation], [not_found], [upstream], [precondition], [cancelled].

Claude Code plugin

The CLI on PATH is all ketch needs — but if you want Claude Code to wire up the MCP server and the bundled skill in one step, this repo doubles as a plugin marketplace:

claude plugin marketplace add 1broseidon/ketch
claude plugin install ketch@ketch

The plugin registers ketch mcp serve as an MCP server and installs the ketch research skill. It does not bundle the binary: ketch >= v0.10.0 must be on PATH (brew install ketch or go install github.com/1broseidon/ketch@latest).

pi package

For pi, which does not speak MCP, the repo is also a pi package: the same bundled skill plus ketch_search, ketch_code, ketch_docs, ketch_scrape, ketch_crawl and ketch_tag tools that shell out to ketch … --json, with the same per-call options and error prefixes as the MCP tools, and a /ketch-doctor command:

pi install git:github.com/1broseidon/ketch

Like the Claude Code plugin it never bundles the binary — ketch must be on PATH. Details in plugins/pi/.

Contributing

Bug reports, documentation, and improvements are welcome. See the contribution guide for pull request guidelines and provider admission criteria. Please discuss new providers in an issue before implementing them.

The extraction benchmark measures content preservation and CLI latency across 500 pinned pages from 70 websites; it lives in its own repository and builds a sibling ketch checkout. Known extraction misses remain visible; passing its regression check is not a release approval.

Sponsors

ketch is free and MIT-licensed. If it saves you or your team time, you can sponsor its development. Sponsors never get product changes, default settings, or vendor-specific code.

License

MIT