# Search that shows its work
**Category:** ๐ป Developer Tools
**Repository:** https://github.com/AG-Bureau/mcp-search
**Views:** 0
**Installs:** 0
**Upvotes:** 0
**Directory Page:** https://allmcps.com/mcp/search-that-shows-its-work
## Description
Self-hosted web search that reports how much of each answer to believe
## Claude Desktop Quick Installation
Heuristic fallback โ verify the package name and runner against the repository README before running it. Uses `npx` (confidence: low):
```json
"mcpServers": {
"search-that-shows-its-work": {
"command": "npx",
"args": ["-y","search-that-shows-its-work"]
}
}
```
## Documentation & README
# search
**A self-hosted MCP server for web search that reports how much of each answer to believe**
[](https://registry.modelcontextprotocol.io)
[](LICENSE)
[](adapter/Dockerfile)
[](#-install)
[](https://glama.ai/mcp/servers/AG-Bureau/mcp-search)
---
A search tool fails in ways that look exactly like success. An engine answers with
somebody else's subject. A page returns text that is an anti-bot shield. Sixteen
sources turn out to be two engines counted eight times. None of that raises an
error, and the model on the other end builds on it.
**This server's job is to make those cases distinguishable, in fields you can
branch on.** It runs on your machine, over your own metasearch instance, with your
own model key โ or none at all.
## ๐ง Tools
| Tool | What it does | Required | Notable options |
|---|---|---|---|
| `web_search` | Finds pages **and reads the top ones** โ one call, links with their text | `query` | `read: false` for links only ยท `read_top` how many to read ยท `min_engines` to force breadth ยท `engines` to name the engines yourself |
| `web_read` | Reads pages by address: text, PDF, or a scan recognised by a vision model | `urls` | `mode: browser` for JS-rendered pages ยท `expect` to assert what must be there ยท `offset` to continue |
| `web_image_search` | Finds images: the address of the FILE and, separately, of the page it sits on | `query` | `max_results`, `page` |
| `web_screenshot` | A PNG of a page **plus its text from the same visit**, so the two can be cross-checked | `url` | `max_chars` for how much text ยท `full_page` ยท `expect` |
| `web_deep_search` | Composes its own queries, reads in waves, and answers from several sources โ saying what it could not confirm | `question` | `waves` |
Full argument reference, response shapes and failure modes: **[HOWTO-CALL.md](https://github.com/AG-Bureau/mcp-search/blob/HEAD/HOWTO-CALL.md)**.
## ๐ฆ Install
From an open repository page to a working answer. Nothing is assumed to be on
your disk already:
```bash
git clone https://github.com/AG-Bureau/mcp-search
cd mcp-search
cp .env.example .env
echo "SEARXNG_SECRET=$(openssl rand -hex 32)" >> .env
echo "BROWSER_WS_SECRET=$(openssl rand -hex 16)" >> .env
echo '{}' > door-tokens.json && mkdir -p tls
docker compose -f docker-compose.yml -f wiring/expose-localhost.yml up -d --build
curl -s http://127.0.0.1:8081/healthz
```
The fourth and fifth lines are not decoration. Without a value in
`SEARXNG_SECRET` the `up` command refuses โ and that refusal is deliberate: with
no key of its own the metasearch does not fail, it comes up with a publicly known
one from its image template, silently. `BROWSER_WS_SECRET` is the secret path of
the browser sidecar's endpoint, and `up` refuses without it too: on a guessable
path, anything that can open a WebSocket to the sidecar โ any container on the
compose network, and any page the browser itself opens โ gets a browser of its
own, outside every check the module applies to the pages it reads. The compose
file hands the same value to the sidecar and, at the end of `BROWSER_WS_URL`, to
the adapter; neither prints it.
The sixth line creates the two paths the compose file mounts for caller
authentication and TLS โ both off by default. A path that does not exist is
created by Docker as a root-owned DIRECTORY, and a token "file" that is a
directory reads as an unreadable list (see Configuration below).
Both are in `.gitignore`: they hold secrets once you use them.
The overlay publishes the port **on loopback only**. A published container port
does not go through the host firewall's usual chain, so exposing it more widely
is a separate, deliberate step โ see [Deployment](#-deployment-and-exposure).
### Two transports
MCP has two, and they answer different questions. **HTTP** โ the commands above โ
is for a server that is already running somewhere. **stdio** is the protocol's
default: the client starts the server as a process and talks to it through the
pipes, which is how most desktop clients and wrappers work.
```bash
docker compose exec -T ag-search python /app/server.py --stdio # or MCP_TRANSPORT=stdio
# outside compose: SEARXNG_URL=http:// python adapter/server.py --stdio
```
One JSON-RPC object per line in, one answer per line out. The mode is chosen
explicitly and never guessed from whether a terminal is attached โ that sign
merely sits next to the subject, and one day it answers for a case nobody meant.
In stdio mode **stdout is the protocol**: answers and nothing else, with the log
on stderr. One stray line of anything else breaks the client reading it.
**Each process started this way is an instance of its own**, even inside the
running container: its browser sessions (`BROWSER_MAX_SESSIONS`), its engine and
domain pacing and its `/stats` counters are separate from the HTTP server's.
Several stdio clients on one sidecar add up in its `pids_limit` like several
adapters do.
The sidecars do not depend on the choice; the metasearch does: `SEARXNG_URL`
must reach one, or every search refuses with `cause: upstream` (the default,
`http://searxng:8080`, is a name only the compose network knows). Started by a
client with no compose project around it, the module names what else is missing
instead of pretending: the browser path reports `not_wired_up`, and `trouble`
carries `pool_unmeasured` โ the engine pool was never computed from observation.
## โฌ๏ธ Upgrading to 0.3.5 โ no new variable
From 0.3.4: the same command as the install block, with the same `-f` files.
Only the adapter image changes (`ag-mod-search/adapter:0.3.5`, used by the
adapter and the prober); the sidecar stays `ag-mod-search/browser:1.49.1-3`.
Two things you may notice: the door now logs one line per request (the
caller's address, the path, the status and the caller's token name โ never the
token, the body or the query string), and a model call that runs out of the
time its caller had now returns a refusal that says so instead of running on.
From 0.3.3 or earlier, the 0.3.4 steps below apply too.
## โฌ๏ธ Upgrading to 0.3.4 โ no new variable, one stricter one
From 0.3.3: the same command as the install block, with the same `-f` files.
Only the adapter image changed; the sidecar stays `ag-mod-search/browser:1.49.1-3`.
Check `AG_DOOR_AUTH` first: a value that is not `off`, `warn` or `on` (in any
case, spaces around it ignored) now stops the adapter from starting, where it
used to mean `off`. From 0.3.2 or earlier, the 0.3.3 steps below apply too.
## โฌ๏ธ Upgrading to 0.3.3 โ one new required variable
`BROWSER_WS_SECRET` must be in `.env` before `up`: add it as the install block
does, with `openssl rand -hex 16`. Then run the install block's own command,
with the same `-f` files you installed with:
`docker compose -f docker-compose.yml -f wiring/expose-localhost.yml up -d --build`.
It rebuilds both images and recreates the adapter, the prober and the sidecar,
since the image or the environment of each changed. Name no services on it:
`up โฆ browser ag-search` leaves the prober on the old image, and leaving out
the `-f` overlay recreates the adapter without its published port. The sidecar
image is `ag-mod-search/browser:1.49.1-3`; with the old sidecar, or with only one
side recreated, the browser path reads "not responding".
**A malformed secret stops the whole module.** Compose checks only that the value
is not empty. Characters outside `A-Z a-z 0-9 _ -` (base64 gives `+ / =`) make the
sidecar exit with 78 in a loop; the adapter waits for a healthy sidecar, so it
never starts and `up` says "dependency failed to start".
If you run the door in `warn` or `on`, also read "Moving to `on`" under
Configuration: the prober now takes a token, a token with no scopes (or with
`scopes` written as a string) may call no tool, and `/healthz?deep=1` needs a
token that may search or read.
## โฌ๏ธ Upgrading from 0.2.x โ the answer changed shape
**If you already run 0.2.1 or earlier, read this before updating.** Nothing here
is a new feature you may ignore; it is what your existing calls will return
differently.
**The search and image answers carry fewer fields by default**, and they say so
in their name: `ag.search/3` and `ag.images/3` instead of `/2`. A caller that
branched on `contract` will break loudly, which is the intent โ a field that
simply vanished would read as "nothing was wrong" in most languages.
| in 0.2.x | in 0.3 |
|---|---|
| `search_aborted`, `engines_unasked` | `trouble.search_aborted`, `trouble.engines_unasked` |
| `unresponsive_engines` | `trouble.unresponsive_engines` |
| `engines_irrelevant` | `trouble.engines_irrelevant` |
| `pool_source: "seed"` | `trouble.pool_unmeasured`, with the reason |
| `arguments_adjusted` | `trouble.arguments_adjusted` |
| `corroborated_by_url`, `corroborated_by_domain` โ always present, `1` on the cheap path | absent when only one engine found results: there `1` meant "nobody else was asked", not a measurement |
| `count`, `query`, `page`, `read`, `read_top`, `pages_*`, `timing_ms`, `engines_skipped`, `engines_used`, `tiers_used`, `pool_source`, `pool_reason` | returned when you ask: `verbose: true` |
`trouble` is always present and empty when nothing went wrong, so `if not
trouble` replaces the four separate checks. Nothing was deleted from the module โ
the accounting moved behind a request.
**Booleans are now parsed rather than cast.** In 0.2.x `"read": "false"` over MCP
read the pages anyway โ eight times the wall clock โ while the plain door
understood the same word correctly. Both doors now accept `true/false`, `1/0`,
`yes/no`, `on/off`, `y/n`, `t/f` and Python's `True/False`, case-blind. **A value that cannot be read
turns the flag off and is named in `arguments_adjusted`**, and an EMPTY value
counts as unreadable: `read=""` no longer buys the expensive default in silence.
**`read_top: 0` means "no preference"**, not "read nothing" โ for nothing, use
`read: false`. The schema used to declare `minimum: 1` while accepting zero.
The field-by-field table is in the contracts:
[`contracts/ag.search.v3.md`](https://github.com/AG-Bureau/mcp-search/blob/HEAD/contracts/ag.search.v3.md) and
[`contracts/ag.images.v3.md`](https://github.com/AG-Bureau/mcp-search/blob/HEAD/contracts/ag.images.v3.md), section "What changed
against `/2`".
Reading (`ag.read/2`), deep search (`ag.deep/2`) and screenshots (`ag.shot/1`)
kept their numbers: they gained fields, and an addition breaks nobody.
## โ๏ธ Configuration
| Variable | Required | What it is |
|---|---|---|
| `SEARXNG_SECRET` | **yes** | Session key for the metasearch. Any long random string that is not from somebody's history. |
| `BROWSER_WS_SECRET` | **yes** | The browser sidecar's secret endpoint path, 16-256 characters of `A-Z a-z 0-9 _ -` (`openssl rand -hex 16`). The sidecar exits at start without it, and the compose file refuses first. A value with other characters passes the compose file and stops the whole module: the sidecar exits with 78, and the adapter, which waits for a healthy sidecar, never starts. To rotate, change it and run the same `up -d` as at install, with the same `-f` files: both `browser` and `ag-search` are recreated, because the environment of both changed. |
| `LLM_API_KEY` | no | Key for any OpenAI-compatible endpoint. **Secret.** |
| `LLM_API_BASE` | no | Base URL of that endpoint. Take it from your provider's documentation, not by analogy โ the obvious guess can answer with a billing error such as `429: Insufficient balance`, because some providers serve a subscription on a different path of the same domain. |
| `LLM_MODEL_TEXT` | no | Model that plans queries and composes answers. No default is shipped: a default would silently ask your provider for a model it may not have. |
| `LLM_MODEL_VISION` | no | Model that reads scanned PDFs. Unset, such documents return an explicit refusal naming the reason. |
| `LLM_DISABLE_THINKING` | no | Set for providers whose reasoning budget swallows the answer, leaving it empty with `finish_reason: length`. |
| `LLM_MAX_CONCURRENT` | no | How many model calls may be in flight at once, ours to hold rather than the provider's to refuse (default 20). A provider enforces its limit by REFUSING the extra call โ after the queries were composed and the pages read, where it costs the most; waiting a turn at the start costs a pause. Since 0.3.5 the wait is bounded: a call that finds no free slot within its own time is refused by us (`no free model slot`) without calling the model, a scan is then `not_recognised`, and deep search answers `busy`. Set it BELOW your provider's stated ceiling: the provider counts the account, and one key often serves several instances. |
| `LLM_RATE_LIMIT_WAIT_S` | no | How long to wait when the provider answers 429 without a usable `Retry-After` โ absent, zero, negative or a date (default 5, never less than 1). A retry after zero seconds is the same request again. |
| `LLM_RATE_LIMIT_CODES` | no | Your provider's own error codes that mean "too many" besides HTTP 429, comma-separated. Compared exactly against `error.code` (or a top-level `code`) in the error body; the body's words are never searched. Empty, the default: only 429 counts. Any 429 counts โ a billing refusal a provider sends as 429 is reported as `busy` too, with the provider's words in `error`. |
| `DEEP_RATE_LIMIT_MAX_WAIT_S` | no | The longest wait deep search will sit through for its one retry after a rate limit (default 10); a longer wait answers `cause: busy` instead. |
| `AG_DOOR_AUTH` | no | Who may call: `off` (default, no check โ what the module always did), `warn` (serve the tools, but count the calls that arrive without a usable token โ except `/healthz?deep=1`, which `warn` refuses too, see below), `on` (refuse them: 401 with no token, 403 when the token may not call that tool). Case and surrounding spaces do not matter; unset or empty is `off`. **Any other value stops the server from starting**, with the value and the three allowed ones in the message: before 0.3.4 a typo such as `enforce` or `0n` silently meant `off`. Move off โ warn โ on: a door that starts in `on` cuts off the callers you have not given tokens to yet. |
| `AG_DOOR_TOKENS_IN_CONTAINER` | no | The token list's path INSIDE the container. The compose file mounts `./door-tokens.json` at `/run/door-tokens.json`, so set `/run/door-tokens.json`; unset, no token list is read. The file is `{"": {"name": "...", "scopes": ["read"]}}`. **Secret, and never in git.** Scopes are tool names, the groups `read` and `paid`, or `*`. **A missing or empty scope list grants nothing, and so does a `scopes` that is a string rather than a list** โ only an explicit `"*"` inside a list grants everything โ and `/stats` lists such tokens by name under `tokens_granting_nothing` (the string ones also under `tokens_scopes_not_a_list`). `paid` is `web_deep_search`, and holding it is also what lets `web_read` spend the vision model on a scanned PDF: in `on`, a token without `paid` still reads, but a scan comes back `not_recognised (scan)` with the missing scope named. The door reads the name and the scopes out of it and never echoes the token โ not in a refusal, not in the counters. Plain `/healthz` stays open in every mode; `/healthz?deep=1`, which runs a real search and a real read, needs in `warn` and `on` a token that may call `web_search` or `web_read` (401 without a token, 403 with one that may do neither). |
| `PROBE_DOOR_TOKEN` | no | The prober's token for the door: one from the token list with the scope `web_read` and nothing else. Needed before `warn` โ without it every read probe counts as `unauthenticated`, so that counter never stops growing โ and in `on`, where the door refuses the probes: the prober then records nothing for them and says why in its log. Empty sends no token. **Secret**: `.env` only. |
| `AG_DOOR_TLS_CERT`, `AG_DOOR_TLS_KEY`, `AG_DOOR_TLS_PORT` | no | TLS on a port of its own, beside the plain one. Paths INSIDE the container โ the compose file mounts `./tls` at `/run/tls` โ and the port to listen on. Set all three or none: any two, or a certificate that will not load, stops the door from starting. The main port always stays plain, so the compose health check keeps working. `wiring/expose-localhost.yml` does not publish the TLS port. |
| `READ_CONTACT` | no | Contact placed in the `User-Agent` when fetching pages. Defaults to this repository; set your own if you run this at scale. |
| `READ_LANGUAGES` | no | `Accept-Language` when reading. Unset by default โ the language of the pages you read is not ours to choose. |
**A token file that cannot be READ is not an empty token file.** Seen on a real
deployment: the file was mounted into the container with mode 600 owned by root
while the container runs as an unprivileged user. The door opened
nothing, refused nothing, and looked correctly configured โ the inability to look
had been handed back as the result of looking. `/stats` therefore carries three
separate fields: `door_auth` (the mode), `tokens_configured` (is a file named at
all) and `tokens_readable` (did it yield at least one token โ a readable `{}`
reads `false` too). An unreadable list refuses
everyone in `on` โ the safe direction for a file we could not read โ and in
`warn` it serves everyone, which is exactly why the two cases must not share one
field. A missing path is worse still: Docker creates it as a DIRECTORY, which
reads like an unreadable file โ so create the file before the first `up` (the
install block above does) and check that it is a file.
**Who owns the secret files.** The adapter runs as uid 10001. Once the token
file holds tokens, and for a TLS key: `chown 10001:10001 door-tokens.json
tls/key.pem && chmod 600 door-tokens.json tls/key.pem`. The tempting fix for the
failure above, `chmod 644`, makes the tokens and the private key readable to
every user of the host.
**Moving to `on`.** Give the prober a token first (`PROBE_DOOR_TOKEN`), then go
`off` โ `warn`, and switch to `on` once `unauthenticated` in `/stats` has stopped
growing โ with the prober on a token, what is left there is callers you have not
given one to yet. Since 0.3.5 the log says who they are: every request prints
one line, `[door] -> in ms, caller:
` (`no token` without one, `unknown token` for one the token list
does not have โ the refusal a caller gets does not tell the two apart, the log
does; `-` where no door verdict was taken: a request refused before it could be
read, a method the door does not serve such as `HEAD` or `OPTIONS`, or a POST
anywhere but `/mcp`), with `?โฆ` where the request had a query string and the
client's own `X-Forwarded-For`, unverified, when it sent one. The token, the
other headers, the body and the query string are never logged.
The browser's session and watchdog limits are set in `.env`; see
[`.env.example`](https://github.com/AG-Bureau/mcp-search/blob/HEAD/.env.example), which explains each one where you set it. Pacing,
pool size and read limits have measured defaults in `adapter/server.py`,
`adapter/reader.py` and `adapter/pool.py`; `docker-compose.yml` does not pass
them through.
**A model key is optional.** Search, reading, image search and screenshots are
HTTP requests and spend no model tokens. A model is called in exactly two places,
and both are named in the answer: `web_deep_search`, and recognising a PDF with no
text layer โ which happens when you ask to read such a document, never behind
your back in `web_search`. `web_deep_search` reads the pages it picks with
recognition on, so a scan among them is recognised as part of that search's own
model spend. With the door `on`, both need a token with the `paid` scope.
## ๐ฏ What a bundled search tool does not do
**Cost you control.** One argument changes the answer by an order of magnitude:
| call | payload | time | model tokens |
|---|---|---|---|
| `read: false`, 6 links | 3.8 KB | 0.6 s | **0** |
| `read_top: 1`, 3 links | 8.8 KB | 1.6 s | **0** |
| `read_top: 3`, 6 links | 9.8 KB | 6.3 s | **0** |
| `web_deep_search` | full account | 36 s | 6 calls |
*Measured on one machine, one query. Take the shape, not the digits.* A consumer
measured the same fork from outside and got 4.8-10.5 s against 0.7 s โ the shape
holds, the digits depend on the pages the query happens to find.
**The choice is made before the call, not after the bill.** `read: false` when
you are mapping what exists or working under a narrow context ceiling; the
default when you want the text of the top results and would otherwise fetch it
yourself. `read_top: 0` means "no preference", not "read nothing".
**The engine list maintains itself.** A hand-written list goes stale in silence:
an engine that was the best returns nothing weeks later and says nothing about it.
Ours was revised three times in a single day โ each revision against the previous
one, each correct on its own data. The problem was never the engines: a decision
freezes while observation goes on.
So the list is not written here. A prober asks every known engine, continuously,
with questions whose correct answer is known in advance, and the pool is the best
few by reference hit share โ recomputed on its own. Verified by falsification: a
planted bad run took an engine out of the pool **with no code change**, and
restoring the run brought it back by itself.
Until enough observation accumulates, the pool is a seed list and every answer
says so: `trouble.pool_unmeasured` by default, `pool_source` itself under
`verbose`.
**Failure is distinguishable from success.** Four ways an engine can fail, and
what shows each:
| how it fails | what shows it |
|---|---|
| answers with a refusal: captcha, rate limit, ban | `trouble.unresponsive_engines` |
| silently returns nothing | the difference between `engines_asked` and `engines_answered` |
| answers a different question | `trouble.engines_irrelevant` โ its results are already discarded |
| substitutes the subject with a better-indexed namesake | `engines_trust`, earned against references |
The same applies to reading: seven distinct outcomes, and a page that returned a
shield is `stub`, not empty text.
## ๐ฉบ If the module stops and nothing looks wrong
One failure of this system does not show up in anything you would normally watch,
so it is written down here rather than left to be rediscovered.
**The symptom.** Containers report unhealthy, `docker compose up` refuses to
raise the adapter, and `docker exec` fails with `runc: nsexec ... setns`. Load,
memory and disk are all fine.
**The check, one command:**
```bash
ps -eo comm | sort | uniq -c | sort -rn | head
```
If `headless_shell` is in the thousands, the browser sidecar has accumulated
processes โ a browser that was opened and never closed keeps its renderers alive,
and past a few thousand the container runtime can no longer start a process at
all. Since both health checks work by `exec`, everything then reports sick at
once, and the cause looks like the last thing you changed.
**The cure:** `docker compose up -d --force-recreate browser`. Measured on our own
instance: 4619 processes before, 126 after.
**What 0.3.1 changed.** The processes were never alive: chromium's children are
orphaned when a visit ends, and a PID 1 that does not call `wait()` leaves each
of them a zombie holding its slot โ forty after ten visits, measured. The sidecar
image now runs a real init that reaps them (and `init: true` in the compose file
says the same for anyone who replaces the image), and the module closes every
browser on every path including the ones that fail.
### The second silent failure: every browser read hangs, then the disk fills
**The symptom.** `web_read` with the browser, and `/healthz` itself, stop
answering while search still works; if several adapters share one sidecar,
all of their browser reads fail; the machine's disk then fills with no log growing.
**The check โ count THREADS, not processes:**
```bash
docker stats --no-stream --format '{{.Name}} {{.PIDs}}' | grep browser
```
`pids_limit` counts every thread of the container. An idle Chromium tree is
about 42 threads, run-server's own base about 14; at the cap every new browser
fails to create a thread and crashes, while the sidecar's own health check โ a
TCP connect โ stays green. `ps` shows a handful of processes and nothing wrong.
**What 0.3.2 changed**, after exactly this on 2026-09-23 โ a burst of concurrent
reads filled the cap, eleven sessions hung for six hours in stages that have no
timeout, the health probe hung behind a lock, and each crashing Chromium wrote a
core dump into the container layer until the disk was full:
* every browser session runs in a child process of its own and is killed with
its whole process group when it outlives its budget plus
`BROWSER_KILL_GRACE_S` (default 10 s); the caller gets `not_reached` saying so,
and saying whether the sidecar or the page was to blame โ a page whose script
never yields is the page's fault and does not mark the sidecar sick;
* one adapter instance holds at most `BROWSER_MAX_SESSIONS` sessions at once
(default 2, health probes included); a read over that is told the browser is
busy after at most `BROWSER_SLOT_WAIT_S` (default 5 s) instead of starting a
browser the sidecar cannot run; the time spent waiting comes off the session,
whose budget is cut to what the call has left โ only a session that hangs
anyway is killed `BROWSER_KILL_GRACE_S` (10 s) after that, and one that
answered at the last moment gets up to 3 s more to exit, so a call can overrun
its ceiling by up to 13 s with the defaults; and a slot whose session may
have left a browser behind on the sidecar (killed during the launch, or a close
that hung) is held back for `BROWSER_COOLDOWN_S` (default 150 s) while the
sidecar's watchdog clears it;
* `/healthz` never launches a browser and never waits for one: the browser state
is the outcome of the last real session, with its age, plus a port check, and
a real probe runs in the background at most every `BROWSER_PROBE_S` (default
600 s) โ or every health tick while the last verdict is "not responding", so
a recovery shows within minutes rather than ten; until the first probe after
a start it reads `not checked yet: โฆ` and counts as degraded;
* the sidecar's main command is a watchdog (`browser/supervise.py`) that kills
any Chromium browser older than `BROWSER_MAX_AGE_S` (default 120 s โ the
longest legitimate session is 42.5 s), whatever the client did;
* the adapter, the prober and the browser set `ulimits: core: 0`, so none of
them writes core dumps, and the adapter image has an init of its own.
**The cure, if it happens anyway:** `docker compose up -d --force-recreate
browser`. The adapters need nothing: since 0.3.2 their `/healthz` keeps
answering, and their browser state turns back to `alive` by itself within a
health tick of their session slots coming free. Slots whose sessions were
killed while the sidecar was wedged are held back for `BROWSER_COOLDOWN_S`
(150 s), so the whole recovery takes up to about three minutes โ measured on a
test instance: 167 s from recreating the sidecar to `alive`, with no read in between.
`GET /pages` runs a real browser session there and then once a slot is free;
`docker compose restart ag-search` skips the wait if it matters more. An adapter
older than 0.3.2 is the exception: its `/healthz` can hang, and it needs
`docker compose up -d --force-recreate ag-search` โ and an upgrade.
## โ๏ธ What it does with `robots.txt`, and why you must decide
**The module REPORTS a site's rules and does not enforce them.** Every read
carries `robots`: `allowed`, `disallowed_by_site`, or `not_checked` when the file
could not be read. A page a site forbids is still fetched, and the answer says so.
**`allowed` means we read the rules and they permit this path** โ not merely that
we saw no ban. A site that says the file is not there (`404`) has no rules, and
that is `allowed`. A file that did not arrive at all โ `5xx`, a timeout, a torn
connection, `401`/`403` โ or arrived in an encoding we could not unpack is
`not_checked`. Those are different events and the field keeps them apart: no data
means "not checked", never "permitted".
That is a decision, not an omission, and it belongs to whoever runs this rather
than to the module. Two reasons. Whether a tool called by a person obeys
robots.txt is the operator's call โ a rule written for crawlers indexing the web
is not obviously a rule for fetching one page a user asked for. And the reference
behaviour โ treat `401`/`403` on robots.txt as a ban โ produces false bans on
ordinary sources, because the same sites answer `401` to everyone from behind an
anti-bot service. We do not call that a ban. We do not call it permission either:
the answer is `not_checked`, which says what actually happened โ the rules exist
or do not, and we did not get to see them.
**So the gate is yours to add.** If your use requires obeying robots, branch on
the field: `robots == "disallowed_by_site"` means the site says no. If you obey
it, treat `not_checked` as a stop too โ it means we could not read the rules, not
that there are none.
## ๐ How it works
- **[ALGORITHM.md](https://github.com/AG-Bureau/mcp-search/blob/HEAD/ALGORITHM.md)** โ what happens, step by step, on each call.
- **[contracts/](https://github.com/AG-Bureau/mcp-search/blob/HEAD/contracts/)** โ the call contracts, versioned separately from
the code that implements them.
- **[measures/](https://github.com/AG-Bureau/mcp-search/blob/HEAD/measures/)** โ dated measurements: which engines were alive, what
the load ladder gives, what the transport change bought. Numbers, with what was
measured and when.
- **[contracts/ag.search.v3.md](https://github.com/AG-Bureau/mcp-search/blob/HEAD/contracts/ag.search.v3.md)** โ and its
neighbours: what each capability promises, plus the table of what changed
against `/1` and `/2` for anyone with stored answers to read.
## ๐ Deployment and exposure
`wiring/expose-localhost.yml` publishes the adapter on `127.0.0.1` only, and it
is the only overlay that ships. Anything wider is an overlay you write yourself,
and the header of `wiring/expose-localhost.yml` says what to check first: Docker
passes traffic to published ports through `FORWARD` after DNAT, while the
firewall's own chain sits before its hooks โ so a firewall that says "closed" can
be open to the internet on a published port.
A search server open to the outside is an open proxy that goes to the network in
the machine owner's name.
**Known limits of this release**, named so that none reads as absent:
- **The browser path keeps pages out of the perimeter with scripts and routing
inside the browser, not with the network.** Three cases stay open, and
they differ. DNS rebinding is not closed and the page is not withheld: the
inside/outside verdict is kept for the session, and the browser connects to
whatever its own lookup returns. A redirect hop that went inside, and a
prefetch from speculation rules the page's scripts cannot reach (inside a
closed declarative shadow root, or an open one the parser made after
pausing on its host), are seen and the page is withheld, but only after the
request was made. A prefetch from a popup the page opens is not watched, and
that page is not withheld. Since 0.3.5 the rules a page's scripts can reach
are removed. The full list is in
[contracts/ag.read.v2.md](https://github.com/AG-Bureau/mcp-search/blob/HEAD/contracts/ag.read.v2.md); closing it needs a
proxy or egress rules on the sidecar.
- **The door log can name the wrong forwarded-for.** On a keep-alive
connection, a request refused before its headers are parsed (400, 414, 431,
505) is logged with the previous request's `X-Forwarded-For`. The field is marked
unverified either way; this is open in 0.3.5 and fixed in the next release.
- **Deep search words its own queue's refusal as the provider's.** When no
model slot comes free in time, the answer's `cause` is `busy`, as documented,
but its `error` text says the model is rate limited rather than naming
`LLM_MAX_CONCURRENT`. Only the text is wrong.
- **`BROWSER_WS_SECRET` is visible to root inside the sidecar.** tini and the
sidecar's supervisor keep it in the environment they were started with, and
root in that container reads it through `/proc`. The sidecar runs as root and
its browsers with `--no-sandbox`, so a page that took over a renderer could
read it; it would then already be root in the sidecar the secret protects.
- **The call ceiling of `web_read` does not bound everything beneath it.** A
browser session that hangs, and one scan page's render that began just
before its stop, can end past it. Since 0.3.5 the wait for a free model slot
and the model's whole answer are bound by it. See the limits table in
[HOWTO-CALL.md](https://github.com/AG-Bureau/mcp-search/blob/HEAD/HOWTO-CALL.md).
## โ
Tests
```bash
IMAGE=ag-mod-search/adapter:0.3.5 bash tests/in-image.sh
```
Four suites โ the protocol and search against a fake metasearch, reading against
a fake site, the computed pool against a database built in memory, and the parse
of a model provider's answers against canned replies. **Not one of
them makes a single outbound request**, and the runner holds that with
`--network none` rather than on trust: for reading it matters more than for
search, because a test that went to the internet would spend the very resource
the tool protects โ the reputation of the one address it calls from.
They run inside the built image rather than on the machine where the code is
edited: the PDF parser lives in the image, and a suite run outside would skip
everything that touches it. The skip is not silent โ the check goes red with a
note saying where to run it.
What these suites cannot check is written down in
[tests/README.md](https://github.com/AG-Bureau/mcp-search/blob/HEAD/tests/README.md).
## ๐ค Contributing
A capability, engine or heuristic is not accepted until its **reference
attribute** is declared โ a property of the correct answer that the thing being
tested could not have told us itself โ and a pool of checked questions is
attached. See [CONTRIBUTING](https://github.com/AG-Bureau/.github/blob/main/.github/CONTRIBUTING.md).
## ๐ License
[GNU Affero General Public License v3.0](https://github.com/AG-Bureau/mcp-search/blob/HEAD/LICENSE). Run it, change it, build on
it. If you make it available to others OVER A NETWORK, the changes you made go
back out under the same licence โ that is the one obligation, and running a
service counts as making it available.
For whoever cannot live with that clause, a commercial licence is a question to
ask rather than a fork to make.