# footnote-mcp

**Category:** 💻 Developer Tools  
**Repository:** https://github.com/KazKozDev/footnote-mcp  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/footnote-mcp

## Description
Source-grounded web research: search, extraction, verification, and browser automation.

## Claude Desktop Quick Installation
Heuristic fallback — verify the package name and runner against the repository README before running it. Uses `npx` (confidence: low):

```json
"mcpServers": {
  "footnote-mcp": {
    "command": "npx",
    "args": ["-y","footnote-mcp"]
  }
}
```

## Documentation & README

<p align="center">
  <img src=".github/banner.png" alt="footnote-mcp" width="350">
</p>

<!-- mcp-name: io.github.KazKozDev/footnote-mcp -->

An MCP server for source-grounded web research. It searches the web, fetches and
extracts pages, pulls structured data out of tables/files/APIs, and — the part that
sets it apart — **verifies that a claim is actually supported by its source** instead of
trusting a snippet. 45 tools over stdio MCP, driven by any MCP client (Claude Desktop,
Cursor) or by the companion [Scholiast](https://github.com/KazKozDev/scholiast) research agent.

The design priority is *trustworthiness over convenience*: search snippets are treated as
discovery only, every fetched page is cached with provenance, and claims are checked
against the source text before they count. It also degrades gracefully — with no API keys
and no config it still works (scraped search + an automatic headless-browser fallback +
an offline verification heuristic); keys and env vars only make it better.

## Quick start

From [PyPI](https://pypi.org/project/footnote-mcp/) (Python ≥ 3.10):

```bash
pip install footnote-mcp
python -m playwright install chromium   # the headless browser used by the fetch fallback
footnote-mcp                            # start the server (speaks MCP over stdio)
```

Or from source:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e .                        # installs the `footnote-mcp` console script + deps
python -m playwright install chromium   # the headless browser used by the fetch fallback
footnote-mcp                            # start the server (speaks MCP over stdio)
```

`footnote-mcp` now waits for an MCP client on stdio. Point a client at it by dropping this
into its MCP settings (Claude Desktop: `claude_desktop_config.json`; Cursor: `~/.cursor/mcp.json`):

```json
{
  "mcpServers": {
    "footnote": { "command": "footnote-mcp" }
  }
}
```

## Hosted MCP endpoint (Render)

This repository also includes an authenticated Streamable HTTP deployment. The
stdio command above remains the recommended local option; the hosted endpoint is
for MCP clients that support remote servers.

1. Push this repository to GitHub, then create a Render **Blueprint** from it.
   Render reads [`render.yaml`](render.yaml), builds the included Dockerfile
   (including Chromium and Tesseract), and exposes the health check at `/healthz`.
2. Choose a unique service name, then set `FOOTNOTE_MCP_PUBLIC_URL` in Render to
   its exact public origin, for example `https://my-footnote-mcp.onrender.com`.
   `FOOTNOTE_MCP_API_KEY` is generated by the Blueprint; it is the owner key.
   Keep it secret and use it only for administration/testing.
3. Connect an MCP client to `https://my-footnote-mcp.onrender.com/mcp` with:

```http
Authorization: Bearer <FOOTNOTE_MCP_API_KEY>
```

The API key is required: a public, unauthenticated research and browser server
would let strangers consume its outbound traffic and call its tools. The service
also validates its configured public host and browser Origin to protect the MCP
endpoint from DNS rebinding.

Render Free is suitable for demos, not production: it sleeps after 15 minutes of
inactivity, cold starts take about a minute, and both outbound bandwidth and
instance hours are limited. The filesystem is ephemeral, so do not rely on the
source cache or browser profile persisting across restarts.

### Giving access to other people

Never distribute the owner key. Create a separate random key for each person:

```bash
python -m footnote_mcp.keygen
```

Then update `FOOTNOTE_MCP_API_KEYS` in Render's Environment settings and
redeploy. Its value is JSON, where each user has a key and a personal
requests-per-minute limit:

```json
{
  "alice": {"key": "fn_alice_key_here", "rpm": 20},
  "bob": {"key": "fn_bob_key_here", "rpm": 10}
}
```

Give each person only their own value. To revoke access, remove that user from
the JSON and redeploy; the other keys keep working. Limits are held in memory,
which is appropriate for this one-instance Free service and reset on restart.

No API keys are required to start — search falls back to zero-key Bing, DuckDuckGo, Brave,
and Wiby discovery. Add
keys later under `"env"` (see [Search backends](#search-backends)). Pass `--headed` to watch
the browser tier work.

Optional runtime variables are documented in [.env.example](.env.example). Copy it to
`.env` for local shells, or paste selected variables into your MCP client config:

```json
{
  "mcpServers": {
    "footnote": {
      "command": "footnote-mcp",
      "env": {
        "TAVILY_API_KEY": "..."
      }
    }
  }
}
```

To run without installing, straight from the source tree:

```bash
PYTHONPATH=src python -m footnote_mcp
```

## Verifying claims — the differentiator

The reason to use this over a plain search tool is `evidence_entailment` and friends:
they tell a claim a source *supports* from one it does not. `benchmarks/run_benchmark.py`
measures that on a labeled set of claim/source pairs (and demos `corroborate_claim` and
`locate_claim_span`):

```bash
python benchmarks/run_benchmark.py                    # offline heuristic (deterministic)
python benchmarks/run_benchmark.py --backend ollama   # LLM judge (needs ollama)
```

Offline-heuristic result on the labeled set ([benchmarks/REPORT.md](benchmarks/REPORT.md)):

| Set | n | Accuracy | Unsupported-claim catch rate | Precision on "supported" |
|-----|---|----------|------------------------------|--------------------------|
| Data domain (numeric + factual) | 15 | 100% | 100% | 100% |
| Overall (incl. semantic) | 18 | 83% | 78% | 80% |

On its design domain — numeric and factual data claims — the offline heuristic never
blesses an unsupported claim and never misses one. Its blind spot is purely-semantic
negation/paraphrase; for those, `evidence_entailment` with `backend="ollama"` (a local LLM
judge) closes the gap. Run the `--backend ollama` line above to score that path on your own
machine.

## Tools

<details>
<summary><b>Discovery and reading</b> (12 tools)</summary>

| Tool | Description |
|------|-------------|
| `web_search` | Configured providers plus zero-key Bing, DuckDuckGo, Brave, and Wiby; Marginalia is explicit-only. Snippets are discovery only. |
| `web_search_recent` | Search restricted to a recency window (day/week/month/year). |
| `web_deep_search` | Iteratively close evidence gaps across web/papers/encyclopedia/GitHub/archive sources; extracts tables/files, verifies individual facts, and returns an evidence ledger plus diagnostic funnel. |
| `web_read` | Fetch one URL, extract text, classify source quality, persist cache metadata. |
| `papers_search` | Search Crossref and arXiv through one normalized, zero-key paper contract. |
| `encyclopedia_search` | Search Wikipedia/Wikidata entities or run read-only Wikidata SPARQL. |
| `github_search` | Search public repositories, issues, code, or commits; authentication is optional. |
| `archive_search` | Find URL captures through Wayback Machine and Common Crawl, optionally extracting archived text. |
| `web_archive_fetch` | Find the closest Wayback Machine snapshot for a dead/changed URL. |
| `web_fetch_authenticated` | Fetch a page that needs cookies or custom headers. |
| `web_crawl` | Breadth-first crawl from a start URL, on-host by default (≤ 50 pages). |
| `generate_search_queries` | Generate operator queries (`site:`, `filetype:csv`, API/data-table variants). |

</details>

<details>
<summary><b>Structured data</b> (9 tools)</summary>

| Tool | Description |
|------|-------------|
| `web_extract_tables` | Parse HTML tables into `columns`/`rows` with source-URL provenance. |
| `web_detect_downloads` | Detect linked CSV/TSV/XLS/XLSX/PDF/JSON/XML files. |
| `web_parse_file` | Download and parse CSV/TSV/XLS/XLSX/PDF/JSON. |
| `web_fetch_json` | Fetch direct API/JSON endpoints into parsed JSON. |
| `check_date_completeness` | Validate required date coverage (day/week/month). |
| `resolve_units` | Detect currencies, currency pairs, measurement units. |
| `validate_unit_rows` | Reject rows with incompatible units or currency pairs. |
| `reconcile_time_series` | Align series on a key, compute deltas, flag missing keys/outliers. |
| `export_dataset` | Write consolidated rows to a `csv`/`xlsx`/`json` file. |

</details>

<details>
<summary><b>Source quality and verification</b> (8 tools)</summary>

| Tool | Description |
|------|-------------|
| `classify_source` | Classify official / aggregator / blog / forum / interactive / blocked / error. |
| `evidence_entailment` | Strict claim-vs-source checker: `heuristic`, `auto`, `ollama`, optional `local_nli`. |
| `corroborate_claim` | Triangulate a claim across excerpts (corroborated / conflicting / single_source / …). |
| `locate_claim_span` | Locate supporting sentence(s) with char offsets and a containment score. |
| `source_cache_get` / `source_cache_put` | Inspect and write persistent source-cache entries. |
| `build_research_debug_report` | Compact report of queries, URLs, source quality, verification gaps. |
| `startup_health_check` | Check parser, OCR, browser, and cache dependencies. |

</details>

<details>
<summary><b>Controlled extraction recipes</b> (6 tools)</summary>

When generic parsers fail, synthesize a sandboxed parser:

| Tool | Description |
|------|-------------|
| `tool_spec_propose` | Propose a task-specific extraction recipe spec. |
| `tool_code_generate` | Generate a starter `extract(source_text, input_payload)` recipe. |
| `tool_code_validate` | Validate recipe code against a static safety allowlist. |
| `tool_code_run_sandboxed` | Run validated code in a limited subprocess (JSON output only). |
| `tool_promote` | Save a validated recipe as reusable memory (no server edit). |
| `recipe_registry` | Manage promoted recipes: `list` / `get` / `run` / `delete`. |

</details>

<details>
<summary><b>Browser fallback</b> (10 tools)</summary>

A controlled Chromium session for JS-heavy or interactive pages:

| Tool | Description |
|------|-------------|
| `web_navigate` · `web_snapshot` · `web_click` · `web_type` · `web_extract` · `web_scroll` | Drive a page via stable element refs. |
| `browser_set_date_range` · `browser_extract_tables` · `browser_extract_tables_for_date_range` | Set a date range, submit, extract visible tables. |
| `web_screenshot` | Save a PNG and optionally OCR text locked inside the image. |

</details>

## Search backends

`web_search` routes through a provider layer. A configured zero-key SearXNG instance is tried
first, followed by keyed providers and finally zero-key Bing, DuckDuckGo, Brave, and Wiby.
Marginalia remains available as an explicit provider. Results are
normalized to one shape regardless of backend. Every provider is relevance-filtered and
deduplicated before cross-provider merging; repeated URLs from the same provider do not receive
an agreement bonus.

| Provider | Env vars | Notes |
|----------|----------|-------|
| SearXNG | `FOOTNOTE_SEARXNG_URL` (or `SEARXNG_URL`) | Zero-key JSON API; instance must enable JSON output. |
| Tavily | `TAVILY_API_KEY` | LLM-oriented search API. |
| Brave | `BRAVE_API_KEY` | Independent web index. |
| Google | `GOOGLE_API_KEY` + `GOOGLE_CSE_ID` | Programmable Search (Custom Search JSON API). |
| Bing + DuckDuckGo + Brave | none | Default fallback; scraped, no key. |
| Wiby | none | Public JSON endpoint; result metadata includes required Wiby attribution. |
| Marginalia | none | Shared public API; result metadata preserves its `CC-BY-NC-SA 4.0` license. |

`auto` (default) queries every configured provider plus the latency-bounded zero-key fallbacks
and merges the complete result set. Marginalia is excluded from `auto` because its shared public
endpoint can be slow; use `provider="auto+marginalia"` to include it in the merged search, or
`provider="marginalia"` to isolate it. Force one isolated backend with the `provider` argument
(`searxng`/`tavily`/`brave`/`google`/`wiby`/`marginalia`/`scrape`). Brave and DuckDuckGo enter a
temporary cooldown after rate limiting; override the defaults with
`FOOTNOTE_BRAVE_COOLDOWN_SECONDS` and `FOOTNOTE_DDG_COOLDOWN_SECONDS`.

### Specialized zero-key discovery

The public MCP surface is organized by user intent rather than by HTTP API:

| Intent tool | Backends | Routing notes |
|-------------|----------|---------------|
| `papers_search` | Crossref + arXiv | `source=auto` queries both; force either backend when needed. |
| `encyclopedia_search` | Wikipedia + Wikidata | Entity search by default; optional read-only SPARQL for structured facts. |
| `github_search` | GitHub REST search | Public zero-key requests work at GitHub's unauthenticated rate limit; `GITHUB_TOKEN` is optional. |
| `archive_search` | Wayback + Common Crawl | Accepts a URL/host pattern. `fetch_text=true` attempts archived-content extraction. |

All four return `title`, `url`, `snippet`, `published`, `authors`, `source`, and
`source_type` where those fields apply. `web_deep_search` accepts an optional `sources`
array (`web`, `papers`, `encyclopedia`, `github`, `archive`). With an empty array it always
uses general web discovery and adds specialized sources when the query signals their intent.

`web_deep_search` is a separate, slower research loop. Set `model` (or
`FOOTNOTE_RESEARCH_MODEL`) to enable requirement decomposition, gap-specific query planning,
and strict fact extraction. It maintains a serializable research state and evidence ledger,
expands fetch/chunk budgets across iterations, parses HTML tables and linked CSV/XLS/XLSX/PDF/JSON
files, and verifies `subject`, `metric`, `period`, `value`, and `unit` against an exact source
quote before admitting an item. The result includes `answer_ready`, unresolved requirements,
per-iteration diagnostics, and the cumulative funnel
`candidates → deduplicated_documents → relevant_documents → successful_fetches → extracted_facts → verified_evidence`.

**Semantic reranking.** Pass `semantic: true` to `web_search` to reorder by meaning rather
than keyword overlap: it over-fetches, embeds query and results with a local ollama model,
and sorts by cosine similarity (each result gains `semantic_score`). Best-effort — if ollama
is unavailable the original order is returned. Model: `FOOTNOTE_EMBED_MODEL` (default `bge-m3`).

## Fetching & anti-bot ladder

`web_read` fetches through an escalation ladder ([scraper.py](src/footnote_mcp/scraper.py)):
the cheapest method runs first and escalates only when a result looks blocked or empty. A
block/quality detector decides when to escalate; a per-domain rate limiter, circuit breaker,
and negative cache keep it polite. The tier used and the full attempt trace come back in
`fetch_tier` / `scrape_tiers`.

| Tier | Method | Enabled by |
|------|--------|-----------|
| 1 | HTTP (curl_cffi TLS impersonation) | always |
| 2 | HTTP through a rotating proxy | `FOOTNOTE_PROXIES` set |
| 3 | Headless Chromium (runs JavaScript) | `FOOTNOTE_BROWSER_FALLBACK=1` (default on) |
| 4 | Chromium through a proxy | proxies + browser |
| 5 | Hosted scrape API (Firecrawl / ScrapingBee) | `FOOTNOTE_SCRAPE_API` set |

With nothing configured it is the plain HTTP path plus an automatic browser fallback for
JavaScript-rendered pages.

| Env var | Default | Purpose |
|---------|---------|---------|
| `FOOTNOTE_BROWSER_FALLBACK` | `1` | Escalate blocked/JS pages to headless Chromium. |
| `FOOTNOTE_PROXIES` | _(none)_ | Comma-separated proxy URLs; sticky per domain with health tracking. |
| `FOOTNOTE_SCRAPE_API` | _(none)_ | `firecrawl` or `scrapingbee` (needs the matching API key). |
| `FOOTNOTE_DOMAIN_RPS` / `_BURST` | `3` / `5` | Per-domain rate limit (token bucket). |
| `FOOTNOTE_BREAKER_THRESHOLD` / `_COOLDOWN` | `5` / `120` | Per-domain circuit breaker. |
| `FOOTNOTE_NEGCACHE_TTL` | `300` | Seconds to remember a blocked URL. |
| `FOOTNOTE_RETRY_AFTER_MAX_SECONDS` | `30` | Longest a request will block waiting out a 429/503 before handing the refusal back. |
| `FOOTNOTE_HTTP_CACHE` | `1` | Store `ETag`/`Last-Modified` and revalidate with conditional requests. |
| `FOOTNOTE_HTTP_CACHE_MAX_BYTES` | `1000000` | Largest body kept for revalidation. |
| `FOOTNOTE_THIN_CONTENT_CHARS` | `200` | Below this extracted length, a script-heavy page counts as a JS shell. |

The rate limit, circuit breaker and negative cache apply to **every** outbound request, not
only to pages fetched through the ladder: they live in `politeness.py` and are taken inside
`fetch._get`, which each tool's HTTP call funnels through. A `429` or `503` is waited out
(honoring `Retry-After`) rather than retried immediately; `web_crawl` stops at the first
refusal; and parallel fetching runs across hosts, never several workers at one host.

## Runtime data

```text
~/.footnote-mcp/source_cache/        # persistent page cache (with provenance)
~/.footnote-mcp/source_cache/http/   # ETag/Last-Modified bodies for conditional requests
~/.footnote-mcp/research_memory.json # persistent research memory
```

Override the cache location with `FOOTNOTE_SOURCE_CACHE=/path/to/cache footnote-mcp`.

`check_date_completeness` supports the calendars `calendar`, `business_day`, `crypto_24_7`,
`forex_weekday`, `us_business_day`, and `ru_business_day` (pass explicit `holidays` for
source-specific ones; the `us_`/`ru_` variants use the optional `holidays` package).

## Other install paths

**Docker** bundles Chromium and tesseract — nothing else to install:

```bash
docker build -t footnote-mcp .
docker run -i --rm footnote-mcp        # the client launches this; see MCP config below
```

Published images are available from GitHub Container Registry:

```bash
docker run -i --rm ghcr.io/kazkozdev/footnote-mcp:0.2.6
docker run -i --rm ghcr.io/kazkozdev/footnote-mcp:latest
```

```json
{
  "mcpServers": {
    "footnote": {
      "command": "docker",
      "args": ["run", "-i", "--rm", "ghcr.io/kazkozdev/footnote-mcp:latest"]
    }
  }
}
```

**pipx / uvx** (isolated install of the entry point):

```bash
pipx install /path/to/footnote-mcp          # or: pipx install git+<repo-url>
uvx --from /path/to/footnote-mcp footnote-mcp   # ad-hoc, no install
```

**OCR.** PDF/image OCR uses `pytesseract` + the system `tesseract` binary (`brew install
tesseract` on macOS). **Local NLI backend** for `evidence_entailment` `backend="local_nli"`:
`pip install -r requirements-nli.txt` (model via `FOOTNOTE_NLI_MODEL`). Either way,
`startup_health_check` reports what is actually available. Runtime dependency ranges
are declared in `pyproject.toml` and mirrored in `requirements.txt`.

## Tests

```bash
pip install -r requirements-dev.txt
python -m pytest -q          # offline unit + smoke tests; no network or keys needed
```

`tests/test_mcp_smoke.py` launches the server over real MCP stdio and exercises the tools
end to end against a local HTTP fixture; the rest are offline unit tests of the parsers,
fetch ladder, search providers, and dispatch. The live search test is opt-in:

```bash
RUN_LIVE_WEB_TESTS=1 python -m pytest -m live
```

CI runs the same suite (`.github/workflows/tests.yml`).

## License

MIT — see [LICENSE](LICENSE).