whetstone

Two learners grinding each other sharp: a student that must earn every promotion, and an examiner that must earn the right to judge it — with every label anchored to an oracle that cannot practically be wrong.

whetstone is a self-improving research harness built around one rule: nothing is believed until something exact has tried to kill it. A verifier-gated student (QLoRA adapters on a local 4B) trains on experience milled from root oracles — exhaustive graph enumeration, simulated-annealing counterexample search, exact game simulators, Monte-Carlo consensus, Stockfish, KataGo — and is graded by an evolving exam bank that quarantines anything the student trained on, learns which items discriminate, retires the saturated, and feeds its retired exams back as training fuel.

Selected results (full ledger with methods, nulls, and receipts in FINDINGS.md; artifacts in results/):

• Settled an open finite conjecture: degree-descending greedy coloring is NOT optimal on connected triangle-free non-bipartite graphs (minimal counterexample: 11 vertices, 13 edges — found by annealing, minimized, exact).
• Quantified the verifier-horizon regress: 99/121 conjectures with precision 1.0 at n<=6 are false at n=7/8; acceptance horizons must sit strictly inside stress horizons or self-training loops train on artifacts.
• Measured salience vs relevance: the most salient memory is the counterfactually useful one 4% of the time; a query-conditioned relevance score, 81%.
• Ran a full unattended continual-learning loop: cold-start 0/8 -> 8/8 in one gated round; the multi-domain generation cracked a cross-domain frontier its own examiner had exposed (base 1/33 -> gen-2 4/33, first MIS pass).
• Closed the same-bank scale comparison locally: stock Qwen2.5 1.5B->32B rose 8->20/48 while every stock model scored 0/24 on graph repairs; the task-routed verifier-trained FastContext-4B scored 28/48 and earned PASS over its 22/48 base (6 gains, 0 regressions, exact p=.03125), repeated identically three times.
• Trained the first trajectory-disjoint Gen-4 engine student on 1,313 fresh Stockfish/KataGo examples. Three fresh loads moved 7/69 -> 8/69, but the item-level gate correctly BLOCKED promotion: 3 gains, 2 regressions, exact p=1.0, including one historically stable regression. No promotion is claimed.
• The private promotion exam bank is deliberately NOT in this repo: publishing it would let future models train on it — the leakage rule at internet scale.

Licensed AGPL-3.0. Commercial licensing available from the author.

Install the dependency-light core (CLI and stateless toolbox) with pip install .. Optional integrations are grouped as agents, engines, and local-models; pip install ".[all]" installs every integration.

The public HTTP MCP endpoint serves both the sessionless 2026-07-28 protocol (server/discover plus per-request metadata and headers) and the initialize-based 2025-06-18, 2025-03-26, and 2024-11-05 revisions. The stdlib server remains dependency-free; mcp>=2,<3 is needed only for SDK clients and the stdio adapter.

Tagged releases contain archive-built wheels, source archives, SPDX 2.3 SBOMs, and SHA256SUMS; GitHub Actions attaches build-provenance attestations to every release artifact. For v0.8.0, verify the wheel against the repository with:

gh release download v0.8.0 -R CarlSR9001/whetstone -p "*.whl" -p "*.spdx.json" -p "SHA256SUMS"
gh attestation verify .\branching_continual_verification-0.8.0-py3-none-any.whl -R CarlSR9001/whetstone

The product layer

The research harness above is operable as a product: one CLI, adapters for any system under exam, pluggable verifiers, a CI contract, and agent-native surfaces. Full guide in docs/product.md.

$env:PYTHONPATH='src'
python -m bcv.cli init                                   # bank + whetstone.toml
python -m bcv.cli mint --domain code --max-items 8       # hidden-check exam items
python -m bcv.cli grade --system v2 --command "python my_agent.py"
python -m bcv.cli grade --system v2 --acp "my-acp-agent" # any Agent Client Protocol agent
python -m bcv.cli gate --baseline v1 --candidate v2      # exit code IS the verdict: 0 PASS, 2 HOLD, 3 BLOCK
python -m bcv.cli sweep                                  # retire saturated items downward
python -m bcv.cli redteam                                # hostile self-test; nonzero exit on escape
python -m bcv.cli serve --port 8977                      # localhost JSON service
python -m bcv.cli mcp                                    # MCP server (also in .mcp.json)
python -m bcv.cli inspect --exam exam.jsonl --exposure exposure.jsonl --baseline v1.json --candidate v2.json
python -m bcv.cli hosted-report-card --command "python my_agent.py" --out report-card.json
python -m bcv.cli open-bench-run --baseline-command "python v1.py" --candidate-command "python v2.py"
python -m bcv.cli verify-receipt --receipt report-card.json
python -m bcv.cli toolbox --port 8988                    # eight stateless tools on localhost

Grading an external endpoint burns every exposed item — permanently — via the bank's exposure accounting; the promotion report carries paired evidence, an exact McNemar p-value, the bank's own resolution statement, and a SHA-256 commitment to bank state. No surface, human or agent, ever serves exam item contents.

The stateless toolbox turns eight mechanisms already in the repo into usable file-in/receipt-out surfaces: Whetstone Inspector, Eval Leak Auditor, Promotion Gate, Bank Health, SafePatch, Counterexample Hunter, Memory Relevance Debugger, and Agent Replay Console. Each run exposes the decision path, an inspectable visual certificate, a SHA-256 receipt, and a task-specific downloadable artifact. Leakage analysis keeps exact identity, finite-corpus behavioral equivalence, and text-similarity review as separate evidence tiers; only the first two quarantine automatically. The public instance at whetstone.cyberelf.link accepts only caller supplied disposable data; it has no examiner bank, accounts, cookies, or upload persistence. Private exams belong in the CLI or a customer-controlled instance.

The same public service now includes Open Promotion Bench, a paired scope-integrity track. open_bench_start gives a baseline and candidate the same six procedurally varied virtual-repository tasks; open_bench_submit checks requested state plus edit-scope conservation and counts gains, regressions, and ties before issuing PASS/HOLD/BLOCK. A submitter may explicitly publish a self-attested sanitized receipt. The ledger stores manifests, transitions, counts, and hashes only—never task contents or answer patches—so the existing workbench/report-card non-retention boundary remains intact.

Hosted report-card and Open Bench receipts are challenge-bound and signed with an Ed25519 SSHSIG key. The one-command runners above fetch the HTTPS key bundle, verify the issuer, challenge, content commitment, signature, and expiry, then write the receipt. A valid signature authenticates Whetstone's grading result and deployed build; it does not authenticate a caller-supplied model or harness name. Runners fail closed on unsigned receipts unless --allow-unsigned is explicitly used for a local development service. Busy report-card submissions are retryable without consuming the session.

Support-panel minting currently fails closed by default: the lexical checker passed its clean smoke corpus but failed 13/13 adversarial cases, so it cannot mint promotion-capable support items until an independently reviewed hard calibration is supplied explicitly.

The sixty-second demo

The whole promotion gate, live, on CPU, in about ten seconds:

$env:PYTHONPATH='src'; python -m bcv.demo_investor

It mints exam items at the verifier frontier, quarantines the ones a toy student trained on (row-identity leakage check against a real buffer on disk), grades two systems through the checker specs — a memorizer that echoes its training data and a candidate that proposes repairs — computes per-item discrimination, issues a gains-vs-regressions promotion decision, retires saturated items into the downward-only training flow, and writes an append-only ledger. The two systems are stored answer policies (this keeps the demo off the GPU); every mint, quarantine, and grade is the production code path, computed live. A canonical transcript is in results/demo_investor_transcript.txt.

Research context