# openhonest/umbra [Health: Active]

**Category:** 💻 Developer Tools  
**Repository:** https://github.com/openhonest/umbra  
**GitHub Stars:** 3  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/openhonest-umbra

## Description
Reports what your test suite is structurally unable to see and proves each gap with a real failing test, across six languages.

## Claude Desktop Quick Installation
Remote MCP endpoint (confidence: high). Install path detected from listing signals. Add as a URL/SSE server in your client:

```json
"mcpServers": {
  "umbra": {
    "url": "https://slopaudit.org)**"
  }
}
```

## Documentation & README

# Umbra

**Reveals what your tests are not telling you.**

Umbra is an apophatic[^definition] code auditor. Every other tool reports what it found: coverage says which lines ran, a linter says which lines smell, a green suite says all the tests pass everything they can see. None of them report what the tests are structurally unable to see. That blind region, the umbra, is where agent-written code fails.

Umbra names the shadow and proves it. For a Python, TypeScript, JavaScript, Java, Rust, or Go module and its test file(s). Umbra maps what the tests actually exercise, finds the reachable and meaningful cases nothing asserts, and reports a **Silence index**: the honest complement to coverage. Higher means more blind spots. Not what ran. What nothing could see. It then writes failing tests that will close the gap, and examines your functions for purity.

Built with Codex, using GPT-5.6. Track: **Developer Tools**. GPT-5.6 runs only where structure runs out and only meaning remains.

Umbra is the newest instrument in the **[Slop Audit](https://slopaudit.org)** (Open Honest Foundation), the same catch-what-metrics-miss-by-construction discipline taken from an enterprise codebase down to a single module and its test suite. Both rest on one premise: name what the instrument cannot see. Umbra's page: **[slopaudit.org/umbra.html](https://slopaudit.org/umbra.html)**.

## Install

Umbra runs two ways: as a **Codex plugin**, its home, auditing code the moment the agent writes it (see "Use from Codex" below); and as a standalone **CLI**. Either way it audits six languages (Python, TypeScript, JavaScript, Java, Rust, Go).

The CLI is a Python package, run with `uv`. From a clone:

```bash
uv tool install .        # or: uv run umbra --help
```

## Usage

Umbra runs two ways. Its home is the Codex plugin; the CLI is the standalone path. Both are shown here.

### In Codex (the plugin)

Once the plugin is registered (see "Use from Codex" below for the one-time setup and the full tool reference), you drive it in plain language. The surface is four tools, and here is what you can ask for:

- **Audit a module and its tests.** *"Audit this module and its test file with Umbra."* Codex calls `umbra_audit`, reports coverage beside the Silence index, and names every blind spot. Deterministic, and needs no API key.
- **Prove the gaps.** For each blind spot the audit returns one isolated proof request. Codex writes a single failing test from it and calls `umbra_verify_proposal`; Umbra's execution gate keeps only a test that genuinely fails or makes the function error, and rejects a malformed one with the exact field to fix. Umbra never writes into your test file, so adopting a surviving proof is your choice.
- **Get the report.** *"Write the report."* Codex calls `umbra_write_report` and presents the self-contained HTML alongside the portable Markdown. Local and deterministic; sends nothing anywhere.
- **Ask what a term means.** *"What does Umbra mean by the Silence index?"* Codex calls `umbra_glossary` and relays Umbra's own fixed definition instead of improvising one. Omit the term to list every defined term.
- **Stay fully offline.** The audit is deterministic by default (`model_backend: off`) and makes zero API calls. Only proof generation can use a model, and only the host sampler unless you opt in: the OpenAI backend stays off until you set `confirm_openai` (or the disclosed `UMBRA_OPENAI_FALLBACK`).

### On the command line

Point Umbra at a source module and its test file, in any supported language:

```bash
uv run umbra audit path/to/module.py --tests path/to/test_module.py
```

Umbra prints the Coverage-versus-Silence report and writes the generated failing test using the audited language's file extension. It reads each test file to identify its framework and runs the matching one: pytest for Python; Vitest or Node's built-in `node:test` for TypeScript and JavaScript; JUnit or a throw-on-failure `main` harness for Java; the `#[test]` suite for Rust, whether inline in the module or in a separate file; and `go test` for Go, including named and external test packages. Coverage comes from each language's own tool: coverage.py, Vitest or Node V8, JaCoCo, cargo-llvm-cov, and the Go cover profile. A framework Umbra does not recognize, or a run it cannot complete, is reported as not measured with the reason attached, never a guessed number.

### Command reference

**`umbra audit [MODULE]`** — audit a single source file (with `--tests`), a Python project directory, or a Git diff (`--since`).

| Flag | Default | Effect |
|---|---|---|
| `MODULE` (positional) | — | A Python, TypeScript, JavaScript, Java, Rust, or Go source file, or a Python project directory. Omit when using `--since`. |
| `--tests PATH` | — | Test file for a single audited source. Project and diff modes discover the suite themselves. |
| `--since REF` | — | Audit only the module and test pairs changed since this Git ref; writes `umbra-diff-report.md`. |
| `--fail-on-silence-increase N` | — | Diff mode only: exit non-zero when the aggregate Silence index rises by more than N points. |
| `--no-model`, `--deterministic` | off | Run fully offline, with zero API calls. |
| `--json` | off | Emit one machine-readable audit JSON object to stdout; the human report goes to stderr. |
| `--json-out PATH` | — | Also write that JSON object to a file. |
| `--harden` | off | Turn the audit into a convergence loop (see below). |
| `--harden-call-budget N` | 12 | Maximum model calls across the whole harden run. |
| `--harden-iteration-budget N` | 4 | Maximum re-audit passes. |
| `--harden-calls-per-iteration N` | 1 | Model calls before each re-audit. |

**`umbra benchmark`** — run the reproducible proof-cycle benchmark on the fixed Python gap set.

| Flag | Default | Effect |
|---|---|---|
| `--runs N` | 3 | Independent trials per fixed gap. |
| `--attempt-cap N` | 4 | Maximum proposal attempts per gap trial. |
| `--output PATH` | `umbra-proof-cycle-scorecard.md` | Markdown scorecard path. |

**Environment variables** (CLI and plugin)

| Variable | Effect |
|---|---|
| `OPENAI_API_KEY` | Enables the model layer for proof generation (CLI), or the disclosed OpenAI backend (plugin). |
| `UMBRA_OPENAI_FALLBACK` | Set to `1` to authorize the plugin's disclosed OpenAI fallback, and only after host sampling is unavailable. |
| `UMBRA_TIMEOUT_SECONDS` | Per-subprocess time limit for audited and generated code (default 10). |
| `UMBRA_JAVA_TOOLS` | Directory holding the JDK, JaCoCo agent, and JUnit console launcher for Java audits; falls back to the repository `tools/` directory. |

### Harden mode: close the gaps, not just name them

Add `--harden` to turn the audit into a loop. Umbra generates the tests that close each open facet, keeps the passing ones as regression guards and the failing ones as bug proofs, re-audits, and repeats until no closeable silence remains. The Silence index is the loop's convergence target, so "done" is definable, unlike coverage-chasing.

```bash
uv run umbra audit path/to/module.py --tests path/to/test_module.py --harden
```

Harden generates tests, so it uses the model layer (or Codex). The call, iteration, and per-iteration budgets are bounded by `--harden-call-budget`, `--harden-iteration-budget`, and `--harden-calls-per-iteration`. Run without `--harden` for a fully deterministic audit.

### Python project mode

Give `audit` a repository directory to audit every real Python source module against the entire discovered pytest suite. Umbra runs that suite once under branch coverage, attributes resolvable calls from every test file to their source module, and writes `umbra-project-report.md` with per-module results plus the repository aggregate. Re-export-only `__init__.py` files are skipped. A module that cannot be parsed, or whose suite does not complete, is listed as unaudited and is excluded from the aggregate.

```bash
uv run umbra audit path/to/repository --no-model
```

Project mode is deterministic and sends no data to any API. It currently supports Python repositories with flat or `src/` layouts and conventional `tests/` or `test/` filenames. A suite with failing tests remains measurable when coverage exists: the report names the pass count, fail count, and exit status. A run that produces no coverage is explicitly unmeasured and excluded from the index.

### Time limit for audited code

Umbra gives each subprocess that runs audited or generated code a 10-second limit. It stops the whole process group, so a timed-out test or replay cannot leave child processes behind. Set `UMBRA_TIMEOUT_SECONDS` to a positive number of seconds to adjust that limit:

```bash
UMBRA_TIMEOUT_SECONDS=30 uv run umbra audit path/to/module.py --tests path/to/test_module.py
```

A timed-out replay becomes an `honesty_unverified` result with the timeout reason. A timed-out coverage run is explicitly unmeasured and does not receive a Silence index.

**Reports made for humans.** Not the usual wall of text a test report buries you in: a short, readable account of what your suite cannot see, with the one failing test that proves it.

## JSON output

For scripts and CI, add `--json`. Standard output is one JSON object only; the human report is sent to standard error, while the Markdown and HTML reports are still written as usual.

```bash
uv run umbra audit examples/cart.py --tests examples/test_cart.py --no-model --json
```

The stable version-1 keys are `schema_version`, `module_path`, `tests_path`, `language`, `coverage_percent`, `coverage_status`, `silence_index`, `silences`, `unverified_properties`, `generated_proofs`, `honest_maps`, and `model`. `silence_index` contains the percent, numerator, denominator, and facet counts. Every silence records its function, kind, location, explanation, and whether it is closeable. Every generated proof records its gap, status, rendered test, captured output, and proposal. For a language whose coverage is not measured, `coverage_status` is `"not_measured"` and `coverage_percent` is `null`.

Use `--json-out path/to/audit.json` to write the same document to a file. It can be combined with `--json`.

In project mode, `--json` emits one aggregate object with `audit_kind: "project"`, the suite status, per-module audited or unaudited status, aggregate Silence index counts, and the by-kind breakdown.

## Data sent to OpenAI

When `OPENAI_API_KEY` is present and the model layer runs, Umbra sends OpenAI one isolated function source, its type contract, and one located gap description for each proposed proof. It does not send the repository, test files, a whole-module site list, or any other functions.

Run fully offline, with zero API calls:

```bash
uv run umbra audit path/to/module.py --tests path/to/test_module.py --no-model
```

`--deterministic` is an alias for `--no-model`.

## Audit a Git diff

Audit only the conventional module and test pairs touched since a Git ref:

```bash
uv run umbra audit --since origin/main --no-model
```

Umbra writes `umbra-diff-report.md`. It reports each pair's prior and current Silence index, the newly introduced blind spots, an aggregate delta, and files it could not pair, including deletions and unconventional test names. With the model enabled, only newly located gaps receive isolated proof requests.

The repository action is [action.yml](https://github.com/openhonest/umbra/blob/HEAD/action.yml). The sample pull-request workflow is [umbra-diff.yml](https://github.com/openhonest/umbra/blob/HEAD/.github/workflows/umbra-diff.yml): it sets up Python and `uv`, comments with `umbra-diff-report.md`, and can fail when the aggregate Silence index increase exceeds `silence-threshold`. Python is the guaranteed runner. Other supported-language files are reported only when their existing structural path can run; the comment names exactly which languages Umbra audited.

## Use from Codex

This repository ships a local Codex MCP plugin at `plugins/umbra-codex`. Use it whichever way you run Codex.

### Plugin tools (MCP)

The plugin exposes exactly four tools. The usual loop is `umbra_audit` → `umbra_verify_proposal` (once per returned proof request) → `umbra_write_report`; `umbra_glossary` is independent and answers term questions at any time.

| Tool | Parameters | Returns |
|---|---|---|
| `umbra_audit` | **`module_path`**, **`tests_path`** (required). `model_backend` (`off` \| `codex` \| `openai`, default `off`), `confirm_openai` (default `false`), `proof_cap` (int, default 3), `generate_proof` (bool, alias for `model_backend: codex`) | Coverage percent and status, the Silence index with its located silences, up to `proof_cap` **isolated proof requests** (one function source, its type contract, and one gap each), a `model_status`, and the Markdown and HTML report artifact paths. |
| `umbra_verify_proposal` | **`module_path`**, **`tests_path`**, **`proof_index`** (from the audit's proof requests), **`proposal`**: `concrete_input`, `expected_property`, `plain_explanation` | An `outcome` of `retained`, `discarded`, `rejected`, or `verification_error`; the rendered test and captured output; a `rejection` naming the field to fix when the proposal is malformed; and a `re_audit` hint. **Retained** means a runnable test genuinely failed or made the function error. A passing proposal is discarded. |
| `umbra_write_report` | **`module_path`**, **`tests_path`** (required). `output_path` (optional base, or `.md`/`.html` path), `retained_proofs` (optional override) | Absolute paths to both written artifacts: a plain, portable Markdown report and a self-contained, shareable HTML report. The server includes only gate-retained proofs for that pair. Deterministic; sends nothing to any API. |
| `umbra_glossary` | `term` (optional; omit to list every term with a short gloss) | Umbra's fixed authoritative definition and sources for the term, or the full term index. An unknown term is named plainly, never guessed. No model call; sends nothing anywhere. |

**Proof backends** (`umbra_audit` → `model_backend`):

- **`off`** (default): fully deterministic, zero API calls, no key required. Returns the isolated proof requests for Codex to write and verify itself.
- **`codex`**: requests host-side MCP sampling, so Codex writes each proof under its own reasoning.
- **`openai`**: uses a locally configured `OPENAI_API_KEY`, and only after `confirm_openai: true`. Never pass a key as a tool argument. The disclosed `UMBRA_OPENAI_FALLBACK=1` is a separate opt-in, used only when host sampling is unavailable.

If a model is reached at all, every backend sends OpenAI only one isolated function source, its type contract, and one located gap per request — never the repository, test files, or other functions.

**Terms `umbra_glossary` defines.** Silence index, closeable facet, closeable silence, undeclared domain, model-ready gap, runtime property, render failure, `honesty_unverified`, unexercised branch, candidate input region, unasserted return contract, exception path, mutation, determinism, purity, idempotency, apophatic, umbra, Slop Audit, call-stack map, boundary_in, orchestrator, pure, boundary_out, `.hd` format, execution gate, and coverage versus silence.

In the **Codex CLI**, register Umbra's MCP server directly (add `--env UMBRA_JAVA_TOOLS=/your/java/tools` for Java JUnit audits; see Platform support):

```bash
codex mcp add umbra -- sh -c 'cd /path/to/umbra/plugins/umbra-codex && uv run --project ./pyproject.toml python ./server.py'
```

In **Codex inside ChatGPT**, register the repository as a plugin marketplace (its manifest is `./.agents/plugins`) and install **Umbra Codex** from the Plugins page, then restart Codex. The plugin exposes `umbra_audit`, which takes `module_path`, `tests_path`, and an optional proof backend. Its deterministic audit needs no API key. It returns capped `proof_requests` in structured output. Each request contains only one isolated function source, its type contract, and one located gap description. It never sends the repository, the test file, or other functions.

The plugin bundles an Umbra wheel and a plugin-local `uv` project. It can start from the Codex plugin cache or any copied directory without importing this repository. Its first launch creates a local uv environment and may download transitive dependency wheels; later launches reuse that environment.

For each proof request the audit returns, Codex writes one candidate failing test from that request alone and calls `umbra_verify_proposal` with the proposal and the request index. Umbra runs the verification gate and retains the proof only when it rendered into a runnable test that genuinely fails or makes the audited function error during execution. A malformed input or property is rejected with the field to correct, so Codex can resubmit it instead of presenting it as proof. Generating and verifying a proof runs in a sandbox and changes nothing: Umbra proves a gap, it never writes the test into your suite. Adopting a surviving proof is your own choice.

Codex writes each proof with its own reasoning through the verify step above, so no host-side MCP sampling is required (Codex does not support it). Requesting the inline path with `generate_proof: true` instead asks the host to run the model, finds sampling unavailable, and returns the deterministic result. A separate inline OpenAI backend is off by default: set `UMBRA_OPENAI_FALLBACK=1` and configure `OPENAI_API_KEY` only if you explicitly authorize it. When used, Umbra reports that the fallback was authorized and that each request sent only one isolated function source, its type contract, and one located gap. It never sends the repository, test files, or other functions.

After the proofs, Codex generates both report artifacts with `umbra_write_report` and presents the self-contained HTML visual. Report writing is local and deterministic, sends no data to any API, and never gates proof generation; it returns absolute paths for you to open.

### Optional OpenAI backend

Never paste an API key into a Codex message or MCP tool call. Configure `OPENAI_API_KEY` in the local environment that starts Codex or the Umbra MCP server, then select `model_backend: "openai"` and `confirm_openai: true` in `umbra_audit`. Without that explicit confirmation, the plugin makes zero API calls. Alternatively, `UMBRA_OPENAI_FALLBACK=1` authorizes the disclosed fallback only after Codex host sampling is unavailable. The same isolated function, contract, and gap boundary applies.

## Platform support

Umbra runs on macOS and Linux. Two things bound where it runs and what it can measure, and it names both rather than pretending otherwise:

- **macOS and Linux today, not Windows.** Umbra runs each piece of audited code in its own process group and stops the whole group on timeout, using POSIX process groups (`os.killpg`, a new session, `SIGKILL`). Windows has no equivalent path yet, so a timeout would fail there. A Windows port using job objects is future work.
- **Non-Python coverage needs that language's own toolchain, and providing it is the user's job, not the plugin's.** Python audits need only Python. TypeScript and JavaScript need Node; a Vitest suite is resolved from the audited project's own `node_modules`, while a `node:test` suite needs only Node. Java needs a JDK, the JaCoCo agent, and, for JUnit suites, the JUnit console launcher; all three are found through the directory named by the `UMBRA_JAVA_TOOLS` environment variable, falling back to the repository `tools/` directory. Rust needs `cargo-llvm-cov` and LLVM; Go needs `go`. Where a toolchain, or a detected framework's tool, is absent, that language's suite is reported as unmeasured with the reason, never guessed.

## What Umbra does not do

Apophasis cuts both ways: the tool that names the shadow in your tests names its own.

- **It proves. It does not fix.** Umbra writes the failing test that proves a gap. It never edits your code and never decides what the code should do. The fix is yours.
- **It finds one kind of blindness, not every bug.** A quiet Umbra means nothing structurally locatable is unseen. It is not a promise your code is correct.
- **On open inputs it tests the edges, not everything.** Bounded types are enumerated; unbounded ones (any int, any string) get boundary cases, not totality.
- **A generated test's expectation is a plausible guess, kept only when it fails.** A surviving failure proves a real discrepancy exists; it does not claim to know your intended contract. You judge which side is wrong.
- **Runtime honesty is not uniform across languages yet.** Line coverage and branch facets are now measured for all six languages, each via its native tool (coverage.py, Vitest V8, JaCoCo, cargo llvm-cov, `go test`), and a branch that the coverage data cannot attribute is marked `honesty_unverified` rather than guessed. What is not uniform is runtime honesty: mutation, purity, and idempotency are verified for Python. JavaScript verifies argument mutation and same-input output determinism, and attempts idempotency only when the suite asserts a re-application. TypeScript, Java, Rust, and Go report those properties as unverified rather than inventing them.

## Try it (judges: nothing to rebuild)

Bundled examples run in one command:

```bash
uv run umbra audit examples/cart.py --tests examples/test_cart.py
```

The suite is green at 100% coverage; Umbra reports the silence and ends on a real failing test it wrote.

The Java fixture is equally self-contained:

```bash
uv run umbra audit examples/Cart.java --tests examples/CartTest.java
```

The Rust fixture is equally self-contained and needs no Cargo project:

```bash
uv run umbra audit examples/cart.rs --tests examples/cart_test.rs
```

`examples/delivery.rs` shows the bounded-enum rule: each enum variant is a finite input region, so Umbra can name every untested variant rather than sampling one.

The Go fixture runs without a module or test framework:

```bash
uv run umbra audit examples/cart.go --tests examples/cart_check.go
```

## Benchmark: a challenge to beat

Umbra's proof loop is a measurement instrument, and `umbra benchmark` turns it into a fixed, reproducible, model-agnostic benchmark: given one located gap under a strict schema and an execution gate, how many debugging cycles does a model need to produce a failing test that genuinely runs and fails?

Twenty fixed Python gaps span input regions, unexercised branches, and unasserted exception paths. Two NaN gaps are labeled inherently unprovable, since no valid Python literal expresses NaN, and are excluded from the model's rates. Every outcome comes from real execution, not a self-report.

Run the reference backend (GPT-5.6) with `OPENAI_API_KEY` set:

```sh
uv run umbra benchmark --runs 3
```

Reference run (GPT-5.6, 60 trials): **100% first-try valid, 0.00 mean debugging cycles.** That is the number to beat. To run it against your own model, see `benchmark/README.md`.

## Docs

- `SUBMISSION.md`: problem, solution, approach, and how Codex and GPT-5.6 were used.
- `docs/BUILD-NOTES.md`: the Codex collaboration log and the honest velocity count.
- `docs/SPEC.md`: the technical target Codex builds to.
- `docs/HONEST-CODE-PRINCIPLES.md`, `docs/HONEST-TEST.md`: the standards Umbra is built on.
- `benchmark/README.md`: the reproducible proof-cycle benchmark.

## License

Code is **Apache-2.0**, matching the Honest Framework: see `LICENSE`.

Documentation content (`docs/`, `SUBMISSION.md`, and the explanatory prose here) is **CC-BY-NC-4.0**: see `LICENSE-DOCS`. Code examples inside the documentation are part of the software and carry the Apache 2.0 licence.

Public repo. (Fallback if kept private: share with testing@devpost.com and build-week-event@openai.com.)

## Submission facts

- Track: **Developer Tools**
- Entrant: **Open Honest Foundation** (incorporated Missouri nonprofit); representative: Adam Z. Wasserman
- Deadline: 2026-07-21, 5:00 PM PT (7:00 PM CDT)
- Codex `/feedback` session ID (core build): `019f62bc-61bc-7631-8402-dfba41870f89`
- Repo: public, Apache-2.0 (`github.com/openhonest/umbra`)

[^definition]: https://github.com/openhonest/umbra/blob/HEAD/Apophatic means describing something by what it is not. It comes from apophatic theology (the *via negativa*), which approaches the divine only by negation. Umbra applies the same move to tests: it reports the cases they cannot see, not the ones they catch.