Umbra

Reveals what your tests are not telling you.

Umbra is an apophatic¹ code auditor. Every other tool reports what it found: coverage says which lines ran, a linter says which lines smell, a green suite says all the tests pass everything they can see. None of them report what the tests are structurally unable to see. That blind region, the umbra, is where agent-written code fails.

Umbra names the shadow and proves it. For a Python, TypeScript, JavaScript, Java, Rust, or Go module and its test file(s). Umbra maps what the tests actually exercise, finds the reachable and meaningful cases nothing asserts, and reports a Silence index: the honest complement to coverage. Higher means more blind spots. Not what ran. What nothing could see. It then writes failing tests that will close the gap, and examines your functions for purity.

Built with Codex, using GPT-5.6. Track: Developer Tools. GPT-5.6 runs only where structure runs out and only meaning remains.

Umbra is the newest instrument in the Slop Audit (Open Honest Foundation), the same catch-what-metrics-miss-by-construction discipline taken from an enterprise codebase down to a single module and its test suite. Both rest on one premise: name what the instrument cannot see. Umbra's page: slopaudit.org/umbra.html.

Install

Umbra runs two ways: as a Codex plugin, its home, auditing code the moment the agent writes it (see "Use from Codex" below); and as a standalone CLI. Either way it audits six languages (Python, TypeScript, JavaScript, Java, Rust, Go).

The CLI is a Python package, run with uv. From a clone:

uv tool install .        # or: uv run umbra --help

Usage

Umbra runs two ways. Its home is the Codex plugin; the CLI is the standalone path. Both are shown here.

In Codex (the plugin)

Once the plugin is registered (see "Use from Codex" below for the one-time setup and the full tool reference), you drive it in plain language. The surface is four tools, and here is what you can ask for:

• Audit a module and its tests. "Audit this module and its test file with Umbra." Codex calls umbra_audit, reports coverage beside the Silence index, and names every blind spot. Deterministic, and needs no API key.
• Prove the gaps. For each blind spot the audit returns one isolated proof request. Codex writes a single failing test from it and calls umbra_verify_proposal; Umbra's execution gate keeps only a test that genuinely fails or makes the function error, and rejects a malformed one with the exact field to fix. Umbra never writes into your test file, so adopting a surviving proof is your choice.
• Get the report. "Write the report." Codex calls umbra_write_report and presents the self-contained HTML alongside the portable Markdown. Local and deterministic; sends nothing anywhere.
• Ask what a term means. "What does Umbra mean by the Silence index?" Codex calls umbra_glossary and relays Umbra's own fixed definition instead of improvising one. Omit the term to list every defined term.
• Stay fully offline. The audit is deterministic by default (model_backend: off) and makes zero API calls. Only proof generation can use a model, and only the host sampler unless you opt in: the OpenAI backend stays off until you set confirm_openai (or the disclosed UMBRA_OPENAI_FALLBACK).

On the command line

Point Umbra at a source module and its test file, in any supported language:

uv run umbra audit path/to/module.py --tests path/to/test_module.py

Umbra prints the Coverage-versus-Silence report and writes the generated failing test using the audited language's file extension. It reads each test file to identify its framework and runs the matching one: pytest for Python; Vitest or Node's built-in node:test for TypeScript and JavaScript; JUnit or a throw-on-failure main harness for Java; the #[test] suite for Rust, whether inline in the module or in a separate file; and go test for Go, including named and external test packages. Coverage comes from each language's own tool: coverage.py, Vitest or Node V8, JaCoCo, cargo-llvm-cov, and the Go cover profile. A framework Umbra does not recognize, or a run it cannot complete, is reported as not measured with the reason attached, never a guessed number.

Command reference

umbra audit [MODULE] — audit a single source file (with --tests), a Python project directory, or a Git diff (--since).

umbra benchmark — run the reproducible proof-cycle benchmark on the fixed Python gap set.

Environment variables (CLI and plugin)

Harden mode: close the gaps, not just name them

Add --harden to turn the audit into a loop. Umbra generates the tests that close each open facet, keeps the passing ones as regression guards and the failing ones as bug proofs, re-audits, and repeats until no closeable silence remains. The Silence index is the loop's convergence target, so "done" is definable, unlike coverage-chasing.

uv run umbra audit path/to/module.py --tests path/to/test_module.py --harden

Harden generates tests, so it uses the model layer (or Codex). The call, iteration, and per-iteration budgets are bounded by --harden-call-budget, --harden-iteration-budget, and --harden-calls-per-iteration. Run without --harden for a fully deterministic audit.

Python project mode

Give audit a repository directory to audit every real Python source module against the entire discovered pytest suite. Umbra runs that suite once under branch coverage, attributes resolvable calls from every test file to their source module, and writes umbra-project-report.md with per-module results plus the repository aggregate. Re-export-only __init__.py files are skipped. A module that cannot be parsed, or whose suite does not complete, is listed as unaudited and is excluded from the aggregate.

uv run umbra audit path/to/repository --no-model

Project mode is deterministic and sends no data to any API. It currently supports Python repositories with flat or src/ layouts and conventional tests/ or test/ filenames. A suite with failing tests remains measurable when coverage exists: the report names the pass count, fail count, and exit status. A run that produces no coverage is explicitly unmeasured and excluded from the index.

Time limit for audited code

Umbra gives each subprocess that runs audited or generated code a 10-second limit. It stops the whole process group, so a timed-out test or replay cannot leave child processes behind. Set UMBRA_TIMEOUT_SECONDS to a positive number of seconds to adjust that limit:

UMBRA_TIMEOUT_SECONDS=30 uv run umbra audit path/to/module.py --tests path/to/test_module.py

A timed-out replay becomes an honesty_unverified result with the timeout reason. A timed-out coverage run is explicitly unmeasured and does not receive a Silence index.

Reports made for humans. Not the usual wall of text a test report buries you in: a short, readable account of what your suite cannot see, with the one failing test that proves it.

JSON output

For scripts and CI, add --json. Standard output is one JSON object only; the human report is sent to standard error, while the Markdown and HTML reports are still written as usual.

uv run umbra audit examples/cart.py --tests examples/test_cart.py --no-model --json

The stable version-1 keys are schema_version, module_path, tests_path, language, coverage_percent, coverage_status, silence_index, silences, unverified_properties, generated_proofs, honest_maps, and model. silence_index contains the percent, numerator, denominator, and facet counts. Every silence records its function, kind, location, explanation, and whether it is closeable. Every generated proof records its gap, status, rendered test, captured output, and proposal. For a language whose coverage is not measured, coverage_status is "not_measured" and coverage_percent is null.

Use --json-out path/to/audit.json to write the same document to a file. It can be combined with --json.

In project mode, --json emits one aggregate object with audit_kind: "project", the suite status, per-module audited or unaudited status, aggregate Silence index counts, and the by-kind breakdown.

Data sent to OpenAI

When OPENAI_API_KEY is present and the model layer runs, Umbra sends OpenAI one isolated function source, its type contract, and one located gap description for each proposed proof. It does not send the repository, test files, a whole-module site list, or any other functions.

Run fully offline, with zero API calls:

uv run umbra audit path/to/module.py --tests path/to/test_module.py --no-model

--deterministic is an alias for --no-model.

Audit a Git diff

Audit only the conventional module and test pairs touched since a Git ref:

uv run umbra audit --since origin/main --no-model

Umbra writes umbra-diff-report.md. It reports each pair's prior and current Silence index, the newly introduced blind spots, an aggregate delta, and files it could not pair, including deletions and unconventional test names. With the model enabled, only newly located gaps receive isolated proof requests.

The repository action is action.yml. The sample pull-request workflow is umbra-diff.yml: it sets up Python and uv, comments with umbra-diff-report.md, and can fail when the aggregate Silence index increase exceeds silence-threshold. Python is the guaranteed runner. Other supported-language files are reported only when their existing structural path can run; the comment names exactly which languages Umbra audited.

Use from Codex

This repository ships a local Codex MCP plugin at plugins/umbra-codex. Use it whichever way you run Codex.

Plugin tools (MCP)

The plugin exposes exactly four tools. The usual loop is umbra_audit → umbra_verify_proposal (once per returned proof request) → umbra_write_report; umbra_glossary is independent and answers term questions at any time.

Proof backends (umbra_audit → model_backend):

• off (default): fully deterministic, zero API calls, no key required. Returns the isolated proof requests for Codex to write and verify itself.
• codex: requests host-side MCP sampling, so Codex writes each proof under its own reasoning.
• openai: uses a locally configured OPENAI_API_KEY, and only after confirm_openai: true. Never pass a key as a tool argument. The disclosed UMBRA_OPENAI_FALLBACK=1 is a separate opt-in, used only when host sampling is unavailable.

If a model is reached at all, every backend sends OpenAI only one isolated function source, its type contract, and one located gap per request — never the repository, test files, or other functions.

Terms umbra_glossary defines. Silence index, closeable facet, closeable silence, undeclared domain, model-ready gap, runtime property, render failure, honesty_unverified, unexercised branch, candidate input region, unasserted return contract, exception path, mutation, determinism, purity, idempotency, apophatic, umbra, Slop Audit, call-stack map, boundary_in, orchestrator, pure, boundary_out, .hd format, execution gate, and coverage versus silence.

In the Codex CLI, register Umbra's MCP server directly (add --env UMBRA_JAVA_TOOLS=/your/java/tools for Java JUnit audits; see Platform support):

codex mcp add umbra -- sh -c 'cd /path/to/umbra/plugins/umbra-codex && uv run --project ./pyproject.toml python ./server.py'

In Codex inside ChatGPT, register the repository as a plugin marketplace (its manifest is ./.agents/plugins) and install Umbra Codex from the Plugins page, then restart Codex. The plugin exposes umbra_audit, which takes module_path, tests_path, and an optional proof backend. Its deterministic audit needs no API key. It returns capped proof_requests in structured output. Each request contains only one isolated function source, its type contract, and one located gap description. It never sends the repository, the test file, or other functions.

The plugin bundles an Umbra wheel and a plugin-local uv project. It can start from the Codex plugin cache or any copied directory without importing this repository. Its first launch creates a local uv environment and may download transitive dependency wheels; later launches reuse that environment.

For each proof request the audit returns, Codex writes one candidate failing test from that request alone and calls umbra_verify_proposal with the proposal and the request index. Umbra runs the verification gate and retains the proof only when it rendered into a runnable test that genuinely fails or makes the audited function error during execution. A malformed input or property is rejected with the field to correct, so Codex can resubmit it instead of presenting it as proof. Generating and verifying a proof runs in a sandbox and changes nothing: Umbra proves a gap, it never writes the test into your suite. Adopting a surviving proof is your own choice.

Codex writes each proof with its own reasoning through the verify step above, so no host-side MCP sampling is required (Codex does not support it). Requesting the inline path with generate_proof: true instead asks the host to run the model, finds sampling unavailable, and returns the deterministic result. A separate inline OpenAI backend is off by default: set UMBRA_OPENAI_FALLBACK=1 and configure OPENAI_API_KEY only if you explicitly authorize it. When used, Umbra reports that the fallback was authorized and that each request sent only one isolated function source, its type contract, and one located gap. It never sends the repository, test files, or other functions.

After the proofs, Codex generates both report artifacts with umbra_write_report and presents the self-contained HTML visual. Report writing is local and deterministic, sends no data to any API, and never gates proof generation; it returns absolute paths for you to open.

Optional OpenAI backend

Never paste an API key into a Codex message or MCP tool call. Configure OPENAI_API_KEY in the local environment that starts Codex or the Umbra MCP server, then select model_backend: "openai" and confirm_openai: true in umbra_audit. Without that explicit confirmation, the plugin makes zero API calls. Alternatively, UMBRA_OPENAI_FALLBACK=1 authorizes the disclosed fallback only after Codex host sampling is unavailable. The same isolated function, contract, and gap boundary applies.

Platform support

Umbra runs on macOS and Linux. Two things bound where it runs and what it can measure, and it names both rather than pretending otherwise:

• macOS and Linux today, not Windows. Umbra runs each piece of audited code in its own process group and stops the whole group on timeout, using POSIX process groups (os.killpg, a new session, SIGKILL). Windows has no equivalent path yet, so a timeout would fail there. A Windows port using job objects is future work.
• Non-Python coverage needs that language's own toolchain, and providing it is the user's job, not the plugin's. Python audits need only Python. TypeScript and JavaScript need Node; a Vitest suite is resolved from the audited project's own node_modules, while a node:test suite needs only Node. Java needs a JDK, the JaCoCo agent, and, for JUnit suites, the JUnit console launcher; all three are found through the directory named by the UMBRA_JAVA_TOOLS environment variable, falling back to the repository tools/ directory. Rust needs cargo-llvm-cov and LLVM; Go needs go. Where a toolchain, or a detected framework's tool, is absent, that language's suite is reported as unmeasured with the reason, never guessed.

What Umbra does not do

Apophasis cuts both ways: the tool that names the shadow in your tests names its own.

• It proves. It does not fix. Umbra writes the failing test that proves a gap. It never edits your code and never decides what the code should do. The fix is yours.
• It finds one kind of blindness, not every bug. A quiet Umbra means nothing structurally locatable is unseen. It is not a promise your code is correct.
• On open inputs it tests the edges, not everything. Bounded types are enumerated; unbounded ones (any int, any string) get boundary cases, not totality.
• A generated test's expectation is a plausible guess, kept only when it fails. A surviving failure proves a real discrepancy exists; it does not claim to know your intended contract. You judge which side is wrong.
• Runtime honesty is not uniform across languages yet. Line coverage and branch facets are now measured for all six languages, each via its native tool (coverage.py, Vitest V8, JaCoCo, cargo llvm-cov, go test), and a branch that the coverage data cannot attribute is marked honesty_unverified rather than guessed. What is not uniform is runtime honesty: mutation, purity, and idempotency are verified for Python. JavaScript verifies argument mutation and same-input output determinism, and attempts idempotency only when the suite asserts a re-application. TypeScript, Java, Rust, and Go report those properties as unverified rather than inventing them.

Try it (judges: nothing to rebuild)

Bundled examples run in one command:

uv run umbra audit examples/cart.py --tests examples/test_cart.py

The suite is green at 100% coverage; Umbra reports the silence and ends on a real failing test it wrote.

The Java fixture is equally self-contained:

uv run umbra audit examples/Cart.java --tests examples/CartTest.java

The Rust fixture is equally self-contained and needs no Cargo project:

uv run umbra audit examples/cart.rs --tests examples/cart_test.rs

examples/delivery.rs shows the bounded-enum rule: each enum variant is a finite input region, so Umbra can name every untested variant rather than sampling one.

The Go fixture runs without a module or test framework:

uv run umbra audit examples/cart.go --tests examples/cart_check.go

Benchmark: a challenge to beat

Umbra's proof loop is a measurement instrument, and umbra benchmark turns it into a fixed, reproducible, model-agnostic benchmark: given one located gap under a strict schema and an execution gate, how many debugging cycles does a model need to produce a failing test that genuinely runs and fails?

Twenty fixed Python gaps span input regions, unexercised branches, and unasserted exception paths. Two NaN gaps are labeled inherently unprovable, since no valid Python literal expresses NaN, and are excluded from the model's rates. Every outcome comes from real execution, not a self-report.

Run the reference backend (GPT-5.6) with OPENAI_API_KEY set:

uv run umbra benchmark --runs 3

Reference run (GPT-5.6, 60 trials): 100% first-try valid, 0.00 mean debugging cycles. That is the number to beat. To run it against your own model, see benchmark/README.md.

Docs

• SUBMISSION.md: problem, solution, approach, and how Codex and GPT-5.6 were used.
• docs/BUILD-NOTES.md: the Codex collaboration log and the honest velocity count.
• docs/SPEC.md: the technical target Codex builds to.
• docs/HONEST-CODE-PRINCIPLES.md, docs/HONEST-TEST.md: the standards Umbra is built on.
• benchmark/README.md: the reproducible proof-cycle benchmark.

License

Code is Apache-2.0, matching the Honest Framework: see LICENSE.

Documentation content (docs/, SUBMISSION.md, and the explanatory prose here) is CC-BY-NC-4.0: see LICENSE-DOCS. Code examples inside the documentation are part of the software and carry the Apache 2.0 licence.

Public repo. (Fallback if kept private: share with testing@devpost.com and build-week-event@openai.com.)

Submission facts

• Track: Developer Tools
• Entrant: Open Honest Foundation (incorporated Missouri nonprofit); representative: Adam Z. Wasserman
• Deadline: 2026-07-21, 5:00 PM PT (7:00 PM CDT)
• Codex /feedback session ID (core build): 019f62bc-61bc-7631-8402-dfba41870f89
• Repo: public, Apache-2.0 (github.com/openhonest/umbra)