Umbra

Reveals what your tests are not telling you.

Umbra is an apophatic[^definition] code auditor. Every other tool reports what it found: coverage says which lines ran, a linter says which lines smell, a green suite says all the tests pass everything they can see. None of them report what the tests are structurally unable to see. That blind region, the umbra, is where agent-written code fails.

Umbra names the shadow and proves it. For a Python, TypeScript, JavaScript, Java, Rust, or Go module and its test file(s). Umbra maps what the tests actually exercise, finds the reachable and meaningful cases nothing asserts, and reports a Silence index: the honest complement to coverage. Higher means more blind spots. Not what ran. What nothing could see. It then writes failing tests that will close the gap, and examines your functions for purity.

Built with Codex, using GPT-5.6. Track: Developer Tools. GPT-5.6 runs only where structure runs out and only meaning remains.

Umbra is the newest instrument in the Slop Audit (Open Honest Foundation), the same catch-what-metrics-miss-by-construction discipline taken from an enterprise codebase down to a single module and its test suite. Both rest on one premise: name what the instrument cannot see. Umbra's page: slopaudit.org/umbra.html.

Install

Umbra runs two ways: as a Codex plugin, its home, auditing code the moment the agent writes it (see "Use from Codex" below); and as a standalone CLI. Either way it audits six languages (Python, TypeScript, JavaScript, Java, Rust, Go).

The CLI is a Python package, run with uv. From a clone:

uv tool install .        # or: uv run umbra --help

Usage

Umbra runs two ways. Its home is the Codex plugin; the CLI is the standalone path. Both are shown here.

In Codex (the plugin)

Once the plugin is registered (see "Use from Codex" below for the one-time setup and the full tool reference), you drive it in plain language. The surface is four tools, and here is what you can ask for:

• Audit a module and its tests. "Audit this module and its test file with Umbra." Codex calls umbra_audit, reports coverage beside the Silence index, and names every blind spot. Deterministic, and needs no API key.
• Prove the gaps. For each blind spot the audit returns one isolated proof request. Codex writes a single failing test from it and calls umbra_verify_proposal; Umbra's execution gate keeps only a test that genuinely fails or makes the function error, and rejects a malformed one with the exact field to fix. Umbra never writes into your test file, so adopting a surviving proof is your choice.
• Get the report. "Write the report." Codex calls umbra_write_report and presents the self-contained HTML alongside the portable Markdown. Local and deterministic; sends nothing anywhere.
• Ask what a term means. "What does Umbra mean by the Silence index?" Codex calls umbra_glossary and relays Umbra's own fixed definition instead of improvising one. Omit the term to list every defined term.
• Stay fully offline. The audit is deterministic by default (model_backend: off) and makes zero API calls. Only proof generation can use a model, and only the host sampler unless you opt in: the OpenAI backend stays off until you set confirm_openai (or the disclosed UMBRA_OPENAI_FALLBACK).

On the command line

Point Umbra at a source module and its test file, in any supported language:

uv run umbra audit path/to/module.py --tests path/to/test_module.py

Umbra prints the Coverage-versus-Silence report and writes the generated failing test using the audited language's file extension. It reads each test file to identify its framework and runs the matching one: pytest for Python; Vitest or Node's built-in node:test for TypeScript and JavaScript; JUnit or a throw-on-failure main harness for Java; the #[test] suite for Rust, whether inline in the module or in a separate file; and go test for Go, including named and external test packages. Coverage comes from each language's own tool: coverage.py, Vitest or Node V8, JaCoCo, cargo-llvm-cov, and the Go cover profile. A framework Umbra does not recognize, or a run it cannot complete, is reported as not measured with the reason attached, never a guessed number.

Command reference

umbra audit [MODULE] — audit a single source file (with --tests), a Python project directory, or a Git diff (--since).

umbra benchmark — run the reproducible proof-cycle benchmark on the fixed Python gap set.

Environment variables (CLI and plugin)

Harden mode: close the gaps, not just name them

Add --harden to turn the audit into a loop. Umbra generates the tests that close each open facet, keeps the passing ones as regression guards and the failing ones as bug proofs, re-audits, and repeats until no closeable silence remains. The Silence index is the loop's convergence target, so "done" is definable, unlike coverage-chasing.

uv run umbra audit path/to/module.py --tests path/to/test_module.py --harden

Harden generates tests, so it uses the model layer (or Codex). The call, iteration, and per-iteration budgets are bounded by --harden-call-budget, --harden-iteration-budget, and --harden-calls-per-iteration. Run without --harden for a fully deterministic audit.

Python project mode

Give audit a repository directory to audit every real Python source module against the entire discovered pytest suite. Umbra runs that suite once under branch coverage, attributes resolvable calls from every test file to their source module, and writes umbra-project-report.md with per-module results plus the repository aggregate. Re-export-only __init__.py files are skipped. A module that cannot be parsed, or whose suite does not complete, is listed as unaudited and is excluded from the aggregate.

uv run umbra audit path/to/repository --no-model

Project mode is deterministic and sends no data to any API. It currently supports Python repositories with flat or src/ layouts and conventional tests/ or test/ filenames. A suite with failing tests remains measurable when coverage exists: the report names the pass count, fail count, and exit status. A run that produces no coverage is explicitly unmeasured and excluded from the index.

Time limit for audited code

Umbra gives each subprocess that runs audited or generated code a 10-second limit. It stops the whole process group, so a timed-out test or replay cannot leave child processes behind. Set UMBRA_TIMEOUT_SECONDS to a positive number of seconds to adjust that limit:

UMBRA_TIMEOUT_SECONDS=30 uv run umbra audit path/to/module.py --tests path/to/test_module.py

A timed-out replay becomes an honesty_unverified result with the timeout reason. A timed-out coverage run is explicitly unmeasured and does not receive a Silence index.

Reports made for humans. Not the usual wall of text a test report buries you in: a short, readable account of what your suite cannot see, with the one failing test that proves it.

JSON output

For scripts and CI, add --json. Standard output is one JSON object only; the human report is sent to standard error, while the Markdown and HTML reports are still written as usual.

uv run umbra audit examples/cart.py --tests examples/test_cart.py --no-model --json