HaltProof

Emergency-shutdown orchestration for Slurm, Kubernetes, and IPMI clusters, with an Ed25519-signed, hash-chained attestation record of exactly what ran.

Signed, dry-run-by-default shutdown orchestration for Slurm, Kubernetes, and IPMI clusters, with a tamper-evident audit trail.

HaltProof is an emergency-shutdown orchestration and cryptographic audit-proof layer for compute clusters. It does not implement a new low-level shutdown mechanism. It coordinates cluster primitives you already trust (Slurm, Kubernetes, IPMI/BMC) and produces a signed, tamper-evident record of exactly what was targeted, what ran, and who authorized it.

pip install haltproof-cli

(Full install options, including the npm wrapper, are in Install below.)

Why

Operators already have the tools to drain a Slurm partition, cordon a Kubernetes node pool, or power-fence a physical host over IPMI. What's usually missing is:

1. One consistent interface across those tools during an incident, instead of three different command sets under pressure.
2. A dry-run-by-default safety rail, so a mistyped target group doesn't take down the wrong nodes.
3. A signed, tamper-evident record of what happened: who ran it, what commands were issued, against which nodes, whether each step succeeded, and whether the record itself has been altered or has a piece missing.

HaltProof is that layer. It calls out to scontrol, kubectl, and ipmitool (or mcp-invokable equivalents) and wraps every invocation in an Ed25519-signed, hash-chained attestation.

Install

pip install haltproof-cli

An npm wrapper is also published for Node.js-based tooling and agent runtimes:

npm install -g haltproof-cli

[!WARNING] The npm package requires the Python haltproof-cli package to already be installed and on PATH. It is a thin execFileSync wrapper, not a reimplementation. If it can't find the Python CLI, it prints an actionable error and exits non-zero rather than silently doing something else.

Table of Contents

• Features
• Quickstart
• Commands
• Architecture
• Security model
• MCP server (for AI agents)
• Comparison
• What Is HaltProof, and Why Does It Exist
• FAQ
• Contributing
• License

Features

• Dry-run required by default. haltproof halt prints exactly which commands it would run, against which nodes, and still writes a signed attestation of that plan, unless you pass --confirm. The gate lives at a single chokepoint (haltproof.backends.runner.run_step) that all three backends route through, so it can't be quietly bypassed by one backend's code path.
• One interface across three real backends. kubernetes (via kubectl), slurm (via scontrol), and ipmi (via ipmitool) each implement the same four-operation ClusterBackend interface: drain, isolate_network, power_fence, status. A backend that doesn't support an operation, for example Kubernetes has no physical power-fencing primitive, returns a SKIPPED result with an explanation instead of raising, so a halt across a mixed environment still produces one complete attestation.
• Ed25519-signed, hash-chained attestations. Every halt, dry-run or executed, appends one signed record to a local append-only NDJSON log. Each record embeds prev_hash, the content hash of the record before it, so haltproof verify --chain can catch a deleted or reordered entry that a single record's own signature check would miss.
• Structured output on every command. --json (or --format json) is available on halt, verify, status, and keygen, so a script or an agent can parse the result without screen-scraping human-formatted text.
• A real MCP server, built for agents. haltproof mcp-server exposes halt, verify, verify_chain, status, and keygen as MCP tools over stdio, built on the official mcp Python SDK. It calls the same haltproof.core functions the CLI calls, so an agent invoking the MCP tool gets the identical dry-run-by-default behavior a human gets from the CLI.
• Tested. 76 tests, 90% overall line coverage, run and confirmed in this audit with pytest -v --cov=haltproof --cov-report=term-missing. Backend tests mock subprocess calls; none of them require a real cluster.

Quickstart

[!IMPORTANT] haltproof halt is a no-op without --confirm. It prints the plan and writes a signed dry-run attestation, but nothing destructive runs until you pass --confirm explicitly.

# Generate a signing key for attestations (do this once).
haltproof keygen --output ~/.config/haltproof/ed25519_key

# See what backend HaltProof auto-detected on this host.
haltproof status

# Dry-run a halt against an explicit node list. Nothing runs yet.
haltproof halt gpu-pod-a --nodes node-1,node-2

# Same halt, actually executed.
haltproof halt gpu-pod-a --nodes node-1,node-2 --confirm

# Verify a past attestation's signature and print its timeline.
haltproof verify 1

# Verify the entire attestation log's hash chain instead of a single
# record: this catches a deleted or reordered entry that a
# single-record signature check alone would miss.
haltproof verify --chain --attestation-log ~/.local/share/haltproof/attestations.jsonl

Every command also supports --json for agent-parseable output:

haltproof status --json
haltproof halt gpu-pod-a --nodes node-1 --json

Config file (optional)

Target groups, the preferred backend, and default paths can live in a haltproof.toml file (checked in the current directory, then ~/.config/haltproof/config.toml, or at a path given by --config / HALTPROOF_CONFIG):

backend = "kubernetes"
operator_id = "ops-team"
attestation_log_path = "/var/log/haltproof/attestations.jsonl"

[groups]
gpu-pod-a = ["node-1", "node-2", "node-3"]

[kubernetes]
namespace = "training"

[ipmi]
user = "admin"

Every config value has a matching CLI flag, and an explicit flag always wins over the config file.

Commands

Reference below is regenerated from the CLI's actual --help output.

haltproof halt TARGET_GROUP

Drain, isolate, and power-fence TARGET_GROUP. Dry-run is the default; nothing destructive runs without --confirm.

Options:
  --nodes TEXT             Comma-separated explicit node list, overrides
                           config group lookup.
  --backend TEXT           Backend to use: kubernetes, slurm, or ipmi.
  --confirm                Actually execute the halt. Without this, dry-run
                           only.
  --reason TEXT            Reason recorded for drain operations.
  --operator-id TEXT       Operator identity to record. Defaults to OS user.
  --config TEXT            Path to a haltproof.toml config file.
  --attestation-log TEXT   Path to the attestation log file.
  --attestation-key TEXT   Path to the Ed25519 private key used to sign the
                           attestation.
  --no-sign                Skip signing (not recommended); still logs the
                           record unsigned.
  --remote-collector TEXT  Optional URL to POST the signed attestation to.
  --format [human|json]    Output format.
  --json                   Shorthand for --format=json.

haltproof verify [ATTESTATION_REF]

Verify one attestation record's signature, or an entire log's hash chain. ATTESTATION_REF is a path to a record file, or an attestation id / sequence number to look up in the log. Pass --chain instead of ATTESTATION_REF to verify a whole --attestation-log: every record's signature holds, and none has been deleted or reordered.

haltproof status

Show backend auto-detection results and, optionally, target-group health.

haltproof keygen

Generate an Ed25519 keypair for attestation signing. The private key is written with 0600 permissions and is never printed or logged.

Options:
  --output TEXT          Path to write the private key to.
  --force                Overwrite an existing key at the output path.
  --format [human|json]  Output format.
  --json                 Shorthand for --format=json.

haltproof mcp-server

Read the full README on GitHub →