WorkloadTruth

Install • Quickstart • CLI reference • Comparison • FAQ

Classify a GPU workload as TRAINING, INFERENCE, or IDLE from telemetry alone. No code changes to the workload, no self-reported job labels.

Every GPU scheduler in common use today, including run:ai, Slurm, and Kubernetes GPU operators, asks you to declare whether a job is training or inference at submission time. None of them check. WorkloadTruth reads GPU telemetry (utilization, memory pattern, power draw) and answers the question independently, so a mislabeled or misbehaving job doesn't go unnoticed.

Install

# Real NVIDIA GPU telemetry (requires an NVIDIA driver on the host)
pip install "workloadtruth-cli[nvml]"

# Try it without a GPU, using the synthetic backend
pip install workloadtruth-cli

# npm launcher (thin wrapper around the PyPI package, see "Why two registries")
npx workloadtruth-cli --help

[!NOTE] The npm package is a launcher, not a standalone install. npx workloadtruth-cli execs the real workloadtruth binary from PATH, so the PyPI package (pip install workloadtruth-cli) must already be installed first.

Quickstart

# No GPU required. Classify a synthetic "training" telemetry trace.
$ workloadtruth classify --backend synthetic --profile training --samples 10 --interval 0
workload_type : TRAINING
confidence    : 1.00
gpu_index     : 0
samples       : 10 over 9.0s
reasons:
  - avg GPU utilization 87.8% >= training threshold 65.0%
  - low GPU utilization variance (std=3.4) <= training ceiling 15.0
  - memory growing 120.0 MiB/sample >= training threshold 5.0
  - low power-draw variance (std=9.4W) <= training ceiling 25.0W

# Real hardware
$ workloadtruth classify --backend nvml --samples 10 --interval 1 --json

--json on every command switches to machine-readable output for scripts and agents.

Quick summary

• Use it for: catching cost-misallocated GPU jobs (a job billed as low-priority "inference" that's actually running full training) and unauthorized workload changes (an inference endpoint that starts training on live traffic without sign-off)
• What it's not: a compliance or regulatory-audit tool. No regulation currently requires this kind of monitoring, see What WorkloadTruth is not below
• Prior art: builds on and cites arXiv:2606.19262 (ICML 2026), see Relationship to prior research

How classification works

WorkloadTruth currently ships a rule-based classifier only: a set of documented, inspectable thresholds over four features extracted from a telemetry window (average and variance of GPU utilization, memory-growth slope, average and variance of power draw). Every threshold lives as a named constant in src/workloadtruth/classifier/rules.py with a comment explaining its intuition. Nothing is a black box.

--experimental (an ML-based classifier) is present as a flag but fails loudly with an explanation rather than shipping a fake result. arXiv:2606.19262's trained model and dataset were never published, and this project has no NVIDIA GPU in its build environment to collect real training data. An ML classifier ships here only once it's trained on a real, disclosed dataset and independently shown to beat the rule-based baseline, not before.

Benchmark

workloadtruth benchmark measures the classifier's accuracy against documented synthetic GPU telemetry traces (see synthetic_backend.py for the exact profile parameters), both clean and under a deliberate evasion transform that mimics an operator disguising a training job as inference (capping utilization swings, flattening memory growth, injecting artificial idle gaps).

This is run on synthetic data, not live NVIDIA hardware, so it is not directly comparable to arXiv:2606.19262's real-hardware numbers. Both are reported below, side by side, never blended into one figure.

The overall evasion number hides the real finding. Broken out by workload type, WorkloadTruth's rule-based classifier is currently completely fooled on evasive training workloads (0% accuracy), while staying accurate on evasive inference and idle traces (100% each). Capping GPU utilization alone is enough to make a training job read as inference to this classifier today.

This is the concrete, disclosed gap the benchmark suite exists to surface and track, not a footnote. A rule-based classifier that only looks at utilization variance is trivially evadable. Closing this gap, through multi-signal fusion or a real trained classifier once data exists, is the roadmap, not a solved problem. Reproduce it yourself:

workloadtruth benchmark --trials 300 --window 30 --json

CLI reference

$ workloadtruth --help
Usage: workloadtruth [OPTIONS] COMMAND [ARGS]...

  Classify a GPU workload as INFERENCE, TRAINING, or IDLE from telemetry
  alone.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  benchmark   Run the evasion-robustness benchmark against synthetic...
  classify    One-shot classification of the current GPU workload.
  mcp         Start an MCP server exposing classify/benchmark/verify-log...
  verify-log  Verify the hash chain of a local audit log has not been...
  watch       Continuously classify and append hash-chained entries to...

Every command supports --json. Full flag reference: workloadtruth <command> --help.

MCP Server

WorkloadTruth ships a Model Context Protocol server so an AI agent (Claude, Cursor, or any MCP-compatible client) can classify GPU workloads, run the evasion-robustness benchmark, and verify the audit log directly, without a human invoking the CLI by hand.

Install the extra:

pip install "workloadtruth-cli[mcp]"

[!NOTE] The mcp extra requires Python 3.10+, stricter than WorkloadTruth's own 3.9 floor. pip install "workloadtruth-cli[mcp]" will fail to resolve on Python 3.9. Every other feature (classify, watch, benchmark, verify-log) works on Python 3.9.

Add it to your MCP client's config (for Claude Desktop, claude_desktop_config.json). The server is started via the workloadtruth mcp subcommand, not a separate console script:

{
  "mcpServers": {
    "workloadtruth": {
      "command": "uvx",
      "args": ["--from", "workloadtruth-cli", "workloadtruth", "mcp"]
    }
  }
}

The server exposes three tools over stdio:

Read the full README on GitHub →