kvm-pilot

Smart hands for your AI agents. A write-capable, multi-plane (KVM + BMC + SSH) MCP server for controlling physical machines — gated, verified, audited.

kvm-pilot lets an agent drive a headless box through POST, firmware, the bootloader, and an OS install with no agent on the target: it works at the pixel level through an IP-KVM (PiKVM, the GL.iNet GLKVM fork GL-RM1 / GL-RM1PE, BliKVM), at the structured-state level through a BMC (Redfish on iDRAC/iLO/OpenBMC, IPMI on BMCs that predate Redfish), at the firmware level through Intel AMT/vPro (a BIOS/POST/GRUB screenshot + power + SOL on a business laptop an HDMI-capture KVM can't see boot on), and over SSH once an OS is up. A pluggable vision subsystem reads a KVM screenshot and tells you what boot phase the machine is in — bios_menu, grub_menu, installer_progress, login_prompt, crash_screen, and so on — and a safety layer gates every destructive operation behind operator opt-ins and per-call approvals.

Vision runs on Claude or any local OpenAI-compatible VLM (LM Studio, Ollama, vLLM, llama.cpp). Point it at a model on your own GPU and the screenshots never leave your network and cost nothing per frame.

How it works

kvm-pilot runs a see → decide → act loop, and the screen is its only sensor: it pulls a screenshot from the KVM, a vision model classifies the boot phase, and kvm-pilot acts back through the KVM's keyboard and power. Because it works at the pixel level, there is no agent on the target — the same loop drives POST, firmware, the bootloader, and an OS install.

A real, unedited run against a GLKVM on the home fleet — an honest preflight, a headless snapshot (watch it wake the on-demand encoder), a gated dry-run power-cycle, and the boot console the agent actually saw:

Quickstart

One install gives you the whole product — the kvm-pilot CLI, the kvm-pilot-mcp MCP server, and the bundled Claude skill — nothing to clone. The current release line is a pre-release: install it with --pre, which is explicit and keeps selecting the beta line once a stable release ships (0.1.0a1 is yanked and much older than this README — don't pin it).

pip install --pre kvm-pilot                    # CLI + skill + MCP server + WebSocket events
pip install --pre "kvm-pilot[totp]"            # + 2FA / TOTP support (pyotp)
kvm-pilot install-skill                        # optional: put the bundled skill where Claude Code loads it

Driving a KVM from an AI agent (MCP)

claude mcp add kvm-pilot -s user \
    -e KVM_PILOT_PROFILE=mykvm -e KVM_PILOT_MCP_READ_ONLY=1 -- \
    kvm-pilot-mcp

KVM_PILOT_MCP_READ_ONLY=1 is the recommended first rung of the trust ladder — the agent can see everything and touch nothing until your hardware is verified. The Getting started guide covers credentials, Claude Desktop JSON config, sample prompts, and climbing the ladder. The server is published to the official MCP registry as io.github.DustinTrap/kvm-pilot, so registry-aware hosts can discover and install it by name. Agents: the repo root carries an llms.txt doc map.

Scripting from Python

from kvm_pilot import KVMClient
from kvm_pilot.vision import ScreenAnalyzer, make_backend

kvm = KVMClient("192.168.8.1", "admin", "secret")

# Classify the current screen with Claude (model auto-resolved at runtime)
analyzer = ScreenAnalyzer(kvm, make_backend("anthropic"))
print(analyzer.classify().phase)

# Or run entirely on a local VLM — nothing leaves your network
local = make_backend("local", base_url="http://127.0.0.1:1234/v1", model="qwen2.5-vl-7b")
analyzer = ScreenAnalyzer(kvm, local)

# Block until the box reaches the GRUB menu, then pick the first entry
analyzer.wait_for_state("grub_menu", timeout=120)
kvm.press_key("Enter")

For the latest unreleased tree:

pip install "kvm-pilot[totp,ws] @ git+https://github.com/DustinTrap/kvm-pilot"

CLI

kvm-pilot info     --host 192.168.8.1 --user admin --ask-passwd   # prompt (no echo)
kvm-pilot capabilities --profile homelab                 # what this driver supports
kvm-pilot snapshot screen.jpg --profile homelab
kvm-pilot --timeout 60 power-cycle --profile homelab --dry-run   # log, don't send
kvm-pilot eject --profile homelab                        # detach virtual media
kvm-pilot events --profile homelab --count 5             # stream device events
kvm-pilot watch grub_menu --profile homelab \
    --backend local --vision-url http://127.0.0.1:1234/v1 --vision-model qwen2.5-vl-7b

The CLI prompts for confirmation before any destructive action (power, virtual media — including uploads — keyboard/mouse injection, GPIO). Use --yes to skip prompts in automation, or --dry-run to log intended actions without sending them — dry-run short-circuits before the prompt, so it never blocks waiting for input. --timeout (HTTP per-request timeout) is a global flag and goes before the subcommand; watch keeps its own --timeout for the vision wait deadline.

Profiles like homelab live in ~/.config/kvm-pilot/config.toml. See docs/cli.md for the full command table (every subcommand, the capability it needs, and its gating), and docs/configuration.md for the config-file format, every KVM_PILOT_* environment variable, and the precedence between flags, env, and profiles.

GLKVM setup note: on GL.iNet firmware the PiKVM REST API is disabled by default (every /api/* call 404s, surfaced as a clear ApiDisabledError), and a firmware upgrade can re-disable it. Enable it in /etc/kvmd/nginx-kvmd.conf and pin the driver with --driver glkvm / driver = "glkvm" — full steps in the troubleshooting guide.

The tool surface, by plane

The same capability protocols span three actuation planes, so one agent workflow can mix pixels, structured BMC state, and shell access — with every destructive effect gated per class:

A machine with no KVM and no BMC at all is itself a first-class target: driver = "ssh" plus ssh_host (the machine's own address, #248) stands kvm-pilot on the recovery ladder's in-band rung — every device capability honestly absent, and the healthcheck saying plainly that there is no out-of-band recovery path there.

The canonical per-tool reference — annotations, effect gates, approval lifecycle — is the MCP server README; the CLI covers the full surface in docs/cli.md.

Status & maturity