ultranix-mcp

**ultranix-mcp is the enterprise-grade, secure Linux desktop-automation layer for AI agents.**It gives Model Context Protocol (MCP) clients - Claude Desktop, Claude Code, Cursor, Windsurf, and any MCP-enabled assistant - the ability to see, click, type, and drive a Wayland desktop: mouse, keyboard, screenshots, OCR, icon finding, window management, and accessibility-tree inspection.

ultranix-mcp is Wayland-native by design: compositor protocols first, uinput/evdev second, XDG Desktop Portals last - behind the same governance-and-trust surface as its siblings (ultramac on macOS, ultrawin on Windows): audit logging, rate limiting, input sanitization, and AES-256-GCM-encrypted action history. It is the first Linux desktop MCP to combine a cross-compositor fallback ladder, a full governance surface, and a tri-OS sibling contract - organisations can let agents control a Linux desktop without giving up control themselves.

**Status:**v1.4.0 implemented. Phases 0-5 of ROADMAP.md have shipped, plus the v1.1.0 wave (layer-shell overlay, X11-native providers, PipeWire portal capture, opt-in Sentry, OCR cache, additional metrics), the v1.2.0 breadth wave (clipboard tools, plugin tool-macros, screen_record, sway/Wayfire/river/KDE/GNOME session detection, per-backend cargo features, framed history v2), the v1.3.0 policy wave (runtime access-control policy with per-key roles, --readonly/--allow-tools/--deny-tools, per-backend invocation metrics, optional HMAC-signed audit lines), and the v1.4.0 reach wave (Wayfire/river/GNOME window rungs, live screen_stream capture, plugin-exposed dynamic tools, OCI image + -bin package) - see CHANGELOG.md for per-release notes. The verified target environment is CachyOS (Arch) + Hyprland on Wayland, PipeWire, xdg-desktop-portal-hyprland, and a live AT-SPI2 bus, on Rust 1.98.1.

Features

• Precision Mouse Control- click, double-click, drag, scroll, button-state control, position queries, and smooth path movement via the zwlr_virtual_pointer_v1 protocol. No root, no helper daemons.
• Advanced Keyboard Input- type text and drive key states through virtual-keyboard-unstable-v1, with full modifier and keymap handling.
• Intelligent Vision- in-process wlr-screencopy-unstable-v1 capture, ONNX Runtime OCR (ort crate), and OWL-ViT icon finding. Region screenshots, color sampling (color_at), and session spatial focus (set_spatial_focus scopes screenshot/find_text_on_screen/find_icon), and real screen_highlight overlays via zwlr_layer_shell_v1 (translucent, click-through; -32010 ProviderUnavailable on compositors/sessions without layer-shell - see docs/TOOLS.md).
• Window Management- list, focus, move, resize, close, and inspect windows through Hyprland's hyprctl IPC socket (hyprctl -j JSON: clients, activewindow, dispatch, workspaces), sway's own IPC protocol on $SWAYSOCK, Wayfire's ipc/ipc-rules plugins on $WAYFIRE_SOCKET, the pinned riverctl subprocess + wlroots foreign-toplevel composite on river (riverctl keeps focused-view close and relative move/resize deltas; the toplevel protocol supplies enumeration and per-window focus/close/min/max/ fullscreen), wlr-toplevel as the shared wlroots fallback rung and the sole rung on unknown wlroots sessions, kdotool on KDE, the "Window Calls" Shell extension on GNOME (when installed), or wmctrl on X11 sessions.
• UI Inspection- full accessibility-tree access over AT-SPI2 (atspi crate): UI-tree dumps, focused-element queries, element search, and wait-for-element synchronization.
• Clipboard & Plugins- clipboard_get/clipboard_set/ clipboard_clear over wl-clipboard (Wayland) or xclip/xsel (X11), writes consent-gated; and declarative plugin tool-macros - ~/.ultranix-mcp/plugins/*.json manifests of catalog-tool steps run via plugin_list/plugin_run/plugin_reload, each step re-entering the secured dispatch path. A manifest tool section (v1.4.0) registers the plugin as a first-class tools/list entry with a generated inputSchema, dispatched through the same secured plugin_run pipeline.
• Screen Recording & Live Capture- screen_record captures a frame every interval_ms for up to duration_ms into a fresh rec-<ulid> dir plus a manifest.json (hard caps: 600 frames, 512 MiB); screen_stream (v1.4.0) runs a continuous start/status/latest/ stop rolling-window capture under stream-<ulid> (≤1800 frames, ≤512 MiB, oldest evicted) with latest returning the newest frame in screenshot's image shape - since/wait_ms turn it into a long-poll (park up to 30 s for a frame newer than the watermark) instead of busy-polling.
• Enterprise Security- uxcp_* API-key auth on HTTP, 10 req/s token bucket, input sanitization, command/path whitelists, AES-256-GCM-encrypted action history, and JSONL audit logging. See SECURITY.md.

Architecture

ultranix-mcp is a single Rust 2024 binary on the tokio runtime, built on rmcp - the official Model Context Protocol Rust SDK - with native stdioand streamable-HTTP (:3010) transports.

The desktop-automation layer is organised as provider traits behind dependency injection(the pattern proven in ultrawin's src/traits.rs): every capability is an Option<Arc<dyn Trait>>, so missing compositor features, absent portals, or headless CI degrade gracefully instead of failing hard. Mock providers implement the same traits, which keeps the full tool surface testable without a Wayland session.

1. wlroots-native- in-process Wayland protocols (Hyprland, sway, Wayfire, river, and most unknown wlroots compositors; no root)
2. grim/slurp (capture) + uinput/evdev (input)- whitelisted helper binaries and kernel-level input for non-wlroots sessions
3. XDG Desktop Portal- Screenshot and RemoteDesktop over zbus (universal fallback, subject to portal consent; the only route on KDE/GNOME Wayland, which implement neither wlr-screencopy nor the wlr virtual-input protocols)

Window management rides compositor IPC where it exists: hyprctl on Hyprland, sway's i3-flavoured IPC ($SWAYSOCK, shipped at v1.2.0) on sway, Wayfire's ipc/ipc-rules plugins ($WAYFIRE_SOCKET, v1.4.0) on Wayfire, riverctl + zwlr_foreign_toplevel_manager_v1 on river (riverctl drives focused-view close and relative-delta move/resize; foreign-toplevel enumerates windows and addresses them as wlr-toplevel-N for focus/close/min/max/fullscreen), and wlr-toplevel as the shared wlroots fallback rung - the sole window rung on unknown wlroots sessions (niri, labwc, ...), kdotool (KWin scripting - Wayland and X11 alike) on KDE, the "Window Calls" Shell extension over D-Bus on GNOME (v1.4.0, extension required - org.gnome.Shell.Eval is deliberately unused), and wmctrl on other X11 sessions.

On X11 sessions the X11-native rungs shipped at v1.1.0 resolve instead: scrot capture, xdotool input (both still ahead of portal/uinput), and wmctrl window management on non-Hyprland X11.

graph TB
    subgraph "Client Layer"
        AI[AI Assistant / Agent]
        MCP[MCP Client]
    end

    subgraph "Transport Layer"
        STDIO[STDIO<br/>never requires auth]
        HTTP[Streamable HTTP :3010<br/>uxcp_* API key]
    end

    subgraph "Security Layer"
        AUTH[API Key Auth]
        RATE[Rate Limiter<br/>10 req/s token bucket]
        SAN[Input Sanitization<br/>command + path whitelists]
    end

    subgraph "Core (rmcp + tokio)"
        SERVER[ultranix-mcp server]
        TOOLS[40 tools - 6 categories]
    end

    subgraph "Providers - Option&lt;Arc&lt;dyn Trait&gt;&gt;"
        CAP[CaptureProvider]
        INP[InputProvider]
        UIA[UIAutomationProvider]
        WIN[WindowProvider]
        VIS[VisionProvider]
        BRW[BrowserProvider]
        CLIP[ClipboardProvider]
    end