open-compute-mcp

npm launcher for the open-compute MCP server — model-agnostic computer-use tools exposed over the Model Context Protocol (MCP).

EN | DE

📦 View on npm → • 📋 Security Policy • ⚖️ Licenses • 🤖 LLM Context (llms.txt)

Quick Navigation

• ✨ Key Capabilities
• 🏗️ Architecture
• 🛠️ Tools (16)
• 🚀 Use with an MCP Client
• 🔄 Safe Interaction & Signal Lifecycle
• ⚙️ Configuration
• 🔒 Safety & Security
• 🌐 ellmos-ai Ecosystem

[!NOTE] AI Assistant / Agent Integration: This repository contains an llms.txt file providing structured, machine-readable specifications of tools, safety modes (OC_SAFETY_MODE), and client configuration examples for RAG crawlers and autonomous agent frameworks.

The MCP client is the reasoner (no API key, model-agnostic): it calls capture to see the screen, then acts with do / click_name / invoke. This is the keyless Mode-A loop of open-compute, but as native tool-calls.

Key Capabilities

1. State-bound Perception & Window Targeting: Captures/trees return one-shot observation IDs; window enumeration returns stable window/process IDs and issued tokens. WGC remains the GPU-window fallback.
2. Fail-closed Action Execution: Coordinates consume one observation and exact window binding; UIA names resolve exact-first; text is segmented with focus checks and character-count postconditions.
3. Leased Signal Overlay & Abort Control: The glowing border/cursor signal has owner/session metadata, a bounded TTL, turn-end cleanup, and immediate human abort.
4. Multimodal Collaboration & Voice Notes: Push-to-talk voice recording (talk), screen chat messaging (chat), directory monitoring (watch_dir), and macro replay (rec_replay).

Architecture

graph TD
    A["AI Reasoner<br/>(Claude / Antigravity / Cursor)"] -- "MCP stdio (JSON-RPC)" --> B["npx open-compute-mcp<br/>(Node.js Launcher)"]
    B -- "Spawns via uvx" --> C["open-compute Python Engine<br/>(GitHub @ main)"]
    C -- "Screenshots / WGC" --> D["Windows Display"]
    C -- "UIA / Mouse / Keys" --> E["Windows Desktop Apps"]
    C -- "Glowing Border & Cursor" --> F["Signal Overlay UI"]

    subgraph Safety Gate
        C -. "OC_SAFETY_MODE<br/>(confirm / read_only / allow_all)" .-> C
        C -. "OC_DENY<br/>(hard action blacklist)" .-> C
    end

This package is a thin launcher. It contains no server logic — it spawns the Python open-compute server (pulled from GitHub) and pipes MCP stdio through. Real screen capture and input require the interactive Windows desktop session.

Requirements

• Python 3.10+ and uv on the host. The default launch uses uvx to fetch open-compute (with the mcp extra) from GitHub on first run — the mcp extra tracks the GitHub repo, so this works regardless of PyPI release timing.
• Windows for real capture/input (mss + UIA). Other platforms import the tools but cannot drive a desktop.

Tools

All coordinates are normalized 0..1 relative to the virtual desktop. Tool descriptions are localized in six languages (de/en/es/ja/ru/zh) via OC_LANGUAGE.

do also accepts the hold primitives mouse_down / mouse_up / key_down / key_up for press-and-hold sequences (rubber-band selection, modifier-held clicking, game input); anything still held is released when the server stops. capture(window=...) falls back to Windows.Graphics.Capture when a plain grab of a hardware-composited window (Roblox Studio, Blender, a GPU-accelerated browser) comes back all-black — install the wgc extra for that.

Safe Interaction & Signal Lifecycle

The v0.8 Python engine enforces observe → one action → automatic refresh. Keep the full descriptor or window_token from list_windows, then pass it as expected_window together with the latest observation_id from capture or tree. click_name/invoke require that issued window too. Reuse, changed state, focus mismatch, covered windows, and ambiguous UIA targets are rejected before input. type returns requested/sent character counts and complete/partial status without echoing the text. Signals have a hard TTL and are removed at action turn end unless keep_signal=true.

An explicit signal_show starts the engine's configured pre-action grace period. The static grace color is distinct from the mode color and the visible text counts down Start in N Sekunden once per second. At zero, both phase and color switch once to active. signal_status exposes the same phase, remaining seconds, current color, and screenreader label. Duration, grace color, and text template come from OC_SIGNAL_GRACE_SECONDS / OC_SIGNAL_CONFIG; 0 skips the countdown. The design uses no flashing, pulsing, or motion animation.

sequenceDiagram
    autonumber
    actor Reasoner as AI Reasoner (Claude / AGY)
    participant Launcher as Node.js Launcher (open-compute-mcp)
    participant Engine as Python Engine (open-compute)
    participant UI as Windows Desktop / UIA
    actor Operator as Human Operator

    Note over Reasoner,Operator: Phase 1: Visual Perception & State Inspection
    Reasoner->>Launcher: capture(window?) / tree()
    Launcher->>Engine: Forward stdio JSON-RPC
    Engine->>UI: Grab Screen (mss/WGC) or Read UIA Tree
    UI-->>Engine: Frame Image / Semantic Element Tree
    Engine-->>Launcher: Observation ID + normalized response/image
    Launcher-->>Reasoner: State-bound visual observation

    Note over Reasoner,Operator: Phase 2: Signal Overlay Activation
    Reasoner->>Launcher: signal_show(mode="control")
    Launcher->>Engine: Invoke Signal Overlay
    Engine->>UI: Render static grace color + Start in N seconds
    UI-->>Operator: Text countdown + accessible window name
    Engine->>UI: At zero, switch once to the mode color