computer-use-mcp ("realhands")

An MCP server that lets Claude operate your real computer the way a human does — moving the actual mouse, clicking, typing, and reading the actual screen.

Unlike OpenAI Operator, browser-use, or Playwright agents (which spin up a separate, isolated, logged-out Chrome), this drives the physical OS cursor and keyboard. So it works in your own Chrome with your own logged-in sessions — and in every other app — because it's just a human at the keyboard, as far as any website can tell.

Status: LIVE and battle-tested. Registered with Claude Code as the user-scope MCP realhands (tool mcp__realhands__computer) and ✓Connected since 2026-06-02. On 2026-06-09 it drove the user's real, logged-in Chrome through a complete Google Play Console deployment (app upload, release notes, submission) end-to-end.

The server is registered as realhands rather than computer-use because the name "computer-use" is reserved in Claude Code.

How it works

Claude (Desktop or Code) is the agent loop. You type a task; Claude calls the single computer tool in a see → think → act cycle:

See — screenshot returns the real screen (downscaled to ~1280px for grounding accuracy) → Think — Claude picks the next action + pixel coordinates → Act — the server moves the real mouse / types on the real keyboard → a fresh screenshot comes back automatically after every action, and it repeats.

Two Windows-specific details make clicks land accurately (src/screen.py):

• DPI awareness — SetProcessDpiAwareness(2) is set at import time so screenshot pixels == pyautogui cursor coordinates even under display scaling (125% / 150% / …).
• Stateless coordinate scaling — screenshots are downscaled (LANCZOS) to at most COMPUTER_USE_MAX_DIM on the longest side before sending; incoming click coordinates are scaled back up to real pixels. The scale factor is a pure function of monitor geometry + MAX_DIM, so mapping never depends on which screenshot ran last. Coordinates are clamped inside the target monitor so a stray click can't fly off-screen. Each axis is rounded down to a whole 28px vision patch, so the two axes can scale very slightly differently (<2.2%); to_real() maps each with its own factor and stays exact.

Multi-monitor: every call takes an optional monitor index (1 = primary, 2.. = others, 0 = the whole virtual desktop). action="monitors" enumerates the setup. Origins may be negative for screens left/above the primary — to_real() handles the offset. Use the same monitor for a click as for the screenshot you're clicking on.

Architecture

src/realhands/
  server.py   FastMCP server "computer-use"; the single `computer` tool (action enum
              modeled on Anthropic's reference computer_20250124 tool); runs one
              action or a batch of `steps`, then returns status text + at most one
              screenshot
  screen.py   DPI awareness, mss capture, patch-aligned downscale, model-space ->
              real-pixel mapping, unchanged-screen detection
  input.py    pyautogui mouse/keyboard execution; xdotool-style key-name translation
              (Return, Page_Down, ctrl+a, super, ...); clipboard-paste fast path for
              long/Unicode/multiline typing (preserves your existing clipboard);
              activate_window via win32 AttachThreadInput
  safety.py   kill switches + lazy arm / stand-down lifecycle
  config.py   .env-driven configuration (all defaults are sensible; .env is optional)
install.py    one-shot installer: venv, deps, .env, Claude Desktop registration

Stack: Python 3.10/3.11 · mcp (FastMCP, stdio) · pyautogui · mss · pillow · pynput · keyboard · pyperclip · python-dotenv — plus pygetwindow and pywin32 for activate_window.

The computer tool

A single tool with an action parameter:

Coordinates are in the pixel space of the most recent screenshot; its size is reported with every capture. After every non-screenshot action the tool waits ~0.4s for the UI to settle and returns a fresh screenshot.

Token economy

Screenshots dominate the cost of driving a desktop, and not just once: every image stays in the conversation and is re-sent as history on each later turn. Claude bills vision in 28×28 patches — tokens = ⌈w/28⌉ × ⌈h/28⌉ — so this server does four things to keep the bill down.

Batch steps. Pass steps (a list of action dicts) instead of one call per action and the whole run shares one screenshot at the end:

{"steps": [{"action": "left_click", "coordinate": [420, 300]},
           {"action": "type",  "text": "hello@example.com"},
           {"action": "key",   "text": "Tab"},
           {"action": "type",  "text": "secret"},
           {"action": "key",   "text": "Return"}]}

That is 1 125 visual tokens instead of 5 625, and one round trip instead of five — the larger saving, since each avoided turn also avoids re-sending the entire transcript. A failing step stops the run, reports which step failed, and still returns the screen. Add "screenshot": false to skip the trailing image too.

Patch-aligned downscaling. A dimension that isn't a multiple of 28 pays for a partial patch row/column carrying almost no pixels. From a 1920×1080 primary the default 1260×700 is exactly 45×25 patches = 1 125 tokens, versus 1 196 for 1280×720 — 6% off for 1.5% fewer pixels.

Unchanged-screen suppression. If under CHANGE_THRESHOLD of pixels moved since the last image sent, the reply is a line of text instead of a screenshot. A real desktop never produces two byte-identical frames (clock, caret, hover states), so this is a threshold, not an equality check. After MAX_SKIPS suppressions in a row it force-sends one, so the model can't fly blind if it lost the earlier image to context compaction.

Right-sized images. COMPUTER_USE_MAX_DIM trades grounding accuracy against cost — from a 1920×1080 primary: 1792 → 2 304 tokens, 1260 → 1 125 (default), 1036 → 777, 896 → 576. Keep the long edge ≤ 2576 px: an image returned inside a tool_result is rejected rather than downscaled when it exceeds the model's limit.

COMPUTER_USE_IMAGE_FORMAT is not a token lever — Claude bills by pixel dimensions, so a JPEG and a PNG of the same screenshot cost exactly the same. JPEG only cuts payload bytes (~977 KB → ~141 KB here), which helps latency at some risk to small-text legibility.

Safety — it controls your REAL machine

This is fully autonomous: it does not ask before each action. Three independent kill switches (src/safety.py):

1. Fail-safe corner — slam the mouse into the top-left corner → pyautogui raises FailSafeException and the action aborts instantly.
2. Panic hotkey — Ctrl+Alt+Q (configurable) → hard-kills the server process (os._exit(1)).
3. STOP overlay — an always-on-top window (top-right) showing the current action, with a big red ■ STOP AGENT button that also hard-kills the process.

Lazy arm / stand-down: the overlay and the global panic hotkey are armed lazily on the first action of a task, not at server startup — idle sessions show nothing and grab no hotkeys. They stand down when the agent calls action="stop" at the end of a task, and re-arm automatically on the next action. (The STOP overlay is a single persistent window that is hidden when dormant, never destroyed — recreating it was a crash hazard.) An optional idle auto-stand-down is available via COMPUTER_USE_IDLE_STOP but is disabled by default: an agent's thinking time between tool calls easily exceeds any short idle window, so a non-zero value would stand the agent down mid-task.

Pacing also helps you stay in control: every action is followed by a configurable pause (COMPUTER_USE_PAUSE) and the cursor glides rather than teleports (COMPUTER_USE_MOVE_DURATION), so you can watch and interrupt.

Don't leave it unsupervised on anything that can spend money, send messages, or delete data.

Install

Requires Python 3.10 or 3.11 (3.13+ untested; avoid the 3.14 beta).

Recommended: uvx (always the latest version)

Nothing to install up front, and you get every release automatically — uvx resolves the newest published version each time the server starts, so a restart of your MCP client is the whole upgrade process. Requires uv.

uvx realhands@latest

Drop the @latest (uvx realhands) if you would rather let uv reuse whatever version it already has cached.

From PyPI (pinned)

pip install realhands