Oswright

What the oswright MCP server does

The oswright MCP server exposes desktop automation tools to an MCP-compatible AI client. Agents can inspect the display, identify visible text with OCR, locate template images, move or click the mouse, enter text, press keys, and wait for interface changes. It can also list, focus, minimize, close, and capture individual windows.

The tool set supports both low-level coordinate actions and higher-level text-driven actions. For example, an agent can locate a label with OCR, fill a field, submit a form, and verify the resulting screen. Clipboard read/write tools provide another way to move text between the agent and desktop applications. launch_app starts an application directly and can wait for it to load.

The project describes support for Windows through Win32 APIs, with Linux and macOS support using pynput and platform-specific OCR fallbacks. The listed automation surface is especially relevant to applications that do not provide an API or that are difficult to drive through browser automation.

How the oswright MCP server works

The server communicates over MCP using a standard stdio configuration. Each action can return a current screenshot, while screen analysis can use OCR, image matching, or Windows UI Automation where available. Text lookup tools return coordinates and confidence values, allowing an agent to act on visible labels rather than hard-coded positions.

Its incremental perception model keeps screen state between observations and rescans areas that changed. The README also describes cached OCR results, screen memory, speculative perception, adaptive waiting, and a resolution cascade that attempts cheaper lookup methods before more expensive ones. These mechanisms are intended to reduce repeated full-screen reads during multi-step tasks.

Coordinates are represented as physical pixels for DPI-aware interaction. The repository includes benchmark and test material for measuring latency, token usage, and task completion, but the reported measurements are machine- and workload-specific rather than universal guarantees.

Setup and configuration

Python 3.10 or newer is required. The documented installation uses uvx:

{
  "mcpServers": {
    "oswright": {
      "command": "uvx",
      "args": ["oswright"]
    }
  }
}

If uvx is unavailable, install the package with pip install oswright and configure the command as oswright, or run it with python -m oswright. The README provides setup instructions for Claude Desktop, Claude Code, VS Code, Cursor, Windsurf, and Cline. It also documents a custom stdio setup for Goose.

Tools and capabilities

The oswright MCP server includes tools for:

• Capturing the full screen, a region, or a specific window.
• Reading all visible text, finding text, and waiting for text to appear or disappear.
• Finding template images with OpenCV matching.
• Clicking, double-clicking, right-clicking, hovering, dragging, scrolling, and moving the mouse.
• Typing text, pressing keys or key combinations, and filling one or multiple fields.
• Reading the mouse position and screen dimensions, including monitor count.
• Listing, focusing, minimizing, and closing windows.
• Reading from and writing to the system clipboard.
• Launching applications and optionally waiting for startup.

Most interaction tools return a screenshot after acting, while screenshot and OCR tools provide visual or structured observations for the next step.

Limitations and notes

Desktop automation depends on the target application's visible state, screen content, window titles, and supported platform mechanisms. OCR can report confidence values and coordinates rather than semantic certainty, and image matching depends on a suitable template. The README notes that accessibility and pixel-based perception have different coverage: accessibility may miss web content or some Win32 controls, while pixel-based methods can miss elements that are easier to identify through accessibility.

The Windows implementation uses built-in Windows OCR and does not require the larger PyTorch installation described for some non-Windows OCR paths. A physical display is needed for desktop-driving tests; those tests skip when no display is available. The project does not provide a remote browser or application API: it operates the local desktop exposed to the process.