droidsight

An MCP server that drives a real Android device over ADB — and hands the agent back the screen its action produced.

One native binary. The server has no Python, Appium, scrcpy, or ffmpeg dependency and runs no Node at all — npx below is only a convenient installer — so the runtime is the binary plus adb. A background H.264 screenrecord stream is decoded in process, so the screen that follows an action is attached to the tool result automatically instead of costing a second round trip to ask what happened.

Every image carries the coordinate space it was produced in, so a model can tap what it just looked at without guessing a scale factor. When the accessibility tree comes back empty — Flutter, React Native, canvas, games — OCR and template matching still find the target.

This is a powerful local controller, not a sandbox. Run it only against a device and an MCP client you trust.

Install

npx -y @edgecasehuman/droidsight

That downloads a prebuilt binary for your platform and needs no build tools. The server is also listed in the official MCP Registry as io.github.edgecasehuman/droidsight.

To compile it instead, from crates.io:

cargo install droidsight

Or from source:

cargo build --locked --release --bin droidsight

Both compiling routes need NASM on the PATH. The H.264 decoder is built from C++ by openh264-sys2, which assembles its hot paths with NASM, and no platform ships it by default -- without it the build fails inside a transitive dependency rather than in this crate. Install it with apt install nasm, brew install nasm, or winget install NASM.NASM. The npx route above needs none of this.

The server is target/release/droidsight. It speaks newline-delimited JSON-RPC 2.0 over stdin and stdout.

Client configuration

{
  "mcpServers": {
    "android": {
      "command": "npx",
      "args": ["-y", "@edgecasehuman/droidsight"],
      "env": {
        "DROIDSIGHT_DEVICE_SERIAL": "<serial from `adb devices`>"
      }
    }
  }
}

ADB must be reachable through PATH, ANDROID_SDK_ROOT, ANDROID_HOME, or an explicit DROIDSIGHT_ADB_PATH. If more than one authorized device is visible the server refuses to guess; set DROIDSIGHT_DEVICE_SERIAL.

Tools

Thirty-one tools are published by default. Two more appear only when DROIDSIGHT_ALLOW_SHELL=1 is set, and are listed last.

Seeing the screen

Driving the device

Applications

Device and system state

Diagnostics and data

Capture and presence

Only with DROIDSIGHT_ALLOW_SHELL=1

Coordinate space

Screenshots are downscaled to 720 pixels wide by default, so a coordinate read off the returned pixels is usually not a device coordinate. Every tool that returns an image also returns a metadata.image object describing how to convert one:

{
  "image": {
    "width": 720, "height": 1600,
    "device_width": 1080, "device_height": 2400,
    "origin_x": 0, "origin_y": 0,
    "scale": 1.5,
    "coordinate_space": "image",
    "note": "To convert a coordinate (x, y) read from this image into a device coordinate: device_x = 0 + x * 1.5, device_y = 0 + y * 1.5."
  }
}

When coordinate_space is device, the image is already 1:1 and coordinates can be used directly. Request max_width: 1440 to reduce or remove downscaling. Crops additionally report a non-zero origin_x/origin_y that must be added after scaling.

Taps, uiautomator hierarchy bounds, OCR boxes, and template matches are all in device coordinates. The background stream runs screenrecord without --size, so decoded frames are native resolution and share that one space.

You can avoid the arithmetic entirely. The mcp_android_vision_query tool's elements action returns an indexed snapshot with a precomputed center per element, and its tap_element action takes that snapshot_id and an index.

Environment variables

Continuous vision cache

One background H.264 screenrecord stream starts at process startup and the latest decoded RGB frame is kept in memory. Screenshot tools and input observations read that cache, so a snapshot is available immediately once the first frame decodes. A static screen's last frame stays valid while the stream runs; it does not expire merely because the encoder stopped emitting unchanged pixels.