Openreality

The Agentic OS for any hardware
Open Reality gives your AI agent the ability to use any hardware. Vision + Action.

Website · Get started · Things you can ask · Self-host · MCP docs · Contributing

_{Demo v1: a real Claude Code (Opus) session against the built-in offline simulator, beside a synchronized scene panel. Video: docs/demo/demo-video-v1.mp4. Rendered from docs/demo/ with VHS.}

_{Live reconstruction from a handheld video: the camera path and the 3D scene build up together. Scan demo and the initial technical idea: VGGT-SLAM by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). Open Reality's reconstruction core builds on their work.}

Utilities

Quickstart

Add the tools to your AI assistant:

claude mcp add openreality -- npx -y openreality-mcp serve

codex mcp add openreality -- npx -y openreality-mcp serve

{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": ["-y", "openreality-mcp", "serve"]
    }
  }
}

/plugin marketplace add reality-opened/openreality
/plugin install openreality@openreality

Then sign in once (opens your browser, stores a revocable API key on your machine):

npx -y openreality-mcp login

Scan a room with your phone at open-reality.io, or just ask your assistant to upload a video file. Full per-client setup: open-reality.io/mcp.

Examples

Once connected, talk to your assistant like this:

"Upload ~/Videos/kitchen.mp4 and reconstruct it."

"What objects are in my latest scan, and how big is the room?"

"The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."

"Plan a path from the door to the desk and describe it."

"Export this scan as robot-training data and save the zip locally."

Self Hosting

The whole workflow is self-hostable. Read server/docs/self-hosting.md; the short version:

git clone https://github.com/reality-opened/openreality
cd openreality/server

# Path A: your own GPU box (one process, local disk)
python -m server.selfhost --data-dir ~/openreality-data

# Path B: your own Modal account (CPU web server + GPU worker)
modal deploy modal_selfhost.py

Self-hosted servers need no account: a single token printed at first start is your login, and the MCP client connects with OPENREALITY_URL plus that token.

[!IMPORTANT] Licensing. This repo is BSD-2-Clause, but the 3D reconstruction model a self-hosted server downloads (VGGT-1B and the VGGT code it runs on) is licensed by its owners as CC BY-NC 4.0, non-commercial use only. Nothing here redistributes it; your server fetches it from the source under their terms. For commercial use, use the hosted service (which runs a commercially licensed model) or get your own license from the model owners.

Repo index

server/ and core/ are curated mirrors of our private working repos, synced by hand; each carries a MIRROR.md that says exactly what is included and how it is synced. mcp/ is developed in this repo directly.

Technical details

flowchart LR
    A["📱 phone video"] --> B["server: 3D reconstruction<br/>(GPU job)"]
    B --> C[("persistent scene:<br/>geometry + objects + report")]
    C --> D["41 MCP tools"]
    D --> E["🤖 Claude / Codex / Cursor"]
    E -->|"measure · navigate · export"| D

The MCP process always runs on your machine and holds your credentials; every tool call is a typed REST call to a server (ours or yours). Big files are written to your disk, never pasted into the AI's context. Server refusals and uncertainty labels reach the AI unedited, so it cannot pretend a relative number is metres.

The measurement rule is explained in Why our 3D tools refuse to say "metres" until you prove it.

🙏 Acknowledgements

Open Reality stands on other people's research and code. Thank you to: