# openreality [Health: Active]

**Category:** 💻 Developer Tools  
**Repository:** https://github.com/reality-opened/openreality  
**GitHub Stars:** 487  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/openreality

## Description
Open Reality: phone video to AI-queryable 3D scenes. MCP tools for Claude/Codex/Cursor (npm openreality-mcp), the self-hostable broker, and the VGGT-SLAM library, in one repo. BSD-2-Clause.

## Tools
Capabilities this server exposes over MCP:

- **workspace_list_scenes** — List all persisted scenes in the signed-in Open Reality account (newest first). Also reports which scan_ids are NEW since this machine last listed — the account is the sync layer: scans made on the phone or the web appear here. Use scene_card next for anything you want context on.
- **workspace_upload_video** — Upload a local video file (.mp4/.mov/.avi/.mkv/.webm, ≤1 GB) into the account and start a reconstruction job. Returns {job_id, scan_id} immediately (202); follow with workspace_job_wait. The scene appears in workspace_list_scenes when the job is done.
- **workspace_upload_recording** — Upload a robot recording (DimOS memory2 .db, ≤2 GB) into the account. Runs the recordings pipeline (recon → odometry-consistency score → QC gate); the scene persists as source="robot_recording" with the QC report under derived/demo/recordings/. Returns {job_id, scan_id}; follow with workspace_job_wait.
- **workspace_upload_splat** — Import a local .ply/.spz Gaussian splat as a scene (source="imported_splat", honestly degraded report). ≤64 MiB uploads inline and returns {scan_id, gaussian_count} synchronously; larger files go through the chunked lane and return {job_id, scan_id} to poll with workspace_job_wait. Non-gaussian PLYs are refused (422 not_a_gaussian_splat).
- **workspace_job_status** — One status read of an ingest/export job (GET /api/workspace/jobs/<job_id>): {status: queued|running|done|error, stage?, scan_id?, error?}.
- **workspace_job_wait** — Poll an ingest/export job until it is done or errored (or the timeout passes). Returns the final status plus the stage trail observed.
- **scene_card** — The Tier-1 context card for a scene: source, metric/anchor state, room + summary, closed-world object inventory, ground frame, and workspace artifacts (agent runs, measurements, paths). Small by design — pull details with the other scene tools.
- **scene_list_objects** — The closed-world object inventory from the persisted record (facts.objects): model query, operator relabel, confidence, world center/extent. These uids/labels are the ONLY objects that exist — never invent others.
- **scene_measure_distance** — Distance between two WORLD-frame points (POST /measure). The response units are "m" ONLY when the scene has a metric anchor — otherwise "relative". ALWAYS report the units and scale_source with the value; never say metres for relative units.
- **scene_measure_angle** — Angle at the vertex (second point) of three WORLD-frame points (POST /measure, kind=angle). Returns degrees.
- **scene_plan_path** — Plan a path through scanned free space to an object (by uid) or a world point (POST /nav/plan). Provide exactly one of object_uid | point_world. Honest failures: 422 no_geometry / no_floor / unreachable_goal (with a nearest-reachable suggestion). The response provenance line ("not certified navigation") must ride along with any use. The per-frame camera flythrough (poses) is left on disk by default and the waypoint list comes back as an evenly-sampled preview — both counts and the full-trajectory artifact key ride along; artifact_fetch(doc_key) for the whole thing.
- **scene_planes** — Floor + wall plane candidates from the scene geometry (POST /planes; cached per-scene). Used for restyle scoping and understanding room structure.
- **scene_ground_frame** — Read the recorded floor/ceiling frame (GET /ground_frame). If vertical_axis_known is false, the numbers are ABSENT on purpose — say so rather than guessing heights.
- **scene_ground_frame_fit** — (Re)fit the floor plane and write the result onto the scene metrics (POST /ground_frame). dry_run=true computes without persisting.
- **scene_anchor** — Metric-anchor calibration (POST /anchor): two picked world points + their real distance in metres. Non-destructive (writes NEW derived/anchor/* artifacts + the derived_latest pointer) and it is what unlocks units:"m" for measure/nav/export. materialize:"job" returns 202 with a job_id to poll via workspace_job_wait.
- **scene_keyframe_image** — One capture keyframe as an image (GET /keyframes/<blob_key>). blob_key comes from the scene detail keyframes[] refs. Real capture photos — cite as photo evidence.
- **scene_synthetic_views** — Renders of the scene's own splat registered as evidence (GET /synthetic_views). These are NOT photographs — always carry the "synthetic view" provenance when citing.
- **scene_synthetic_view_image** — One synthetic view PNG (a RENDER of the splat, not a photo — say so when citing).
- **scene_lod** — Level-of-detail index status for the scene splat (GET /lod). Always 200 — read `status` (ready|none|running|error|not_needed).
- **scene_lod_build** — Spawn the LOD decimation job (POST /lod → 202 {job_id}; 409 lod_job_active when one is already running). Poll with workspace_job_wait.
- **scene_imported_objects** — Status of geometry-derived objects for an imported splat (GET /imported_objects).
- **scene_imported_objects_run** — Cluster an imported splat's geometry into facts.objects (POST /imported_objects). Labels carry geometry/synthetic-view provenance — confidence 0.0 is honest, not a bug.
- **scene_object_complete** — SAM-3D-Objects mesh completion for a segmented object (POST /objects/<uid>/complete → 202 {job_id}, broker-thread lane). Poll with scene_job_wait. Output is GENERATED geometry (honesty envelope in the artifact meta.json).
- **scene_object_variants** — TRELLIS variant generation for a completed object (POST /objects/<uid>/variants → 202 {job_id}). Optional body fields pass through (see server docs). GENERATED assets.
- **scene_segment** — SAM-3 click segmentation (POST /segment). Advanced: the body must match the server contract (click point + view selection — see routes_sam3d.py / the OS UI). Provided as a raw passthrough so the full surface stays reachable.
- **scene_job_status** — One status read of a broker-thread scene job (SAM-3D completion, variants, view renders). NOTE: these jobs honestly 404 across a broker restart.
- **scene_job_wait** — Poll a broker-thread scene job until done/error/timeout.
- **scene_share_access** — Owner-only read of the share-link access log (GET /share/access): per minted link (access_id) — opened, request count, visits (new visit after 30 min idle), first/last seen, returned (a visit >= 24 h after the first), devices, and the Q&A questions the viewer asked. This is the behavioural evidence for customer discovery (L5 = returned). Match access_id to the person you sent the link to. events:true appends the raw event list.
- **scene_share** — Mint a read-only, single-scene share token + embed URL (POST /share; default TTL 30 days). Share tokens can NEVER reach the OS feature routes — viewer only.
- **agent_annotate** — Start the five-phase scene annotation run (survey → labels → description → dimensions → key features) on the server-side scene agent. 202 {run_id}; one active run per scene (409 agent_run_active carries active_run_id — or set attach_if_active). Costs bounded server-side LLM calls. Follow with agent_run_events wait_for_done.
- **agent_replay** — Re-serve a recorded run's persisted event log as a new (replay-badged, $0) run. Use for reviewing what an earlier run found without spending LLM calls.
- **agent_pilot** — Give the pilot agent a free-text instruction (navigate/inspect flows; tools: list_scene_objects, plan_path, measure_distance). 202 {run_id}; poll agent_run_events.
- **agent_chat** — One chat turn with the scene agent (its own bounded run; the server agent may call its scene tools mid-turn). By default waits for the run to finish and returns the event transcript. The agent's numbers carry units/scale_source — repeat them honestly.
- **agent_run_events** — Read a run's event log from a cursor (GET /runs/<run_id>/events?after=). With wait_for_done, polls at ~1 Hz (the web client's cadence) until the run finishes.
- **agent_runs** — The scene's run index (newest first) + which run is live: kind, status, cost_usd, llm_calls, findings. Persisted runs can be replayed for $0 with agent_replay.
- **export_manifest** — What an export WOULD contain (GET /demo/export/manifest): file tree + sizes, complete/absent components, and whether the zip route would work. No build, no writes. Formats: openreality | groot_lerobot_v2 | isaac_usd | mujoco_mjcf (the simulator formats isaac_usd / mujoco_mjcf are metric-gated — they need an anchor).
- **export_prepare** — Build the export zip as a background job (POST /demo/export/prepare → 202 {job_id}; 409 export_job_active when one is running). With wait=true (default) polls the job and returns the prepared artifact (including download_path). Then artifact_fetch it. For MuJoCo prefer export_mujoco, which also downloads and unpacks the scene.
- **export_status** — What is prepared for scene+format (GET /demo/export/prepared). Always 200 with {status: ready|running|error|none}; ready carries the artifact + download_path.
- **export_mujoco** — One call from scene to a loadable MuJoCo model: builds (or reuses a fresh) mujoco_mjcf export, downloads it and unpacks it to disk, and returns the path to scene.xml — load it with mujoco.MjModel.from_xml_path(path) or `python -m mujoco.viewer --mjcf=<path>` (needs mujoco>=3.3). METRIC-GATED: an un-anchored scene is refused with 409 metric_scale_required — run scene_anchor first. Collision is primitives only (a floor plane + one upright box per detected object); the scan mesh is visual-only. A fresh build meshes the scene on a 64 GB container and takes minutes. Bytes never enter context.
- **artifact_fetch** — Download any derived/... artifact (export zips, LOD levels, agent run logs, QC reports, masks) to a local file and return the path — bytes NEVER go into context. Default target: <artifacts_dir>/<scan_id>/<key>.
- **artifact_fetch_splat** — Download the scene's original splat.ply to a local file and return the path. For a render-sized variant prefer the LOD levels (scene_lod → artifact_fetch of a level key).
- **artifact_fetch_cloud** — Download the scene's cloud.ply to a local file and return the path.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "openreality": {
    "command": "npx",
    "args": ["-y","openreality-mcp"],
    "env": {
      "OPENREALITY_URL": ""
    }
  }
}
```

**Requires environment variables:** `OPENREALITY_URL` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation

## What openreality does

openreality turns ordinary phone video into persistent, queryable 3D scenes and exposes the resulting workflow through MCP. An assistant can upload a video or work with a scan made through the Open Reality service, then inspect objects, ask questions about the scene, measure geometry, plan routes, and export data for robotics tools.

The repository combines the npm package `openreality-mcp`, a backend that reconstructs and serves scenes, and a 3D reconstruction library based on the VGGT-SLAM line. The MCP layer is intended for assistants such as Claude, Codex, and Cursor.

## How it works

The MCP process runs locally and keeps the user's credentials. Each tool invocation becomes a typed REST request to either the hosted backend or a server operated by the user. Large generated files are saved to the local machine instead of being inserted into the assistant's context, while backend refusals and uncertainty labels are passed through to the model.

A typical workflow starts with a phone video. The backend performs a GPU reconstruction and creates a persistent scene containing geometry, detected objects, and a report. The assistant can then query that scene or request actions such as measurement, route planning, and export.

Measurements remain in relative units until the scene is calibrated using a known real-world distance. Only after that calibration can results be described as metres, helping prevent unverified scale claims.

## Setup and configuration

Install the MCP entry point through npm and register it with the client of your choice. For Claude Code or Codex, the command is:

```bash
npx -y openreality-mcp serve
```

Claude Desktop and Cursor can use the same command as the configured MCP server, with `npx` as the executable and `-y openreality-mcp serve` as its arguments. Authentication is initiated separately:

```bash
npx -y openreality-mcp login
```

That login opens a browser and stores a revocable API key locally. Self-hosted deployments connect through `OPENREALITY_URL` and a token printed when the self-hosted server first starts.

The backend can run on a user's own GPU machine with a local data directory, or through a deployment on the user's Modal account. The repository also includes an offline simulator with fixture data for development and demonstrations without an account or GPU.

## Tools and capabilities

The MCP surface covers:

- Uploading video and initiating 3D reconstruction.
- Listing and inspecting objects in a scanned scene.
- Measuring distances and angles between points.
- Calibrating relative geometry against a real distance.
- Planning paths through scanned free space to points or objects.
- Asking server-side agents to survey, label, or answer questions about scenes.
- Exporting scans as LeRobot or GR00T-style datasets.
- Producing Isaac Sim or MuJoCo scene exports through the hosted service.
- Working with scene resources and an offline simulator.

The reconstruction library also provides camera tracking, dense geometry, object detection, metric calibration, and splat export as part of the broader repository.

## Limitations and notes

Self-hosting downloads the VGGT-1B reconstruction model and its associated code from their source. Those components are licensed CC BY-NC 4.0 for non-commercial use, even though the repository itself uses BSD-2-Clause. Commercial users should use the hosted service or obtain the required model license.

A scan does not automatically establish physical scale. Distances and angles should be treated as relative until calibration with a real measurement has been completed. Self-hosting also requires either a suitable GPU machine or a Modal account, depending on the deployment path.

_Full upstream README: https://allmcps.com/mcp/openreality/readme_