Marey
Give AI agents eyes for motion.
A screenshot tells an agent what the screen looks like. Marey tells it what changed.
Marey is a Model Context Protocol server that turns a short screen interaction into a timestamped, agent-readable contact sheet — a grid of still frames an agent can inspect as a temporal sequence, with any frame retrievable at full resolution.
That's the whole install. Zero runtime dependencies, nothing to clone.
See it work
You: "When I click More, the project menu opens but most of its text gets cut off. Can you find out why?"
Agent: "Let me record the interaction with Marey." (records while you reproduce the click)
The bug: clicking More opens a readable menu, but a layout update makes it so narrow that most of each action label is cut off.
The contact sheet preserves the failure in three frames: before the click → readable menu → clipped text. The red outline highlights the broken menu width.
An illustrative agent response:
Agent: "At 0.36 s, I can read all three actions. By 0.48 s, the menu has shrunk to roughly the More button's width and the labels are clipped. It stays open throughout. I'd inspect the post-open sizing code: it may be using the trigger's width instead of the menu's content width."
The sequence shows the menu opens at a readable width, then narrows while it remains open. This directs the agent to the post-open layout measurement; source inspection confirms the cause.
Staged demonstration: a browser fixture rendered at fixed timestamps and composed with Marey's own contact-sheet code. The headings, outline, and callout are explanatory annotations added for the README; the full contact sheet contains the raw frames. The fixture reproduces a sizing pass that incorrectly uses the trigger's width for the menu. Regenerate both images with node demo/make-contactsheet.mjs. Requires Chrome, Chromium, or Edge; set CHROME_BIN if it isn't found automatically.
For containers without browser sandbox support, use node demo/make-contactsheet.mjs --no-sandbox.
Why Marey?
AI coding agents are good at understanding screenshots, but motion is still awkward. A bug such as:
- a control point jumping while it is dragged,
- a menu flashing and immediately closing,
- a canvas updating in the wrong order,
- an animation stuttering between states,
cannot be understood from a single screenshot. Marey bridges that gap: because it speaks MCP, the agent requests the recording itself and receives the resulting image directly in context.
No GIF inspection. No manually extracting frames. No dragging a dozen screenshots into chat.
It is named after Étienne-Jules Marey, a pioneer of chronophotography — the study of motion through sequences of images. Marey applies the same idea to AI agents: instead of handing a model a video it cannot reliably inspect frame by frame, it converts motion into a visual sequence the model can reason about.
How it works
- An MCP client asks Marey to record the screen.
- Marey captures frames at a chosen frame rate.
- Each frame is numbered and timestamped.
- Marey composes the frames into a contact sheet.
- The contact sheet is returned to the agent as MCP image content.
- Raw frames remain available on disk for closer inspection or re-stitching.
The idea is deliberately simple:
motion becomes one image containing time.
Tools
Marey exposes ten MCP tools:
| Tool | What it does |
|---|---|
record | Records the screen for a fixed duration and returns a timestamped contact sheet |
start_recording | Begins an open-ended recording the user controls |
stop_recording | Stops the open-ended recording and returns the contact sheet |
capture | Captures a single screenshot |
get_frame | Returns one frame from a recording at full resolution |
list_windows | Lists window IDs, titles, bounds, and hidden state |
observe | Records until the user finishes through the local control page |
replay | Keeps a rolling buffer and returns clips around user markers |
status | Reports recording state |
get_frames | Returns a contact sheet for a time range, with optional crop |
record
| Parameter | Default | Description |
|---|---|---|
seconds | 5 | Recording duration |
fps | 15 | Target frames per second; achieved rate and gaps are reported |
region | primary | primary, virtual, or window |
title | — | Unambiguous visible window-title substring when region is window |
windowId | — | Exact native window ID from list_windows; takes precedence over title |
rect | — | {x,y,w,h} in target-relative pixels; restricts the captured source during recording |
delay | 0 | Delay before capture begins |
detail | overview | Legibility preset: overview, high, or max (see below) |
cols | 4 | Number of thumbnails per contact-sheet row (overrides detail) |
thumbWidth | 480 | Thumbnail width in pixels (overrides detail) |
The result contains:
- the contact sheet as MCP image content,
- a short text summary with capture metadata and the full-resolution path of every frame, so an agent can open the exact frame it needs,
- raw frames saved locally for later inspection.
Resolution and the detail preset
A single returned image has a fixed resolution budget, and a contact sheet
splits that budget across its columns. So legibility comes from fewer, wider
cells — not from thumbWidth alone (past a point, a large sheet is just
downscaled again by the client). The detail preset picks a sensible
columns/width pair:
detail | Layout | Use when |
|---|---|---|
overview (default) | 4 cols · 480px | You want many frames at a glance |
high | 2 cols · 760px | UI text / fine detail must be readable |
max | 1 col · 1280px | You need the closest thing to the raw frame |
Two more levers when detail still falls short:
- Capture a
windowinstead of the full screen. A 2560px desktop shrunk into a 480px thumbnail loses ~5× of its detail; an 800px window barely shrinks at all. - Open the raw frame. Every frame is saved at full resolution under
captures/<timestamp>/, and therecordresponse lists each one's path. Reading a single raw frame is better than re-recording.
start_recording / stop_recording
record is fixed-duration — the agent decides how long. When you control
the timing (you will drag something, open a menu, trigger an animation and the
duration is unpredictable), use the open-ended pair instead:
- The agent calls
start_recordingon your cue (same parameters asrecordexceptseconds:fps,region,title/windowId,rect,delay,detail,cols,thumbWidth). - You perform the interaction.
- The agent calls
stop_recording, which composes and returns the contact sheet — identical output torecord.
Only one recording may be active at a time. Frames are written to disk as they are captured, and a safety cap stops a forgotten session before it grows without bound.
capture
Captures one frame immediately using the same region-selection semantics as record.
get_frame
Returns a single frame from a prior recording at full resolution, as image
content over MCP. The contact sheet is a downscaled overview; when it is too
small to read fine detail, call get_frame with the recording directory and the
frame number (both listed in the record / stop_recording response), or a
direct frame path. Because the frame is returned through the protocol, this works
even for clients with no filesystem access.
list_windows
Returns visible window titles, and geometry where available, so an agent can choose a target for window capture.
Example
Once Marey is connected to an MCP client, interaction can be as simple as:
Use Marey to record 6 seconds of my editor window at 4 fps with high detail while I drag an element, then tell me what changes between frames.
The agent receives the complete sequence as a single image and can reason about the transition rather than only the initial state. If any frame needs a closer look, the full-resolution originals are listed in the response and saved under captures/.
Command-line use
Marey runs standalone from the repo — the easiest way to try the whole workflow,
including the interactive observe and replay modes, without wiring up an
MCP client. Clone it first (see From source); every command is
node src/cli.mjs <command>.
Diagnose capture — confirm the backend actually works on this machine before anything else:
Record a fixed clip — you choose the duration up front:
Capture an unpredictable moment — when you control the timing:
Inspect a frame — zoom into any frame at full resolution without re-recording:
Window capture grabs the window's own surface (via PrintWindow), so it
works even when the target is behind other windows. A minimized or off-screen
window has nothing to render, so Marey reports that rather than capturing junk —
restore the window (node src/cli.mjs windows flags which are hidden).
Inspecting evidence quality and transitions
Recording results include requested and achieved FPS, mean spacing, maximum
frame gap, completion reason, warnings, errors, and the resolved target ID and
bounds. New MCP clients (protocol 2025-06-18 or later) also receive these as
structuredContent; older clients receive the same quality information as text.
Partial recordings remain inspectable and failed recordings return isError:true
alongside any available images. stop_recording retrieves evidence even if the
backend already stopped automatically.
A contact sheet selects neighborhoods around the strongest transitions: the immediate before, changed, and following frames. Larger budgets also retain the beginning and end of the recording. Small budgets favor transition context over distant endpoints. Every stored frame is analyzed sequentially, so analysis does not discard a brief event through preliminary time sampling. Raw frames remain available even when the contact sheet must shrink.
Use get_frames to inspect a narrower time range together:
Times are inclusive acquisition timestamps in milliseconds. This returns one budgeted contact sheet, the displayed frame indices/times, and source quality metadata. It fails explicitly if no frames exist in the requested range.
For a smaller recording source (rather than an output-only crop), use:
Hidden windows are rejected. Ambiguous titles require an exact ID. Linux window capture is a screen-rectangle grab and can be occluded; Windows uses the window's own surface. Bounds describe target resolution at capture startup; frame metadata includes actual dimensions when a window changes size.
CLI equivalents:
A note on frame rate
fps is the target rate (default 15). A short event can still fall
between samples; an absent frame is not proof that a flicker did not occur.
Check achieved FPS and maximum frame gap before making timing claims.
X11 uses one persistent FFmpeg process when ffmpeg and xdotool are available.
macOS uses FFmpeg's AVFoundation screen input when available; Screen Recording
permission is required. Frames use acquisition PTS, not PNG arrival times. Windows
keeps its existing single PowerShell stream. Smaller capture rectangles reduce
encoding and transfer work; Windows window-surface capture must still render the
full window before cropping.
Wayland and hosts without a continuous backend use native single-frame commands, scheduled against deadlines so capture/encoding time is not added to every frame interval. Results explicitly report that fallback. Installing FFmpeg enables continuous X11/macOS capture, but does not establish its achievable rate.
On Linux/X11 in development validation, a 320×240 capture reached 20 fps with 50ms frame gaps, and an isolated real-pixel test captured a 100ms flash. This is a tested configuration, not a guarantee for other hosts or resolutions.
Installation
Requirements
- Node.js 18+
- A supported screen-capture backend
Zero runtime dependencies. Marey ships with an empty dependencies block —
npm install pulls nothing. Everything is built on Node builtins:
- PNG decode/encode — pure JavaScript over the builtin
zlib(src/png.mjs); nosharp,jimp, orpngjs. - Thumbnails, compositing, and frame labels — a pure-JS image buffer and a
hand-coded 5×7 bitmap font (
src/image.mjs,src/font.mjs); no image or font library. - MCP protocol — JSON-RPC 2.0 over stdio, hand-rolled
(
src/jsonrpc.mjs); no MCP SDK. - Screen capture —
child_processdriving the native or command-line backend available on the host (src/capture.mjs).
npx fetches and runs Marey on demand, so there is nothing to install globally
and no path to configure.
Connect to an MCP client
Claude Code
Verify with claude mcp list or /mcp.
Claude Code plugin
The plugin bundles the MCP server and a skill that teaches Claude when and how to use Marey for visual debugging:
Claude Desktop or another MCP client
Add Marey to the client's MCP configuration:
From source
To hack on Marey, clone it and point your client at src/server.mjs:
Capture backends
Marey auto-detects an available screen-capture backend.
| Platform | Full / monitor capture | Window capture |
|---|---|---|
| Windows 10/11 | Built-in PowerShell + System.Drawing | Built-in |
| Linux · X11 | ImageMagick/scrot for screenshots; FFmpeg + xdotool for continuous capture | ffmpeg + xdotool + xwininfo |
| Linux · Wayland | grim | Compositor-dependent |
| macOS | Built-in screencapture; optional FFmpeg AVFoundation stream | Not yet supported |
On X11, window listing and validation use xdotool and xwininfo; xrandr
identifies the primary monitor when available. ImageMagick output is forced to
8-bit, non-interlaced RGB so monochrome screens remain decodable. Wayland supports
virtual capture or an explicit rectangle; it rejects window capture and cannot
infer a primary output. macOS supports primary/rectangles, not a virtual desktop.
Native Windows/macOS changes require validation on those platforms; the new
real-pixel regression is currently Linux/X11-only.
Run ordinary tests with npm test. For real X11 coverage, install xvfb, ffmpeg,
imagemagick, xdotool, x11-utils, x11-xserver-utils, and x11-apps, then run
MAREY_NATIVE_TESTS=1 node --test test/native-capture.test.mjs. It creates and
cleans up its own display. CI runs it separately from the portable tests.
Output
Recordings are stored under captures/:
The raw frames make it possible to generate a different contact-sheet layout without recording the interaction again.
Design principles
Agent-first
Marey is an MCP server rather than just a screen-recording CLI. The agent can request the visual evidence it needs.
Still images over video
The output is intentionally model-friendly: a numbered, timestamped sequence of frames in one image.
Small surface area
A handful of tools cover the core workflow: record, capture, inspect.
Cross-platform core
Frame composition stays platform-independent while screen capture is delegated to the best backend available on the host.
Zero dependencies
The entire pipeline — PNG codec, image compositing, frame labelling, and the MCP protocol itself — is built on Node builtins. Nothing is pulled from npm, so there is no supply chain to audit, no install step beyond cloning, and no version drift in third-party packages.
Short, deterministic recordings
Fixed duration and frame rate make captures reproducible and easy for agents to request. An optional delay gives the user time to focus the target window before recording starts.
Why the name?
Étienne-Jules Marey used chronophotography to make motion visible by decomposing it into successive images.
Marey does the same thing for AI agents.
License
MIT — see LICENSE.