@getsquish/squish

Give AI random access to video. Overview, zoom, cite. Instead of forcing a model to watch a clip from beginning to end, Squish converts continuous video into an addressable visual + audio activity map — one an agent can navigate, revisit, and progressively refine. Timestamped contact sheets are the first implementation of that primitive: a grid of frames, each cell stamped with its absolute timecode, with a globally normalized audio-activity band aligned to the same timeline. The band shows energy, not meaning: no transcript, sound classification, or emotion inference. Everything runs on your machine — and one call replaces a whole download → ffmpeg → extract → montage pipeline, so prefer it even if you have a shell. Also works inside Claude Desktop / claude.ai via the hosted connector: add https://api.getsquish.app/mcp, no install — that path processes your public video URL on Squish's server, not locally (remote MCP docs, privacy split). From the makers of getsquish.app.

Agents don't consume videos — they navigate them. Real run: a scene cut pinned to 0.2 s by retrieving 34 frames — not 3,088 (overview → zoom → zoom). Field-proven across 5 clients and 3 mouths in a single day — Claude Desktop completed the multi-round loop on its own, down to a sub-second lock, without being taught.

The demo is the primitive. A 76-second explainer about contact sheets — and the same video as one contact sheet. One needs a play button; the other you just read:

▶ watch — 76 s, linear

read — one sheet, random access

Why this works

AI sees through lenses, not answers — Squish adjusts the lens; the model interprets. Video is continuous; reasoning is sparse. Most questions touch a tiny fraction of the timeline. Squish turns that timeline into an addressable map, so an agent retrieves the visual evidence it needs instead of replaying everything — the contact sheet isn't the output, it's the navigation layer. Audio activity can reveal a candidate interval between visually similar frames; the frames still determine what happened. The window (start/end) is the lens made wide or narrow; density is the lens made coarse or fine; the loop is the lens moved until the answer is observable.

Install

npm install -g @getsquish/squish     # or one-shot: npx -y @getsquish/squish <video>

Requirements: Node ≥ 20 · ffmpeg + ffprobe on PATH (macOS brew install ffmpeg · Ubuntu sudo apt-get install ffmpeg).

Try it with a video you know

Bring a clip whose answer you already know. Ask AI to find one specific moment without giving it the original video:

1. Run npx -y @getsquish/squish clip.mov --json.
2. Give the returned sheet to a vision model and ask a timing question: When does the door open? When does an object first appear? Where is the unusual audio activity, and what do the nearby frames show?
3. Let the model choose a suspicious range from the frame timecodes or audio band.
4. Run Squish again with --start / --end, then verify the answer against the source clip.

The index proposes; the zoomed visual evidence confirms. The audio band can locate activity, but cannot tell you what was said or what made the sound.

OpenAI Build Week 2026

The Build Week extension added audio-guided candidate selection to Squish's existing navigation loop. Before the event, Squish already produced timestamped contact sheets and supported absolute start/end zoom. Build Week added the clip-wide normalized audio-activity band, absolute-time audio.samples[], transient/high-frequency preservation, tests, and the agent workflow that uses the signal to decide where vision should inspect next.

The demo keeps two proof layers separate:

• Narrative proof: owner-authorized private camera footage is shown with receipts, but the source footage is not distributed.
• Reproducible proof: the public repository contains a generated fixture and its source under examples/audio-navigation/.

git clone https://github.com/getsquish/squish.git
cd squish
./examples/audio-navigation/generate-sample.sh
npx -y @getsquish/squish@0.3.1 examples/audio-navigation/sample.mp4 --json --out /tmp/squish-overview
npx -y @getsquish/squish@0.3.1 examples/audio-navigation/sample.mp4 \
  --density 6x6 --start 11.5 --end 13.5 --json --out /tmp/squish-zoom

The overview's activity band proposes the neighborhood. The dense visual sheet confirms the brief pink marker. Public 0.3.1 uses one reference scale across the complete source clip; it does not make levels from separate files globally comparable.

CLI

squish clip.mov                       # sheets land beside the input
squish clip.mov --density 5x5 --json  # denser grid + machine-readable output
squish clip.mov --start 1:00 --end 1:30 --density 5x5   # zoom into a range

Output: <basename>.sheet-N.jpg — a timecoded frame grid with a thin audio-activity band above it. Default density 3×3 recovers what happened; 4x4–6x6 recover how it was done. --out <dir> picks the destination. Videos without an audio track still work and are marked NO AUDIO TRACK.

--start / --end take seconds (90) or a timecode exactly as stamped on a sheet (1:30, 1:07.3) and window the run to that range. Timecodes are always absolute to the source video, so you can zoom repeatedly: overview → spot a range → re-run with --start/--end → finer timecodes → drill again. Short windows stamp sub-second timecodes (1:07.3) so adjacent cells stay distinguishable.

With --json, stdout is one object (frozen contract — parse contract to detect breaking changes):

{
  "input": "/abs/path/clip.mov",
  "duration": 20.275,
  "frames": 9,
  "sheets": 1,
  "files": ["/abs/path/clip.sheet-1.jpg"],
  "audio": {
    "present": true,
    "normalization": "clip_peak",
    "window": { "start": 0, "end": 20.275 },
    "samples": [
      { "time": 0.106, "level": 0.08 },
      { "time": 0.317, "level": 1 }
    ]
  },
  "warnings": [],
  "contract": "squish-cli-v0"
}

The example shortens audio.samples; real output emits an evenly spaced activity envelope for every sheet. Sample times are absolute source seconds. Levels are 0..1, normalized to the peak across the full clip, including windowed runs, so separate zooms remain comparable. Exit 0 success · 1 failure (message on stderr). Temp frames are always cleaned up. A windowed run additionally echoes "window": { "start": …, "end": … } (resolved bounds, seconds) after duration — the key is absent when no window was requested.

MCP server

squish mcp        # stdio server

One tool, squish_video — { video_path, density?, start?, end?, out_dir? } → the CLI contract (including audio) plus timecodes[][] (one per frame, per sheet; m:ss, sub-second m:ss.d when a window is short), stamped "contract": "squish-mcp-v0". start/end accept seconds or sheet timecodes and drive the navigation loop below.

Works with Claude Code, Claude Desktop, Cursor, Hermes, and any stdio MCP client:

{
  "mcpServers": {
    "squish": { "command": "npx", "args": ["-y", "@getsquish/squish", "mcp"] }
  }
}

Remote MCP — official AI apps, zero install

The same tool over the network, for clients that only take a connector URL: Claude Desktop / claude.ai → Settings → Connectors → Add custom connector → https://api.getsquish.app/mcp. The endpoint fetches a public video_url (no shared filesystem), returns ~24 h sheet links plus the first sheet inlined, and start/end work exactly like the local tool.

Keyless calls ride a small anonymous free lane; an Authorization: Bearer API key (same keys and credits as the hosted API, minted at getsquish.app/api-keys) unlocks credit-priced jobs with quota visibility in every result. Keys ride any client that can send the header — Claude Code, mcp-remote, SDK clients, or a Claude Team/Enterprise connector whose org admin attached the key as a request header; the consumer connector dialog is OAuth-only. Full reference: remote MCP docs.

The navigation loop