🎞️ clickcast — Playwright browser automation, screenshots, and GIF/MP4/WebP recording for AI agents

Give AI agents visual + structured feedback about live web UIs — and give humans deterministic demo reels while you're at it.

What's new (v0.3.0) — live agent control via a new MCP server, pixel-level clickcast diff, accessibility semantics fused with the pixel-grid overlay, an official CI GitHub Action, day-one Homebrew/apt packaging, and a self-healing first run. See CHANGELOG.md for the full notes.

See docs/ONE_PAGE_NAVIGATION_ORDER_TIPS.md for the nine principles behind why this reel reads as legibly as it does — and the scenario template you can copy for your own reels.

clickcast drives a real browser through a website and hands back two things:

1. A watchable reel — GIF / MP4 / WebP / raw frames.
2. A machine-readable JSON sidecar — every step's selector, timings, per-step frame paths, discovered elements, and post-action page state (title, URL, console errors, failed requests). Versioned. See docs/feedback-schema.md.

Point it at a URL and it will auto-discover the interactive elements and build a tour for you, or hand it a small YAML scenario for a scripted, repeatable walkthrough.

Install

One line, ready to run immediately after:

pip install clickcast && clickcast install --with-deps chromium && clickcast doctor

Broken down — pip install clickcast (requires Python ≥ 3.10) gets you the CLI; clickcast install downloads Chromium (~one-time, ~180MB — kept out of the pip package itself since it's versioned independently and every project doesn't need every engine); --with-deps also pulls the system libraries Chromium needs (Linux only, may prompt for sudo); clickcast doctor confirms everything above actually worked.

Forgot the second step? Any command that needs a browser (auto, run, shot, elements, mcp) detects a missing engine itself and offers to install it right then instead of failing — say yes once and it retries automatically. That prompt only fires in an interactive terminal; CI/scripted runs fail fast with the exact fix command instead of hanging on stdin.

Homebrew (macOS/Linux)

brew install --build-from-source ./Formula/clickcast.rb   # works today, clone this repo first
brew install AlexKay28/clickcast/clickcast                # tap not bootstrapped yet -- see docs/packaging/homebrew.md

apt (Debian/Ubuntu, .deb)

bash scripts/build_deb.sh 0.2.9 && sudo dpkg -i dist-deb/clickcast_0.2.9_amd64.deb   # works today, clone this repo first
sudo apt install clickcast                                                           # hosted repo not bootstrapped yet -- see docs/packaging/apt.md

Both native packages skip the Chromium download (~180MB, versioned independently of clickcast) and never install a second ffmpeg (clickcast already bundles one via imageio[ffmpeg]) -- run clickcast install --with-deps chromium once after either install path. Full design rationale, what's live today vs. what needs one-time bootstrapping, and the exact bootstrap steps: docs/packaging/homebrew.md, docs/packaging/apt.md.

npm (clickcast + clickcast-mcp)

Two npm packages -- thin Node wrappers whose postinstall provisions an isolated Python venv and pip-installs the pinned PyPI clickcast, since there's no way to ship clickcast's Playwright/Pillow/ffmpeg runtime as pure JS. Built specifically because the MCP ecosystem's install pattern is npx <package>, not pip install:

npx -y clickcast-mcp                 # MCP entry point -- execs `clickcast mcp`
npx clickcast --version              # general CLI wrapper

Both are published on the npm registry. To run against your working copy instead of the published version:

cd npm/clickcast-mcp && npm pack && npm install -g ./clickcast-mcp-*.tgz && clickcast-mcp --help
cd npm/clickcast      && npm pack && npm install -g ./clickcast-*.tgz      && clickcast --version

Full design rationale (including why the shared provisioning code is a vendored copy rather than a file: dependency), what's live today vs. what needs bootstrapping, and the exact bootstrap steps: docs/packaging/npm.md.

For AI agents — copy this prompt into your chat

Paste the block below into your coding agent (Claude Code, Cursor, Copilot Chat, Codex, etc.) to teach it clickcast in one message. The agent will install the tool, verify the environment, generate a visual + machine-readable report for your project, and know how to gate CI on the results.

You are going to use `clickcast` (https://pypi.org/project/clickcast/) to visually verify
this project's frontend. It produces a watchable GIF reel + a JSON sidecar with
per-step selectors, timings, page state, error codes, and a graph of the app's
pages. Follow this exact sequence:

1. INSTALL
   pip install clickcast
   clickcast install --with-deps chromium   # first time only, ~180 MB
   clickcast doctor                          # bail out if this fails

2. LOAD FULL CAPABILITIES
   Run `clickcast skill --json` and read the output. It lists every subcommand,
   flag, docs URL, and sidecar schema pointer. Prefer this over guessing.

3. RUN A QUICK TOUR OF THE APP
   If we have a running URL (dev server / staging / production):
     clickcast auto <URL> --for-humans --emit-events --out tour.gif
   If we have a specific flow to verify, write a YAML scenario and run:
     clickcast run tour.yml --emit-events --out tour.gif
   `--for-humans` gives a legible reel for the user to watch;
   `--emit-events` prints a machine-readable JSONL line you can parse.

4. READ THE SIDECAR (JSON at `<gif>.json`, schema v3)
   For each step, gate on the structured fields — NOT regex over prose:
     - `status`      : "ok" | "failed" | "skipped"
     - `error_code`  : "timeout" | "locator_missing" | "cross_origin" |
                       "navigation_error" | "selector_ambiguous" | "other"
     - `skip_reason` : "optional_no_reaction" | "pre_action_failed" |
                       "element_vanished" | "cross_origin_bounce"
     - `page_state`  : title, url_after, console_errors, page_errors,
                       network_failed
   The top-level `graph` block gives page nodes + navigation edges you can
   use to reason about the app's shape, not just the sequence you ran.

5. WATCH STDERR FOR ADVISORIES
   clickcast prints `⚠ <message> — see <docs-url>` lines for known anti-
   patterns (nav-heavy tour, click without DOM reaction, very short reel,
   cross-origin bounce, incoherent cursor styling). Each has a stable
   kebab-case id you can dedupe or gate on.

6. FOR CI REGRESSION GATES
   `clickcast assertions <sidecar>.json --baseline golden.json` diffs the
   run against a committed baseline; nonzero exit on drift. Byte-identical
   across runs (timestamps, frame paths, and URL query strings excluded).

Reference docs (all in-repo, load lazily as needed):
  - Sidecar shape:   https://github.com/AlexKay28/clickcast/blob/main/docs/feedback-schema.md
  - Human-legible reel authoring: https://github.com/AlexKay28/clickcast/blob/main/docs/ONE_PAGE_NAVIGATION_ORDER_TIPS.md
  - Agent integration:  https://github.com/AlexKay28/clickcast/blob/main/docs/ai-integration.md

If something is unclear, run `clickcast <subcommand> --help` before asking me.

Once the agent has this, ask it something concrete like "run clickcast auto against http://localhost:3000 and tell me which clicks had DOM reactions" — it now has everything it needs.

Live agent control (MCP)

Everything above is batch mode: record a whole tour, then read back a GIF + sidecar. clickcast mcp is the live counterpart — an MCP server that drives one action at a time (goto/click/type/scroll/...) and hands back clickcast's richer per-call payload (annotated frame, page_state, grid coordinates, an enumerated error_code) instead of a bare screenshot, so an agent can react before deciding the next step.

pip install 'clickcast[mcp]'

{
  "mcpServers": {
    "clickcast": { "command": "clickcast", "args": ["mcp"] }
  }
}

Reach for mcp when an agent needs to explore and react live; reach for auto/run for a repeatable, one-shot artifact (CI, docs, release notes). Full tool reference, client config for Claude Code / Claude Desktop, and the schema doc: docs/mcp.md · docs/mcp-tool-schema.md.

First run — 30 seconds

clickcast auto https://example.com --out tour.gif

Produces two files:

• tour.gif — the reel
• tour.gif.json — the AI-consumable sidecar (schema_version: 1, spec at docs/feedback-schema.md)

For a walkthrough of how an LLM agent consumes both, see docs/ai-integration.md.

Three modes