# claude-video-vision [Health: Active]

**Category:** 🎥 Multimedia Process  
**Repository:** https://github.com/jordanrendric/claude-video-vision  
**GitHub Stars:** 1454  
**npm Downloads (last month):** 7499  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/claude-video-vision

## Description
MCP server that gives Claude Code the ability to watch and understand videos — extracts frames via ffmpeg and processes audio via multiple backends

## Tools
Capabilities this server exposes over MCP:

- **video_watch** — Extract frames and process audio from a local video file or YouTube URL. Returns frames (as base64 images or text descriptions) + transcription + audio analysis for Claude to understand the video content. IMPORTANT: For videos longer than 30 seconds, call video_analyze FIRST to get structural data (scene changes, silence, transcription) before calling this tool — use that data to set smart segments with variable FPS. When the video is long enough to trigger audio chunking, the result's `audio.warnings` array (if present) reports chunk-boundary decisions and any per-chunk retries or failures — surface these to the user.
- **video_info** — Get metadata about a local video file or YouTube URL without processing it (duration, resolution, codec, etc.)
- **video_setup** — Check dependencies for video perception (ffmpeg, whisper, gemini api, optional yt-dlp for YouTube URLs).
- **video_configure** — Configure video perception preferences (backend, resolution, fps, whisper model, etc.)
- **video_analyze** — Analyze local video file or YouTube URL structure using ffmpeg filters. Returns scene changes, silence intervals, motion levels, and more. Use this before video_watch to plan which segments need detailed frame extraction. Does not extract frames. When transcription is enabled and the video is longer than the configured chunk trigger, the audio is chunked and transcribed in parallel; the result's `analysis.audio_warnings` array (when present) describes chunk-boundary decisions, retries, or failures — surface these to the user. If `analysis.incomplete` is present, the ffmpeg pass stopped early and filter results only cover the source up to `analysis.incomplete.analyzed_until` — tell the user instead of treating later sections as empty.
- **video_detail** — Drill into specific segments of a local video file or YouTube URL. Extracts frames at variable FPS/resolution per segment. Separates extraction from viewing: use segments to extract, view/view_sample to control which frames are returned as images. When enable_index is on, frames are cached and deduplicated across calls.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "claude-video-vision": {
    "command": "npx",
    "args": ["-y","claude-video-vision@latest"]
  }
}
```

## Documentation

## What claude-video-vision does

claude-video-vision connects Claude Code to local video files and YouTube URLs. It supplies the visual and audio inputs needed for video understanding rather than generating the interpretation itself. Frames are returned as base64 images or text descriptions, while audio results can include timestamped transcription and audio-event analysis.

The server supports adaptive inspection. A request for a short moment can use a narrow time range and higher frame rate, while a long lecture can be sampled more sparsely. YouTube downloads use yt-dlp and can preserve source metadata and available captions.

## How it works

ffmpeg handles video metadata, frame extraction, and structural analysis. Audio can be processed through one of three backends:

- Gemini API, using `GEMINI_API_KEY`.
- Local Whisper through whisper.cpp or Python `openai-whisper`.
- OpenAI Whisper API, using `OPENAI_API_KEY`.

For videos longer than 30 seconds, run `video_analyze` first. Its results can identify scene changes, silence intervals, motion levels, and other structure so a later `video_watch` or `video_detail` call can choose more appropriate segments. `video_analyze` does not extract frames.

Longer audio may be split into chunks and transcribed in parallel. Results can include warnings about chunk boundaries, retries, or failed chunks. An incomplete analysis identifies the point where ffmpeg stopped; later portions should not be treated as empty.

YouTube audio results identify whether the transcription came from manual subtitles, automatic captions, or the configured audio backend. Manual English subtitles are preferred when available.

## Setup and configuration

The server requires Node.js 24 or newer and ffmpeg. Install yt-dlp when YouTube URLs are needed. Backend-specific setup is required for cloud processing, while local processing requires Whisper tooling; models download automatically on first use. The setup wizard checks dependencies and guides backend and frame configuration.

Claude Code can install the plugin from its marketplace, after which the MCP package is fetched with `npx` on first use. Local development can instead run the plugin from a cloned repository. The `/setup-video-vision` command opens the configuration workflow, and `/watch-video` accepts a file or YouTube URL with an optional question.

Settings are stored in `~/.claude-video-vision/config.json`. Options include audio backend, Whisper engine and model, frame mode and format, resolution, default frame rate, maximum frame count, frame-description model, indexing, and retention periods. Supported frame formats are JPEG, PNG, and WebP. Whisper models are stored under `~/.claude-video-vision/models/`.

## Tools and capabilities

- `video_info` reads duration, resolution, codec, and other metadata without processing the video.
- `video_setup` checks video-perception dependencies and provides installation guidance.
- `video_configure` changes processing preferences.
- `video_analyze` examines scene changes, silence, motion, and related structure without extracting frames.
- `video_watch` combines frame extraction with audio processing and returns visual and audio results.
- `video_detail` focuses extraction on selected segments, with variable frame rates and resolutions. Its `view` and `view_sample` controls determine which extracted frames are returned as images. Optional indexing caches and deduplicates frames across calls.

## Limitations and notes

The server is a perception layer; Claude remains responsible for interpreting the returned frames and audio. YouTube support depends on yt-dlp and available captions or an enabled audio backend. Cloud backends require their corresponding API credentials and may incur usage costs; local Whisper avoids API usage but requires local installation and model resources.

For long videos, skipping structural analysis can lead to less informed segment selection. Audio warnings and incomplete-analysis details should be surfaced to users because they can affect transcription coverage.

_Full upstream README: https://allmcps.com/mcp/claude-video-vision/readme_