---
name: cli-hub-matrix-video-creation
description: >-
  Capability-based multi-tool matrix for video production. Agents pick providers
  (CLI-Anything harnesses, public CLIs, Python libs, native binaries, cloud APIs)
  per capability rather than marching through fixed stages, including storyboard
  planning, story/audio direction, source triage, internet video/music
  search/download, capture/generation, analysis, sound design, high-end caption
  design, NLE/render doctor investigation, review, and packaging.
---

# Video Creation Matrix (v3 — capability-based)

This matrix describes **capabilities** the agent can compose on demand — not a fixed pipeline. A "video creation" workflow picks a *recipe* (which capabilities it needs) and, per capability, picks a *provider* from the task requirements and preflight facts below.

Schema: [`docs/cli-matrix/matrix_registry.schema.md`](../../docs/cli-matrix/matrix_registry.schema.md).

## Install (installable portion)

```bash
cli-hub matrix install video-creation   # installs registered matrix CLIs only
cli-hub matrix info    video-creation   # inspect providers & recipes
cli-hub matrix preflight video-creation # check available providers in this environment
```

Not everything in this matrix is installed by `cli-hub matrix install`. Cloud APIs, Python packages, native binaries, third-party public CLIs, and external skills are first-class providers too, but install them only after the task actually needs that provider.
Do not hand-write `pip install ...#subdirectory=...` for CLI-Anything matrix members; install the supported harnesses through `cli-hub matrix install video-creation`, then use preflight to see what else is already available.

---

## Provider selection constraints (agent: evaluate per capability)

1. Use preflight as an availability report, not as a provider selector.
2. Choose providers from the user's goal, quality bar, budget, offline needs, credential state, install cost, and requested workflow.
3. Treat registry/provider order as documentation order only; do not assume the first provider is the correct one.
4. Install Python libs, native binaries, harness CLIs, public CLIs, or agent skills only when they fit the task constraints.
5. Escalate to paid or metered APIs only when the user has supplied credentials or explicitly consents. Never silently call a paid API.
6. If a task falls into a genre with a mandatory provider rule, the provider is not optional. Do not substitute a cheaper/easier fallback just because it is locally convenient.

Offline context? Filter to `offline: true` providers only.

---

## Preflight (run once per session, cache the result)

Run the built-in matrix preflight first:

```bash
cli-hub matrix preflight video-creation --json
cli-hub matrix preflight video-creation --capability composite.assemble
cli-hub matrix preflight video-creation --offline
```

If you need raw checks or are running without the latest `cli-hub`, use the manual block:

```bash
cli-hub list --json
python - <<'PY'
import importlib.util
for m in ("moviepy","whisper","pydub","PIL","edge_tts","pysrt","pysubs2","yt_dlp","spotdl","scenedetect","paddleocr","twelvelabs"):
    print(m, importlib.util.find_spec(m) is not None)
PY
for b in ffmpeg ffprobe sox convert magick screencapture yt-dlp spotdl scdl bandcamp-dl you-get lux BBDown scenedetect mediainfo ffmpeg-quality-metrics paddleocr hyperframes; do command -v "$b" >/dev/null && echo "$b: yes" || echo "$b: no"; done
for e in RUNWAY_API_KEY KLING_API_KEY PIKA_API_KEY SEEDANCE_API_KEY \
         ELEVENLABS_API_KEY MINIMAX_API_KEY OPENAI_API_KEY GOOGLE_CLOUD_PROJECT \
         ASSEMBLYAI_API_KEY DEEPGRAM_API_KEY \
         SUNO_API_KEY UDIO_API_KEY IDEOGRAM_API_KEY STABILITY_API_KEY \
         TWELVELABS_API_KEY GOOGLE_APPLICATION_CREDENTIALS; do
  [ -n "${!e}" ] && echo "$e: set" || echo "$e: unset"
done
```

---

## Suggest-to-user template (agent uses verbatim when escalating)

```
To enable <capability> via <provider>, please set <ENV_VAR>.
  Cost: <cost notes>
  Quality: <quality tier>
Reply 'skip' to fall back to <next provider>.
```

Examples:

- *To enable cinematic AI video via Runway Gen-4, please set `RUNWAY_API_KEY`. Cost: ~$0.05/sec as of 2026-04. Quality: sota. Reply 'skip' to fall back to `generate-veo-video` or `jimeng` if configured.*
- *To enable ByteDance Seedance video generation, please set `SEEDANCE_API_KEY`. Cost: metered per-clip. Quality: sota for realistic motion. Reply 'skip' to fall back to `jimeng` (Dreamina) which shares the ByteDance model family.*

---

## Capabilities

### `script.storyboard` — brief to creative direction, script, shot list, timing, and asset plan

Use this before generation, search, capture, or assembly when the user gives a vague concept or asks for a complete video. By default, do the planning directly as agent work: produce a structured brief, global creative direction, narrative/emotional arc, audio arc, script/narration, shot list, timing map, asset requirements, and reviewable storyboard before spending time on downloads or generation.

For any non-trivial video, read [`references/story-structure-audio.md`](references/story-structure-audio.md) and save `creative_direction.md` before final assembly. This is mandatory for trailers, sports/music montages, film commentary, found-footage edits, product launch videos, and any output where flat random clips or boring music would fail the brief.

Hard gate: do not begin final assembly for a non-trivial video until `creative_direction.md` exists. `plan.md` is not a substitute. The file must include target duration or requested duration range, output language, a shot-role table with time, beat, source/shot role, audio event, caption/title role, and failure risk; reject plans where the ending/final act has no payoff, climax, reveal, useful recap, or deliberate unresolved hook appropriate to the genre. Language rule: if the user specifies an output language, use it for all agent-authored viewer-facing content; otherwise use the language the user is using in the conversation.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| Agent-native planning | agent-native | none | free | high | yes |
| `storyboard-creation` skill | agent-skill | installed skill | free | high | yes |
| `remotion-best-practices` skill | agent-skill | installed skill | free | high for code-driven motion video | yes |

Selection:

- Start with agent-native planning for normal scripts, shot lists, timing maps, and asset plans.
- Include `creative_direction.md` with the one-sentence promise, story arc, emotional curve, audio arc, cut-density curve, visual motif, source roles, and no-flatness guardrails.
- Use `storyboard-creation` only when you need explicit storyboard-panel conventions, camera-angle grammar, continuity checks, or animatic planning.
- Use `remotion-best-practices` only when the storyboard will be implemented as Remotion/React motion-video code.

### `video.search` — discover candidate internet footage

Use this before `video.download` when the user asks for found footage, B-roll, public-domain clips, stock footage, YouTube/Bilibili material, or named movie/TV/game/anime moments. Prefer free/open sources first and record source URL, license, creator, and attribution requirement before editing.

For found-footage deliverables or platform-origin claims, read [`references/source-triage.md`](references/source-triage.md) and classify each candidate as direct platform source, verified platform-origin transport, or weak mirror.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| Web search + source filters | web-search | online search access | free | good | no |

Search discipline:

- For reusable/commercial-safe B-roll, start with Wikimedia Commons, YouTube Creative Commons, or other browser-searchable stock/public-domain pages and keep attribution metadata beside every downloaded file.
- For Bilibili/YouTube reference or fan-edit workflows, search specific scene names rather than generic terms. Add quality modifiers such as `1080p`, `4K`, `HD`, `BD`, `蓝光`, or `高清`.
- For Bilibili, use targeted web search such as `site:bilibili.com "Game of Thrones" S3E09 BV`; standalone `/video/BV...` uploads are usually easier to process than geo-restricted bangumi URLs.
- Do not treat downloadability as permission. If license or user authorization is unclear, ask before using the footage in a deliverable.

### `video.download` — download/import web video into the workspace

Use this after `video.search` identifies candidate URLs, or directly when the user gives URLs. Keep all raw downloads in one `sources/` directory, save `sources.json` with URL/license/creator/provenance, and normalize filenames before downstream editing.

For internet footage, `sources.json` must record the platform URL, transport URL if different, command, cookie-file path if used, local file, probe summary, selected ranges, source role, license/rights notes, and quality caveats.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `yt-dlp` | public-cli | `yt-dlp` + `ffmpeg` | free | high | no |
| `you-get` | public-cli | `you-get` bin | free | good | no |
| `lux` | public-cli | `lux` + `ffmpeg` | free | good | no |
| `BBDown` | public-cli | `BBDown` + `ffmpeg` | free | high for Bilibili | no |

Operational notes:

- Prefer `yt-dlp -f "bestvideo[height>=1080]+bestaudio/best" --merge-output-format mp4` for YouTube/Bilibili URLs when quality matters. Use cookies only when the user has authorized access to the content.
- For Bilibili audio-only extraction, avoid `yt-dlp -x --audio-format mp3`; download the raw m4a and convert with `ffmpeg`, then verify volume before using it.
- Use `BBDown` when Bilibili-specific metadata, subtitles, danmaku, playlists, or high-quality member streams are central to the task.

### `music.search` — discover existing songs, BGM, or clean audio sources

Use this before `music.download` when the user wants an existing song, soundtrack cue, royalty-free track, platform audio, or a specific cover/version. Keep the search about the requested music, not just any available audio. Record title, artist, platform URL, uploader, source type, version notes, rights/licensing, and attribution in `music_sources.json`.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| Web search + source filters | web-search | online search access | free | good | no |
| `yt-dlp` search extractors | public-cli | `yt-dlp` | free | good | no |
| `spotdl` metadata/search | public-cli | `spotdl` | free | good for Spotify-linked songs | no |

Search discipline:

- For clean audio, look for the artist's official upload, label upload, official audio/MV, verified lyric video, or a standalone audio upload with no extra descriptors.
- If the user asks for a specific version such as `女生版`, duet, piano, acoustic, live, DJ, karaoke, or instrumental, verify the title/uploader/metadata names that exact version before committing.
- Reject candidate titles that imply the wrong source: `remix`, `cover`, `fan edit`, `AMV`, `MAD`, `mashup`, compilation, trailer mix, or unrelated soundtrack/OST unless the user explicitly asked for that variant.
- For tie-in songs and promo tracks, assume dialogue/voiceover/SFX bleed is possible. Search alternatives with `{artist} {song} 纯音乐`, `{song} 无对白`, `{song} 歌词版`, `official audio`, or `lyric video` before asking the user to accept a risky source.
- Do not treat downloadability as permission. If rights, license, or user authorization is unclear, ask before using the music in a deliverable.

### `music.download` — download/import existing music into the workspace

Use this after `music.search` identifies a candidate or directly when the user supplies a music URL/local file. Keep raw downloads in `sources/music/`, save `music_sources.json`, and create a normalized working file for editing.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `yt-dlp` audio download | public-cli | `yt-dlp` + `ffmpeg` | free | high | no |
| `spotdl` | public-cli | `spotdl` + `ffmpeg` | free | good for Spotify-linked metadata | no |
| `scdl` | public-cli | `scdl` | free | good for SoundCloud | no |
| `bandcamp-dl` | public-cli | `bandcamp-dl` | free | good for Bandcamp | no |
| local file import + `ffmpeg` | native | `ffmpeg` | free | high | yes |

Operational notes:

- Use only music the user is authorized to use or that is clearly licensed for the deliverable. Do not bypass DRM, paywalls, or access controls.
- For general audio URLs, download source audio first, then convert explicitly:

```bash
yt-dlp -f "bestaudio[ext=m4a]/bestaudio" -o "sources/music/audio_raw.%(ext)s" "URL"
ffmpeg -i sources/music/audio_raw.m4a -vn -c:a libmp3lame -q:a 0 sources/music/audio.mp3
```

- For Bilibili audio-only downloads, do **not** use `yt-dlp -x --audio-format mp3`; it can produce a valid-looking but nearly silent MP3. Download the raw m4a, convert with `ffmpeg`, then verify volume:

```bash
yt-dlp -f 30280 -o "sources/music/audio_raw.%(ext)s" "BILIBILI_URL"
ffmpeg -i sources/music/audio_raw.m4a -c:a libmp3lame -q:a 0 sources/music/audio.mp3
ffmpeg -i sources/music/audio.mp3 -af volumedetect -f null - 2>&1 | grep mean_volume
```

Reject files with mean volume below roughly `-40dB` unless silence is expected.

- Sample-listen at least three points before editing, especially for promo/OST/tie-in songs:

```bash
ffplay -ss 30 -t 5 -autoexit sources/music/audio.mp3
ffplay -ss 90 -t 5 -autoexit sources/music/audio.mp3
ffplay -ss 150 -t 5 -autoexit sources/music/audio.mp3
```

- Normalize or loudness-match only after verifying the source is the right song/version and has no dialogue/SFX bleed.

### `visual.capture` — record screen / webcam / window

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `cli-anything-openscreen` | harness-cli | harness installed | free | high | yes |
| `cli-anything-obs-studio` | harness-cli | OBS installed | free | high | yes |
| `ffmpeg -f x11grab` / `avfoundation` | native | `ffmpeg` | free | high | yes |
| `screencapture` | native | macOS | free | high | yes |
| `mss` / `pyautogui` + `cv2` | python | pkgs | free | good | yes |

### `visual.generate` — produce a video clip from prompt/reference

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `generate-veo-video` | public-cli | `generate-veo` bin + Google creds | metered | high | no |
| `jimeng` | public-cli | `dreamina` bin + Dreamina login | metered | high | no |
| Runway Gen-4 | api | `RUNWAY_API_KEY` | paid | sota | no |
| Kling | api | `KLING_API_KEY` | paid | high | no |
| Pika | api | `PIKA_API_KEY` | paid | good | no |
| Seedance | api | `SEEDANCE_API_KEY` | paid | sota | no |

### `audio.capture` — record and clean audio tracks

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `cli-anything-audacity` | harness-cli | Audacity installed | free | high | yes |
| `sox` / `ffmpeg` | native | binary | free | high | yes |
| `pydub` / `soundfile` / `librosa` / `noisereduce` | python | pkgs | free | good | yes |

### `audio.synthesize` — text-to-speech / voice

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `minimax-cli` | public-cli | bin + MiniMax key | metered | high | no |
| `elevenlabs` | public-cli | bin + `ELEVENLABS_API_KEY` | paid | sota | no |
| OpenAI TTS | api | `OPENAI_API_KEY` | metered | high | no |
| Google Cloud TTS | api | `GOOGLE_CLOUD_PROJECT` | metered | high | no |
| `edge-tts` | python | pkg | free | good | no |

### `music.generate` — generated music / BGM

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| `suno` | public-cli | bin + Suno account | metered | sota | no |
| `minimax-cli` | public-cli | bin + MiniMax key | metered | high | no |
| Udio | api | `UDIO_API_KEY` | paid | sota | no |

Use `music.search` + `music.download` instead when the user asks for an existing song, official upload, platform audio, soundtrack cue, royalty-free track, or a specific cover/version.

Music and SFX must follow the story/audio arc in `creative_direction.md`. For polished 60+ second videos, choose a real main music strategy first: either AI-generated music from a music provider, downloaded relevant/authorized music via `music.search` + `music.download`, or strong source ambience when the genre is documentary/ambient. Avoid one flat loop from start to finish; plan section changes such as intro, buildup, drop, dip, final lift, source-audio reveal, or final resolve.

### `sound.design` — hits, risers, score sections, mix dynamics, and final audio arc

Use this when a video needs trailer hits, whooshes, risers, drones, heartbeat gaps, sub drops, crowd/source accents, or locally generated score elements. Do not hide sound design inside `music.generate`; a music bed is not a designed mix.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| Agent-native sound plan | agent-native | `references/sound-design.md` | free | high | yes |
| `ffmpeg` / `sox` procedural stems | native | binary | free | good | yes |
| `pydub` / `numpy` procedural stems | python | pkgs | free | good | yes |
| Generated music provider | public-cli/api | chosen `music.generate` provider | metered/paid | high | no |
| Downloaded/authorized music | public-cli/skill | `music.search` + `music.download` or `music-downloader` skill | varies | high when relevant | no |

Deliverables for polished edits: `sound_design.md` with stems, cue times, story function, ducking notes, and section loudness targets; separate WAV stems when generated locally; and per-section loudness checks. The ending/final act should have intentional audio shape such as escalation, silence/hold, hit/drop, source-audio reveal, or resolve when the genre calls for it. Read [`references/sound-design.md`](references/sound-design.md) for sports, commentary, trailer, and final-act patterns.

Source-audio gate: before mixing downloaded/captured clip audio with new music or narration, classify every used range as `silent_or_mute`, `ambience_keep`, `dialogue_keep`, `music_only`, `mixed_music_speech`, or `needs_separation`. Keep one foreground voice and one intentional music bed at a time. If source speech/music overlaps new narration/music, mute, duck, make the source foreground, run separation, or reject the range; do not hide the conflict behind "source texture."

Procedural-audio gate: locally generated audio is acceptable for short SFX, UI ticks, impacts, pulses, and risers, but it must not become the default main music bed for polished 60+ second videos. Prefer AI-generated music or downloaded relevant/authorized music for the main bed. Locally generated noise, risers, and whooshes must be filtered, enveloped, gain-staged, and sample-reviewed. Do not use raw Gaussian/full-band noise as a music bed or repeated transition effect; hiss/sizzle in the promoted final is a critical issue even if `silencedetect` looks normal.

### `media.analyze` — segment, label, OCR, and search footage

Use this after download/capture and before edit planning when there are many clips or when the edit depends on finding specific shots. Output should be a scene library with time ranges, keyframes, visible text, people/objects/actions where available, usability notes, and searchable tags.

| Provider | Kind | Requires | Cost | Quality | Offline |
|---|---|---|---|---|---|
| PySceneDetect `scenedetect` | public-cli | `scenedetect` + `ffmpeg` | free | high | yes |
| Google Cloud Video Intelligence | api | GCP creds | metered | sota | no |
| TwelveLabs video search/index | api | `TWELVELABS_API_KEY` + `twelvelabs` pkg | metered | sota | no |
| PaddleOCR on sampled keyframes | public-cli/python | `paddleocr` pkg or bin | free | good overall