The TWV video pipeline got its biggest quality jump yet — real audio, human-paced narration, and karaoke captions — without touching the deterministic core.
A 3-agent audit + viral-channel research produced an 8-phase roadmap. This session shipped Phases 0, 1, 2 and most of 3: reliability fixes, real Envato SFX/music at broadcast loudness, role-based voiceover pacing, word-by-word captions, and three new short-form layouts. The single loudest amateur tell (synthesized sine-wave SFX + one music bed at −25 LUFS) is gone. Two items were deliberately deferred because they risk the number↔voice sync gate the whole engine is built on.
StatusShipped + verifiedScopescripts/twv-automationPhases0,1,2 + most of 3Date2026-07-03
ReasoningLập luận
6-layer thinking chainChuỗi tư duy 6 tầng
L1
Raw requestYêu cầu gốc
"bạn rà soát lại skill tạo video của TWV xem có thể nâng cấp những gì để tạo nên 1 video mượt mà, chất lượng, cinematic, chuyên nghiệp... hình ảnh visual, hiệu ứng cho đến content mọi khía cạnh; ứng dụng best practices của các kênh viral; flow làm việc cũng tối ưu cả chất lượng lẫn token."
L2
Reframe — what was really neededDiễn giải lại — nhu cầu thật
The ask wasn't "add effects" — it was "close the gap between our current output and a professional channel, everywhere at once." The real need: find where perceived quality is actually capped, fix those specific ceilings, and do it without rewriting a pipeline that already works. Then keep going ("làm tiếp") through the voice and visual layers.
L3
ConstraintsRàng buộc
- The render architecture is deliberately deterministic — the LLM only writes scripts, everything else is ffmpeg + Remotion math. Must not break that.
- Voicebox (local TTS) was OFFLINE this session (http 000), so longform voiceover changes can't be live-rendered — only validated by deterministic timing tests + the shorts path that bakes voiceover already.
- The entire engine locks visuals to voice: every spoken number must be on screen within 0.6s. Any change that desyncs audio from the cut endangers that gate.
- Free-tier quotas (Gemini 250 vision calls/day) and a solo operator — no paid subscriptions beyond the Envato Elements the user already owns.
- Two SKILL.md files described pipelines that no longer exist (Canva, ElevenLabs) — forcing a re-discovery tax every session.
L4
Options weighedCác lựa chọn cân nhắc
- Rewrite the renderer for a "cinematic" look — highest ceiling, but throws away a working, token-cheap engine and can't be verified in one session. Cost: enormous risk, low confidence.
- Chase visible effects first (transitions, LUTs) — feels productive, but the real amateur tell was audio, not visuals. Cost: polishing the wrong layer.
- Fix the measured ceilings in dependency order — reliability → audio → voice → visual, each verified before the next. Cost: slower, unglamorous phase 0, but everything after stands on solid ground. ✅
L5
Principle invokedNguyên tắc áp dụng
Fix the ceiling that actually caps perceived quality, in the order that lets each fix be verified — audio is half of "cinematic" and the cheapest to fix; reliability and doc-truth come first because everything later is built on them.
L6
Pick + recognition signalLựa chọn + dấu hiệu nhận biết
Chose the phased, dependency-ordered path: Phase 0 reliability + truth, Phase 1 real audio, Phase 2 voice humanization, Phase 3 visual. Rejected the rewrite (unverifiable, discards a working engine) and the effects-first path (wrong layer). Recognition signal: when a request is "make everything better," resist starting with the visible surface — measure where quality is actually capped, then fix ceilings in the order that keeps each fix testable.
ArchitectureKiến trúc
How the upgraded engine fits together
Two render paths share one asset + audio spine. The short-form path renders through Remotion; the long-form path composites through ffmpeg. This session added a single source of truth for timing, a shared audio manifest, and role-driven voiceover — so both paths speak the same language.
| Technical termThuật ngữ kỹ thuật | Plain nameTên dễ hiểu | RoleVai trò |
|---|---|---|
timing.py | One clock everyone reads | The cut-offset / shot-start math used to live copied across four files; one wrong copy made the sync gate lie. Now assemble, sync-qa, quality-gate and verify all import the same functions — proven numerically identical to the old code. |
sfx-manifest.json + music_pool.py | Real sound, rotated | Logical SFX names (whoosh/pop/riser…) resolve to real Envato samples with variant rotation so back-to-back cuts don't repeat; music picks by mood (tension/optimism/data/warm) least-recently-used, so the channel never repeats a track. |
narrate.py roles | Voice that breathes | Each voiceover paragraph gets a role (hook/explain/stat/punch/cta) that sets its own pace and the silence around it — the read speeds through a hook and takes a real 0.7s breath before a number lands. |
Captions.tsx | Karaoke captions | Words light up one at a time across the line's cue-accurate window — the 2026 standard for the 85% who watch muted. |
yt-retention.js | Start collecting the truth | Pulls YouTube retention curves per video into content/retention/ from today, so the future analytics loop (Phase 7) has history to learn from. |
Read it in 30 seconds: the top row is where a video is authored; the middle row is the three shared spines this session hardened (timing, audio, data); the bottom row is the two render paths that now both draw from those spines. Nothing in the deterministic core was replaced — the upgrades slot into the existing math.
Lesson to carry forward: when two pipelines drift apart, the fix isn't to sync them by hand each time — give them one shared module to import. timing.py turned a recurring class of "the gate lies" bugs into an impossible state.
PipelineQuy trình
What shipped, in dependency order
0
Reliability + truth
timing.py single-source; staleness bug (stale overlaid master) + auto-fix clobber guard (never replace curated footage) fixed; two stale SKILL.md files rewritten to match reality; YouTube retention ingestion wired.
1
Real audio
15 real Envato SFX with variant rotation; mood-tagged music pool with least-recently-used rotation; −14 LUFS broadcast loudness bus ported to shorts (was ~−25 LUFS). Verified: a test clip measured exactly −14.0 LUFS.
2
Voice humanization
Per-beat role pacing (hook fast, stat slow + breath, punch slowest), emphasis via chatterbox, per-scene Gemini style hints for shorts. Verified by stubbed-TTS timing test (contiguous scenes, exact length match).
3
Visual / cinematic
Karaoke captions + 3 new short-form layouts (all render-verified); motivated-transition lint (caught real issues); alternating Ken-Burns (footage-verified); 3 grade presets. J/L cuts + 3D still-grid deferred with reasons.
4
Phase 4–7
Script craft (hook lint, open loops, loop endings, data-driven thumbnail specs), batched Gemini vision, /twv-demo skill, analytics feedback loop.
DecisionsQuyết định
Layered decision cardsCác quyết định theo tầng
Fix audio before chasing visual effects
L1"cinematic, chuyên nghiệp... hình ảnh visual, hiệu ứng"
L2The word "cinematic" pulls toward LUTs and transitions, but the audit found the loudest amateur tell was the soundtrack: synthesized sine-wave SFX and one reused music bed, mixed ~11 dB quieter than platform standard.
L3Envato Elements is already paid for; no budget for new subscriptions. Deterministic core must stay.
L4(a) Buy a paid SFX library — needless cost. (b) Keep synth SFX, polish visuals — polishing the wrong layer. (c) Curate real SFX/music from Envato + port the loudness bus — cheap, high-impact. ✅
L5Audio is half of "cinematic" and the cheapest half to fix — a −25 LUFS clip sounds amateur no matter how it's graded.
L6Built shared SFX + music manifests and a −14 LUFS bus. Rejected paid libraries (already own Envato) and visuals-first (wrong ceiling). Recognition signal: before grading footage, listen — if the audio is quiet or synthetic, that's the ceiling.
Keep Voicebox, humanize it — don't migrate to ElevenLabs
L1Research flagged ElevenLabs v3 as the 2026 standard for human-sounding TTS.
L2The real goal is a voice that doesn't sound robotic — which is mostly about pacing variance and breaths, not the engine. ElevenLabs costs per character; Voicebox is free with a cloned TWV voice.
L3Voicebox is local and free but was offline this session; monotone 180-wpm read is the flat tell.
L4(a) Migrate to ElevenLabs v3 — best raw quality, ongoing cost, credit dependency. (b) Hybrid (ElevenLabs for flagship only) — two engines to maintain. (c) Keep Voicebox, add role-based pacing + breaths + emphasis — free, closes 70–80% of the gap. ✅ (user's call)
L5Human-sounding narration is 80% pacing and breath, 20% engine — buy the 80% for free before paying for the 20%.
L6Role-driven pacing in narrate.py (hook 192 wpm, stat 165 + a 0.7s breath, punch 160). Rejected ElevenLabs migration (cost, credits) and hybrid (maintenance). Recognition signal: when "the voice sounds robotic," fix rhythm and pauses first — the engine swap is the expensive last resort.
Defer J/L cuts and speed ramps — they threaten the sync gate
L1The roadmap listed J/L cuts and speed ramps as cinematic-polish items.
L2Both deliberately desync audio from the visual cut, or change a clip's duration — and the entire engine guarantees every spoken number is on screen within 0.6s. These features fight the one invariant that makes the pipeline trustworthy.
L3Voicebox offline → longform voiceover changes can't be live-rendered and validated this session.
L4(a) Ship them anyway — risk silently breaking the number-sync gate. (b) Build behind a flag, unverified — dead code that lies about being done. (c) Defer with a written reason until they can be render-validated. ✅
L5Never ship a change that can break your load-bearing invariant when you can't verify it — an unverified cinematic flourish isn't worth a video where the numbers drift off the voice.
L6Deferred both, documented the reason in decisions.md. Rejected shipping-blind and flag-hiding. Recognition signal: if a "polish" feature touches the one guarantee your system is built on, gate it behind verification, not a feature flag.
Karaoke by line-window, not word-alignment
L1Research: word-by-word captions add 25–40% retention in the first 5 seconds.
L2True karaoke needs each word's timestamp — but the captions are distilled ("10% baseline"), not verbatim to the spoken words, so per-word alignment to the transcript is unreliable.
L3The caption line's START is already cue-accurate (pinned to the spoken cue). Word timestamps exist but don't map to the distilled text.
L4(a) Force-align distilled words to transcript words — fragile, mismatches. (b) Add per-word timing data in the bridge — extra data plumbing. (c) Reveal the line's own words evenly across its cue-accurate window — robust, zero new data. ✅
L5You don't need per-word truth to feel synced — a reliable line window plus even reveal reads as karaoke without the fragility of exact alignment.
L6Rewrote Captions.tsx to reveal words across the line window (spoken lit, upcoming dim, key-word gold). Rejected force-alignment (fragile) and new timing data (unnecessary). Recognition signal: when exact alignment is unreliable, anchor to the interval you DO trust and interpolate inside it.
FilesTệp
Artifact mapBản đồ tệp tạo ra
| PathĐường dẫn | WhatLà gì | Who reads itAi dùng |
|---|---|---|
scripts/twv-automation/longform/timing.py | NEW — single source of truth for xfade-chain timing math | Imported by assemble.py, sync-qa.py, quality-gate.py, verify.py |
scripts/twv-automation/longform/assemble.py | Staleness fix, transition lint, 3 grade presets, alternating Ken-Burns, manifest-driven SFX | Long-form ffmpeg compositor |
scripts/twv-automation/longform/narrate.py | Per-beat role pacing, breaths before stats, emphasis passthrough | Long-form voiceover (Voicebox) |
scripts/twv-automation/longform/quality-gate.py + run.py | Auto-fix clobber guard (never replace curated/pinned footage) | QA gate + orchestrator |
scripts/twv-automation/music_pool.py | NEW — mood-tagged, least-recently-used music selector | Both render paths |
.../remotion/src/clip/components/Captions.tsx | Word-by-word karaoke reveal | Short-form TwvClip |
.../remotion/src/clip/scenes/SplitCompare|ChecklistBeat|QuoteCard.tsx | NEW — 3 short-form layouts (VN-vs-X, checklist, pull-quote) | Short-form scene dispatch |
scripts/twv-automation/yt-api/yt-retention.js | NEW — pulls YouTube retention curves per video | Future analytics loop (Phase 7) |
~/beup-workspace/assets/sfx/sfx-manifest.json + music-manifest.json | NEW — real Envato audio catalogs with rotation | assemble.py + twv-clip-bridge.mjs |
docs/decisions.md | Two dated entries logging the full roadmap + Phase 2/3 decisions | Project decision log |
Pending manual actions — required before fully liveViệc cần làm thủ công — trước khi hoàn tất
- Run `node scripts/twv-automation/yt-api/yt-auth.js` once to re-consent (adds the YouTube Analytics scope) — then `yt-retention.js pull` starts collecting retention history.
- Download 4 Envato items to replace synthesized fallbacks (real riser/impact/ding SFX + a tension music track) per ~/beup-workspace/assets/sfx/ENVATO-DOWNLOAD-CHECKLIST.md, then run `node sync-sfx.mjs`.
- Open the Voicebox app and run one longform `run.py` to live-validate the new role-based voiceover pacing (it was offline this session, so voiceover changes were only validated by deterministic timing tests).
Self-testTự kiểm tra
Check your understanding
Why was the same cut-timing math extracted into timing.py instead of left copied across four files?
Why were J/L cuts and speed ramps deferred rather than shipped?
In your own words: why keep Voicebox and add pacing/breaths instead of migrating to ElevenLabs v3?
In your own words: why does the karaoke caption reveal words by the line's time window rather than aligning each word to the transcript?
Mastery checklist — tick what you can explain unpromptedBảng tự đánh giá — tích những gì bạn tự giải thích được