Build a Video to Shorts Pipeline
Step 2 of 9 · An afternoon to build, then about an hour of machine time per recording
Transcribe and Strip the Dead Air
Get the recording onto your machine, transcribe it with word level timestamps, and cut the silence and filler. Those word timings are the source of truth for every cut in this pipeline.
A word timestamped transcript, an edit decision list that logs the reason for every cut, and a rough cut with the dead air gone.
Two things happen in this step and the order matters. You transcribe with word level timestamps, not sentence level, because every later step cuts on word boundaries. Then you use those timings to remove silence and filler.
Get the file onto your machine
Keep a manifest so a rerun does not download a multi gigabyte file twice, and keep an exclusion list so an unrelated giant upload sitting in the same folder never enters the pipeline by accident.
Write pipeline/ingest/ingest.py.
It downloads a recording from a shared cloud folder into work/ and keeps a
manifest so already downloaded files are skipped.
Requirements:
- Use gdown (the files are link accessible, so no OAuth flow needed)
- work/manifest.json maps file id to {downloaded: true, path}
- If the id is in the manifest AND the file exists on disk, print
"already downloaded" and return the path instead of downloading again
- Keep an EXCLUDED_IDS set at the top. If an id is in it, exit with a clear
message. That is how I keep giant unrelated uploads out of the pipeline.
- Usage: python pipeline/ingest/ingest.py <file_id> <output_name>
Then run it on this file id: [YOUR FILE ID], saving to work/ep-test.mp4Transcribe with word level timestamps
This is the most important artifact in the pipeline. Get it wrong and nothing downstream cuts cleanly. Two settings matter: word timestamps on, and a voice activity filter so long silences are not transcribed as hallucinated text.
Write pipeline/transcribe/transcribe.py using faster-whisper.
Requirements:
- model name from argv[2], falling back to env WHISPER_MODEL, default large-v3
- WhisperModel(model, device="auto", compute_type="auto")
- transcribe with word_timestamps=True, vad_filter=True, and
vad_parameters of min_silence_duration_ms 500
- Write TWO outputs next to the input video:
<stem>.words.json segments each with s, e, text, and a words array where
every word is {w, s, e, p}, p being probability, plus
top level language and duration
<stem>.srt plain SRT so I can eyeball the transcript
- Round all times to 3 decimals
- Print progress every 50 segments so I know it is alive
- Usage: python pipeline/transcribe/transcribe.py work/ep-test.mp4 [model]
Run it on work/ep-test.mp4 with model "base" first so I get a fast result to
sanity check, then tell me how long the largest model will take on this file.Cut the dead air and the filler
The rule that keeps this from sounding robotic: only cut filler words that are flanked by pauses. An um in the middle of fluent speech is load bearing, and cutting it produces an audible jump. An um surrounded by silence is dead weight.
| Setting | Value | Effect |
|---|---|---|
| GAP_CUT | 1.6s | Silences longer than this get cut |
| PAD | 0.25s | Breathing room kept each side of a cut so it does not clip the next word |
| FILLER_PAUSE | 0.35s | A filler word is only cuttable if pauses this long surround it |
| FILLERS | um, uh, erm, hmm, mm | Token list, matched case insensitively with punctuation stripped |
Write pipeline/cut/cut.py.
It reads <stem>.words.json, builds an edit decision list, and applies it with
ffmpeg.
Constants at the top:
GAP_CUT = 1.6 cut silences longer than this
PAD = 0.25 breathing room kept each side of a cut
FILLERS = {"um","uh","erm","hmm","mm"}
FILLER_PAUSE = 0.35 a filler is cuttable ONLY if pauses this long flank it
Logic:
1. Flatten all words from all segments into one list.
2. Mark a filler for removal ONLY when the gap before AND the gap after both
meet FILLER_PAUSE. Mid sentence fillers must be left alone, they create
audible jump cuts.
3. Build keep intervals across the remaining words, cutting any gap longer
than GAP_CUT, and trim leading and trailing silence.
4. Every cut records a reason string, for example "filler um", "dead air
3.2s", "leading silence", "trailing silence".
Outputs:
<stem>.edl.json duration, kept_seconds, removed_seconds, keeps, cuts
<stem>.roughcut.mp4 the applied cut
Render with a single ffmpeg select and aselect filter built from the keep
intervals, libx264 crf 20 preset veryfast, aac audio, faststart.
Support a no-render flag so I can inspect the EDL before burning CPU.
Usage: python pipeline/cut/cut.py work/ep-test.mp4 [--no-render]
Run it with the no-render flag first and show me the summary line.Questions
Which transcription model should I use?
Use the small one while you are building so each run takes minutes instead of an hour. Switch to the largest available for anything you will publish. Word timings from the small models drift enough to clip word starts, which shows up as clipped audio in every short.
Why keep an edit decision list instead of just rendering?
Because the reason for every cut is logged. When the scoring step later says pacing is off, you can read exactly what was removed and change one threshold instead of guessing.
Talk it through
Stuck on this step?
Free 30 minutes. Bring the error, the transcript, or the clip that will not pass the gate, and we will work through it.