AV Processor 0.2.2

Portable audio/video synchronization, interrupted-camera recovery, transcription, QA, and final naming pipeline.

Start here: Quick Start

• Windows details: Windows Setup
• Weekly process and recovery controls: Operating Procedure

Scope

Current implementation covers:

• input probing via ffprobe
• local source pairing by inferred recording start/end time
• audio extraction to normalized mono 48k WAV via ffmpeg
• sync offset detection via:
  • multi-peak transient matching
  • fallback envelope cross-correlation
• mux stage that outputs:
  • external/dedicated audio as track 1
  • camera scratch audio as track 2
• JSON + debug artifact output
• one-command run workflow for one or multiple camera segments
• automatic transcript-based segment alignment and freeze-frame gap recovery
• portable whisper.cpp CUDA, ROCm, Metal, and CPU tool discovery
• NVENC, AMF, QSV, VAAPI, VideoToolbox, and libx264 stitch encoders
• final transcript-derived provisional title metadata

Planned / partial next:

• upload/job manifest workflow
• Copyparty-compatible intake and download adapter
• trim boundary detection
• title/speaker/series metadata extraction
• final output renaming pass
• posting metadata generation

Repo layout

src/av_match/
  cli.py
  probe.py
  extract_audio.py
  sync_detect.py
  mux.py
  ffmpeg_utils.py
sync_detect.py          # convenience entrypoint

Install / run

Extract the release ZIP, then run the platform setup script.

Windows:

Set-ExecutionPolicy -Scope Process Bypass -Force; irm https://av-proc.jezzick.net/install.ps1 | iex

From an extracted release ZIP:

powershell -ExecutionPolicy Bypass -File .\setup.ps1
.\av-processor.ps1 doctor

Linux/macOS:

chmod +x setup.sh av-processor.sh
./setup.sh
./av-processor.sh doctor

Process a normal recording:

.\av-processor.ps1 run --video "D:\Incoming\camera.mp4" --audio "D:\Incoming\board.mkv" --output-dir "D:\AV-Jobs" --title "The Gift"

For interrupted video, repeat --video in chronological order:

.\av-processor.ps1 run --video "D:\Incoming\part-1.mp4" --video "D:\Incoming\part-2.mp4" --audio "D:\Incoming\board.mkv" --output-dir "D:\AV-Jobs"

Sync detect CLI

Intended Job Shape

The final interface should be upload/job based rather than folder-scan based. Folder scan is a local recovery convenience.

Normal job inputs:

• one or more chronological camera/video files with scratch audio
• one separate AV setup audio file

Normal job outputs:

• final rendered video
• transcript files
• sync report
• final QA report
• metadata draft
• optional YouTube upload receipt

The video scratch audio must be retained as a first-class fallback track because some services need it to repair missing or damaged dedicated audio.

Copyparty can be used as an intake/download adapter:

• upload raw video and audio into a job folder
• worker starts once both assets are present
• finished video and artifacts are exposed for download
• YouTube upload remains optional and credential-gated

Process Job Manifest

The process command is the compatibility bridge between today's local CLI and the later upload/worker model. It runs from a JSON manifest, writes artifacts into a job work directory, and reuses existing stage outputs unless forced.

Example:

PYTHONPATH=src python3 -m av_match.cli process \
  --job examples/job.local.json \
  --dry-run

Run the job:

PYTHONPATH=src python3 -m av_match.cli process \
  --job examples/job.local.json

Force individual stages:

PYTHONPATH=src python3 -m av_match.cli process \
  --job examples/job.local.json \
  --force-sync \
  --force-render \
  --force-qa

Minimal manifest shape:

{
  "job_id": "20260607-sws-the-gospel",
  "series": "SWS",
  "title_override": "The Gospel",
  "source": {
    "video": "/path/to/camera-video.mp4",
    "audio": "/path/to/av-audio.mkv"
  },
  "storage": {
    "kind": "local",
    "work_dir": "/path/to/job-work-dir"
  },
  "sync": {
    "method": "auto",
    "search_window_sec": 180,
    "max_offset_sec": 180,
    "ignore_leading_silence": true
  },
  "render": {
    "profile": "stream_copy",
    "keep_scratch_audio": true,
    "scratch_fallback": true,
    "scratch_volume": 0.12,
    "crossfade_sec": 2.0,
    "limiter": 0.95
  },
  "validation": {
    "transcribe_sources": true,
    "require_transcript_alignment": true,
    "auto_correct_gross_sync": true,
    "max_sync_disagreement_sec": 2.0
  },
  "publish": {
    "youtube": false
  }
}

Current process support:

• explicit single or multi-segment source paths
• optional source discovery through source.discover_dir
• sync report
• stream-copy dual-audio mux
• final output probe
• final alignment check when scratch fallback audio is retained
• source scratch, source dedicated, and final-track GPU transcription through whisper.cpp
• transcript-anchor alignment report and semantic/waveform sync decision
• automatic local waveform refinement around a transcript-derived coarse offset
• independent per-segment transcript alignment and waveform refinement
• frozen-frame camera-gap recovery and duplicate-overlap trimming
• automatic attenuated scratch intro/tail fallback with crossfades and limiting
• saved render plan with dedicated availability and fallback transition times
• summary.json and qa.json

Weekly Reliability Protocol

The weekly Sunday-service workflow should treat the first process output as a candidate, not a final render. Peak matching is useful for an initial guess, but it has produced false matches when the scratch and board recordings contain similar repeated phrases or very different room/board dynamics.

Required stages for each weekly job:

1. Intake

   • Select the newest camera video and AV recorder file, or use explicit source paths.
   • Probe both files and record duration, size, codecs, and source paths.
   • Keep the camera scratch audio as a first-class reference/fallback track.

2. First pass

   • Run process with GPU transcription enabled.
   • Preserve the first-pass sync report even if it later proves wrong.
   • Do not rename or publish from this pass until validation is complete.

3. Transcription gate

   • Transcribe the default/dedicated output track with whisper.cpp on ROCm GPU device 0.
   • Transcribe the scratch/reference track too, either full-length or at minimum enough excerpts to cover:
     • opening / first audible speech
     • early body section
     • mid-service section
     • late body section
     • closing / tail
   • Compare transcript anchors, not just peak offsets. The same content should appear at the same final timeline positions.

4. Sync validation gate

   • Run waveform/envelope correlation over multiple windows after the board audio is active.
   • Treat correlation as advisory when board and scratch dynamics differ heavily.
   • Prefer transcript anchors when peak/correlation checks disagree with obvious speech content.
   • A release candidate should usually have stable checked windows within about +/-0.05s when waveform correlation is reliable.

5. Boundary validation gate

   • Confirm the default track has audible content at the opening.
   • Confirm the default track has audible content through the closing section.
   • If dedicated audio starts late, use attenuated scratch audio as the default-track intro and crossfade into dedicated audio.
   • If dedicated audio ends early, use attenuated scratch audio as the default-track tail and crossfade from dedicated audio.
   • Retain the unmodified scratch audio as track 2.

6. Level validation gate

   • Measure intro, crossfade, body, and tail volume.
   • Attenuate scratch fallback segments so they do not jump louder than the board audio.
   • Use a limiter after mixing fallback and dedicated tracks.

7. Finalization

   • Render a corrected audio-sync-vN release candidate when any gate fails.
   • Regenerate the full final transcript and SRT from the final default track with GPU device 0.
   • Create the generic and title-named hard links only after validation.
   • Update the vault note with source files, final output, transcript paths, correction decisions, and validation anchors.

Practical naming convention:

• YYYYMMDD-sws.mp4 is the current final hard link.
• YYYYMMDD-SWS-Title.Case.mp4 is the title-named final hard link.
• YYYYMMDD-sws.audio-sync-vN.mp4 records corrected candidates.
• Older candidates are retained for audit/debug unless disk pressure requires cleanup.

Known failure modes and required response:

• Peak match trims the wrong direction: transcribe both tracks and derive the offset from matching phrases.
• Default audio misses the intro: scratch intro plus crossfade into dedicated audio.
• Default audio goes silent before the end: scratch tail plus crossfade out of dedicated audio.
• Scratch fallback is too loud: attenuate scratch fallback, then limit the mixed track.
• Waveform QA disagrees with transcript anchors: trust transcript anchors for content placement and use waveform only for fine tuning where stable.
• Long adelay creates a sparse or oversized AAC silence packet: prefix the delayed track with frame-sized anullsrc audio through concat; verify packet duration and seeked extraction before release.

The process command now implements these core gates. A job only reports ready when the source transcript alignment is reliable, the sync decision is approved, the requested scratch track is present, and the final transcript exists. Otherwise it still produces a reviewable candidate and reports needs_review.

Optional render overrides for exceptional recordings:

{
  "render": {
    "scratch_volume": 0.06,
    "crossfade_sec": 2.0,
    "scratch_intro_until_sec": 12.0,
    "scratch_tail_from_sec": 1840.0
  }
}

scratch_intro_until_sec is the end of the opening crossfade. scratch_tail_from_sec is the start of the closing crossfade. When omitted, the processor derives both boundaries from the selected offset and source durations.

Planned process stages:

• Copyparty job adapter
• YouTube upload adapter

Transcription

Transcription uses GPU device 0 by default. binary and model should normally remain auto; platform setup places them under the package tools/ directory.

Use whisper.cpp with ROCm GPU device 0:

PYTHONPATH=src python3 -m av_match.cli transcribe \
  --input /path/to/final.mp4 \
  --out-dir /path/to/job/transcript \
  --stem 20260621-sws

Use --audio-stream 1 to transcribe the retained scratch track from an output file directly. Normal process jobs transcribe the camera and dedicated source tracks independently before rendering.

Default transcription settings:

• engine: whisper_cpp
• binary: auto (portable package, environment override, or PATH)
• model: auto (tools/models/ggml-small.en.bin)
• GPU: enabled
• GPU device: 0
• language: en
• outputs: txt, srt, vtt, json, csv

CPU transcription is opt-in only:

PYTHONPATH=src python3 -m av_match.cli transcribe \
  --input /path/to/final.mp4 \
  --out-dir /path/to/job/transcript \
  --stem 20260621-sws \
  --cpu

Environment overrides are available for managed installations:

• AV_PROCESSOR_HOME
• AV_PROCESSOR_WHISPER_BIN
• AV_PROCESSOR_WHISPER_MODEL

Source Pair Discovery

AV Processor can scan a local directory and find likely camera/video plus AV/audio pairs. The pairing rule is intentionally simple and explainable:

• infer each media file's start time from filename timestamp, creation_time metadata, or mtime - duration
• compute finish time as start + duration
• pair files when both start times and finish times are within the threshold
• rank pairs by the larger boundary delta

This fits the Sunday-service workflow because phone video and AV recorder files usually start and stop within a few minutes of each other even when their internal audio needs sync correction.

PYTHONPATH=src python3 -m av_match.cli discover-pairs \
  --dir /home/joseph/Downloads \
  --threshold-sec 300 \
  --timezone America/Chicago

The default threshold is 300s / five minutes.

python3 sync_detect.py \
  --video input.mp4 \
  --audio external.wav \
  --out sync_report.json \
  --method auto \
  --search-window-sec 30 \
  --max-offset-sec 30 \
  --peak-threshold 4.0 \
  --min-peak-gap-ms 150 \
  --min-matched-peaks 3 \
  --debug-dir debug/

Manual mode:

python3 sync_detect.py \
  --video input.mp4 \
  --audio external.wav \
  --method manual \
  --offset-sec -1.642 \
  --out sync_report.json

Semantics

offset_sec = external_peak_time - camera_peak_time

• positive offset: external audio starts later than camera audio
• negative offset: external audio starts earlier than camera audio

Output report

Example:

{
  "method": "spike_multi",
  "selected": true,
  "offset_sec": -1.642,
  "confidence": 0.994,
  "matched_peaks": 7,
  "residual_median_ms": 6.8,
  "residual_p95_ms": 14.2,
  "drift_ppm_estimate": 18.0,
  "camera_peak_count": 12,
  "external_peak_count": 10,
  "search_window_sec": 30,
  "sample_rate": 48000,
  "debug_files": [
    "camera_peaks.csv",
    "external_peaks.csv",
    "matched_peaks.csv",
    "envelope_summary.json"
  ]
}

Mux CLI

PYTHONPATH=src python3 -m av_match.cli mux \
  --video input.mp4 \
  --audio external.wav \
  --offset-sec -1.642 \
  --out final_with_dual_audio.mp4

Behavior:

• output video stream is copied from camera source
• audio track 1 = dedicated/external audio aligned by offset
• audio track 2 = original camera scratch audio
• track 1 is default

Current real-world artifact status

Completed synced output currently exists at:

• /home/joseph/Downloads/AV-Match-out-v3/20260517-SWS-The.Lordship.of.Christ.mp4

Earlier versions also exist at:

• /home/joseph/Downloads/AV-Match-out-v2/IMG_9883.avmatch.dual-audio.v2.mp4
• /home/joseph/Downloads/AV-Match-out/IMG_9883.avmatch.dual-audio.mp4

Sync diagnostics for current v3 output:

• /home/joseph/Downloads/AV-Match-out-v3/sync_report.json

Transcription smoke-test artifacts currently exist at:

• /home/joseph/Downloads/AV-Match-transcribe-test/sample30.wav
• /home/joseph/Downloads/AV-Match-transcribe-test/sample30.txt
• /home/joseph/Downloads/AV-Match-transcribe-test/sample30.srt
• /home/joseph/Downloads/AV-Match-transcribe-test/sample30.vtt
• /home/joseph/Downloads/AV-Match-transcribe-test/sample30.tsv
• /home/joseph/Downloads/AV-Match-transcribe-test/sample30.json

Note:

• a fully renamed final deliverable does not exist yet
• a full finished transcript export does not currently exist yet
• /home/joseph/Downloads/AV-Match-full-transcript/ was created but no completed transcript artifacts are present there right now

Transcript-derived title

Current inferred lesson title from transcript:

• The Lordship of Christ

This is transcript-inferred, not yet confirmed from title card / bulletin metadata.

Naming convention for final output pass

Final basename target shape:

• YYYYMMDD-CODE-Speaker-Title

Examples:

• 20260517-SWS-Tim.Kernan-The.Lordship.of.Christ.mp4
• 20260517-SWS-The.Lordship.of.Christ.mp4

Current series code definitions:

• SWS = Sunday Weekly Sermon
• MMW = Men's Midweek
• WMW = Women's Midweek
• CMW = Congregational Midweek
• CD = Campus Devo

Recommended runtime naming rules:

1. infer date from source metadata or source file
2. require or provide series code
3. infer title from transcript
4. optionally provide speaker
5. if speaker missing, omit it cleanly

Recommended CLI additions later:

• --series SWS
• --speaker "Tim Kernan"
• --title "The Lordship of Christ"
• auto-slugify to dotted title tokens

Whisper usage

Installed CLI on this host:

• /home/joseph/.local/bin/whisper

Example:

whisper \
  --model turbo \
  --device cpu \
  --language en \
  --task transcribe \
  --output_dir /path/to/out \
  --output_format all \
  /path/to/audio_or_video_file

Notes

This is intentionally the AV Processor lane only; kept separate from Brain/Odoo/Nextcloud audit work.