Add CLI progress reporting: Running detect/trim used to sit completely silent through the whole pipeline, including the ASR step -- the slowest part on CPU and the one most likely to actually run -- with no indication anything was happening. Found via a direct user complaint: 'it would be nice if the cli tool emitted some sort of progress vs sitting there blankly for unknown time.' detect.detect() now takes an optional progress callback, called with a short message at each major stage (fetching audio, official SRT, ASR, reference correlation, sibling-episode fetching); defaults to a no-op so existing callers and tests are unaffected. cli.py wires this to a single, in-place-overwriting stderr line (so it never pollutes piped/ redirected stdout) for detect, sweep, and random-sample. Also documents the pre-2015 outro-wording fix (markers.json, already committed separately) in docs/DESIGN.md's Notable fixes, with a screenshot showing the actual speech-over-music overlap in Audacity that motivated it.
43 KiB
Design
Detailed technical background for hpr-intro-stripper. For usage, see the
top-level README.md.
Goals & non-goals
Goals:
- Given an HPR episode number (or a range), produce the "inner" audio — the host's originally submitted recording, with HPR's own added intro and outro removed — individually or in batch.
- Work unattended against the live HPR archive: fetch whatever metadata/audio/transcript it needs itself, rather than assuming a pre-populated local mirror.
- Run on modest hardware. No step requires a GPU or a specific speech-
recognition setup. Dependency weight is comparable to a repo that says "install these
packages from your distro" —
ffmpeg, Python 3,numpy,sqlite3,awk, optionally one CPU ASR tool — not a PyTorch/CUDA stack as a hard requirement. - Be safe by default: never silently produce or redistribute a trimmed derivative of a No-Derivatives-licensed episode; never touch audio the detector isn't confident about; always leave an audit trail (what was cut, by which method, with what confidence) alongside trimmed output.
Non-goals:
- Not a bulk mirror of HPR's audio. No episode audio or bulk transcript corpus lives in this repo — only code, a small marker-phrase library, and a per-episode cut-point cache (gitignored).
- Not a replacement for HPR's own transcription pipeline. Where HPR already publishes a transcript/SRT, this tool uses it rather than re-deriving it.
- Not an attempt to perfectly restore the exact original submitted file byte-for-byte — the goal is the host's spoken content, cleanly bounded, not bit-identical reconstruction.
Why this is harder than "just trim N seconds"
The intro/outro have never been one fixed-length clip across HPR's 18-year history, and the boundary moved more than once:
- Pre-~2015 (roughly episode <1700): no spoken announcement at all — just a shared, unchanging music jingle before the host's own cold open. Confirmed by cross-correlating two episodes' openings sample-for-sample: hpr0500 vs hpr0600 matched at r≥0.99 for exactly 15.0s, then diverged sharply. This era is the one case where "fixed timing" really is trivial, if you know the length.
- ~2015 onward: HPR began auto-generating a spoken per-episode metadata announcement (episode number, date, title, series, flag, summary) read before the host's own greeting. Its length is not fixed — it tracks how much text there is to read — and its exact wording drifted at least twice more. ASR testing across 9 real episodes (2008–2025) measured intro lengths from 13s to 48s depending on title/summary length.
- The outro turns out to be fixed after all — but only once you know which asset is current (see below): a single unchanging 40.0-second clip in the current era, not something to measure per episode, but it has presumably changed at least once already (branding, hosting-sponsor wording) and will again.
So "trim N seconds" is right for the outro (per era) and wrong for the intro (always, by construction) — any workable tool has to detect the intro boundary per-episode, and detect the outro's current fixed length rather than hard-coding one.
What HPR's own pipeline already does
Read directly from repo.anhonesthost.net/HPR/hpr-tools (cloned locally
and read, not guessed):
-
workflow/hpr-get-and-transcribe.bashruns, on each newly submitted raw episode file, before intro/outro are ever added:whisper --output_format srt --model tiny --language en --output_dir "${show_dir}" "${this_media}"This is HPR's own standard: the smallest Whisper model, on the raw submission, as routine janitor tooling — strong precedent for what this project assumes is a reasonable dependency to ask of a contributor's machine. This repo's own ASR fallback matches it exactly (see "ASR dependency strategy" below).
-
workflow/hpr-process_episode.bashassembles the final published SRT by concatenatingintro.srt(template) + the body SRT from the step above (time-shifted by the real rendered intro duration, read viamediainfo) +outro.srt(time-shifted again). The exact cut points are computed server-side and deterministic — they're just not published as a separate field anywhere. -
workflow/intro.srt(vendored intoreferences/intro.srt) is a 5-line template with placeholders for: episode number + date, title, series, flag + license, summary. Nominal total length 27.32s — but this is a default template; the real TTS-rendered audio varies with actual text length (matches the 13–48s range measured by ASR). -
workflow/outro.srt+workflow/outro.flac(vendored intoreferences/) are completely static — no per-episode substitution at all.outro.flacmeasured viaffprobeat exactly 40.0 seconds.theme.flac(the intro music bed, vendored alongside it) measured at 40.52 seconds — the raw bed length available to loop/crop under the TTS speech, not the per-episode intro duration. -
hpr_hub/cms/say.phpbuilds the exact spoken announcement text — thesynopsisfield — by plain string concatenation from theeps,hosts,licenses, andminiseriestables (the same schema the fetchedhpr.sqlgives this tool).hpr-process_episode.bash'screate_tts_summary()then feeds that text verbatim intopiper --model en_US-ljspeech-high.onnxwith no further processing, andgenerate_intro()mixes the resulting speech withtheme.flacadditively (sox -m, a mix, not a sequential splice) — so the real intro audio is theme-plus-speech layered together for most of its length, not theme-then-speech.
Implication: for the current era, the outro is not something to
detect per episode at all — it's a known, fixed-length clip, safe to
match directly against the vendored outro.flac. The intro is the only
part that genuinely requires per-episode detection, though the vendored
theme.flac still gives a useful independent boundary signal (see below).
A documentation/code discrepancy worth noting plainly:
hpr_documentation/branding.md states the intro "will always be 30
seconds long and in some edge cases may be slightly longer." The actual
generation code enforces no such cap (create_tts_summary() has no length
limit; generate_intro()'s sox -m just takes the longer of its two
inputs), and empirical ASR measurements (13–48s) match the code's
behaviour, not the documented policy. Treated here as a sign the docs lag
the implementation in places, not a sign the measurements are wrong.
Data acquisition
Audio and sibling files. Confirmed working endpoint (verified directly, not guessed):
https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hprNNNN/hprNNNN.{mp3,ogg,opus,txt,srt,json}
Episode number zero-padded to 4 digits, directory and filename both
hprNNNN. HPR publishes every episode in mp3, ogg, and opus
simultaneously — format is a configurable preference with fallback
(default order: mp3 → ogg → opus, prioritizing broad recognition and
compatibility over opus's better compression; override with --format on
the CLI or config.AUDIO_FORMAT_PREFERENCE), never hardcoded, since
ffmpeg/ffprobe handle all three natively at no extra dependency cost.
Design rule, not just a bug workaround: never hardcode a storage backend
host. The proxy has been observed redirecting to a DigitalOcean Spaces
bucket (*.digitaloceanspaces.com) — and hitting that bucket directly can
return an AccessDenied XML error even for files the proxy serves fine
(confirmed: hpr0400, hpr1000). HPR's hosting is explicitly designed to
be swappable (see the CCDN architecture doc's own multi-origin,
DNS-round-robin design), so a URL seen once in a redirect or an error
message is an observation, never a contract. This tool only ever
constructs URLs against ccdn.php and follows whatever it redirects to;
it never remembers or reconstructs a backend host itself. AccessDenied,
403, 404, and a 200-with-empty-body are all treated as the same outcome —
"not available right now, fall back to the next signal" — never
special-cased per status code.
No public bulk endpoint exists.
HPR/hpr_documentation/ccdn/ccdn_architecture.md documents HPR's CCDN as
a private origin-to-mirror-node design: volunteer-run edge nodes rsync
over SSH from a private origin, not open to the public, with directory
browsing explicitly disabled. The practical approach is a polite,
resumable, per-episode HTTP fetch loop — not an attempt to find or build a
bulk-sync path. Concretely: every request to HPR's servers goes through a
single gate (fetch._wait_for_politeness_gap()) enforcing a minimum
config.REQUEST_DELAY_SECONDS (default 0.5s) between requests, regardless
of which function issued them, and fetch_audio/fetch_official_srt
skip the network entirely once a file is cached. Retry-with-backoff on a
transient failure is not implemented — a failed request is currently
treated the same as "not available," falling through to the next
detection signal rather than being retried.
Official transcripts/SRT exist, but coverage is patchy — confirmed by direct testing, not assumed. Even episodes from 2024–2025 sometimes return a genuine transcript, sometimes a 200-with-empty-body placeholder, and older ones sometimes 404 or 403. The tool always fetches and checks actual byte content, never infers availability from the episode page or from era alone.
Metadata. Fetched from the canonical source —
https://www.hackerpublicradio.org/hpr.sql — the exact URL
hpr_generator's own utils/lib_utils.sh (download_hpr_sql) pulls
from, converted to SQLite with the same vendored mysql2sqlite AWK script
HPR's own update-hpr.sh uses. This repo ships no snapshot and assumes no
pre-existing local copy; a bootstrap step downloads the current dump and
refreshes it periodically, so every run reflects the live database.
Cut-point detection strategy
Four layered signals, tried in order, cheapest/most-certain first. Every episode gets whichever signal actually works for it, plus a record of which one was used.
-
Official SRT, when present and non-empty. Gives the boundary for free — but a present SRT that fails to yield an answer for one side (e.g. only the intro marker matches, not the outro) doesn't suppress the ASR-anchor fallback for the side it couldn't resolve; each side is tracked and filled independently. Every accepted result is also sanity-checked against whichever reference-correlation/metadata signal is independently available — a disagreement surfaces as a flag on the result rather than being silently resolved in either direction (see
DISAGREEMENT_THRESHOLD_SECONDSindetect.py). -
Metadata-duration arithmetic, as a cross-check for the modern era. The
eps.durationfield appears to represent the raw submitted (pre-intro/outro) length — validated against three real test episodes (hpr3700: 882s vs. 880.98s measured; hpr3900: 704s vs. 700.34s; hpr4300: 1105s vs. 1101.7s), all within a few seconds. Does not hold for older episodes (hpr2100: 280s vs. 333.23s measured, a 53s gap) — likely because this exact invariant inhpr-process_episode.bashis itself a newer part of the pipeline. Used only as corroboration for recent episodes, never as sole truth for old ones. -
For the intro boundary specifically: synopsis-alignment (
markers.find_intro_end_via_synopsis), the primary method, with fixed-phrase marker matching (hpr_stripper/markers.json) as the fallback. Extracts only the first ~120s and last ~70s viaffmpeg— never the whole episode — transcribes with Whisper, then sequence- aligns the transcript (difflib.SequenceMatcher, word level) against the deterministically-known expected text for this exact episode (metadata.synopsis_text(), thesay.phpreproduction from "What HPR's own pipeline already does"), rather than matching against a fixed phrase list. This tolerates scattered word-level ASR errors throughout the transcript, not just at phrase boundaries a marker happens to cover, and requires a confirmed match near the very start of the transcript before trusting any boundary (the same anchoring principle the audio-correlation signal below uses) — a transcript garbled specifically near its opening correctly returns no result rather than a guess built from a coincidental match found deeper in (seetests/test_synopsis_alignment.pyfor both outcomes side by side).Once the per-episode announcement is confirmed, the boundary extends through whatever
markers.jsonrecognizes immediately afterwards — a sponsor ad-read has no deterministic per-episode text of its own to align against, so the marker list remains the right tool there, not for the announcement itself.hpr_stripper/markers.py's fixed-phrase matching is the fallback for episodes with no reconstructable expected text. It splits markers into "strong" (exact-substring, or — for the single long, distinctive station-name marker specifically — a validated high-threshold fuzzy match; see its docstring for the exact thresholds and why they're safe) and "greeting" (can extend an already-established run, but never start one on its own, since hosts legitimately greet listeners with "welcome to Hacker Public Radio" organically too). -
Cross-correlation, used three ways:
- Against the vendored reference assets (
references/theme.flac,references/outro.flac) — the cleanest signal for the current era.outro.flacis the complete rendered outro; correlating it directly against an episode's tail is simpler and more universal than comparing to a sibling episode, and self-detects when HPR changes the asset (correlation quality drops across the board, signalling "go vendor the new one").theme.flacis only the music bed, additively mixed with speech in the real intro — correlating it still works as a boundary detector (it stays high wherever the theme is actually present underneath the speech, and drops sharply once the bed ends and the host's own raw recording begins), and gives a useful floor: if the theme-correlation run is longer than what the ASR anchor found, the theme bed is still playing past the recognized speech (short-text episodes), and the floor wins. - Against a sibling episode, as the primary method for the intro in the pre-~2015 fixed-jingle era (no reference asset exists for that era yet). Checks a small radius of nearby episodes (±2) and takes the best match found, not a median or consensus across them — most immediate neighbours often don't happen to share this episode's specific jingle variant at all, so requiring agreement would discard a genuine match outvoted by neighbours that simply aren't evidence of anything (e.g. hpr0400 correlates at ~0 against its three closest neighbours but cleanly at 29.25s against the fourth). A single clean, sustained high-correlation run is strong evidence on its own; only the case where nothing nearby correlates at all is flagged inconclusive.
- As an anomaly flag: when an episode's measured match length against its neighbours is a big outlier, flag it rather than trust it. This is exactly what caught Linux Inlaws correctly in testing — hpr3009 vs. a normal sibling (hpr3010) matched for only 2.5s before diverging. Their extra theme music plays after the standard HPR greeting, as their own content, and must never be touched.
- Against the vendored reference assets (
Nothing in this pipeline ever guesses: an episode with no usable signal at
all is flagged for manual review (intro_flag / outro_flag in the
result), never given a plausible-looking wrong answer.
ASR dependency strategy
Decided: the shipped default is plain openai-whisper --model tiny,
not whisper.cpp. Rationale: consistency with tooling already in
people's workflows — this is literally what HPR's own
hpr-get-and-transcribe.bash already runs — and it's what the original
tested prototype already used. whisper.cpp remains a reasonable
pluggable alternative for anyone who wants an even smaller footprint
(hpr_stripper/asr.py is written as a one-engine-at-a-time dispatch
specifically so a second backend can be added without touching callers),
but it is not the default.
Real CPU benchmarks (openai-whisper, CPU-only, capped to 4 cores via
taskset, on a 90-second clip, on a modern desktop CPU — AMD Ryzen 9
3950X):
| Model | Wall time |
|---|---|
| tiny | 11.7s |
| base | 23.0s |
| small | 57.0s |
A 2016 dual-core laptop would likely run 2–4× slower per the cores-and-
clock gap — plausibly 30–50s for tiny — but that's for a 90-second clip,
never the whole episode: this design only ever transcribes short head/tail
clips, when ASR is needed at all (i.e. when neither the official SRT nor
the metadata-arithmetic cross-check gave a usable answer).
License handling
The HPR database has a real licenses table, not one blanket license —
confirmed directly from hpr.db:
| License | Episodes |
|---|---|
| CC-BY-SA | 3,923 |
| CC-BY-NC-SA | 772 |
| CC-0 | 33 |
| CC-BY | 8 |
| CC-BY-NC-ND | 6 |
| CC-BY-NC | 5 |
| CC-BY-ND | (listed, 0 currently assigned) |
| MIT | (listed) |
The two No-Derivatives variants (CC-BY-ND, CC-BY-NC-ND) legally forbid
producing or redistributing a modified/derivative version of the work —
which a trimmed audio file is — without separate rightsholder permission.
6 episodes currently carry CC-BY-NC-ND.
Every pipeline run reads the episode's license from metadata and
carries it through to the result and the trimmed output's sidecar file.
By default, hpr_stripper/trim.py's should_trim() refuses ND-licensed
episodes (--force overrides this, not recommended) — detection can still
run and show the cut points, but the trim step won't silently produce
output for distribution.
This goes further than just the 6 ND episodes. Per
hpr_documentation/branding.md, on the "Who?" segment of the intro: "We
are required by the CC-BY-SA... license to credit our hosts, so we do."
The intro is HPR's actual mechanism for satisfying CC-BY-SA host
attribution — so stripping it removes a legally required notice for
every CC-BY-SA/CC-BY episode, not only the ND-licensed ones. That's why
trim.write_sidecar() always writes a small JSON file alongside trimmed
output recording the host, episode number, date, license, and original
URL, regardless of which license applies — trimmed audio stays
self-documenting if it's ever shared separately from this tool.
Historical/ethos note: HPR dropped the CC-BY-NC license as an option in a
2025 policy change (see hpr_documentation/policies.md) and clarified
that contributions default to CC-BY-SA 4.0 unless otherwise stated — the
772 CC-BY-NC-SA / 5 CC-BY-NC episodes are a legacy category, not one
expected to keep growing.
Repo structure & pipeline stages
hpr-intro-stripper/
├── README.md # minimal: usage, requirements, brief how-it-works
├── docs/
│ └── DESIGN.md # this file
├── references/ # vendored from HPR's own hpr-tools repo (CC0/CC-BY-SA, not this repo's code license)
│ ├── theme.flac
│ ├── outro.flac
│ ├── intro.srt
│ ├── outro.srt
│ ├── mysql2sqlite # the same AWK converter HPR's own tooling uses
│ └── LICENSE_NOTICE.md
├── hpr_stripper/
│ ├── config.py # the one place to change defaults
│ ├── markers.json # versioned, plain-text boilerplate phrase library
│ ├── markers.py # fuzzy matching against markers.json
│ ├── metadata.py # canonical hpr.sql fetch + sqlite conversion + queries
│ ├── fetch.py # audio/transcript/SRT fetch via ccdn.php only
│ ├── srt.py # minimal SRT parser (stdlib only)
│ ├── audio.py # ffmpeg/ffprobe wrappers, cross-correlation
│ ├── asr.py # pluggable ASR backend dispatch
│ ├── detect.py # the layered detection pipeline
│ ├── ledger.py # resumable per-episode progress tracking
│ ├── trim.py # atomic trim + license gating + sidecar metadata
│ └── cli.py # detect / sweep / random-sample / trim / report
├── tools/
│ └── visualize_clip.sh # renders a waveform/spectrogram for a clip, for inspecting a cut point directly
└── cache/ # gitignored: hpr.db, downloaded audio, transcripts, ledger.db, trimmed output
What lives in git: code, the marker-phrase library, the vendored reference assets (small, clearly licensed), and this documentation.
What never lives in git: episode audio, bulk transcripts, the
metadata database, and trimmed output — all fetched or produced into
cache/ on demand, so cloning the repo stays small regardless of how much
of the archive someone has processed.
Config surface: hpr_stripper/config.py — one file selecting the ASR
backend/model, audio format preference, fetch/clip sizes, and cache
location.
Testing & resumability strategy
Two first-class test modes, not ad hoc spot checks:
- Random-sample test (
cli.py random-sample): picks N episodes at random, stratified across the three known eras (pre-2015, ~2016–2020, ~2020+) so a lucky or unlucky sample can't hide an era-specific regression. Fast, for day-to-day development checks. - Full-corpus sweep test (
cli.py sweep): runs detection (not trimming) across every episode in the archive. Slow — a real, continuous run of 21 episodes from the hardest category (pre-2015, no official transcript, full ASR + sibling-correlation fallback) averaged 32.1 seconds/episode on the hardware this was developed on; scaled to the full ~4,750-episode archive that's on the order of a day or two of continuous wall-clock time, not weeks, though a weaker CPU or the politeness delay (see "Data acquisition") would push it higher — and explicitly designed to be interrupted. - Interrupt/resume was smoke-tested for real, not just unit-tested:
a
sweepwas started, sent a realSIGINTmid-episode, resumed (and confirmed it skipped the already-completed episodes rather than reprocessing them), interrupted again, and resumed to completion. No lost work, no stray subprocesses left running after either interrupt, no corrupted ledger state.
Resumability is a hard requirement, because a half-hour-plus batch run will get interrupted (Ctrl+C, laptop sleep, SSH drop):
- Idempotent, per-episode units of work. Every stage treats one
episode as the atomic unit (
ledger.is_done()checked before any work starts). - Atomic writes, never partial files.
fetch.pyandtrim.pyboth write to a.tmppath andos.replace()into place only on success. - A persistent progress ledger (
hpr_stripper/ledger.py, SQLite): one durable, committed row per episode, written immediately after that episode finishes. Ctrl+C at minute 29 of a 30-minutesweepcosts at most the one in-flight episode. - Safe to re-run after changing settings. The ledger stamps each row
with the marker-library version and ASR engine/model that produced it
(
config.MARKERS_VERSION);ledger.is_done()only returns true if those still match the current settings, so bumping the marker version or switching ASR models re-processes exactly the episodes affected, not everything.
Fit with HPR's own ethos & principles
Read directly from hpr_documentation (branding.md, policies.md) and
from the generation code itself, not inferred:
- The intro's spoken metadata exists partly for accessibility, not just
branding.
branding.md: "We started to do it because some of our Visually Impaired users appreciated knowing what show is playing." Stripping it removes a real accessibility aid for listeners who rely on audio cues rather than a visual player UI. This tool is an optional alternate listening format for people who've chosen to opt out of that cue — never a claim that HPR's own intro design is a defect to fix. - The intro is also how HPR satisfies its CC-BY-SA attribution requirement — see "License handling" above.
- A documentation/code discrepancy, flagged rather than smoothed over — see "What HPR's own pipeline already does" above (the "30 seconds" claim vs. the actual uncapped generation code).
- The exact spoken text is fully reconstructable from the database —
hpr_hub/cms/say.php's plain string concatenation, reproduced inmetadata.synopsis_text()and used directly by the primary detection path (see "Cut-point detection strategy"). An audio-level counterpart — resynthesizing the expected speech with the same Piper voice (en_US-ljspeech-high.onnx) and correlating it against the real episode, removing the need to run ASR at all — remains open; see "Open items". - The project is careful, consensus-driven, and averse to touching
canonical identifiers.
policies.md: "The janitors really do not like to move shows around once they are posted" — doing so breaks RSS readers and CCDN sync. This tool stays strictly read-only against the official archive: it never proposes renaming, renumbering, or otherwise modifying HPR's own canonical releases, and names its own output so it can never be confused with the original (hprNNNN. trimmed.ext, never literallyhprNNNN.mp3). - Harm is handled by response episodes and janitor review, not
unilateral editing.
policies.md: the remedy for a problematic show is a response episode or raising it with the janitors — never silently altering what's published. This tool's framing matches that norm. - General ethos match. The CCDN architecture doc's explicit
preference for minimal, boring, FLOSS, Debian-stable tooling over
frameworks ("The system intentionally avoids: Databases, PHP, Python
web applications, Kubernetes, Redis...") lines up with the
dependency-weight goal here —
ffmpeg+ Python stdlib +numpy, no heavy framework, ASR optional.
Edge cases & known limitations
- Shows with their own bonus intro music (Linux Inlaws and similar). Their extra theme plays after the standard HPR greeting — it's the hosts' own content, not station boilerplate, and must never be stripped. The cross-correlation anomaly check guards against this (see "Cut-point detection strategy"); phrase/synopsis matching independently stops at the right place anyway, since both only recognize station-specific content.
- Episodes with no usable signal at all are flagged for manual review,
never guessed. Two distinct cases: an organic host cold-open that
happens to echo the station's greeting pattern without any real TTS
announcement present (hpr0500, see
markers.py's docstring), and a transcript too garbled near its opening for synopsis-alignment to anchor confidently (hpr1892, seetests/test_synopsis_alignment.py). Both are accepted limitations, not bugs: some transcripts are too degraded for any automated method to safely resolve. markers.jsonstill needs maintaining for two things synopsis-alignment can't cover on its own: sponsor ad-read scripts (a fixed, reused script per sponsor — HPR names only three in total, anhonesthost.com, Internet Archive, and rsync.net; an rsync.net script hasn't been observed yet, so it isn't guessed at) and the outro boilerplate, neither of which has per-episode deterministic text to align against. The TTS announcement itself no longer needs marker updates for ordinary wording drift — synopsis-alignment handles that directly.- Official SRT segment timing isn't always exact even when present —
the checked-in
intro.srttemplate's timestamps are nominal defaults, not measured per episode (real intro lengths observed: 13–48s against a 27.32s nominal template). - The
duration-field arithmetic cross-check only holds for the modern era — validated within a few seconds for three recent test episodes, off by 53s for an older one. Used only as corroboration, never as sole truth.
Why not just detect music, generally?
The intro genuinely splits into two different sub-problems that want
different tools, not one. Finding where the music bed ends is a real
acoustic question, and audio correlation (against theme.flac) is the
right tool for it. Finding where station-inserted speech ends — the
TTS announcement, and any sponsor ad-read that follows it — is not: a
sponsor ad-read is audio-feature-identical to a host's own speech (same
microphone, same format, no music, nothing a spectral or music-presence
classifier could key on), so only understanding what's being said can
draw that line. This half of the problem is bounded, not open-ended —
HPR's wording has drifted a handful of times in 18 years, and it names
exactly three sponsors in total — which is also why a generic "small gap
after a confirmed match still counts" heuristic isn't used: it would also
swallow a host's own content whenever they start talking immediately
after the TTS block with no gap at all, which is the common case, not the
exception.
Notable fixes along the way
Kept brief, for context on why some of the above looks the way it does —
full detail (specific episodes, before/after measurements) lives in git
history and tests/, not duplicated here:
-
Detection results are cross-checked against independent signals (described above under "Cut-point detection strategy") — this was documented before it was actually implemented, caught by a code-review pass.
-
markers.jsonhas been through 4 wording-era revisions and several ASR-mis-hearing aliases, each found by ametadata_arithmeticdisagreement flag on a real episode duringrandom-sampletesting, confirmed by rendering the clip withtools/visualize_clip.sh. -
fuzzy_contains()once compared a raw (un-normalized) marker string against normalized text, so a marker containing punctuation (e.g.archive.org) could never match — found by a failing test. -
The theme-correlation signal originally used a simple leading contiguous run, which couldn't bridge a masked-then-resumed correlation pattern (TTS speech mixed additively over the theme bed) — replaced by
audio.last_high_correlation_run_seconds(). -
The sponsor-ad-read continuation originally filtered segments by end time instead of start time, silently discarding a real extension.
-
The sibling-correlation fallback originally took a median across nearby episodes rather than the best match found, which silently discarded a genuine match whenever most immediate neighbours simply didn't share this episode's specific jingle variant (found by testing real episode numbers a user reported, not by inspection) — replaced by
_best_sibling_estimate(). -
--formatwasn't exposed on the CLI at all (config-only), and the default order preferredopusfirst; both found by a user actually running the tool. Reordered tomp3, ogg, opusand exposed as--format, and reported timestamps are now rounded to hundredths of a second rather than showing meaningless float precision. -
trim --episode Nwithout a priordetectfor that episode used to silently do nothing at all — no output, no error, exit code 0. Now errors clearly, telling the user to rundetectfirst; a batch trim over a range still treats "not yet detected" as normal and reports a count rather than erroring per-episode. -
leading_match_seconds's old gap-tolerance rule treated any two consecutive borderline-low correlation windows as fatal, even with a long, clean, sustained match starting immediately after them. Found by comparing the same episode's own audio across two different codecs (mp3 vs opus): a minor codec-boundary artifact in the first ~0.75s made the old rule report 0.25s for what was actually 7+ windows of near-perfect correlation right after (confirmed with a full, non-windowed cross-correlation: 0.9979 overall, zero lag). Replaced with the same qualifying-run logiclast_high_correlation_run_secondsalready used (shared via_qualifying_runs()), just taking the first run instead of the last. A real, general robustness fix, found while chasing a separate, still-open question (see "Open items"): whether hpr0900's confirmed-real station jingle has no detectable match because the search radius is too narrow, or a second distinct jingle variant is in circulation that no tested sample happens to share — testing against 17 candidates spanning the whole pre-2015 range, with this fix applied, still found nothing for it. The codec-fragility bug was real and worth fixing on its own merits, but it isn't the explanation for that case. -
_fetch_bytes()treated a transient connection failure (a dropped connection, a DNS hiccup, a bare socket timeout) identically to a definitive 404 — silently, with no retry. Found by a user reporting that an episode'sdetectfell through to a worse, flagged signal on the first run, then resolved cleanly on a forced re-run with no code or data changes — the only explanation being a transient failure on the first run that got permanently written to the ledger. Now retries a couple of times onURLError/TimeoutErrorbefore giving up; an actualHTTPError(the server really did answer, e.g. with a 404) is never retried, since that's real signal, not noise. -
The markers.json note claimed "pre-2015: no spoken announcement at all" — true for the intro, but not the outro: confirmed directly (hpr0900, hpr0300, by ear and by transcript) that pre-2015 episodes have their own spoken outro sign-off, read over a continuing music bed (the same additive-mix pattern as the modern-era intro, not sequential), with its own wording drift the marker set didn't cover — a binref.com/lunarpages.com sponsor script and a plain "thank you for listening" sign-off. This was the real cause of a user-reported bug: an episode's outro fell through to
sibling_xcorr_fallbackand got flagged, blockingtrim, on a fresh clone with every other fix already applied, because the outro side simply had no text-marker answer available at all. Fixed by adding both phrases tooutro_markers.
Open items
-
Audio-level zero-ASR intro alignment. The text-level half is built and is the primary intro-boundary method (
find_intro_end_via_synopsis— see "Cut-point detection strategy"). Still open: resynthesizing the expected speech with the same Piper voice (en_US-ljspeech-high.onnx) and correlating that against the real episode, which would remove the need to run ASR at all for episodes this already handles well. A "remove a dependency" improvement, not a correctness fix — lower priority than open detection gaps. -
Sidecar metadata format — currently a JSON file alongside each trimmed output (
trim.write_sidecar()); ID3 tags are a reasonable addition if trimmed files are likely to circulate without their sidecar. -
The pre-2015 era mixes a shared station jingle with genuine episode-specific content that happens to also be music — "no match found" means two different things, and the tool must be able to tell them apart. hpr0900 opens with a real, ear-confirmed HPR jingle. hpr0949 also opens with ~30s of non-speech content, but for a completely different reason: it's "the first CCHits.net yearly round-up show," an episode literally about playing Creative Commons- licensed music tracks as content — the host's own chosen music, not station boilerplate, confirmed by ear as "other music, not the HPR jingle." Flagging both as inconclusive would be correct for hpr0949 (nothing to strip, full stop) but a real gap for hpr0900 (station content we're failing to match) — distinguishing the two needs a real reference-variant library, not just better aggregation.
A first vendored variant now exists:
references/ pre2015_jingle_variant_b.flac, a clean 32s clip extracted from hpr0900. Getting here took finding and fixing two real bugs, not just adding the clip:-
last_high_correlation_run_seconds'sleading_anchor_windowsdefault (2) was too tight — found directly on hpr0908 vs this reference: near-zero noise for 3 windows (a tiny leading-silence/ trim difference between the files), then 0.99+ for 37 consecutive windows, an obvious real match the old default rejected as starting "too late to trust." Widened to 4, matchingleading_match_seconds's own default (fixed earlier for the same reason). This is also why an exhaustive scan of all 1,699 locally-available pre-2015 episodes against the unfixed function found nothing but hpr0900 itself — the scan wasn't wrong for lack of data, it was wrong because of this bug. -
Even with that fixed, fixed-lag correlation still cannot find every true match. hpr0901 and hpr0902 share hpr0900's jingle — confirmed by ear, and independently by a matching ~34s intro-envelope duration visible across all three in a waveform view — but neither correlates above ~0.3-0.4 against the reference at any single fixed lag or speed ratio (both tested directly). The jingle apparently has internal tempo variation across different episodes' copies, which no single global alignment can follow.
(2) is why chroma-feature DTW matching (
audio.dtw_chroma_match) was added: DTW doesn't assume one global alignment, so it isn't defeated by internal tempo drift, and its warping-path end position gives a real intro_end estimate for the candidate, not just a yes/no. Calibrated on real episodes: confirmed/likely matches (hpr0908, hpr1117, hpr1467, hpr0901, hpr0902) scored normalized cost 0.09-0.17 against the reference; a confirmed non-match (hpr0898, no intro at all) scored 0.94 —config.DTW_MATCH_THRESHOLD(0.35) sits generously between the two. This is the thing a plain chroma classifier (below) can't do: it's matching against a specific reference clip, not asking "does this sound musical in general."detect._check_pre2015_variantsis the main-line check against every vendored variant (fast correlation first, DTW as the per-variant fallback), tried for every pre-2015 episode before falling back to neighbour-matching — not confined to_sibling_fallback's small ±2 neighbour radius. This matters more than it might first appear: the pre-2015 episodes are this tool's core use case, not an edge case — they're the ones most likely to have no official SRT and no other usable signal, so the detection needs to work for any one of them, not just ones lucky enough to have a same-variant neighbour nearby. Confirmed directly: hpr0908, hpr1117, and hpr1467 are nowhere near hpr0900 or each other, so they'd never have been found through_sibling_fallback's radius at all — only through this direct check. (_sibling_fallback's own DTW fallback still exists underneath, for whatever jingle variants aren't vendored yet.)Only one variant is vendored so far; a random 40-episode sample found it in ~5% of episodes (hpr0900, hpr0908, hpr1117, hpr1467), spread across a wide range — consistent with it having been a genuinely common theme, not a one-off, which is promising for vendoring more variants the same way as they're found.
-
-
A reference-free acoustic signal (generic "does this sound like music" classification, not matching against a specific clip) was tried twice and ruled out both times, for the same structural reason. First pass: a chroma_stability / onset-periodicity / spectral-flatness scoring approach (
librosa); see git history for detail.chroma_stabilityreads hpr0900 and hpr0949 as "music-like" over similar ranges, because both genuinely are music — it just can't tell whose. Second pass, after building the reference-variant library above: tried naive chroma cosine similarity (framewise, no DTW) directly against the vendored reference — hpr0899 (not independently confirmed either way) scored 0.993, nearly indistinguishable from the confirmed true match hpr0908's 0.997, while confirmed non-match hpr0898 scored 0.55. Not reliably discriminative. Both failures are the same structural limit already written up in "Why not just detect music, generally?", on the other side of the coin: generic acoustic classification (or a naive framewise similarity) distinguishes tonal from noisy, but not whose content something is, even when compared directly against a reference. Chroma-feature DTW matching against a specific reference clip (above) is a different technique that doesn't have this problem — ruling out generic chroma similarity doesn't rule out DTW, and this distinction is worth keeping straight for future reading. Chromaprint/pyacoustid (Hamming distance over fingerprint hashes) was also tried and discards cleanly (true matches 0.46-0.61 vs non-matches 1.16-15.38) but doesn't tolerate internal tempo variation at all (every true positive it found resolved to lag=0), so it wasn't adopted over DTW for this purpose. -
The intro music bed fades into the host's own talking rather than cutting cleanly, and nothing here actually separates the two — deferred, substantial future work, not attempted. Confirmed directly (hpr0900: host speech starts at 33.14s while the jingle is still audibly present, fading rather than stopping). Every detection method in this tool picks a single cut timestamp; none of them can leave a genuinely clean voice track across a region where music and voice overlap, because there's no single timestamp where music ends and speech begins — the honest trim either keeps a little residual music under the first few words, or risks clipping the start of the host's own speech. The real fix for that is audio source separation (vocal isolation) — a model like Demucs or Spleeter run specifically across the overlap region, keeping the isolated vocal stem and discarding the music stem, rather than picking a single trim point. Not attempted: this is a materially heavier addition than anything else in this tool — a real ML model (PyTorch for Demucs, TensorFlow for Spleeter), GPU strongly recommended for reasonable per-episode runtime, and real processing time even then. That's a direct conflict with this project's "modest dependencies, no mandatory GPU/CUDA stack" stance as a default (see Goals & non-goals), so if it's ever built, it belongs as a strictly optional, explicitly-invoked extra stage — never something
detect/trimreach for automatically — applied only to the narrow fade window itself (a few seconds either side of the chosen cut point), not the whole episode.

