Camera sources drop out at random in OBS and never recover, while the same players stay healthy in browser talkback.
Measured on the live server, not inferred: every OBS plugin subscriber accumulated 862-1099 nackMisses (retransmit requests for packets the SFU had already aged out — unrecoverable loss) and sat at plis == 2 for a multi-hour session, against 0 and adaptive PLI for a browser subscriber in the same room. The pinned client-sdk-cpp (1.10.1) exposes no PLI/keyframe-request API, so a decoder that lost a frame that way had no way to resync for the rest of the show.
Adds StallWatchdog — a pure, fake-clock-testable class that decides when a video subscription has gone too long (2000ms) without a decoded frame reaching OBS, then forces a keyframe by toggling the subscription. Exponential backoff 2s -> 30s, reset on actual recovery.
Verification
Full cmake --build + ctest, all suites green; 3 new headless StallWatchdog tests (fires after threshold, does not fire when muted/disabled, backs off) using a fake clock — 124/124 checks.
ThreadSanitizer warning count unchanged against a pre-change baseline (113 both ways), none tracing into the new code.
Caught a real bug in development: onFrameDelivered() reset the backoff duration but not next_attempt_allowed_, which would have throttled a just-recovered stream against a stale backoff.
What is NOT verified
The setEnabled(false)/setEnabled(true) toggle has never run against a real SFU — there was no live room available. The logic risk is low (it wraps two SDK calls already used elsewhere) but the mechanism that matters is untested. This wants a deliberate live check: force a drop on one source and watch it come back.
Correction to an earlier hypothesis setVideoQuality(HIGH) only bounds spatial/simulcast layer selection, NOT temporal. Congestion-driven frame shedding is the SFU's job regardless, so the HIGH pin is not the cause of the packet-loss symptom and is most likely an inert no-op now that publishing is L1T3. Left unchanged.
Camera sources drop out at random in OBS and never recover, while the same players stay healthy in browser talkback.
**Measured on the live server**, not inferred: every OBS plugin subscriber accumulated 862-1099 `nackMisses` (retransmit requests for packets the SFU had already aged out — unrecoverable loss) and sat at `plis == 2` for a multi-hour session, against 0 and adaptive PLI for a browser subscriber in the same room. The pinned client-sdk-cpp (1.10.1) exposes no PLI/keyframe-request API, so a decoder that lost a frame that way had no way to resync for the rest of the show.
Adds `StallWatchdog` — a pure, fake-clock-testable class that decides when a video subscription has gone too long (2000ms) without a decoded frame reaching OBS, then forces a keyframe by toggling the subscription. Exponential backoff 2s -> 30s, reset on actual recovery.
**Verification**
- Full `cmake --build` + `ctest`, all suites green; 3 new headless `StallWatchdog` tests (fires after threshold, does not fire when muted/disabled, backs off) using a fake clock — 124/124 checks.
- ThreadSanitizer warning count unchanged against a pre-change baseline (113 both ways), none tracing into the new code.
- Caught a real bug in development: `onFrameDelivered()` reset the backoff duration but not `next_attempt_allowed_`, which would have throttled a just-recovered stream against a stale backoff.
**What is NOT verified**
The `setEnabled(false)`/`setEnabled(true)` toggle has never run against a real SFU — there was no live room available. The logic risk is low (it wraps two SDK calls already used elsewhere) but the mechanism that matters is untested. This wants a deliberate live check: force a drop on one source and watch it come back.
**Correction to an earlier hypothesis**
`setVideoQuality(HIGH)` only bounds spatial/simulcast layer selection, NOT temporal. Congestion-driven frame shedding is the SFU's job regardless, so the HIGH pin is not the cause of the packet-loss symptom and is most likely an inert no-op now that publishing is `L1T3`. Left unchanged.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Camera sources in the OBS plugin were dropping out at random and never
recovering, while the same players stayed healthy in browser talkback.
Measured on the live server: every OBS plugin subscriber racked up
862-1099 nackMisses (retransmit requests for packets the SFU had already
aged out of its send buffer -- unrecoverable loss) and sat at plis == 2
for a multi-hour session, versus 0/adaptive-PLI for a browser subscriber
in the same room. The pinned client-sdk-cpp (1.10.1) exposes no
PLI/keyframe-request API at all, so a decoder that lost a frame that way
had no way to resync for the rest of the show.
Add a stall-recovery watchdog: StallWatchdog (session_types.h/.cpp) is a
pure, fake-clock-testable class that decides when a video subscription
has gone too long (2000ms, kStallRecoveryTimeout) without a decoded
frame reaching OBS. LiveKitSession::Impl polls it from a dedicated
thread and, through the existing command queue (never touching the SDK
off the worker thread), toggles the publication's setEnabled(false)/
setEnabled(true) -- the one lever this SDK exposes that makes the SFU
restart delivery, and a restart always begins with a keyframe.
Repeated attempts against the same stall back off exponentially
(2s/4s/8s/16s/30s-capped, mirroring the shape of the adapter's own
reconnect backoff) so a genuinely gone publisher is retried on a bounded
cadence instead of hammered every 2 seconds. A muted, disabled or
unsubscribed track, an audio-only source, or a disconnected session
never arms the watchdog: new onTrackMuted/onTrackUnmuted handlers
suspend and resume its clock, always re-baselining from "now" rather
than a stale timestamp, so un-muting after a long legitimate camera-off
period cannot read as a multi-minute stall. Each attempt, and eventual
recovery, is logged through a new DiagnosticHandler that the OBS adapter
maps onto obs_log at the same severities the file already uses for other
notable events.
Also investigated (not changed): the setVideoQuality(HIGH) pin added for
an earlier simulcast-resize bug. The SDK's own docs and the FFI binary's
wire-protocol strings show setVideoQuality only bounds spatial/simulcast
layer selection (UpdateTrackSettings.quality), never temporal layers or
frame rate, which the SFU's own congestion control governs independently
-- so this pin is unlikely to be the cause of the packet-loss symptom,
and is probably an inert no-op now that publishing is pinned server-side
to a single spatial layer (L1T3). Left in place since that is not fully
unambiguous from the SDK alone. Full writeup in
.stall-recovery-report.md (untracked, not part of this commit).
Adds 3 new headless StallWatchdog tests (fires-after-threshold, does-
not-fire-when-muted/disabled, backs-off-rather-than-loops) to
test_session.cpp; caught a real bug during development where
onFrameDelivered() reset the backoff duration but not the
next-attempt-allowed timestamp, throttling a just-recovered stream
against its own stale backoff. Built and all 6 CTest suites pass
(124/124 checks in test_session); also verified clean under
ThreadSanitizer with warning counts unchanged from the pre-change
baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jknapp
merged commit 969a500dfd into main2026-09-21 16:43:21 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Camera sources drop out at random in OBS and never recover, while the same players stay healthy in browser talkback.
Measured on the live server, not inferred: every OBS plugin subscriber accumulated 862-1099
nackMisses(retransmit requests for packets the SFU had already aged out — unrecoverable loss) and sat atplis == 2for a multi-hour session, against 0 and adaptive PLI for a browser subscriber in the same room. The pinned client-sdk-cpp (1.10.1) exposes no PLI/keyframe-request API, so a decoder that lost a frame that way had no way to resync for the rest of the show.Adds
StallWatchdog— a pure, fake-clock-testable class that decides when a video subscription has gone too long (2000ms) without a decoded frame reaching OBS, then forces a keyframe by toggling the subscription. Exponential backoff 2s -> 30s, reset on actual recovery.Verification
cmake --build+ctest, all suites green; 3 new headlessStallWatchdogtests (fires after threshold, does not fire when muted/disabled, backs off) using a fake clock — 124/124 checks.onFrameDelivered()reset the backoff duration but notnext_attempt_allowed_, which would have throttled a just-recovered stream against a stale backoff.What is NOT verified
The
setEnabled(false)/setEnabled(true)toggle has never run against a real SFU — there was no live room available. The logic risk is low (it wraps two SDK calls already used elsewhere) but the mechanism that matters is untested. This wants a deliberate live check: force a drop on one source and watch it come back.Correction to an earlier hypothesis
setVideoQuality(HIGH)only bounds spatial/simulcast layer selection, NOT temporal. Congestion-driven frame shedding is the SFU's job regardless, so the HIGH pin is not the cause of the packet-loss symptom and is most likely an inert no-op now that publishing isL1T3. Left unchanged.🤖 Generated with Claude Code