fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog #7

Merged

jknapp merged 1 commits from fix/stall-recovery into main 2026-09-21 16:43:21 +00:00

Conversation 0 Commits 1 Files Changed 6

+573 -13

1 Commits

Author	SHA1	Message	Date
shadowdaoandClaude Sonnet 5	3f2933ae3f	fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog ... Camera sources in the OBS plugin were dropping out at random and never recovering, while the same players stayed healthy in browser talkback. Measured on the live server: every OBS plugin subscriber racked up 862-1099 nackMisses (retransmit requests for packets the SFU had already aged out of its send buffer -- unrecoverable loss) and sat at plis == 2 for a multi-hour session, versus 0/adaptive-PLI for a browser subscriber in the same room. The pinned client-sdk-cpp (1.10.1) exposes no PLI/keyframe-request API at all, so a decoder that lost a frame that way had no way to resync for the rest of the show. Add a stall-recovery watchdog: StallWatchdog (session_types.h/.cpp) is a pure, fake-clock-testable class that decides when a video subscription has gone too long (2000ms, kStallRecoveryTimeout) without a decoded frame reaching OBS. LiveKitSession::Impl polls it from a dedicated thread and, through the existing command queue (never touching the SDK off the worker thread), toggles the publication's setEnabled(false)/ setEnabled(true) -- the one lever this SDK exposes that makes the SFU restart delivery, and a restart always begins with a keyframe. Repeated attempts against the same stall back off exponentially (2s/4s/8s/16s/30s-capped, mirroring the shape of the adapter's own reconnect backoff) so a genuinely gone publisher is retried on a bounded cadence instead of hammered every 2 seconds. A muted, disabled or unsubscribed track, an audio-only source, or a disconnected session never arms the watchdog: new onTrackMuted/onTrackUnmuted handlers suspend and resume its clock, always re-baselining from "now" rather than a stale timestamp, so un-muting after a long legitimate camera-off period cannot read as a multi-minute stall. Each attempt, and eventual recovery, is logged through a new DiagnosticHandler that the OBS adapter maps onto obs_log at the same severities the file already uses for other notable events. Also investigated (not changed): the setVideoQuality(HIGH) pin added for an earlier simulcast-resize bug. The SDK's own docs and the FFI binary's wire-protocol strings show setVideoQuality only bounds spatial/simulcast layer selection (UpdateTrackSettings.quality), never temporal layers or frame rate, which the SFU's own congestion control governs independently -- so this pin is unlikely to be the cause of the packet-loss symptom, and is probably an inert no-op now that publishing is pinned server-side to a single spatial layer (L1T3). Left in place since that is not fully unambiguous from the SDK alone. Full writeup in .stall-recovery-report.md (untracked, not part of this commit). Adds 3 new headless StallWatchdog tests (fires-after-threshold, does- not-fire-when-muted/disabled, backs-off-rather-than-loops) to test_session.cpp; caught a real bug during development where onFrameDelivered() reset the backoff duration but not the next-attempt-allowed timestamp, throttling a just-recovered stream against its own stale backoff. Built and all 6 CTest suites pass (124/124 checks in test_session); also verified clean under ThreadSanitizer with warning counts unchanged from the pre-change baseline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-20 19:04:14 -07:00