fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog
Build / macOS (macos-latest) (push) Successful in 1m7s
Build / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / macOS (macos-latest) (pull_request) Successful in 53s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m19s
Build / Windows (windows-latest) (push) Failing after 3m25s
Build / Windows (windows-latest) (pull_request) Failing after 3m0s

Camera sources in the OBS plugin were dropping out at random and never
recovering, while the same players stayed healthy in browser talkback.
Measured on the live server: every OBS plugin subscriber racked up
862-1099 nackMisses (retransmit requests for packets the SFU had already
aged out of its send buffer -- unrecoverable loss) and sat at plis == 2
for a multi-hour session, versus 0/adaptive-PLI for a browser subscriber
in the same room. The pinned client-sdk-cpp (1.10.1) exposes no
PLI/keyframe-request API at all, so a decoder that lost a frame that way
had no way to resync for the rest of the show.

Add a stall-recovery watchdog: StallWatchdog (session_types.h/.cpp) is a
pure, fake-clock-testable class that decides when a video subscription
has gone too long (2000ms, kStallRecoveryTimeout) without a decoded
frame reaching OBS. LiveKitSession::Impl polls it from a dedicated
thread and, through the existing command queue (never touching the SDK
off the worker thread), toggles the publication's setEnabled(false)/
setEnabled(true) -- the one lever this SDK exposes that makes the SFU
restart delivery, and a restart always begins with a keyframe.

Repeated attempts against the same stall back off exponentially
(2s/4s/8s/16s/30s-capped, mirroring the shape of the adapter's own
reconnect backoff) so a genuinely gone publisher is retried on a bounded
cadence instead of hammered every 2 seconds. A muted, disabled or
unsubscribed track, an audio-only source, or a disconnected session
never arms the watchdog: new onTrackMuted/onTrackUnmuted handlers
suspend and resume its clock, always re-baselining from "now" rather
than a stale timestamp, so un-muting after a long legitimate camera-off
period cannot read as a multi-minute stall. Each attempt, and eventual
recovery, is logged through a new DiagnosticHandler that the OBS adapter
maps onto obs_log at the same severities the file already uses for other
notable events.

Also investigated (not changed): the setVideoQuality(HIGH) pin added for
an earlier simulcast-resize bug. The SDK's own docs and the FFI binary's
wire-protocol strings show setVideoQuality only bounds spatial/simulcast
layer selection (UpdateTrackSettings.quality), never temporal layers or
frame rate, which the SFU's own congestion control governs independently
-- so this pin is unlikely to be the cause of the packet-loss symptom,
and is probably an inert no-op now that publishing is pinned server-side
to a single spatial layer (L1T3). Left in place since that is not fully
unambiguous from the SDK alone. Full writeup in
.stall-recovery-report.md (untracked, not part of this commit).

Adds 3 new headless StallWatchdog tests (fires-after-threshold, does-
not-fire-when-muted/disabled, backs-off-rather-than-loops) to
test_session.cpp; caught a real bug during development where
onFrameDelivered() reset the backoff duration but not the
next-attempt-allowed timestamp, throttling a just-recovered stream
against its own stale backoff. Built and all 6 CTest suites pass
(124/124 checks in test_session); also verified clean under
ThreadSanitizer with warning counts unchanged from the pre-change
baseline.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-20 19:04:14 -07:00
co-authored by Claude Sonnet 5
parent 48a74e8c67
commit 3f2933ae3f
6 changed files with 573 additions and 13 deletions
+23
View File
@@ -11,6 +11,7 @@ You may obtain a copy of the License at
#pragma once
#include <chrono>
#include <memory>
#include <string>
@@ -54,6 +55,22 @@ struct SessionConfig {
int connect_timeout_ms = 15000;
};
/// How long the stall-recovery watchdog waits for a decoded video frame
/// before treating the subscription as stalled and forcing a fresh
/// keyframe (see StallWatchdog's comment in session_types.h for why this
/// exists -- unrecoverable packet loss with no PLI/keyframe-request API in
/// the pinned SDK). Long enough that ordinary jitter never trips it (a
/// healthy 30fps subscription delivers a frame at least every ~33ms);
/// short enough a director barely has time to notice before it recovers.
constexpr std::chrono::milliseconds kStallRecoveryTimeout{2000};
/// Ceiling for the backoff between repeated recovery attempts against the
/// SAME stall. Starts at kStallRecoveryTimeout and doubles each attempt, so
/// a genuinely gone publisher (crashed encoder, dead upstream network) is
/// retried every 2s, 4s, 8s, ... 30s rather than hammered every 2 seconds
/// for the rest of the show.
constexpr std::chrono::milliseconds kStallRecoveryMaxBackoff{30000};
/// Wraps livekit::Room for exactly one subscribed slot.
///
/// Threading contract, which the OBS adapter depends on:
@@ -67,6 +84,11 @@ struct SessionConfig {
/// call.
/// - The state handler is invoked from whichever thread observed the
/// change. It must not block and must not call back into this object.
/// - The diagnostic handler (currently just the stall-recovery watchdog,
/// see kStallRecoveryTimeout below) may be invoked from the internal
/// command-queue worker thread or a video reader thread. Same rules as
/// the state handler: must not block, must not call back into this
/// object.
/// - All handlers must be installed before connect(); they are not
/// synchronised against a running session.
class LiveKitSession {
@@ -80,6 +102,7 @@ public:
void setVideoHandler(VideoFrameHandler handler);
void setAudioHandler(AudioFrameHandler handler);
void setStateHandler(SessionStateHandler handler);
void setDiagnosticHandler(DiagnosticHandler handler);
/// Connect and start subscribing. Returns true once the room is up; the
/// selected slot's tracks may still arrive later (or not at all, if the