fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog
Build / macOS (macos-latest) (push) Successful in 1m7s
Build / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / macOS (macos-latest) (pull_request) Successful in 53s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m19s
Build / Windows (windows-latest) (push) Failing after 3m25s
Build / Windows (windows-latest) (pull_request) Failing after 3m0s
Build / macOS (macos-latest) (push) Successful in 1m7s
Build / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / macOS (macos-latest) (pull_request) Successful in 53s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m19s
Build / Windows (windows-latest) (push) Failing after 3m25s
Build / Windows (windows-latest) (pull_request) Failing after 3m0s
Camera sources in the OBS plugin were dropping out at random and never recovering, while the same players stayed healthy in browser talkback. Measured on the live server: every OBS plugin subscriber racked up 862-1099 nackMisses (retransmit requests for packets the SFU had already aged out of its send buffer -- unrecoverable loss) and sat at plis == 2 for a multi-hour session, versus 0/adaptive-PLI for a browser subscriber in the same room. The pinned client-sdk-cpp (1.10.1) exposes no PLI/keyframe-request API at all, so a decoder that lost a frame that way had no way to resync for the rest of the show. Add a stall-recovery watchdog: StallWatchdog (session_types.h/.cpp) is a pure, fake-clock-testable class that decides when a video subscription has gone too long (2000ms, kStallRecoveryTimeout) without a decoded frame reaching OBS. LiveKitSession::Impl polls it from a dedicated thread and, through the existing command queue (never touching the SDK off the worker thread), toggles the publication's setEnabled(false)/ setEnabled(true) -- the one lever this SDK exposes that makes the SFU restart delivery, and a restart always begins with a keyframe. Repeated attempts against the same stall back off exponentially (2s/4s/8s/16s/30s-capped, mirroring the shape of the adapter's own reconnect backoff) so a genuinely gone publisher is retried on a bounded cadence instead of hammered every 2 seconds. A muted, disabled or unsubscribed track, an audio-only source, or a disconnected session never arms the watchdog: new onTrackMuted/onTrackUnmuted handlers suspend and resume its clock, always re-baselining from "now" rather than a stale timestamp, so un-muting after a long legitimate camera-off period cannot read as a multi-minute stall. Each attempt, and eventual recovery, is logged through a new DiagnosticHandler that the OBS adapter maps onto obs_log at the same severities the file already uses for other notable events. Also investigated (not changed): the setVideoQuality(HIGH) pin added for an earlier simulcast-resize bug. The SDK's own docs and the FFI binary's wire-protocol strings show setVideoQuality only bounds spatial/simulcast layer selection (UpdateTrackSettings.quality), never temporal layers or frame rate, which the SFU's own congestion control governs independently -- so this pin is unlikely to be the cause of the packet-loss symptom, and is probably an inert no-op now that publishing is pinned server-side to a single spatial layer (L1T3). Left in place since that is not fully unambiguous from the SDK alone. Full writeup in .stall-recovery-report.md (untracked, not part of this commit). Adds 3 new headless StallWatchdog tests (fires-after-threshold, does- not-fire-when-muted/disabled, backs-off-rather-than-loops) to test_session.cpp; caught a real bug during development where onFrameDelivered() reset the backoff duration but not the next-attempt-allowed timestamp, throttling a just-recovered stream against its own stale backoff. Built and all 6 CTest suites pass (124/124 checks in test_session); also verified clean under ThreadSanitizer with warning counts unchanged from the pre-change baseline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,6 +11,7 @@ You may obtain a copy of the License at
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <chrono>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
@@ -54,6 +55,22 @@ struct SessionConfig {
|
||||
int connect_timeout_ms = 15000;
|
||||
};
|
||||
|
||||
/// How long the stall-recovery watchdog waits for a decoded video frame
|
||||
/// before treating the subscription as stalled and forcing a fresh
|
||||
/// keyframe (see StallWatchdog's comment in session_types.h for why this
|
||||
/// exists -- unrecoverable packet loss with no PLI/keyframe-request API in
|
||||
/// the pinned SDK). Long enough that ordinary jitter never trips it (a
|
||||
/// healthy 30fps subscription delivers a frame at least every ~33ms);
|
||||
/// short enough a director barely has time to notice before it recovers.
|
||||
constexpr std::chrono::milliseconds kStallRecoveryTimeout{2000};
|
||||
|
||||
/// Ceiling for the backoff between repeated recovery attempts against the
|
||||
/// SAME stall. Starts at kStallRecoveryTimeout and doubles each attempt, so
|
||||
/// a genuinely gone publisher (crashed encoder, dead upstream network) is
|
||||
/// retried every 2s, 4s, 8s, ... 30s rather than hammered every 2 seconds
|
||||
/// for the rest of the show.
|
||||
constexpr std::chrono::milliseconds kStallRecoveryMaxBackoff{30000};
|
||||
|
||||
/// Wraps livekit::Room for exactly one subscribed slot.
|
||||
///
|
||||
/// Threading contract, which the OBS adapter depends on:
|
||||
@@ -67,6 +84,11 @@ struct SessionConfig {
|
||||
/// call.
|
||||
/// - The state handler is invoked from whichever thread observed the
|
||||
/// change. It must not block and must not call back into this object.
|
||||
/// - The diagnostic handler (currently just the stall-recovery watchdog,
|
||||
/// see kStallRecoveryTimeout below) may be invoked from the internal
|
||||
/// command-queue worker thread or a video reader thread. Same rules as
|
||||
/// the state handler: must not block, must not call back into this
|
||||
/// object.
|
||||
/// - All handlers must be installed before connect(); they are not
|
||||
/// synchronised against a running session.
|
||||
class LiveKitSession {
|
||||
@@ -80,6 +102,7 @@ public:
|
||||
void setVideoHandler(VideoFrameHandler handler);
|
||||
void setAudioHandler(AudioFrameHandler handler);
|
||||
void setStateHandler(SessionStateHandler handler);
|
||||
void setDiagnosticHandler(DiagnosticHandler handler);
|
||||
|
||||
/// Connect and start subscribing. Returns true once the room is up; the
|
||||
/// selected slot's tracks may still arrive later (or not at all, if the
|
||||
|
||||
Reference in New Issue
Block a user