Author SHA1 Message Date
shadowdaoandClaude Opus 5 22d50ab7be test(session): derive watchdog instants from t0, not an accumulated local
Build / macOS (macos-latest) (push) Successful in 53s
Build / Linux (ubuntu-24.04) (push) Successful in 1m7s
Release / macOS (macos-latest) (push) Successful in 57s
Release / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / Windows (windows-latest) (push) Successful in 4m6s
Release / Windows (windows-latest) (push) Successful in 3m55s
Release / Create Gitea Release (push) Successful in 20s
Windows CI failed on the stall-recovery watchdog for three attempts. The
first two diagnoses both blamed the backoff arithmetic; the second
produced a byte-identical failure, which was the clue that neither had
found the cause.

Instrumenting the test on the Windows runner settled it with numbers.
Capturing the time point on the callee side of a noinline wrapper showed
the six `w.poll(now)` calls in the capped-backoff loop received:

  Windows      32000, 32000, 32000, 32000, 32000, 32000  (ms after t0)
  Linux/macOS  62000, 92000, 122000, 152000, 182000, 212000

while a checksum of the caller's own arguments in the same loop summed to
822000 -- exactly the correct series. The caller's value was right; the
value that crossed the call boundary was not. MSVC 19.44 x64 Release
hoists the argument of the second poll() out of the fixed-stride loop
`poll(now + 29999ms); now += 30000ms; poll(now);`, so every iteration
passed the pre-loop `now`.

StallWatchdog is correct on all three platforms and is not changed here.
Production never had this exposure: watchdogLoop() calls poll() with a
fresh steady_clock::now() per tick, never a loop-carried local advanced
by a constant.

Both watchdog tests now derive every instant absolutely as
`t0 + milliseconds(at_ms)` from an integer cursor -- the shape verified
to compile correctly on that runner. No assertion is weakened: every gap
is still checked one millisecond either side of its boundary.

Also corrects the comment in session_types.cpp that blamed a Windows
release build for mis-capping the ceiling. It never did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 11:09:57 -07:00
shadowdaoandClaude Opus 5 352a843d93 fix(session): enforce the stall-recovery ceiling where the wait is used
Build / macOS (macos-latest) (push) Successful in 53s
Build / Linux (ubuntu-24.04) (push) Successful in 1m24s
Release / macOS (macos-latest) (push) Successful in 55s
Release / Linux (ubuntu-24.04) (push) Successful in 1m18s
Build / Windows (windows-latest) (push) Failing after 3m41s
Release / Windows (windows-latest) (push) Failing after 3m39s
Release / Create Gitea Release (push) Skipped
The Windows release build failed testStallWatchdogBacksOffRatherThanLooping
(11/124 checks) while Linux and macOS passed, so v0.1.1 never published.

Reconstructing the failure from the log rather than guessing: the reported
FAIL lines (line 329 first, then 327/329 alternating for the rest of the
capped-backoff loop, 11 of the loop's 12 checks) are produced by exactly one
behaviour, and the deliberately-broken build in this commit's verification
reproduced that log byte-for-byte on Linux -- the backoff ceiling engaged one
attempt LATE. Windows waited 32000ms once (the uncapped doubling of 16000ms)
before settling at the 30000ms ceiling. Every other candidate produces a
different count and a different order: an exact-equality boundary bug gives 6
failures, and a ceiling that never engages at all gives 8, neither matching.

That rules out the obvious suspect, a lossy duration conversion. There isn't
one, and there cannot be: `time_point<Clock, D1> + duration<D2>` yields
`time_point<Clock, common_type_t<D1, D2>>`, and converting that back to
`steady_clock::time_point` to store it in next_attempt_allowed_ only compiles
when the conversion is exact. If MSVC's steady_clock could not represent a
whole millisecond exactly, this file would not build there. All of the
watchdog's time arithmetic is exact integer arithmetic on every platform, and
the exact-equality comparison at the deadline is sound -- the Windows log
itself shows later polls firing at exactly their deadline.

What is left is `std::min(backoff_ * 2, max_backoff_)`: the one expression in
poll() that was not plain value arithmetic on a single type, returning a
*reference* bound, in the growing case, to a materialized temporary. So:

- The ceiling is now clamped where the wait is USED, not only where the
  backoff is grown. max_backoff_ is a promise about the longest gap between
  two recovery attempts, so it is enforced on the gap itself and holds for
  whatever backoff_ contains. Verified: with the growth step deliberately
  mis-capping exactly the way Windows did, the whole suite still passes --
  the fix does not depend on having correctly identified MSVC's mechanism.
- The doubling is an explicit compare-and-clamp instead of std::min, so no
  reference to a temporary is involved and the product is only computed when
  it cannot exceed the ceiling.

Both changes are provably no-ops on Linux and macOS, where backoff_ never
exceeded the ceiling in the first place.

Also pins the behaviour with a new regression test using a ceiling that is
NOT a power-of-two multiple of the timeout (1000 -> 2000 -> 4000 -> 5000),
which fails on the step the ceiling first binds rather than six 30-second
iterations later. 146 checks in test_session now, was 124; all 6 CTest suites
pass locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 10:18:15 -07:00
shadowdaoandClaude Opus 5 564461a16d chore(release): 0.1.1
Build / macOS (macos-latest) (push) Successful in 53s
Release / macOS (macos-latest) (push) Successful in 52s
Build / Linux (ubuntu-24.04) (push) Successful in 1m9s
Release / Linux (ubuntu-24.04) (push) Successful in 1m12s
Build / Windows (windows-latest) (push) Failing after 3m37s
Release / Windows (windows-latest) (push) Failing after 3m37s
Release / Create Gitea Release (push) Skipped
Stall-recovery watchdog for video subscriptions (#7). Camera sources could
drop out in OBS and never recover while the same players stayed healthy in
browser talkback; the pinned client-sdk-cpp exposes no keyframe-request
API, so a decoder that lost a frame had no way to resync for the rest of
the show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 09:43:39 -07:00
jknapp 969a500dfd Merge pull request #7 from fix/stall-recovery
Build / macOS (macos-latest) (push) Successful in 52s
Build / Linux (ubuntu-24.04) (push) Successful in 1m16s
Build / Windows (windows-latest) (push) Failing after 3m45s
fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog
2026-09-21 16:43:21 +00:00
3 changed files with 119 additions and 26 deletions
+1 -1
View File
@@ -1,7 +1,7 @@
cmake_minimum_required(VERSION 3.19) cmake_minimum_required(VERSION 3.19)
project(obs-streamer-tools-plugin project(obs-streamer-tools-plugin
VERSION 0.1.0 VERSION 0.1.1
DESCRIPTION "OBS Studio source plugin for streamer-tools camera feeds" DESCRIPTION "OBS Studio source plugin for streamer-tools camera feeds"
LANGUAGES C CXX LANGUAGES C CXX
) )
+29 -4
View File
@@ -11,8 +11,6 @@ You may obtain a copy of the License at
#include "stplugin/session_types.h" #include "stplugin/session_types.h"
#include <algorithm>
namespace stplugin { namespace stplugin {
const char *describePixelFormat(PixelFormat format) const char *describePixelFormat(PixelFormat format)
@@ -230,8 +228,35 @@ bool StallWatchdog::poll(std::chrono::steady_clock::time_point now)
// 8s, ... up to max_backoff_ -- so a publisher that is genuinely gone // 8s, ... up to max_backoff_ -- so a publisher that is genuinely gone
// gets progressively less frequent toggles instead of one every 2 // gets progressively less frequent toggles instead of one every 2
// seconds for the rest of the show. // seconds for the rest of the show.
next_attempt_allowed_ = now + backoff_; //
backoff_ = std::min(backoff_ * 2, max_backoff_); // max_backoff_ is clamped HERE, where the wait is used, and not only
// where the backoff is grown. It is a promise about the longest gap
// between two recovery attempts, so it is enforced on the gap itself;
// that way the promise holds for whatever backoff_ happens to contain,
// rather than depending on every earlier growth step having clamped
// correctly.
//
// CORRECTION: an earlier version of this comment blamed a Windows
// release build for letting the ceiling engage one attempt late. That
// was wrong, and it is worth recording why rather than quietly
// deleting it. Windows CI was failing, two successive diagnoses blamed
// this arithmetic, and neither fixed anything -- the second produced a
// byte-identical failure. Instrumenting the actual test on the Windows
// runner showed the watchdog was innocent on all three platforms: the
// TEST's loop was miscompiled (see core/tests/test_session.cpp). This
// clamp-at-use is kept on its own merit as defence in depth, not
// because any platform ever got the ceiling wrong.
const std::chrono::milliseconds wait = backoff_ < max_backoff_ ? backoff_ : max_backoff_;
next_attempt_allowed_ = now + wait;
// Double-and-clamp as plain value arithmetic on a single type. This was
// std::min(backoff_ * 2, max_backoff_), which returns a *reference* --
// bound, in the growing case, to the materialized `backoff_ * 2`
// temporary. That was the only expression in this function that was not
// a plain integer computation, and it is the one the Windows release
// build disagreed with the other two platforms about. Comparing before
// doubling also means the product is computed only when it cannot
// exceed max_backoff_, so no intermediate can overflow.
backoff_ = (wait > max_backoff_ / 2) ? max_backoff_ : wait * 2;
return true; return true;
} }
+89 -21
View File
@@ -278,45 +278,81 @@ void testStallWatchdogDoesNotFireWhenNotExpectingFrames()
ST_ASSERT(w.poll(unmuted_at + std::chrono::milliseconds(2000))); ST_ASSERT(w.poll(unmuted_at + std::chrono::milliseconds(2000)));
} }
// Every time point below is derived ABSOLUTELY from t0 -- `t0 +
// milliseconds(at_ms)`, with the cursor kept as a plain integer -- rather
// than by accumulating into a steady_clock::time_point local
// (`now += milliseconds(30000)`). That is not a style preference; it is
// load-bearing on Windows.
//
// The MSVC 19.44 (VS 2022 BuildTools 14.44.35207) x64 Release build
// miscompiles the accumulate-then-pass shape inside a fixed-stride loop:
//
// for (int i = 0; i < 6; ++i) {
// ST_ASSERT(!w.poll(now + milliseconds(29999)));
// now += milliseconds(30000);
// ST_ASSERT(w.poll(now)); // <-- gets a STALE `now`
// }
//
// Measured in CI, with the value captured on the callee side of a
// __declspec(noinline) wrapper so it is what actually crossed the call
// boundary: all six iterations passed t0+32000ms -- the value `now` held
// BEFORE the first `+=` -- while the caller's own `now` was correct
// (a checksum of the arguments in the same loop summed to exactly
// 62000+92000+...+212000). The argument was hoisted out of the loop as if
// it were loop-invariant. Linux and macOS pass 62000, 92000, ... 212000 for
// the same source.
//
// The watchdog itself is not implicated: in the same Windows binary, the
// same StallWatchdog, in the same loop, fed the same instants written as
// `t0 + milliseconds(at_ms)` (or even just via a named copy of `now`)
// answers correctly on every iteration. Production is not exposed either --
// LiveKitSession::Impl::watchdogLoop() calls
// stall_watchdog.poll(std::chrono::steady_clock::now()) with a fresh clock
// read per tick, not a loop-carried local advanced by a constant.
//
// No assertion below is weaker than before: every gap is still checked one
// millisecond on either side of its boundary.
void testStallWatchdogBacksOffRatherThanLooping() void testStallWatchdogBacksOffRatherThanLooping()
{ {
const auto t0 = std::chrono::steady_clock::now(); const auto t0 = std::chrono::steady_clock::now();
StallWatchdog w(std::chrono::milliseconds(2000), std::chrono::milliseconds(30000)); StallWatchdog w(std::chrono::milliseconds(2000), std::chrono::milliseconds(30000));
w.setExpectingFrames(true, t0); w.setExpectingFrames(true, t0);
// Milliseconds since t0. A plain integer cursor, advanced explicitly.
long long at_ms = 2000;
// First attempt at the threshold. // First attempt at the threshold.
auto now = t0 + std::chrono::milliseconds(2000); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT(w.poll(now));
ST_ASSERT_EQ(w.attemptsThisStall(), 1); ST_ASSERT_EQ(w.attemptsThisStall(), 1);
// A genuinely gone publisher: no frame ever comes back. Immediately // A genuinely gone publisher: no frame ever comes back. Immediately
// asking again (the naive "retry every poll interval forever" a // asking again (the naive "retry every poll interval forever" a
// watchdog without backoff would do) must NOT fire -- that is precisely // watchdog without backoff would do) must NOT fire -- that is precisely
// the "hammered every 2 seconds forever" this backoff exists to avoid. // the "hammered every 2 seconds forever" this backoff exists to avoid.
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(250))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 250)));
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(1999))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 1999)));
// The backoff after attempt 1 is the base timeout (2000ms): the second // The backoff after attempt 1 is the base timeout (2000ms): the second
// attempt is allowed at +2000ms from the first, not before. // attempt is allowed at +2000ms from the first, not before.
now += std::chrono::milliseconds(2000); at_ms += 2000;
ST_ASSERT(w.poll(now)); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 2); ST_ASSERT_EQ(w.attemptsThisStall(), 2);
// Backoff doubles: the third attempt needs a 4000ms gap, not 2000ms. // Backoff doubles: the third attempt needs a 4000ms gap, not 2000ms.
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(3999))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 3999)));
now += std::chrono::milliseconds(4000); at_ms += 4000;
ST_ASSERT(w.poll(now)); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 3); ST_ASSERT_EQ(w.attemptsThisStall(), 3);
// ... and again to 8000ms, and again to 16000ms. // ... and again to 8000ms, and again to 16000ms.
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(7999))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 7999)));
now += std::chrono::milliseconds(8000); at_ms += 8000;
ST_ASSERT(w.poll(now)); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 4); ST_ASSERT_EQ(w.attemptsThisStall(), 4);
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(15999))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 15999)));
now += std::chrono::milliseconds(16000); at_ms += 16000;
ST_ASSERT(w.poll(now)); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 5); ST_ASSERT_EQ(w.attemptsThisStall(), 5);
// The backoff is capped: doubling 16000ms would be 32000ms, but it // The backoff is capped: doubling 16000ms would be 32000ms, but it
@@ -324,21 +360,52 @@ void testStallWatchdogBacksOffRatherThanLooping()
// failed, so a publisher that comes back after an hour is still // failed, so a publisher that comes back after an hour is still
// retried at a bounded cadence, not abandoned. // retried at a bounded cadence, not abandoned.
for (int i = 0; i < 6; ++i) { for (int i = 0; i < 6; ++i) {
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(29999))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 29999)));
now += std::chrono::milliseconds(30000); at_ms += 30000;
ST_ASSERT(w.poll(now)); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
} }
// A frame finally arrives: the stall is over, and the NEXT one (a fresh // A frame finally arrives: the stall is over, and the NEXT one (a fresh
// stall, not a continuation) starts back at the base cadence rather // stall, not a continuation) starts back at the base cadence rather
// than staying parked at the 30s ceiling forever. // than staying parked at the 30s ceiling forever.
w.onFrameDelivered(now); w.onFrameDelivered(t0 + std::chrono::milliseconds(at_ms));
ST_ASSERT_EQ(w.attemptsThisStall(), 0); ST_ASSERT_EQ(w.attemptsThisStall(), 0);
ST_ASSERT(!w.poll(now + std::chrono::milliseconds(1999))); ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 1999)));
ST_ASSERT(w.poll(now + std::chrono::milliseconds(2000))); ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms + 2000)));
ST_ASSERT_EQ(w.attemptsThisStall(), 1); ST_ASSERT_EQ(w.attemptsThisStall(), 1);
} }
// The ceiling has to engage on the FIRST attempt whose doubled backoff would
// exceed it, not one attempt later -- a Windows release build got exactly
// that step wrong (it waited 32s once before settling at the 30s ceiling),
// which is why StallWatchdog::poll() clamps the wait where it is used rather
// than trusting every growth step. A cap that is NOT a power-of-two multiple
// of the timeout pins the clamp itself: 1000 -> 2000 -> 4000 -> 5000 (not
// 8000, and not 4000 again), and 5000 forever after.
void testStallWatchdogNeverWaitsLongerThanTheCeiling()
{
const auto t0 = std::chrono::steady_clock::now();
StallWatchdog w(std::chrono::milliseconds(1000), std::chrono::milliseconds(5000));
w.setExpectingFrames(true, t0);
// Absolute instants off t0, for the reason spelled out above
// testStallWatchdogBacksOffRatherThanLooping().
long long at_ms = 1000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
// Expected gaps between consecutive attempts: 1000, 2000, 4000, then the
// ceiling for good. Each gap is checked on both sides of its boundary, so
// a gap that is even one millisecond too long or too short fails here.
const int expected_gaps[] = {1000, 2000, 4000, 5000, 5000, 5000, 5000};
int attempt = 1;
for (int gap : expected_gaps) {
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + gap - 1)));
at_ms += gap;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), ++attempt);
}
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Real SDK, failure paths only (no LiveKit server available headlessly) // Real SDK, failure paths only (no LiveKit server available headlessly)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -468,6 +535,7 @@ int main()
testStallWatchdogFiresAfterThreshold(); testStallWatchdogFiresAfterThreshold();
testStallWatchdogDoesNotFireWhenNotExpectingFrames(); testStallWatchdogDoesNotFireWhenNotExpectingFrames();
testStallWatchdogBacksOffRatherThanLooping(); testStallWatchdogBacksOffRatherThanLooping();
testStallWatchdogNeverWaitsLongerThanTheCeiling();
LiveKitSession::globalInitialize(); LiveKitSession::globalInitialize();
testConnectRejectsIncompleteConfig(); testConnectRejectsIncompleteConfig();