Compare commits

...
Author SHA1 Message Date
jknapp d097ea4300 Merge pull request 'fix(ci): run apt non-interactively so needrestart cannot hang a runner' (#9) from fix/ci-noninteractive-apt into main
Build / Linux (ubuntu-24.04) (push) Successful in 1m5s
Build / macOS (macos-latest) (push) Successful in 43s
Release / Linux (ubuntu-24.04) (push) Successful in 1m29s
Release / macOS (macos-latest) (push) Successful in 47s
Build / Windows (windows-latest) (push) Successful in 4m9s
Release / Windows (windows-latest) (push) Successful in 3m47s
Release / Create Gitea Release (push) Successful in 27s
2026-10-05 01:49:32 +00:00
shadowdaoandClaude Opus 5.5 a3344c249b fix(ci): run apt non-interactively so needrestart can't hang a runner
Build / macOS (macos-latest) (pull_request) Successful in 45s
Build / Linux (ubuntu-24.04) (push) Successful in 1m3s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m27s
Build / macOS (macos-latest) (push) Successful in 43s
Build / Windows (windows-latest) (pull_request) Successful in 4m10s
Build / Windows (windows-latest) (push) Successful in 4m6s
The v0.1.2 Linux release job sat for 15+ minutes on a self-hosted Debian
runner: after apt-get install, needrestart prompted for which daemons to
restart and waited on a terminal CI does not have.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 18:49:25 -07:00
shadowdaoandClaude Opus 5.5 3e8affea03 chore(release): 0.1.2
Build / macOS (macos-latest) (push) Successful in 44s
Build / Linux (ubuntu-24.04) (push) Successful in 1m8s
Release / macOS (macos-latest) (push) Successful in 48s
Build / Windows (windows-latest) (push) Successful in 4m6s
Release / Windows (windows-latest) (push) Successful in 3m47s
Release / Create Gitea Release (push) Canceled after 0s
Release / Linux (ubuntu-24.04) (push) Canceled after 14m54s
Selective subscription (#8). Every OBS plugin source auto-subscribed to
the whole room, so N sources pulled every camera N times over the
director's one downlink; on 2026-10-04 (8 sources, 7 cameras) all of
them hit congestion together and on-screen cameras dropped out. Each
source now subscribes only to its own slot's camera and mic.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 18:34:35 -07:00
jknapp dc5ffc9938 Merge pull request 'feat(session): subscribe only to the slot's own camera and mic' (#8) from feat/selective-subscribe into main
Build / macOS (macos-latest) (push) Successful in 45s
Build / Linux (ubuntu-24.04) (push) Successful in 1m1s
Build / Windows (windows-latest) (push) Successful in 4m12s
2026-10-05 01:33:34 +00:00
shadowdaoandClaude Opus 5.5 7723082795 feat(session): subscribe only to the slot's own camera and mic
Build / Linux (ubuntu-24.04) (push) Successful in 1m8s
Build / macOS (macos-latest) (push) Successful in 1m9s
Build / macOS (macos-latest) (pull_request) Successful in 44s
Build / Windows (windows-latest) (push) Successful in 7m0s
Build / Windows (windows-latest) (pull_request) Successful in 4m14s
Build / Linux (ubuntu-24.04) (pull_request) Failing after 42m30s
Every plugin source connected with auto_subscribe on, so each one pulled
every camera in the room and discarded all but one. In the 2026-10-04
old-gods-of-appalachia show (8 sources, 7 cameras) each source carried
4-7 Mbps and all 8 hit congestion in the same instant: they share the
director's single downlink, so the SFU starved the cameras on screen.

The session now connects with auto_subscribe off and calls
setSubscribed(true) on exactly the wanted publications: at connect, on
onTrackPublished (re-sweeping the participant, since SDK 1.10.1 delivers
that event's publication null), and after a reconnect. Any subscription
it did not ask for is reported and handed back. Audio-only sources no
longer subscribe-then-disable video; they never subscribe to it.

The integration test gains a bystander camera that must never be
subscribed (fails with 3 unwanted subscriptions before this change) and
a full publisher leave/rejoin as a new participant session.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 18:00:28 -07:00
shadowdaoandClaude Opus 5 22d50ab7be test(session): derive watchdog instants from t0, not an accumulated local
Build / macOS (macos-latest) (push) Successful in 53s
Build / Linux (ubuntu-24.04) (push) Successful in 1m7s
Release / macOS (macos-latest) (push) Successful in 57s
Release / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / Windows (windows-latest) (push) Successful in 4m6s
Release / Windows (windows-latest) (push) Successful in 3m55s
Release / Create Gitea Release (push) Successful in 20s
Windows CI failed on the stall-recovery watchdog for three attempts. The
first two diagnoses both blamed the backoff arithmetic; the second
produced a byte-identical failure, which was the clue that neither had
found the cause.

Instrumenting the test on the Windows runner settled it with numbers.
Capturing the time point on the callee side of a noinline wrapper showed
the six `w.poll(now)` calls in the capped-backoff loop received:

  Windows      32000, 32000, 32000, 32000, 32000, 32000  (ms after t0)
  Linux/macOS  62000, 92000, 122000, 152000, 182000, 212000

while a checksum of the caller's own arguments in the same loop summed to
822000 -- exactly the correct series. The caller's value was right; the
value that crossed the call boundary was not. MSVC 19.44 x64 Release
hoists the argument of the second poll() out of the fixed-stride loop
`poll(now + 29999ms); now += 30000ms; poll(now);`, so every iteration
passed the pre-loop `now`.

StallWatchdog is correct on all three platforms and is not changed here.
Production never had this exposure: watchdogLoop() calls poll() with a
fresh steady_clock::now() per tick, never a loop-carried local advanced
by a constant.

Both watchdog tests now derive every instant absolutely as
`t0 + milliseconds(at_ms)` from an integer cursor -- the shape verified
to compile correctly on that runner. No assertion is weakened: every gap
is still checked one millisecond either side of its boundary.

Also corrects the comment in session_types.cpp that blamed a Windows
release build for mis-capping the ceiling. It never did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 11:09:57 -07:00
shadowdaoandClaude Opus 5 352a843d93 fix(session): enforce the stall-recovery ceiling where the wait is used
Build / macOS (macos-latest) (push) Successful in 53s
Build / Linux (ubuntu-24.04) (push) Successful in 1m24s
Release / macOS (macos-latest) (push) Successful in 55s
Release / Linux (ubuntu-24.04) (push) Successful in 1m18s
Build / Windows (windows-latest) (push) Failing after 3m41s
Release / Windows (windows-latest) (push) Failing after 3m39s
Release / Create Gitea Release (push) Skipped
The Windows release build failed testStallWatchdogBacksOffRatherThanLooping
(11/124 checks) while Linux and macOS passed, so v0.1.1 never published.

Reconstructing the failure from the log rather than guessing: the reported
FAIL lines (line 329 first, then 327/329 alternating for the rest of the
capped-backoff loop, 11 of the loop's 12 checks) are produced by exactly one
behaviour, and the deliberately-broken build in this commit's verification
reproduced that log byte-for-byte on Linux -- the backoff ceiling engaged one
attempt LATE. Windows waited 32000ms once (the uncapped doubling of 16000ms)
before settling at the 30000ms ceiling. Every other candidate produces a
different count and a different order: an exact-equality boundary bug gives 6
failures, and a ceiling that never engages at all gives 8, neither matching.

That rules out the obvious suspect, a lossy duration conversion. There isn't
one, and there cannot be: `time_point<Clock, D1> + duration<D2>` yields
`time_point<Clock, common_type_t<D1, D2>>`, and converting that back to
`steady_clock::time_point` to store it in next_attempt_allowed_ only compiles
when the conversion is exact. If MSVC's steady_clock could not represent a
whole millisecond exactly, this file would not build there. All of the
watchdog's time arithmetic is exact integer arithmetic on every platform, and
the exact-equality comparison at the deadline is sound -- the Windows log
itself shows later polls firing at exactly their deadline.

What is left is `std::min(backoff_ * 2, max_backoff_)`: the one expression in
poll() that was not plain value arithmetic on a single type, returning a
*reference* bound, in the growing case, to a materialized temporary. So:

- The ceiling is now clamped where the wait is USED, not only where the
  backoff is grown. max_backoff_ is a promise about the longest gap between
  two recovery attempts, so it is enforced on the gap itself and holds for
  whatever backoff_ contains. Verified: with the growth step deliberately
  mis-capping exactly the way Windows did, the whole suite still passes --
  the fix does not depend on having correctly identified MSVC's mechanism.
- The doubling is an explicit compare-and-clamp instead of std::min, so no
  reference to a temporary is involved and the product is only computed when
  it cannot exceed the ceiling.

Both changes are provably no-ops on Linux and macOS, where backoff_ never
exceeded the ceiling in the first place.

Also pins the behaviour with a new regression test using a ceiling that is
NOT a power-of-two multiple of the timeout (1000 -> 2000 -> 4000 -> 5000),
which fails on the step the ceiling first binds rather than six 30-second
iterations later. 146 checks in test_session now, was 124; all 6 CTest suites
pass locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 10:18:15 -07:00
shadowdaoandClaude Opus 5 564461a16d chore(release): 0.1.1
Build / macOS (macos-latest) (push) Successful in 53s
Release / macOS (macos-latest) (push) Successful in 52s
Build / Linux (ubuntu-24.04) (push) Successful in 1m9s
Release / Linux (ubuntu-24.04) (push) Successful in 1m12s
Build / Windows (windows-latest) (push) Failing after 3m37s
Release / Windows (windows-latest) (push) Failing after 3m37s
Release / Create Gitea Release (push) Skipped
Stall-recovery watchdog for video subscriptions (#7). Camera sources could
drop out in OBS and never recover while the same players stayed healthy in
browser talkback; the pinned client-sdk-cpp exposes no keyframe-request
API, so a decoder that lost a frame had no way to resync for the rest of
the show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 09:43:39 -07:00
jknapp 969a500dfd Merge pull request #7 from fix/stall-recovery
Build / macOS (macos-latest) (push) Successful in 52s
Build / Linux (ubuntu-24.04) (push) Successful in 1m16s
Build / Windows (windows-latest) (push) Failing after 3m45s
fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog
2026-09-21 16:43:21 +00:00
shadowdaoandClaude Sonnet 5 3f2933ae3f fix(session): recover stalled video subscriptions with a keyframe-forcing watchdog
Build / macOS (macos-latest) (push) Successful in 1m7s
Build / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / macOS (macos-latest) (pull_request) Successful in 53s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m19s
Build / Windows (windows-latest) (push) Failing after 3m25s
Build / Windows (windows-latest) (pull_request) Failing after 3m0s
Camera sources in the OBS plugin were dropping out at random and never
recovering, while the same players stayed healthy in browser talkback.
Measured on the live server: every OBS plugin subscriber racked up
862-1099 nackMisses (retransmit requests for packets the SFU had already
aged out of its send buffer -- unrecoverable loss) and sat at plis == 2
for a multi-hour session, versus 0/adaptive-PLI for a browser subscriber
in the same room. The pinned client-sdk-cpp (1.10.1) exposes no
PLI/keyframe-request API at all, so a decoder that lost a frame that way
had no way to resync for the rest of the show.

Add a stall-recovery watchdog: StallWatchdog (session_types.h/.cpp) is a
pure, fake-clock-testable class that decides when a video subscription
has gone too long (2000ms, kStallRecoveryTimeout) without a decoded
frame reaching OBS. LiveKitSession::Impl polls it from a dedicated
thread and, through the existing command queue (never touching the SDK
off the worker thread), toggles the publication's setEnabled(false)/
setEnabled(true) -- the one lever this SDK exposes that makes the SFU
restart delivery, and a restart always begins with a keyframe.

Repeated attempts against the same stall back off exponentially
(2s/4s/8s/16s/30s-capped, mirroring the shape of the adapter's own
reconnect backoff) so a genuinely gone publisher is retried on a bounded
cadence instead of hammered every 2 seconds. A muted, disabled or
unsubscribed track, an audio-only source, or a disconnected session
never arms the watchdog: new onTrackMuted/onTrackUnmuted handlers
suspend and resume its clock, always re-baselining from "now" rather
than a stale timestamp, so un-muting after a long legitimate camera-off
period cannot read as a multi-minute stall. Each attempt, and eventual
recovery, is logged through a new DiagnosticHandler that the OBS adapter
maps onto obs_log at the same severities the file already uses for other
notable events.

Also investigated (not changed): the setVideoQuality(HIGH) pin added for
an earlier simulcast-resize bug. The SDK's own docs and the FFI binary's
wire-protocol strings show setVideoQuality only bounds spatial/simulcast
layer selection (UpdateTrackSettings.quality), never temporal layers or
frame rate, which the SFU's own congestion control governs independently
-- so this pin is unlikely to be the cause of the packet-loss symptom,
and is probably an inert no-op now that publishing is pinned server-side
to a single spatial layer (L1T3). Left in place since that is not fully
unambiguous from the SDK alone. Full writeup in
.stall-recovery-report.md (untracked, not part of this commit).

Adds 3 new headless StallWatchdog tests (fires-after-threshold, does-
not-fire-when-muted/disabled, backs-off-rather-than-loops) to
test_session.cpp; caught a real bug during development where
onFrameDelivered() reset the backoff duration but not the
next-attempt-allowed timestamp, throttling a just-recovered stream
against its own stale backoff. Built and all 6 CTest suites pass
(124/124 checks in test_session); also verified clean under
ThreadSanitizer with warning counts unchanged from the pre-change
baseline.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 19:04:14 -07:00
jknapp 48a74e8c67 Merge pull request 'docs: the macOS bundle was fine all along, and record the GUI verification' (#6) from docs/macos-packaging-and-gui-status into main 2026-09-10 12:31:33 +00:00
shadowdaoandClaude Opus 5 ab455a9cfe docs: the macOS bundle was fine all along, and record the GUI verification
Two corrections and one promotion, all from evidence rather than inference.

1. The "macOS packaging gap (known, unfixed)" section was WRONG, and it
   contradicted the release notes for the same build. It claimed the artifact
   is "a bare streamer-tools-camera.so" with a relative libobs install name
   that "will not load in OBS.app as it stands". Downloading and inspecting
   the shipped streamer-tools-camera-v0.1.0-macos.zip shows otherwise:

   - a proper streamer-tools-camera.plugin bundle -- Contents/MacOS/<name> is
     Mach-O MH_BUNDLE (what OBS loads), with Info.plist (BNDL, correct
     CFBundleExecutable), Contents/Resources/locale/en-US.ini, and both
     LiveKit dylibs in Contents/Frameworks/
   - install names are right: the module loads
     @rpath/libobs.framework/Versions/A/libobs and carries
     LC_RPATH @executable_path/../Frameworks, which inside OBS.app resolves to
     OBS.app/Contents/Frameworks; @rpath/liblivekit.dylib resolves through
     LC_RPATH @loader_path/../Frameworks to the bundle's own copy, and
     liblivekit.dylib finds liblivekit_ffi.dylib through LC_RPATH @loader_path.
     Nothing points into a build tree.
   - all three binaries carry LC_CODE_SIGNATURE, which is not optional:
     arm64 macOS refuses to load unsigned code.

   The real macOS limitation is different and now stated: the bundle is
   arm64-only (no x86_64 slice), macOS 13+. Nobody has still ever opened it in
   OBS.app -- well formed and signed is a prior, not a load.

2. Linux and Windows are confirmed working in the OBS GUI: video and audio
   both arrive and hold up across a session, Linux by the project owner and
   Windows by two directors independently, and the plugin carried a live show
   on 2026-09-07. Since listing cameras requires an API call, that also
   retires "the WinHTTP backend has never run against a real streamer-tools
   server".

   Scoped, not inflated: MEASURED A/V sync and latency against the egress path
   are still unverified (no drift reported is not a measurement), as is
   mid-show publisher restart. The "Not verified anywhere" list is now
   deduplicated and says exactly that.

3. The CI table's Windows row still said "Failing, fix pushed and awaiting a
   completed run". It is green, after the WinHTTP deadline fix.

Release notes template updated to match, and v0.1.0's published notes have
been regenerated through it so the public page stops repeating the bare-.so
claim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AzGnvQ6wfD7bw7PZN35ft9
2026-09-10 05:31:09 -07:00
jknapp e763d61ed3 Merge pull request 'fix(http): bound the WinHTTP response-header wait, and stop docs triggering builds' (#5) from fix/winhttp-response-header-timeout into main
Build / macOS (macos-latest) (push) Successful in 49s
Build / Linux (ubuntu-24.04) (push) Successful in 1m49s
Build / Windows (windows-latest) (push) Successful in 4m5s
2026-09-10 01:54:16 +00:00
shadowdaoandClaude Opus 5 17540c75b0 test: assert the 2x-budget ceiling the watchdog now guarantees, not 4s
Build / macOS (macos-latest) (push) Successful in 53s
Build / Linux (ubuntu-24.04) (push) Successful in 1m23s
Build / macOS (macos-latest) (pull_request) Successful in 46s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m19s
Build / Windows (windows-latest) (push) Successful in 3m57s
Build / Windows (windows-latest) (pull_request) Successful in 3m59s
The Windows job went green, but CTest prints test output only on failure, so a
pass says nothing about WHAT cancelled the request. Under the old 4000ms bound
a pass is ambiguous: the watchdog firing at ~1400ms and WinHTTP's own erratic
cancellation (measured at 1490-4506ms for this same 700ms budget) both fit
under it.

So assert the guarantee the code actually makes now -- a hard ceiling of twice
the caller's budget -- at 2500ms, which is 1400ms plus slack for a loaded
runner. If the watchdog stops doing the work, roughly half the attempts land
above this and print their elapsed time and error string, instead of quietly
passing.

Linux is unaffected: curl honours the 700ms budget exactly, 5/5 at ~701ms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AzGnvQ6wfD7bw7PZN35ft9
2026-09-09 18:43:32 -07:00
shadowdaoandClaude Opus 5 b23644fa3e fix(http): enforce a hard deadline on WinHTTP by cancelling the request
Build / macOS (macos-latest) (push) Successful in 51s
Build / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / macOS (macos-latest) (pull_request) Successful in 48s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m8s
Build / Windows (windows-latest) (push) Successful in 3m56s
Build / Windows (windows-latest) (pull_request) Successful in 3m56s
Setting WINHTTP_OPTION_RECEIVE_RESPONSE_TIMEOUT (previous commit) fixed the
original failure -- the request is now always cancelled instead of waiting out
a stall and returning 200 -- but the instrumented CI probe shows it is not
cancelled on TIME. Five attempts with a 700ms budget against a server that
accepts and then stalls 5s returned after 1490, 1529, 2485, 3493 and 4506ms,
every one of them ERROR_WINHTTP_TIMEOUT (12002). One exceeded the test's 4s
bound, which is why Windows CI was still red.

That is the documented behaviour, not a mystery: both receive timeouts are
"checked only when data is received from the socket", so an expired timeout is
not surfaced until the peer sends something. Neither option is a deadline.

It matters because `fetchSlots` is called synchronously on the OBS UI thread,
behind the properties dialog's "Refresh camera list" button
(obs-adapter/src/plugin-main.cpp:486, kPropertiesTimeoutMs = 5000). At the
overshoot ratio measured above, a stalling server freezes that dialog for
something like half a minute -- the exact failure the shortened timeout there
was chosen to avoid.

So: a watchdog thread that closes the request handle once the deadline passes,
which is the documented way to cancel a WinHTTP operation. `RequestDeadline`
owns the handle and both threads close it through an
`atomic::exchange(nullptr)`, so exactly one close ever happens. Failures are
reported as a timeout rather than as a raw GetLastError when the deadline is
what fired.

The ceiling is twice the caller's budget, not the budget itself: resolve,
connect, send and receive each get `timeout` from WinHttpSetTimeouts, so a
slow-but-progressing exchange can legitimately exceed one budget and must not
be cancelled. There is one accepted race, documented at the class: the caller
can load the handle just before the watchdog closes it, turning the call into
ERROR_INVALID_HANDLE instead. Both mean the deadline expired.

Cross-compiled with mingw-w64 (`-fsyntax-only`) rather than waiting on CI to
find syntax errors; also confirmed by preprocessor probe that
WINHTTP_OPTION_RECEIVE_RESPONSE_TIMEOUT is defined in those headers, so the
#ifdef guard is not silently skipping the option. Linux: all 6 suites pass.
Real verification is the Windows job's probe output.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AzGnvQ6wfD7bw7PZN35ft9
2026-09-09 18:37:58 -07:00
shadowdaoandClaude Opus 5 8888e57d08 fix(http): bound the WinHTTP response-header wait, and stop docs triggering builds
Build / macOS (macos-latest) (push) Successful in 50s
Build / Linux (ubuntu-24.04) (push) Successful in 1m14s
Build / macOS (macos-latest) (pull_request) Successful in 49s
Build / Linux (ubuntu-24.04) (pull_request) Successful in 1m16s
Build / Windows (windows-latest) (push) Failing after 3m55s
Build / Windows (windows-latest) (pull_request) Failing after 3m51s
Two things, both prompted by an intermittent Windows CI failure in
test_api_client's testPlatformBackendTimeout: roughly 2 runs in 6, both its
assertions failed together, meaning a request with timeout_ms=700 waited out a
5s server stall and returned 200. Same failure on 2026-09-07 (job 5834) and
2026-09-09 (job 5911), on code that passed on other runs -- pre-existing and
intermittent, not caused by a change.

1. The real bug. `WinHttpSetTimeouts`' receive parameter maps to
   WINHTTP_OPTION_RECEIVE_TIMEOUT, which Microsoft documents as a PER-PACKET
   Winsock-layer read timeout ("applies to fetching each packet of data off
   the socket"). The wait for the response HEADERS is a separate option,
   WINHTTP_OPTION_RECEIVE_RESPONSE_TIMEOUT, which WinHttpSetTimeouts does not
   set and which defaults to 90 SECONDS. So a server that accepts, reads the
   request and then stalls could block the calling thread for a minute and a
   half no matter what the caller passed as timeout_ms -- precisely the
   "blocking an OBS thread indefinitely" failure that test exists to prevent.
   Now set explicitly, guarded by #ifdef so an older SDK still builds.

   That is a genuine defect on its own merits. Whether it is the whole
   explanation for the intermittency is NOT established: the same docs say
   this timeout "is checked only when data is received from the socket", so
   neither option guarantees a hard deadline -- that needs a watchdog calling
   WinHttpCloseHandle, deliberately not done here.

2. Evidence, so the next run says more than pass/fail. The probe now runs 5
   times and prints elapsed ms, ok, status, requests_seen and the backend's
   error string (carrying GetLastError) for every attempt, so one CI run
   yields a failure RATE and an error code. Each attempt gets a FRESH
   loopback server: the server handles one connection at a time on a single
   thread, so reusing it would leave attempts 2..n in the accept backlog --
   never accepted, a different scenario from the one that fails. Verified on
   Linux: 5/5 attempts give up at ~701ms.

Also: build.yml now has paths-ignore for **.md, LICENSE, NOTICE, and the two
release-only files. This is a full three-platform build behind a runner with
capacity:1, and six of them fired for one afternoon of documentation edits.
Nothing that feeds a build or a test is on that list. Tradeoff: a docs-only
push now shows no status at all rather than a green one.

The WinHTTP change cannot be compiled locally (Linux host); CI is its first
build. All 6 suites pass locally on Linux.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AzGnvQ6wfD7bw7PZN35ft9
2026-09-09 18:30:02 -07:00
jknapp 5a39881af8 Merge pull request 'feat(release): publish releases directly, and fix the install path in the notes' (#4) from feat/publish-releases-directly into main
Build / macOS (macos-latest) (push) Successful in 27s
Build / Linux (ubuntu-24.04) (push) Successful in 45s
Build / Windows (windows-latest) (push) Failing after 4m59s
2026-09-09 23:51:05 +00:00
16 changed files with 1227 additions and 168 deletions
+10 -2
View File
@@ -3,5 +3,13 @@
# both need the exact same Linux build dependencies. Edit once, here. # both need the exact same Linux build dependencies. Edit once, here.
set -euo pipefail set -euo pipefail
sudo apt-get update -qq # Non-interactive, or a self-hosted Debian runner hangs forever: after an
sudo apt-get install -y -qq cmake ninja-build libobs-dev libcurl4-openssl-dev # install, needrestart asks which daemons to restart ("1. postfix.service
# 2. none of the above") and waits on a terminal CI does not have. That
# stalled the v0.1.2 Linux release job on 2026-10-05. NEEDRESTART_MODE=a
# answers it; DEBIAN_FRONTEND covers debconf prompts. Passed through sudo
# explicitly, since sudo drops the caller's environment by default.
APT=(sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a apt-get)
"${APT[@]}" update -qq
"${APT[@]}" install -y -qq cmake ninja-build libobs-dev libcurl4-openssl-dev
+14 -10
View File
@@ -25,7 +25,7 @@ set -euo pipefail
MACOS_BUNDLE_FOUND="${MACOS_BUNDLE_FOUND:-false}" MACOS_BUNDLE_FOUND="${MACOS_BUNDLE_FOUND:-false}"
if [ "${MACOS_BUNDLE_FOUND}" = "true" ]; then if [ "${MACOS_BUNDLE_FOUND}" = "true" ]; then
MACOS_NOTE="This archive contains a \`.plugin\` bundle." MACOS_NOTE="This archive contains a \`.plugin\` bundle (verified on v0.1.0: MH_BUNDLE + Info.plist, libobs via \`@rpath\` + \`@executable_path/../Frameworks\`, LiveKit dylibs bundled, all three binaries code-signed). **arm64 only -- no Intel slice**, macOS 13+. Never yet loaded in OBS.app by a human."
else else
MACOS_NOTE="This archive is packaged as a bare \`streamer-tools-camera.so\` (the layout \`build/package/\` currently produces on macOS), **not** an OBS.app-loadable \`.plugin\` bundle. It will not load in the OBS GUI as-is -- see the \"macOS packaging gap\" section of \`README.md\`." MACOS_NOTE="This archive is packaged as a bare \`streamer-tools-camera.so\` (the layout \`build/package/\` currently produces on macOS), **not** an OBS.app-loadable \`.plugin\` bundle. It will not load in the OBS GUI as-is -- see the \"macOS packaging gap\" section of \`README.md\`."
fi fi
@@ -36,18 +36,22 @@ cat > "${NOTES_FILE}" <<EOF
Built from commit \`${SHA}\`. Built from commit \`${SHA}\`.
**Read the per-platform notes below before relying on this.** The module has **Confirmed working on Linux and Windows, including a live show.** The plugin
been loaded in the OBS GUI exactly once -- Windows, OBS 32.2.2 on Windows 11, carried a real broadcast on 2026-09-07. Video and audio both arrive and hold
2026-09-09 -- and only "it loads and registers its source type" is confirmed up across a session: Linux verified by the project owner, Windows by two
there. Whether video renders correctly, A/V sync, end-to-end latency and directors independently.
mid-show publisher restart are all still unverified on every platform. See
"What is verified, and how" in \`README.md\` for exactly what is backed by Still unverified: **macOS in the OBS GUI** (nobody has opened it -- see the
automated tests versus a human watching OBS. table), **measured** A/V sync and end-to-end latency against the existing
egress path (no drift reported, but nothing measured), and whether a publisher
restarting mid-show recovers cleanly on screen. See "What is verified, and
how" in \`README.md\` for what is backed by automated tests versus a human
watching OBS.
| Platform | Archive | Notes | | Platform | Archive | Notes |
|---|---|---| |---|---|---|
| Linux (x64) | \`streamer-tools-camera-${TAG}-linux-x64.zip\` | Functionally complete and verified end to end against a real LiveKit server and a real libobs (see README); OBS GUI itself still unverified | | Linux (x64) | \`streamer-tools-camera-${TAG}-linux-x64.zip\` | Functionally complete, verified end to end against a real LiveKit server and a real libobs (see README), and confirmed working in the OBS GUI |
| Windows (x64) | \`streamer-tools-camera-${TAG}-windows-x64.zip\` | Built and tested by this workflow's Windows job. Confirmed to load in the OBS GUI (32.2.2 / Windows 11, 2026-09-09); nothing past module load is verified. The WinHTTP backend has never been exercised against a real streamer-tools server, only a loopback test server -- see README's Windows CI section | | Windows (x64) | \`streamer-tools-camera-${TAG}-windows-x64.zip\` | Built and tested by this workflow's Windows job, and confirmed working in the OBS GUI by two directors independently (first load: OBS 32.2.2 / Windows 11) -- which also exercises the WinHTTP backend against a real streamer-tools server |
| macOS | \`streamer-tools-camera-${TAG}-macos.zip\` | Built and tested by this workflow's macOS job. ${MACOS_NOTE} | | macOS | \`streamer-tools-camera-${TAG}-macos.zip\` | Built and tested by this workflow's macOS job. ${MACOS_NOTE} |
## Installing ## Installing
+22
View File
@@ -31,7 +31,29 @@ on:
# job is ordinary commits. # job is ordinary commits.
branches: branches:
- "**" - "**"
# Documentation-only changes cannot break a build, and this workflow is a
# full three-platform build (Windows included) behind a runner with
# capacity:1. Six of these fired for one afternoon of README/release-notes
# edits on 2026-09-09. Anything that feeds a build or a test is absent
# from this list on purpose -- release.yml and publish-release.sh only run
# on a `v*` tag, via release.yml's own trigger.
#
# Tradeoff: a docs-only push now shows NO status at all on the branch,
# rather than a green one. If a required-status check is ever added, these
# paths have to be reconsidered.
paths-ignore:
- "**.md"
- "LICENSE"
- "NOTICE"
- ".gitea/workflows/release.yml"
- ".gitea/scripts/publish-release.sh"
pull_request: pull_request:
paths-ignore:
- "**.md"
- "LICENSE"
- "NOTICE"
- ".gitea/workflows/release.yml"
- ".gitea/scripts/publish-release.sh"
jobs: jobs:
linux: linux:
+2 -1
View File
@@ -41,7 +41,8 @@ jobs:
# zip is not guaranteed present on a minimal self-hosted runner # zip is not guaranteed present on a minimal self-hosted runner
# image (unlike GitHub-hosted ubuntu-24.04, which build.yml's # image (unlike GitHub-hosted ubuntu-24.04, which build.yml's
# deps script doesn't need to care about). # deps script doesn't need to care about).
command -v zip >/dev/null || sudo apt-get install -y -qq zip # Non-interactive for the same reason as .gitea/scripts/linux-deps.sh.
command -v zip >/dev/null || sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a apt-get install -y -qq zip
out="streamer-tools-camera-${GITEA_REF_NAME}-linux-x64.zip" out="streamer-tools-camera-${GITEA_REF_NAME}-linux-x64.zip"
root="$(pwd)" root="$(pwd)"
+1 -1
View File
@@ -1,7 +1,7 @@
cmake_minimum_required(VERSION 3.19) cmake_minimum_required(VERSION 3.19)
project(obs-streamer-tools-plugin project(obs-streamer-tools-plugin
VERSION 0.1.0 VERSION 0.1.2
DESCRIPTION "OBS Studio source plugin for streamer-tools camera feeds" DESCRIPTION "OBS Studio source plugin for streamer-tools camera feeds"
LANGUAGES C CXX LANGUAGES C CXX
) )
+61 -49
View File
@@ -24,15 +24,21 @@ The plugin is **functionally complete on Linux and verified end to end there**
real streamer-tools API shape, and pushes decoded frames into real streamer-tools API shape, and pushes decoded frames into
`obs_source_output_video`/`_audio`). `obs_source_output_video`/`_audio`).
**First confirmed OBS GUI load: Windows, 2026-09-09** — the v0.1.0 release **Confirmed working in the OBS GUI on Linux and Windows, including a live
artifact loaded into OBS 32.2.2 on Windows 11 (build 26200) on a director's show.** The plugin carried a real broadcast on 2026-09-07 and was reported to
machine, from work well. Video and audio both arrive and hold up across a session: Linux
`C:\ProgramData\obs-studio\plugins\streamer-tools-camera\bin\64bit\`. verified by the project owner, Windows by two directors independently
That retires "the module will not even load in a real OBS" for Windows. It (2026-09-09/10; the first Windows load was OBS 32.2.2 on Windows 11 build
does **not** yet cover whether video renders correctly, colours, A/V sync or 26200, from
latency — see "Not verified anywhere" below for what is still open. Linux and `C:\ProgramData\obs-studio\plugins\streamer-tools-camera\bin\64bit\`).
macOS have still never been opened in the GUI; macOS builds the real module in Because listing cameras requires an API call, that also retires "the WinHTTP
CI but its artifact is not yet loadable (see the macOS packaging gap under CI). backend has never run against a real streamer-tools server".
What that does **not** cover: measured A/V sync and end-to-end latency against
the existing egress path (no drift reported over a session, but nothing was
measured), mid-show publisher restart, and **macOS in the GUI — still never
opened by anyone**, though its artifact is now known to be correctly packaged
(see macOS packaging below). See "Not verified anywhere" for the current list.
⚠️ **The install directory is not the same on every platform, and getting it ⚠️ **The install directory is not the same on every platform, and getting it
wrong fails silently.** On Windows it is wrong fails silently.** On Windows it is
@@ -169,15 +175,15 @@ plugins. This bit a director on 2026-09-09: a correctly-shaped install under
`AppData\Roaming` produced a log with zero mention of the module. `AppData\Roaming` produced a log with zero mention of the module.
The module resolves the LiveKit libraries from `$ORIGIN` (verified: `ldd` on the staged copy resolves both The module resolves the LiveKit libraries from `$ORIGIN` (verified: `ldd` on the staged copy resolves both
to `bin/64bit/`), not from the build tree. macOS is not this shape; see the to `bin/64bit/`), not from the build tree. macOS is not this shape — it ships
macOS packaging gap under CI. a `.plugin` bundle; see macOS packaging under CI.
## Testing this by hand ## Testing this by hand
**The module has been loaded in the OBS GUI on Windows once (2026-09-09, OBS **Linux and Windows are confirmed working in the GUI — video and audio over a
32.2.2 / Windows 11 26200) — nothing beyond "it loads and registers its source" real session, Linux by the project owner and Windows by two directors
is confirmed there, and Linux and macOS have never been opened in the GUI at independently (2026-09-09/10). macOS has never been opened in the GUI by
all.** To do it on Linux: anyone.** To repeat the Linux run:
``` ```
mkdir -p ~/.config/obs-studio/plugins/streamer-tools-camera mkdir -p ~/.config/obs-studio/plugins/streamer-tools-camera
@@ -233,21 +239,17 @@ livekit-server 1.13.6 in dev mode):
| Two sources in one OBS process | same harness with a second source added: both connect with distinct nonce identities, both receive frames, both tear down cleanly | | Two sources in one OBS process | same harness with a second source added: both connect with distinct nonce identities, both receive frames, both tear down cleanly |
**Not verified anywhere:** **Not verified anywhere:**
- Anything past module load in the OBS GUI. Windows 2026-09-09 confirms the - **macOS in the OBS GUI.** Nobody has opened it. Its artifact is now known to
module loads and its source type appears; whether video actually renders be a correctly-formed, correctly-linked, code-signed `.plugin` bundle
(right way up, right colours), what the A/V sync and latency look like, and (verified by inspecting the shipped v0.1.0 zip — see macOS packaging under
whether a publisher restarting mid-show recovers on screen are all still CI), and it is arm64-only, so Intel Macs are out regardless. "The bundle is
unanswered. Linux and macOS have not been opened in the GUI at all. well formed" is not "OBS loaded it".
- macOS beyond "CI builds and links the real module and the core tests pass". - **Measured** A/V sync and end-to-end latency against the existing egress
Its artifact is a bare `.so` with a relative libobs install name and will path. A live show and several sessions on Linux and Windows produced no
not load in OBS.app — see the macOS packaging gap under CI. reported drift, which is not the same as a measurement — and the timestamp
- Windows beyond "the core library and the WinHTTP backend compile and their caveat above is the reason to want real numbers.
tests pass", from runs predating the current fixes. The WinHTTP backend has - Whether a publisher restarting mid-show recovers cleanly on screen.
never run against a real streamer-tools server, only against the loopback - Token expiry across a session longer than an hour (see below).
test server in `test_api_client`.
- A/V sync and end-to-end latency against the existing egress path.
- Behaviour against the real production streamer-tools server (only against a
stand-in serving the same shapes).
- Token expiry after an hour. Expiry is handled *reactively*: a fatal - Token expiry after an hour. Expiry is handled *reactively*: a fatal
disconnect makes the worker mint a fresh token and reconnect. The design disconnect makes the worker mint a fresh token and reconnect. The design
doc's "proactively refreshed before expiry" is **not** implemented — doc's "proactively refreshed before expiry" is **not** implemented —
@@ -261,8 +263,8 @@ runners available to this repo under the `CyberCoveLLC` org.
| Job | `runs-on` | Runner | State | | Job | `runs-on` | Runner | State |
|---|---|---|---| |---|---|---|---|
| `linux` | `ubuntu-24.04` | `localhost.localdomain` | **Green.** Builds the real adapter against Ubuntu's libobs-dev 30.0.2, runs all six test suites, uploads `build/package` as an artifact | | `linux` | `ubuntu-24.04` | `localhost.localdomain` | **Green.** Builds the real adapter against Ubuntu's libobs-dev 30.0.2, runs all six test suites, uploads `build/package` as an artifact |
| `macos` | `macos-latest` | `home-mac` (Global) | **Green.** Builds libobs 30.0.2 from source, then the real adapter; 6/6 tests; artifact uploaded. But see the macOS packaging gap below | | `macos` | `macos-latest` | `home-mac` (Global) | **Green.** Builds libobs 30.0.2 from source, then the real adapter; 6/6 tests; artifact uploaded as a `.plugin` bundle. Never loaded in OBS.app, and arm64-only — see macOS packaging below |
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) | **Failing, fix pushed and awaiting a completed run.** Every completed run so far has failed; the latest got as far as building libobs and stopped on an OBS-side `OBS::w32-pthreads` target that its own modern CMake path never defines. A bootstrap patch for that gap has been pushed but not yet confirmed by a green run; see below | | `windows` | `windows-latest` | `winvm-builder` (org-scoped) | **Green.** Builds libobs 30.0.2 from source, then the real adapter; 6/6 tests; artifact staged. Was red twice more after the bootstrap was fixed, both times on `test_api_client`'s timeout probe — see "WinHTTP timeouts are not deadlines" below |
The Linux job is pinned to `ubuntu-24.04` rather than `ubuntu-latest`: this The Linux job is pinned to `ubuntu-24.04` rather than `ubuntu-latest`: this
instance's two Linux runners answer `ubuntu-latest` with different releases, instance's two Linux runners answer `ubuntu-latest` with different releases,
@@ -356,26 +358,36 @@ obs-studio, builds libobs from source, builds and links the real adapter,
passes 6/6 tests, and uploads its artifact. `otool -L` on the result shows it passes 6/6 tests, and uploads its artifact. `otool -L` on the result shows it
linked against libobs and `@rpath/liblivekit.dylib`. linked against libobs and `@rpath/liblivekit.dylib`.
### macOS packaging gap (known, unfixed) ### macOS packaging (was described here as broken; it is not)
**The macOS artifact will not load in OBS.app as it stands.** Two reasons, **This section used to claim the macOS artifact was a bare
neither of which CI can catch, because CI only proves it compiles and links: `streamer-tools-camera.so` with a relative libobs install name that "will not
load in OBS.app as it stands". That is wrong, and it contradicted the release
notes for the same build.** Corrected 2026-09-10 by inspecting the shipped
`streamer-tools-camera-v0.1.0-macos.zip` itself:
1. It is a bare `streamer-tools-camera.so`. OBS on macOS loads plugins as - It is a proper bundle: `streamer-tools-camera.plugin/Contents/MacOS/streamer-tools-camera`
`<name>.plugin` bundles (`Contents/MacOS/<name>`, `Contents/Resources/`, (Mach-O **`MH_BUNDLE`**, which is what OBS loads), plus `Info.plist`
an `Info.plist`), which is what obs-plugintemplate's (`CFBundlePackageType BNDL`, `CFBundleExecutable streamer-tools-camera`),
`cmake/macos/helpers.cmake` builds and which this project deliberately did `Contents/Resources/locale/en-US.ini`, and both LiveKit dylibs under
not vendor. `Contents/Frameworks/`.
2. `otool -L` shows the libobs dependency recorded as the relative path - The install names are right, which was the specific doubt. The module loads
`libobs/libobs.framework/Versions/A/libobs`, inherited from the `@rpath/libobs.framework/Versions/A/libobs` and carries
from-source libobs's own install name. A real plugin needs `LC_RPATH @executable_path/../Frameworks` — inside OBS.app that resolves to
`@rpath/libobs.framework/Versions/A/libobs` plus an `LC_RPATH` pointing at `OBS.app/Contents/Frameworks`, where libobs lives. `@rpath/liblivekit.dylib`
`OBS.app/Contents/Frameworks`. resolves through `LC_RPATH @loader_path/../Frameworks` to the bundle's own
copy, and `liblivekit.dylib` finds `liblivekit_ffi.dylib` through its own
`LC_RPATH @loader_path`. Nothing points into a build tree.
- All three binaries carry an `LC_CODE_SIGNATURE` (superblob `0xfade0cc0`),
which is not optional: arm64 macOS refuses to load unsigned code at all.
Fixing this means either vendoring the template's macOS bundle helpers or **The real macOS limitation is different: the bundle is arm64-only.** There is
adding an `install_name_tool` pass and a bundle layout — bounded work, but no x86_64 slice, so Intel Macs cannot load it, and `LSMinimumSystemVersion` is
work that has to be done and checked on an actual Mac. It is deliberately not `13.0`. Shipping a universal binary would mean building both slices and
attempted here rather than guessed at. `lipo`-ing them, on a Mac.
Everything above is static inspection of the artifact. **Nobody has yet opened
it in OBS.app** — well-formed and signed is a strong prior, not a load.
### Where the Windows bootstrap got to ### Where the Windows bootstrap got to
+26 -5
View File
@@ -11,6 +11,7 @@ You may obtain a copy of the License at
#pragma once #pragma once
#include <chrono>
#include <memory> #include <memory>
#include <string> #include <string>
@@ -43,17 +44,31 @@ struct SessionConfig {
bool subscribe_audio = true; bool subscribe_audio = true;
/// False for an audio-only source (the soundboard, say): the wanted /// False for an audio-only source (the soundboard, say): the slot's
/// video track is never attached (no AttachVideo command posted), and /// video track is never subscribed to (see shouldSubscribe), so the SFU
/// its publication is explicitly disabled server-side (RemoteTrack- /// never sends it at all -- not just "decoded and discarded here".
/// Publication::setEnabled(false)) so the SFU stops sending it at all --
/// not just "decoded and discarded here", genuinely not delivered.
bool subscribe_video = true; bool subscribe_video = true;
/// How long connect() waits for the room to come up before giving up. /// How long connect() waits for the room to come up before giving up.
int connect_timeout_ms = 15000; int connect_timeout_ms = 15000;
}; };
/// How long the stall-recovery watchdog waits for a decoded video frame
/// before treating the subscription as stalled and forcing a fresh
/// keyframe (see StallWatchdog's comment in session_types.h for why this
/// exists -- unrecoverable packet loss with no PLI/keyframe-request API in
/// the pinned SDK). Long enough that ordinary jitter never trips it (a
/// healthy 30fps subscription delivers a frame at least every ~33ms);
/// short enough a director barely has time to notice before it recovers.
constexpr std::chrono::milliseconds kStallRecoveryTimeout{2000};
/// Ceiling for the backoff between repeated recovery attempts against the
/// SAME stall. Starts at kStallRecoveryTimeout and doubles each attempt, so
/// a genuinely gone publisher (crashed encoder, dead upstream network) is
/// retried every 2s, 4s, 8s, ... 30s rather than hammered every 2 seconds
/// for the rest of the show.
constexpr std::chrono::milliseconds kStallRecoveryMaxBackoff{30000};
/// Wraps livekit::Room for exactly one subscribed slot. /// Wraps livekit::Room for exactly one subscribed slot.
/// ///
/// Threading contract, which the OBS adapter depends on: /// Threading contract, which the OBS adapter depends on:
@@ -67,6 +82,11 @@ struct SessionConfig {
/// call. /// call.
/// - The state handler is invoked from whichever thread observed the /// - The state handler is invoked from whichever thread observed the
/// change. It must not block and must not call back into this object. /// change. It must not block and must not call back into this object.
/// - The diagnostic handler (currently just the stall-recovery watchdog,
/// see kStallRecoveryTimeout below) may be invoked from the internal
/// command-queue worker thread or a video reader thread. Same rules as
/// the state handler: must not block, must not call back into this
/// object.
/// - All handlers must be installed before connect(); they are not /// - All handlers must be installed before connect(); they are not
/// synchronised against a running session. /// synchronised against a running session.
class LiveKitSession { class LiveKitSession {
@@ -80,6 +100,7 @@ public:
void setVideoHandler(VideoFrameHandler handler); void setVideoHandler(VideoFrameHandler handler);
void setAudioHandler(AudioFrameHandler handler); void setAudioHandler(AudioFrameHandler handler);
void setStateHandler(SessionStateHandler handler); void setStateHandler(SessionStateHandler handler);
void setDiagnosticHandler(DiagnosticHandler handler);
/// Connect and start subscribing. Returns true once the room is up; the /// Connect and start subscribing. Returns true once the room is up; the
/// selected slot's tracks may still arrive later (or not at all, if the /// selected slot's tracks may still arrive later (or not at all, if the
+94
View File
@@ -19,6 +19,7 @@ You may obtain a copy of the License at
// self-consistent -- all live here, and LiveKitSession is the (much thinner) // self-consistent -- all live here, and LiveKitSession is the (much thinner)
// piece that wires real SDK callbacks into them. // piece that wires real SDK callbacks into them.
#include <chrono>
#include <cstddef> #include <cstddef>
#include <cstdint> #include <cstdint>
#include <functional> #include <functional>
@@ -69,6 +70,14 @@ bool isWantedVideoTrack(const std::string &wanted_identity, const std::string &t
bool isWantedAudioTrack(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind, bool isWantedAudioTrack(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
MediaSource source); MediaSource source);
/// Should this session subscribe to this publication at all? The session
/// connects with auto_subscribe off and asks for exactly these, so the SFU
/// never sends this source anything it would only throw away. Without it,
/// every plugin source pulled every camera in the room: eight sources in one
/// OBS meant the director's downlink carried each camera eight times.
bool shouldSubscribe(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
MediaSource source, bool subscribe_video, bool subscribe_audio);
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Session state // Session state
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -137,6 +146,82 @@ private:
bool has_audio_ = false; bool has_audio_ = false;
}; };
// ---------------------------------------------------------------------------
// Stall-recovery watchdog (pure timing/decision logic)
// ---------------------------------------------------------------------------
/// Decides WHEN to force a fresh keyframe on an already-subscribed video
/// track. It does not touch LiveKit or OBS at all -- LiveKitSession::Impl
/// (session.cpp) is what actually carries the decision out
/// (RemoteTrackPublication::setEnabled(false) then setEnabled(true)), and
/// only from its SDK command-queue thread, the same rule every other SDK
/// interaction in that file already follows.
///
/// Why this exists: measured on the live server, comparing subscribers in
/// the same LiveKit room over the same 30-minute window, every OBS plugin
/// connection racked up ~862-1099 nackMisses and ~2200-3100 nackRepeated --
/// nackMisses means the subscriber asked the SFU to retransmit a packet
/// that had already aged out of its send buffer, i.e. unrecoverable loss --
/// while a browser subscriber in the same room saw 0 and 0. A decoder that
/// loses a frame that way cannot resync without a fresh keyframe. Every OBS
/// plugin connection also sat at `plis` == 2 for a multi-hour session (a
/// browser adapts and asks for keyframes normally), and grepping the pinned
/// client-sdk-cpp (1.10.1) headers turns up no PLI/keyframe-request API at
/// all. So today, once that happens, the source just stays broken for the
/// rest of the show -- "drops at random and never recovers". Toggling the
/// subscription off and back on is the one lever this SDK exposes that
/// forces the SFU to stop and restart delivery of the track, and a restart
/// always begins with a keyframe. This class is the "have we gone too long
/// without a frame, and is it still worth trying again" clock behind that
/// lever; see LiveKitSession::Impl::recoverVideo() for where it is pulled.
///
/// Not thread-safe on its own, deliberately -- same contract as
/// SessionStateMachine above: LiveKitSession::Impl owns the lock (in
/// practice the same state_mutex that guards `machine`).
class StallWatchdog {
public:
StallWatchdog(std::chrono::milliseconds timeout, std::chrono::milliseconds max_backoff);
/// Call whenever whether a frame could legitimately arrive right now
/// changes: true once a video track is subscribed (and unmuted), false
/// on detach/unsubscribe/mute/disconnect. Flipping to false always
/// clears all timing state; flipping back to true always starts a
/// brand-new grace period rather than measuring from a stale timestamp.
/// That is what stops an unmute -- or an ordinary publisher swap --
/// from firing the INSTANT it resumes, off a "last frame" that might
/// actually be minutes old: a muted, disabled or unsubscribed track, an
/// audio-only source, or a disconnected session must never trip this.
void setExpectingFrames(bool expecting, std::chrono::steady_clock::time_point now);
/// Call every time a decoded video frame is actually delivered.
void onFrameDelivered(std::chrono::steady_clock::time_point now);
/// Call periodically (finer-grained than the configured timeout).
/// Returns true exactly when a recovery attempt should be made right
/// now; each true also arms the backoff before the next one is even
/// considered, so a caller polling in a tight loop still cannot fire
/// back-to-back attempts against a publisher that never comes back --
/// see the class comment: hammering every 2 seconds forever against a
/// genuinely gone publisher is worse than a frozen source.
bool poll(std::chrono::steady_clock::time_point now);
/// Attempts made since the current stall started (since the last frame,
/// or since expecting-frames most recently became true). Reset by
/// onFrameDelivered and by setExpectingFrames.
int attemptsThisStall() const { return attempts_; }
private:
std::chrono::milliseconds timeout_;
std::chrono::milliseconds max_backoff_;
bool expecting_ = false;
bool have_baseline_ = false;
std::chrono::steady_clock::time_point baseline_{};
std::chrono::milliseconds backoff_{};
std::chrono::steady_clock::time_point next_attempt_allowed_{};
int attempts_ = 0;
};
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Frames handed to the OBS adapter // Frames handed to the OBS adapter
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -177,4 +262,13 @@ using VideoFrameHandler = std::function<void(const VideoFrameData &)>;
using AudioFrameHandler = std::function<void(const AudioFrameData &)>; using AudioFrameHandler = std::function<void(const AudioFrameData &)>;
using SessionStateHandler = std::function<void(SessionState state, const std::string &detail)>; using SessionStateHandler = std::function<void(SessionState state, const std::string &detail)>;
/// Severity for LiveKitSession's own diagnostic log lines (currently just
/// the stall-recovery watchdog). Kept separate from OBS's LOG_* levels and
/// from the SDK's own livekit::LogLevel so core/ stays free of any OBS
/// dependency -- the adapter maps this onto obs_log the same way it already
/// maps livekit::LogLevel (see plugin-main.cpp's livekit log bridge).
enum class DiagnosticLevel { Info, Warning };
using DiagnosticHandler = std::function<void(DiagnosticLevel level, const std::string &message)>;
} // namespace stplugin } // namespace stplugin
+147 -23
View File
@@ -19,8 +19,13 @@ You may obtain a copy of the License at
#include <windows.h> #include <windows.h>
#include <winhttp.h> #include <winhttp.h>
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <cstddef> #include <cstddef>
#include <mutex>
#include <string> #include <string>
#include <thread>
#include <vector> #include <vector>
namespace stplugin { namespace stplugin {
@@ -72,6 +77,79 @@ private:
HINTERNET h_ = nullptr; HINTERNET h_ = nullptr;
}; };
/// Hard deadline for one WinHTTP exchange, enforced by cancelling it.
///
/// Neither receive timeout is a guaranteed deadline: Microsoft documents both
/// as "checked only when data is received from the socket", so an expired
/// timeout is not surfaced until the peer finally sends something. Measured on
/// the Windows CI runner against a server that accepts and then stalls 5s: a
/// 700ms budget returned after 1490, 1529, 2485, 3493 and 4506ms across five
/// attempts -- always cancelled, never on time.
///
/// That overshoot matters because `fetchSlots` is called synchronously on the
/// OBS UI thread behind the properties dialog's "Refresh camera list" button
/// (obs-adapter/src/plugin-main.cpp), with a 5s budget. At the ratio above
/// that is a frozen dialog for half a minute.
///
/// The documented way to force cancellation is to close the handle from
/// another thread; the pending call then fails with
/// ERROR_WINHTTP_OPERATION_CANCELLED. This owns the request handle so that
/// exactly one of the two threads ever closes it: `handle_.exchange(nullptr)`
/// hands the close to whichever gets there first.
///
/// Known, accepted race: the caller may load the handle and have the watchdog
/// close it before the WinHttp* call reads it, in which case the call fails
/// with ERROR_INVALID_HANDLE instead. Both outcomes are "the deadline
/// expired", which is what the caller is told either way.
class RequestDeadline {
public:
RequestDeadline(HINTERNET request, DWORD after_ms) : handle_(request)
{
watchdog_ = std::thread([this, after_ms] {
std::unique_lock<std::mutex> lock(mutex_);
if (cv_.wait_for(lock, std::chrono::milliseconds(after_ms), [this] { return finished_; }))
return; // exchange finished inside the deadline
if (closeOnce())
expired_.store(true);
});
}
~RequestDeadline()
{
{
std::lock_guard<std::mutex> lock(mutex_);
finished_ = true;
}
cv_.notify_all();
if (watchdog_.joinable())
watchdog_.join();
closeOnce(); // no-op if the watchdog got there first
}
RequestDeadline(const RequestDeadline &) = delete;
RequestDeadline &operator=(const RequestDeadline &) = delete;
HINTERNET get() const { return handle_.load(); }
bool expired() const { return expired_.load(); }
private:
bool closeOnce()
{
HINTERNET h = handle_.exchange(nullptr);
if (!h)
return false;
WinHttpCloseHandle(h);
return true;
}
std::atomic<HINTERNET> handle_;
std::atomic<bool> expired_{false};
std::mutex mutex_;
std::condition_variable cv_;
bool finished_ = false;
std::thread watchdog_;
};
class WinHttpClient : public HttpClient { class WinHttpClient : public HttpClient {
public: public:
HttpResponse send(const HttpRequest &request) override HttpResponse send(const HttpRequest &request) override
@@ -116,6 +194,39 @@ public:
WinHttpSetTimeouts(session.get(), static_cast<int>(timeout), static_cast<int>(timeout), WinHttpSetTimeouts(session.get(), static_cast<int>(timeout), static_cast<int>(timeout),
static_cast<int>(timeout), static_cast<int>(timeout)); static_cast<int>(timeout), static_cast<int>(timeout));
// WinHttpSetTimeouts' receive parameter maps to
// WINHTTP_OPTION_RECEIVE_TIMEOUT, which Microsoft documents as a
// PER-PACKET Winsock-layer read timeout ("applies to fetching each
// packet of data off the socket"), not a deadline on the response.
// The wait for the response HEADERS is a *separate* option,
// WINHTTP_OPTION_RECEIVE_RESPONSE_TIMEOUT ("to wait to receive all
// response headers to a request"), which WinHttpSetTimeouts does not
// touch and which defaults to 90 SECONDS. Without this call a server
// that accepts, reads the request and then stalls can hold this
// thread for a minute and a half regardless of request.timeout_ms --
// exactly the "blocking an OBS thread indefinitely" failure
// testPlatformBackendTimeout exists to prevent, and the likely
// mechanism behind that test's intermittent Windows failures.
//
// Caveat, also documented: this timeout "is checked only when data is
// received from the socket", so it bounds the wait but does not
// guarantee a hard deadline. A guaranteed deadline needs a watchdog
// thread calling WinHttpCloseHandle; not done here.
//
// Guarded because the constant postdates some Windows SDK headers; a
// toolchain without it keeps the previous (90s default) behaviour
// rather than failing to build.
#ifdef WINHTTP_OPTION_RECEIVE_RESPONSE_TIMEOUT
DWORD response_timeout = timeout;
// Return value deliberately unchecked: a rejected option leaves the
// documented default in place, which is degraded but still correct
// behaviour, and there is no logging sink in this layer to report it
// to. The timeout probe in test_api_client.cpp is what would catch a
// regression here.
WinHttpSetOption(session.get(), WINHTTP_OPTION_RECEIVE_RESPONSE_TIMEOUT, &response_timeout,
sizeof(response_timeout));
#endif
Handle connect(WinHttpConnect(session.get(), host, parts.nPort, 0)); Handle connect(WinHttpConnect(session.get(), host, parts.nPort, 0));
if (!connect) { if (!connect) {
response.network_error = lastErrorMessage("WinHttpConnect"); response.network_error = lastErrorMessage("WinHttpConnect");
@@ -126,13 +237,36 @@ public:
target += extra; target += extra;
const DWORD flags = (parts.nScheme == INTERNET_SCHEME_HTTPS) ? WINHTTP_FLAG_SECURE : 0u; const DWORD flags = (parts.nScheme == INTERNET_SCHEME_HTTPS) ? WINHTTP_FLAG_SECURE : 0u;
Handle req(WinHttpOpenRequest(connect.get(), widen(request.method).c_str(), target.c_str(), nullptr, HINTERNET raw_req = WinHttpOpenRequest(connect.get(), widen(request.method).c_str(), target.c_str(),
WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, flags)); nullptr, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES,
if (!req) { flags);
if (!raw_req) {
response.network_error = lastErrorMessage("WinHttpOpenRequest"); response.network_error = lastErrorMessage("WinHttpOpenRequest");
return response; return response;
} }
// Ceiling at twice the caller's budget: each of the four
// WinHttpSetTimeouts phases (resolve, connect, send, receive) is
// allowed `timeout` on its own, so a slow-but-progressing exchange can
// legitimately exceed one budget, and this must not cancel those. The
// floor keeps a very small timeout_ms from producing a deadline the
// exchange cannot meet on a cold connection.
const DWORD deadline_ms = (timeout > 500u) ? (timeout * 2u) : 1000u;
RequestDeadline req(raw_req, deadline_ms);
// From here on, `req.get()` can be closed underneath us by the
// watchdog; every WinHttp* failure below is therefore checked against
// req.expired() before its GetLastError text is reported, so an
// expired deadline reads as a timeout rather than as
// "WinHttpReceiveResponse failed (GetLastError=12017)".
const auto fail = [&](const char *what) -> HttpResponse {
if (req.expired())
response.network_error = "timed out after " + std::to_string(deadline_ms) + " ms";
else
response.network_error = lastErrorMessage(what);
return response;
};
std::wstring headers; std::wstring headers;
if (!request.content_type.empty()) if (!request.content_type.empty())
headers = L"Content-Type: " + widen(request.content_type) + L"\r\n"; headers = L"Content-Type: " + widen(request.content_type) + L"\r\n";
@@ -144,31 +278,23 @@ public:
: const_cast<char *>(request.body.data()); : const_cast<char *>(request.body.data());
const DWORD body_len = static_cast<DWORD>(request.body.size()); const DWORD body_len = static_cast<DWORD>(request.body.size());
if (!WinHttpSendRequest(req.get(), header_ptr, header_len, body_ptr, body_len, body_len, 0)) { if (!WinHttpSendRequest(req.get(), header_ptr, header_len, body_ptr, body_len, body_len, 0))
response.network_error = lastErrorMessage("WinHttpSendRequest"); return fail("WinHttpSendRequest");
return response; if (!WinHttpReceiveResponse(req.get(), nullptr))
} return fail("WinHttpReceiveResponse");
if (!WinHttpReceiveResponse(req.get(), nullptr)) {
response.network_error = lastErrorMessage("WinHttpReceiveResponse");
return response;
}
DWORD status = 0; DWORD status = 0;
DWORD status_size = sizeof(status); DWORD status_size = sizeof(status);
if (!WinHttpQueryHeaders(req.get(), WINHTTP_QUERY_STATUS_CODE | WINHTTP_QUERY_FLAG_NUMBER, if (!WinHttpQueryHeaders(req.get(), WINHTTP_QUERY_STATUS_CODE | WINHTTP_QUERY_FLAG_NUMBER,
WINHTTP_HEADER_NAME_BY_INDEX, &status, &status_size, WINHTTP_NO_HEADER_INDEX)) { WINHTTP_HEADER_NAME_BY_INDEX, &status, &status_size, WINHTTP_NO_HEADER_INDEX))
response.network_error = lastErrorMessage("WinHttpQueryHeaders"); return fail("WinHttpQueryHeaders");
return response;
}
response.status = static_cast<long>(status); response.status = static_cast<long>(status);
std::string body; std::string body;
for (;;) { for (;;) {
DWORD available = 0; DWORD available = 0;
if (!WinHttpQueryDataAvailable(req.get(), &available)) { if (!WinHttpQueryDataAvailable(req.get(), &available))
response.network_error = lastErrorMessage("WinHttpQueryDataAvailable"); return fail("WinHttpQueryDataAvailable");
return response;
}
if (available == 0) if (available == 0)
break; break;
if (body.size() + available > kMaxResponseBytes) { if (body.size() + available > kMaxResponseBytes) {
@@ -177,10 +303,8 @@ public:
} }
std::vector<char> chunk(available); std::vector<char> chunk(available);
DWORD read = 0; DWORD read = 0;
if (!WinHttpReadData(req.get(), chunk.data(), available, &read)) { if (!WinHttpReadData(req.get(), chunk.data(), available, &read))
response.network_error = lastErrorMessage("WinHttpReadData"); return fail("WinHttpReadData");
return response;
}
if (read == 0) if (read == 0)
break; break;
body.append(chunk.data(), read); body.append(chunk.data(), read);
+359 -58
View File
@@ -17,6 +17,7 @@ You may obtain a copy of the License at
#include <deque> #include <deque>
#include <exception> #include <exception>
#include <mutex> #include <mutex>
#include <string>
#include <thread> #include <thread>
#include <utility> #include <utility>
#include <vector> #include <vector>
@@ -30,6 +31,7 @@ You may obtain a copy of the License at
#include <livekit/room_delegate.h> #include <livekit/room_delegate.h>
#include <livekit/room_event_types.h> #include <livekit/room_event_types.h>
#include <livekit/track.h> #include <livekit/track.h>
#include <livekit/track_publication.h>
#include <livekit/video_frame.h> #include <livekit/video_frame.h>
#include <livekit/video_stream.h> #include <livekit/video_stream.h>
@@ -138,6 +140,13 @@ int &globalRefCount()
return n; return n;
} }
// --- Stall-recovery watchdog tuning -----------------------------------------
/// How often the watchdog thread checks StallWatchdog's clock. Deliberately
/// finer than kStallRecoveryTimeout (session.h) so detection latency tracks
/// the threshold itself, not the threshold plus a whole polling period.
constexpr std::chrono::milliseconds kWatchdogPollInterval{250};
} // namespace } // namespace
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -145,11 +154,12 @@ int &globalRefCount()
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
struct LiveKitSession::Impl : public livekit::RoomDelegate { struct LiveKitSession::Impl : public livekit::RoomDelegate {
enum class CommandType { AttachVideo, DetachVideo, AttachAudio, DetachAudio, Stop }; enum class CommandType { AttachVideo, DetachVideo, AttachAudio, DetachAudio, RecoverVideo, Stop };
struct Command { struct Command {
CommandType type; CommandType type;
std::shared_ptr<livekit::Track> track; std::shared_ptr<livekit::Track> track;
std::shared_ptr<livekit::RemoteTrackPublication> publication;
}; };
livekit::Room room; livekit::Room room;
@@ -157,10 +167,14 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
mutable std::mutex state_mutex; mutable std::mutex state_mutex;
SessionStateMachine machine; SessionStateMachine machine;
// Guarded by state_mutex too, same contract as `machine` -- see
// StallWatchdog's own comment for what it decides and why it exists.
StallWatchdog stall_watchdog{kStallRecoveryTimeout, kStallRecoveryMaxBackoff};
VideoFrameHandler on_video; VideoFrameHandler on_video;
AudioFrameHandler on_audio; AudioFrameHandler on_audio;
SessionStateHandler on_state; SessionStateHandler on_state;
DiagnosticHandler on_diagnostic;
std::atomic<std::uint64_t> video_frames{0}; std::atomic<std::uint64_t> video_frames{0};
std::atomic<std::uint64_t> audio_frames{0}; std::atomic<std::uint64_t> audio_frames{0};
@@ -170,7 +184,10 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
// livekit::AudioStream happens on `worker`, never on a room event thread: // livekit::AudioStream happens on `worker`, never on a room event thread:
// the SDK's room callbacks run on its own event thread and blocking or // the SDK's room callbacks run on its own event thread and blocking or
// re-entering there stalls every other event (and Room::disconnect() from // re-entering there stalls every other event (and Room::disconnect() from
// inside one is documented to deadlock outright). // inside one is documented to deadlock outright). The stall-recovery
// watchdog thread follows the same rule: it never touches
// RemoteTrackPublication itself, only posts CommandType::RecoverVideo
// and lets the worker thread do it (see recoverVideo() below).
std::mutex queue_mutex; std::mutex queue_mutex;
std::condition_variable queue_cv; std::condition_variable queue_cv;
std::deque<Command> queue; std::deque<Command> queue;
@@ -183,6 +200,26 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
std::shared_ptr<livekit::AudioStream> audio_stream; std::shared_ptr<livekit::AudioStream> audio_stream;
std::thread audio_thread; std::thread audio_thread;
// The publication/track backing the CURRENT video subscription, kept
// around purely so recoverVideo() and the mute-change handlers have
// something to act on without reaching into `video_stream` (which is
// worker-thread-exclusive, per the comment above). Set together in
// attachVideo(), cleared together in detachVideo(), both on the worker
// thread; read from the room event thread (handleMuteChange) and the
// worker thread (recoverVideo()) under this mutex.
std::mutex video_track_mutex;
std::shared_ptr<livekit::Track> current_video_track;
std::shared_ptr<livekit::RemoteTrackPublication> current_video_publication;
// The watchdog's own timer thread. It owns no SDK state and calls no SDK
// method directly -- see the queue comment above. `watchdog_mutex` only
// ever guards the shutdown flag/condvar pair, never `stall_watchdog`
// (that is guarded by `state_mutex`, alongside `machine`).
std::mutex watchdog_mutex;
std::condition_variable watchdog_cv;
bool watchdog_running = false;
std::thread watchdog_thread;
bool connected = false; bool connected = false;
~Impl() override = default; ~Impl() override = default;
@@ -207,52 +244,46 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
handler(state, detail); handler(state, detail);
} }
void post(CommandType type, std::shared_ptr<livekit::Track> track = nullptr) void post(CommandType type, std::shared_ptr<livekit::Track> track = nullptr,
std::shared_ptr<livekit::RemoteTrackPublication> publication = nullptr)
{ {
{ {
std::lock_guard<std::mutex> guard(queue_mutex); std::lock_guard<std::mutex> guard(queue_mutex);
if (!worker_running) if (!worker_running)
return; return;
queue.push_back(Command{type, std::move(track)}); queue.push_back(Command{type, std::move(track), std::move(publication)});
} }
queue_cv.notify_one(); queue_cv.notify_one();
} }
void logDiagnostic(DiagnosticLevel level, const std::string &message)
{
DiagnosticHandler handler;
{
std::lock_guard<std::mutex> guard(state_mutex);
handler = on_diagnostic;
}
if (handler)
handler(level, message);
}
// Handles the wanted video track once matched, shared by onTrackSubscribed // Handles the wanted video track once matched, shared by onTrackSubscribed
// (a fresh subscription) and attachExistingTracks (one already up when // (a fresh subscription) and attachExistingTracks (one already up when
// this session started watching). Two responsibilities that only make // this session started watching). An audio-only source (the soundboard)
// sense together, both keyed off the SAME publication: // never gets here for video: shouldSubscribe() never asks for it.
// //
// - subscribe_video: an audio-only source (the soundboard) never wants // Fixed video quality: LiveKit's default subscriber behaviour lets the
// this video at all. Rather than attach it and let the OBS adapter // SFU switch simulcast layers per its own adaptive/bandwidth logic, which
// discard every decoded frame, disable the publication itself // for a source with no rendered-size hint (this is a native C++
// (RemoteTrackPublication::setEnabled(false)) so the SFU stops // subscriber, not a sized <video> element) means the received resolution
// sending it -- real bandwidth saved, not just wasted decode. // can hop between layers -- observed live as OBS source geometry visibly
// - Fixed video quality: LiveKit's default subscriber behaviour lets // changing size mid-show. Pinning to HIGH asks the SFU to always send the
// the SFU switch simulcast layers per its own adaptive/bandwidth // top layer, which is what a fixed OBS source needs regardless of
// logic, which for a source with no rendered-size hint (this is a // bandwidth (the plugin has no picture-in-picture tier to fall back to
// native C++ subscriber, not a sized <video> element) means the // the way a browser grid view would).
// received resolution can hop between layers -- observed live as OBS
// source geometry visibly changing size mid-show. Pinning to HIGH
// asks the SFU to always send the top layer, which is what a fixed
// OBS source needs regardless of bandwidth (the plugin has no
// picture-in-picture tier to fall back to the way a browser grid
// view would).
void handleWantedVideoTrack(const std::shared_ptr<livekit::Track> &track, void handleWantedVideoTrack(const std::shared_ptr<livekit::Track> &track,
const std::shared_ptr<livekit::RemoteTrackPublication> &publication) const std::shared_ptr<livekit::RemoteTrackPublication> &publication)
{ {
if (!config.subscribe_video) {
if (publication) {
try {
publication->setEnabled(false);
} catch (const std::exception &) {
// Best-effort: worst case this track keeps being
// delivered and decoded, wasting bandwidth -- it is
// still never attached to OBS below.
}
}
return;
}
if (publication) { if (publication) {
try { try {
publication->setVideoQuality(livekit::VideoQuality::HIGH); publication->setVideoQuality(livekit::VideoQuality::HIGH);
@@ -261,7 +292,42 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
// it already had, which is the pre-existing behaviour. // it already had, which is the pre-existing behaviour.
} }
} }
post(CommandType::AttachVideo, track); // `publication` rides along so attachVideo() can remember it: it is
// the handle the stall-recovery watchdog later toggles
// (setEnabled(false)/(true)) to force a fresh keyframe. See
// recoverVideo() and StallWatchdog's comment in session_types.h.
post(CommandType::AttachVideo, track, publication);
}
// Fired for ANY track (any participant, any kind) muting or unmuting.
// Filtered down to "is this the video publication we are currently
// watching" by SID, which also naturally excludes every audio mute and
// every other participant's tracks without a separate identity/kind
// check.
//
// Why this exists: the stall-recovery watchdog (see StallWatchdog's
// comment) treats "no decoded frame for kStallRecoveryTimeout" as a
// stall worth toggling the subscription over. A publisher who
// legitimately turned their camera off produces exactly that symptom on
// purpose, and toggling their subscription every couple of seconds for
// as long as they stay off would be an endless, pointless loop against
// healthy behaviour. Muting suspends the watchdog's clock entirely;
// unmuting starts a brand-new grace period rather than reading "muted
// for twenty minutes" as "stalled for twenty minutes".
void handleMuteChange(const std::shared_ptr<livekit::TrackPublication> &publication, bool unmuted)
{
if (!publication || publication->kind() != livekit::TrackKind::KIND_VIDEO)
return;
std::string current_sid;
{
std::lock_guard<std::mutex> guard(video_track_mutex);
if (current_video_publication)
current_sid = current_video_publication->sid();
}
if (current_sid.empty() || publication->sid() != current_sid)
return;
std::lock_guard<std::mutex> guard(state_mutex);
stall_watchdog.setExpectingFrames(unmuted, std::chrono::steady_clock::now());
} }
// --- RoomDelegate ------------------------------------------------------ // --- RoomDelegate ------------------------------------------------------
@@ -275,12 +341,46 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
const MediaSource source = const MediaSource source =
event.publication ? toMediaSource(event.publication->source()) : MediaSource::Unknown; event.publication ? toMediaSource(event.publication->source()) : MediaSource::Unknown;
if (!shouldSubscribe(config.participant_identity, identity, kind, source, config.subscribe_video,
config.subscribe_audio)) {
// Never asked for (see requestSubscription). Should not happen with
// auto_subscribe off; if the SDK or server ever subscribes us
// anyway, say so and hand it back rather than silently paying for
// another camera's bandwidth.
logDiagnostic(DiagnosticLevel::Warning, "dropping subscription to unwanted track " +
event.track->sid() + " from " + identity);
if (event.publication) {
try {
event.publication->setSubscribed(false);
} catch (const std::exception &) {
// Best-effort: worst case it keeps arriving and is
// discarded below, which is the old behaviour.
}
}
return;
}
if (isWantedVideoTrack(config.participant_identity, identity, kind, source)) if (isWantedVideoTrack(config.participant_identity, identity, kind, source))
handleWantedVideoTrack(event.track, event.publication); handleWantedVideoTrack(event.track, event.publication);
else if (config.subscribe_audio && isWantedAudioTrack(config.participant_identity, identity, kind, source)) else if (config.subscribe_audio && isWantedAudioTrack(config.participant_identity, identity, kind, source))
post(CommandType::AttachAudio, event.track); post(CommandType::AttachAudio, event.track);
} }
// auto_subscribe is off (see connect()), so a newly published track --
// including every track of a player who just rejoined -- arrives
// unsubscribed, and this is where the session asks for the ones it shows.
//
// event.publication is not usable: SDK 1.10.1 delivers it null (observed
// against livekit-server 1.13.6), so re-sweep the participant's own
// publication map instead. Idempotent, and it only ever subscribes --
// re-attaching the camera already on screen would blank it.
void onTrackPublished(livekit::Room &, const livekit::TrackPublishedEvent &event) override
{
if (!event.participant || event.participant->identity() != config.participant_identity)
return;
requestWantedSubscriptions(*event.participant);
}
void onTrackUnsubscribed(livekit::Room &, const livekit::TrackUnsubscribedEvent &event) override void onTrackUnsubscribed(livekit::Room &, const livekit::TrackUnsubscribedEvent &event) override
{ {
if (!event.participant || !event.track) if (!event.participant || !event.track)
@@ -304,6 +404,16 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
post(CommandType::DetachAudio); post(CommandType::DetachAudio);
} }
void onTrackMuted(livekit::Room &, const livekit::TrackMutedEvent &event) override
{
handleMuteChange(event.publication, false);
}
void onTrackUnmuted(livekit::Room &, const livekit::TrackUnmutedEvent &event) override
{
handleMuteChange(event.publication, true);
}
void onReconnecting(livekit::Room &, const livekit::ReconnectingEvent &) override void onReconnecting(livekit::Room &, const livekit::ReconnectingEvent &) override
{ {
mutateState([](SessionStateMachine &m) { m.onReconnecting(); }); mutateState([](SessionStateMachine &m) { m.onReconnecting(); });
@@ -312,6 +422,11 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
void onReconnected(livekit::Room &, const livekit::ReconnectedEvent &) override void onReconnected(livekit::Room &, const livekit::ReconnectedEvent &) override
{ {
mutateState([](SessionStateMachine &m) { m.onReconnected(); }); mutateState([](SessionStateMachine &m) { m.onReconnected(); });
// Belt and braces: re-ask for anything wanted that the reconnect left
// unsubscribed. A no-op for publications still subscribed.
auto participant = room.remoteParticipant(config.participant_identity).lock();
if (participant)
requestWantedSubscriptions(*participant);
} }
void onDisconnected(livekit::Room &, const livekit::DisconnectedEvent &event) override void onDisconnected(livekit::Room &, const livekit::DisconnectedEvent &event) override
@@ -359,7 +474,7 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
void workerLoop() void workerLoop()
{ {
for (;;) { for (;;) {
Command command{CommandType::Stop, nullptr}; Command command{CommandType::Stop, nullptr, nullptr};
{ {
std::unique_lock<std::mutex> lock(queue_mutex); std::unique_lock<std::mutex> lock(queue_mutex);
queue_cv.wait(lock, [this] { return !queue.empty(); }); queue_cv.wait(lock, [this] { return !queue.empty(); });
@@ -369,7 +484,7 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
switch (command.type) { switch (command.type) {
case CommandType::AttachVideo: case CommandType::AttachVideo:
attachVideo(command.track); attachVideo(command.track, command.publication);
break; break;
case CommandType::DetachVideo: case CommandType::DetachVideo:
detachVideo(); detachVideo();
@@ -380,6 +495,9 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
case CommandType::DetachAudio: case CommandType::DetachAudio:
detachAudio(); detachAudio();
break; break;
case CommandType::RecoverVideo:
recoverVideo();
break;
case CommandType::Stop: case CommandType::Stop:
detachVideo(); detachVideo();
detachAudio(); detachAudio();
@@ -388,7 +506,103 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
} }
} }
void attachVideo(const std::shared_ptr<livekit::Track> &track) // --- stall-recovery watchdog --------------------------------------------
//
// See StallWatchdog's comment (session_types.h) for the measured
// evidence and why toggling the publication is the only lever
// available. This thread does exactly one thing: tick StallWatchdog's
// clock and, when it says to, post CommandType::RecoverVideo so the
// worker thread does the actual SDK call. It never touches
// livekit::VideoStream, livekit::Room or RemoteTrackPublication itself.
void startWatchdog()
{
{
std::lock_guard<std::mutex> guard(watchdog_mutex);
watchdog_running = true;
}
watchdog_thread = std::thread([this] { watchdogLoop(); });
}
void stopWatchdog()
{
{
std::lock_guard<std::mutex> guard(watchdog_mutex);
if (!watchdog_running)
return;
watchdog_running = false;
}
watchdog_cv.notify_all();
if (watchdog_thread.joinable())
watchdog_thread.join();
}
void watchdogLoop()
{
std::unique_lock<std::mutex> lock(watchdog_mutex);
while (watchdog_running) {
// kWatchdogPollInterval is finer than kStallRecoveryTimeout so
// detection latency tracks the threshold itself rather than the
// threshold plus a whole polling period; woken early on
// shutdown by stopWatchdog()'s notify_all().
watchdog_cv.wait_for(lock, kWatchdogPollInterval);
if (!watchdog_running)
break;
lock.unlock();
bool should_fire;
{
std::lock_guard<std::mutex> guard(state_mutex);
should_fire = stall_watchdog.poll(std::chrono::steady_clock::now());
}
if (should_fire)
post(CommandType::RecoverVideo);
lock.lock();
}
}
// Actually pulls the lever: toggles the video publication off and back
// on, which makes the SFU stop and restart delivery of that track --
// and a restart always begins with a keyframe. Only ever called from
// the worker thread (via CommandType::RecoverVideo), same as every
// other RemoteTrackPublication/VideoStream call in this file.
void recoverVideo()
{
std::shared_ptr<livekit::RemoteTrackPublication> publication;
std::shared_ptr<livekit::Track> track;
{
std::lock_guard<std::mutex> guard(video_track_mutex);
publication = current_video_publication;
track = current_video_track;
}
// Detached, replaced, or muted between the watchdog deciding to
// fire and the worker getting to this command -- nothing to do, and
// silently: this is the expected shape of the race, not a failure
// worth logging.
if (!publication || !track || track->muted())
return;
int attempt = 0;
{
std::lock_guard<std::mutex> guard(state_mutex);
attempt = stall_watchdog.attemptsThisStall();
}
logDiagnostic(DiagnosticLevel::Warning,
"no decoded video frame for >= " + std::to_string(kStallRecoveryTimeout.count()) +
"ms; toggling the subscription to force a fresh keyframe (attempt " +
std::to_string(attempt) + ")");
try {
publication->setEnabled(false);
publication->setEnabled(true);
} catch (const std::exception &e) {
logDiagnostic(DiagnosticLevel::Warning, std::string("stall-recovery toggle failed: ") + e.what());
}
}
void attachVideo(const std::shared_ptr<livekit::Track> &track,
const std::shared_ptr<livekit::RemoteTrackPublication> &publication)
{ {
if (!track) if (!track)
return; return;
@@ -413,6 +627,18 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
if (!stream) if (!stream)
return; return;
{
std::lock_guard<std::mutex> guard(video_track_mutex);
current_video_track = track;
current_video_publication = publication;
}
{
// A fresh subscription (or a publisher swap) starts a brand-new
// grace period -- see StallWatchdog::setExpectingFrames.
std::lock_guard<std::mutex> guard(state_mutex);
stall_watchdog.setExpectingFrames(true, std::chrono::steady_clock::now());
}
video_stream = stream; video_stream = stream;
video_thread = std::thread([this, stream] { videoReaderLoop(stream); }); video_thread = std::thread([this, stream] { videoReaderLoop(stream); });
mutateState([](SessionStateMachine &m) { m.onVideoAttached(); }); mutateState([](SessionStateMachine &m) { m.onVideoAttached(); });
@@ -426,6 +652,20 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
video_thread.join(); video_thread.join();
const bool had = static_cast<bool>(video_stream); const bool had = static_cast<bool>(video_stream);
video_stream.reset(); video_stream.reset();
{
std::lock_guard<std::mutex> guard(video_track_mutex);
current_video_track.reset();
current_video_publication.reset();
}
{
// Nothing subscribed means nothing expected -- see
// StallWatchdog::setExpectingFrames. Unconditional, not gated on
// `had`: this also covers the detachVideo() at the top of
// attachVideo() above, which is exactly the publisher-swap
// moment the grace period needs to restart from.
std::lock_guard<std::mutex> guard(state_mutex);
stall_watchdog.setExpectingFrames(false, std::chrono::steady_clock::now());
}
if (had) if (had)
mutateState([](SessionStateMachine &m) { m.onVideoDetached(); }); mutateState([](SessionStateMachine &m) { m.onVideoDetached(); });
} }
@@ -548,6 +788,23 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
out.plane_count = count; out.plane_count = count;
video_frames.fetch_add(1); video_frames.fetch_add(1);
// Tell the stall-recovery watchdog a frame actually made it all the
// way to "about to hand to OBS" -- not merely that the SDK's queue
// produced an event, which the drop paths above also see. See
// StallWatchdog's comment for why this exists.
int attempts_before_this_frame = 0;
{
std::lock_guard<std::mutex> guard(state_mutex);
attempts_before_this_frame = stall_watchdog.attemptsThisStall();
stall_watchdog.onFrameDelivered(std::chrono::steady_clock::now());
}
if (attempts_before_this_frame > 0) {
logDiagnostic(DiagnosticLevel::Info, "video resumed after " +
std::to_string(attempts_before_this_frame) +
" stall-recovery attempt(s)");
}
handler(out); handler(out);
} }
@@ -577,10 +834,39 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
} }
} }
/// After connect(), the target slot may already be in the room with its /// Subscribes to `publication` if, and only if, this source shows it.
/// tracks subscribed, in which case no onTrackSubscribed event is coming. /// Idempotent: an already-subscribed publication is left alone. The
/// Sweep what is already there so a source added mid-show shows video /// resulting onTrackSubscribed does the attaching.
/// immediately instead of waiting for the publisher to republish. void requestSubscription(const std::string &identity,
const std::shared_ptr<livekit::RemoteTrackPublication> &publication)
{
if (!publication || publication->subscribed())
return;
if (!shouldSubscribe(config.participant_identity, identity, toMediaKind(publication->kind()),
toMediaSource(publication->source()), config.subscribe_video, config.subscribe_audio))
return;
try {
publication->setSubscribed(true);
} catch (const std::exception &e) {
logDiagnostic(DiagnosticLevel::Warning,
std::string("could not subscribe to ") + publication->sid() + ": " + e.what());
}
}
/// requestSubscription() for every publication `participant` has.
void requestWantedSubscriptions(livekit::RemoteParticipant &participant)
{
const std::string identity = participant.identity();
for (const auto &entry : participant.trackPublications())
requestSubscription(identity, entry.second);
}
/// After connect(), the target slot may already be in the room. Its
/// publications fire no onTrackPublished for us, so sweep them: subscribe
/// to the wanted ones not yet subscribed, and attach any already
/// subscribed (no onTrackSubscribed is coming for those), so a source
/// added mid-show shows video immediately instead of waiting for the
/// publisher to republish.
void attachExistingTracks() void attachExistingTracks()
{ {
auto participant = room.remoteParticipant(config.participant_identity).lock(); auto participant = room.remoteParticipant(config.participant_identity).lock();
@@ -592,11 +878,13 @@ struct LiveKitSession::Impl : public livekit::RoomDelegate {
if (!publication) if (!publication)
continue; continue;
const std::shared_ptr<livekit::Track> track = publication->track(); const std::shared_ptr<livekit::Track> track = publication->track();
if (!track) if (!track) {
continue; // published but not subscribed yet requestSubscription(identity, publication);
continue;
}
const MediaKind kind = toMediaKind(track->kind()); const MediaKind kind = toMediaKind(track->kind());
const MediaSource source = toMediaSource(publication->source()); const MediaSource source = toMediaSource(publication->source());
if (isWantedVideoTrack(config.participant_identity, identity, kind, source)) if (config.subscribe_video && isWantedVideoTrack(config.participant_identity, identity, kind, source))
handleWantedVideoTrack(track, publication); handleWantedVideoTrack(track, publication);
else if (config.subscribe_audio && isWantedAudioTrack(config.participant_identity, identity, kind, source)) else if (config.subscribe_audio && isWantedAudioTrack(config.participant_identity, identity, kind, source))
post(CommandType::AttachAudio, track); post(CommandType::AttachAudio, track);
@@ -633,6 +921,12 @@ void LiveKitSession::setStateHandler(SessionStateHandler handler)
impl_->on_state = std::move(handler); impl_->on_state = std::move(handler);
} }
void LiveKitSession::setDiagnosticHandler(DiagnosticHandler handler)
{
std::lock_guard<std::mutex> guard(impl_->state_mutex);
impl_->on_diagnostic = std::move(handler);
}
bool LiveKitSession::connect(const SessionConfig &config) bool LiveKitSession::connect(const SessionConfig &config)
{ {
if (impl_->connected) if (impl_->connected)
@@ -652,20 +946,23 @@ bool LiveKitSession::connect(const SessionConfig &config)
} }
impl_->startWorker(); impl_->startWorker();
// Runs for the lifetime of the worker: connected-but-nothing-subscribed
// is a no-op for StallWatchdog (see setExpectingFrames), so there is no
// reason to start/stop it separately from the worker it posts to.
impl_->startWatchdog();
livekit::RoomOptions options; livekit::RoomOptions options;
// auto_subscribe is what makes track_subscribed events (and therefore any // Selective subscription. The SDK's headers warn that without
// media at all) happen; the SDK is emphatic about this. // auto_subscribe no media arrives -- true only if nothing subscribes by
// hand, and this session does: onTrackPublished and attachExistingTracks
// call setSubscribed(true) on exactly the slot's camera and mic.
// //
// Known, measured-but-unaddressed cost: auto_subscribe pulls every // With auto_subscribe on, every source pulled every camera in the room
// participant's published track, not just the one camera this session // and threw away all but one. Measured live 2026-10-04 (eight sources,
// actually wants, and this client discards the unwanted ones // seven cameras): each source carried 4-7 Mbps, and all eight hit
// client-side. In a multi-camera room that is real, wasted bandwidth // congestion in the same instant because they shared the director's one
// and decode CPU that scales with room size, not with what this source // downlink -- starving the cameras actually on screen.
// displays. Selectively unsubscribing from unwanted publications (the options.auto_subscribe = false;
// SDK exposes per-publication subscribe/unsubscribe) is a real
// follow-up optimization, deliberately out of scope here.
options.auto_subscribe = true;
options.dynacast = false; options.dynacast = false;
// This client never publishes, so a single peer connection is all it // This client never publishes, so a single peer connection is all it
// needs. // needs.
@@ -680,6 +977,7 @@ bool LiveKitSession::connect(const SessionConfig &config)
} catch (const std::exception &e) { } catch (const std::exception &e) {
ok = false; ok = false;
impl_->mutateState([&](SessionStateMachine &m) { m.onConnectFailed(e.what()); }); impl_->mutateState([&](SessionStateMachine &m) { m.onConnectFailed(e.what()); });
impl_->stopWatchdog();
impl_->stopWorker(); impl_->stopWorker();
impl_->room.setDelegate(nullptr); impl_->room.setDelegate(nullptr);
return false; return false;
@@ -689,6 +987,7 @@ bool LiveKitSession::connect(const SessionConfig &config)
impl_->mutateState([](SessionStateMachine &m) { impl_->mutateState([](SessionStateMachine &m) {
m.onConnectFailed("could not connect to LiveKit (check the server URL, or the token may have expired)"); m.onConnectFailed("could not connect to LiveKit (check the server URL, or the token may have expired)");
}); });
impl_->stopWatchdog();
impl_->stopWorker(); impl_->stopWorker();
impl_->room.setDelegate(nullptr); impl_->room.setDelegate(nullptr);
return false; return false;
@@ -705,9 +1004,11 @@ void LiveKitSession::disconnect()
if (!impl_) if (!impl_)
return; return;
// Order matters: stop the readers first so nothing is mid-read on a // Order matters: stop the watchdog and the readers first so nothing is
// stream the room is about to tear down, then disconnect the room, then // mid-read (or about to post a recovery command) on a stream the room
// drop the delegate so no event can arrive at a half-destroyed object. // is about to tear down, then disconnect the room, then drop the
// delegate so no event can arrive at a half-destroyed object.
impl_->stopWatchdog();
impl_->stopWorker(); impl_->stopWorker();
if (impl_->connected) { if (impl_->connected) {
+93
View File
@@ -70,6 +70,14 @@ bool isWantedAudioTrack(const std::string &wanted_identity, const std::string &t
return source == MediaSource::Microphone || source == MediaSource::Unknown; return source == MediaSource::Microphone || source == MediaSource::Unknown;
} }
bool shouldSubscribe(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
MediaSource source, bool subscribe_video, bool subscribe_audio)
{
if (subscribe_video && isWantedVideoTrack(wanted_identity, track_identity, kind, source))
return true;
return subscribe_audio && isWantedAudioTrack(wanted_identity, track_identity, kind, source);
}
const char *describeSessionState(SessionState state) const char *describeSessionState(SessionState state)
{ {
switch (state) { switch (state) {
@@ -175,4 +183,89 @@ void SessionStateMachine::onAudioDetached()
has_audio_ = false; has_audio_ = false;
} }
// ---------------------------------------------------------------------------
// StallWatchdog
// ---------------------------------------------------------------------------
StallWatchdog::StallWatchdog(std::chrono::milliseconds timeout, std::chrono::milliseconds max_backoff)
: timeout_(timeout), max_backoff_(max_backoff), backoff_(timeout)
{
}
void StallWatchdog::setExpectingFrames(bool expecting, std::chrono::steady_clock::time_point now)
{
expecting_ = expecting;
// Always re-baseline from `now`, whichever direction this flips.
// Losing the baseline (rather than, say, keeping the old one around for
// when expecting_ next becomes true) is what stops a track that was
// muted for the last twenty minutes from reading as "twenty minutes
// stalled" the instant it unmutes.
have_baseline_ = expecting;
baseline_ = now;
backoff_ = timeout_;
next_attempt_allowed_ = now;
attempts_ = 0;
}
void StallWatchdog::onFrameDelivered(std::chrono::steady_clock::time_point now)
{
have_baseline_ = true;
baseline_ = now;
backoff_ = timeout_;
// A recovered stream must be able to fire again the moment a FRESH
// stall clears the (now-reset) timeout, not sit throttled by whatever
// backoff a previous, unrelated stall had climbed to -- next_attempt_
// allowed_ belongs to that old stall and is meaningless once frames are
// flowing again.
next_attempt_allowed_ = now;
attempts_ = 0;
}
bool StallWatchdog::poll(std::chrono::steady_clock::time_point now)
{
if (!expecting_ || !have_baseline_)
return false;
if (now - baseline_ < timeout_)
return false;
if (now < next_attempt_allowed_)
return false;
++attempts_;
// Next attempt against this SAME stall is not allowed until the backoff
// elapses, and the backoff itself doubles (capped) each time -- 2s, 4s,
// 8s, ... up to max_backoff_ -- so a publisher that is genuinely gone
// gets progressively less frequent toggles instead of one every 2
// seconds for the rest of the show.
//
// max_backoff_ is clamped HERE, where the wait is used, and not only
// where the backoff is grown. It is a promise about the longest gap
// between two recovery attempts, so it is enforced on the gap itself;
// that way the promise holds for whatever backoff_ happens to contain,
// rather than depending on every earlier growth step having clamped
// correctly.
//
// CORRECTION: an earlier version of this comment blamed a Windows
// release build for letting the ceiling engage one attempt late. That
// was wrong, and it is worth recording why rather than quietly
// deleting it. Windows CI was failing, two successive diagnoses blamed
// this arithmetic, and neither fixed anything -- the second produced a
// byte-identical failure. Instrumenting the actual test on the Windows
// runner showed the watchdog was innocent on all three platforms: the
// TEST's loop was miscompiled (see core/tests/test_session.cpp). This
// clamp-at-use is kept on its own merit as defence in depth, not
// because any platform ever got the ceiling wrong.
const std::chrono::milliseconds wait = backoff_ < max_backoff_ ? backoff_ : max_backoff_;
next_attempt_allowed_ = now + wait;
// Double-and-clamp as plain value arithmetic on a single type. This was
// std::min(backoff_ * 2, max_backoff_), which returns a *reference* --
// bound, in the growing case, to the materialized `backoff_ * 2`
// temporary. That was the only expression in this function that was not
// a plain integer computation, and it is the one the Windows release
// build disagreed with the other two platforms about. Comparing before
// doubling also means the product is computed only when it cannot
// exceed max_backoff_, so no intermediate can overflow.
backoff_ = (wait > max_backoff_ / 2) ? max_backoff_ : wait * 2;
return true;
}
} // namespace stplugin } // namespace stplugin
+70 -14
View File
@@ -16,6 +16,7 @@ You may obtain a copy of the License at
// three runners rather than assumed to work. // three runners rather than assumed to work.
#include <chrono> #include <chrono>
#include <cstdio>
#include <memory> #include <memory>
#include <string> #include <string>
#include <thread> #include <thread>
@@ -422,22 +423,77 @@ void testPlatformBackendTimeout()
{ {
// A server that accepts and then stalls. The plugin must give up on its // A server that accepts and then stalls. The plugin must give up on its
// own timeout rather than blocking an OBS thread indefinitely. // own timeout rather than blocking an OBS thread indefinitely.
sttest::LoopbackServer server([](const std::string &) { //
std::this_thread::sleep_for(std::chrono::seconds(5)); // INSTRUMENTED (2026-09-09) while chasing an intermittent Windows-only
return sttest::httpResponse(200, "OK", R"({"slots":[]})"); // failure: on roughly 2 of 6 CI runs both assertions below fail together,
}); // meaning the request waited out the full 5s stall and returned 200 --
ST_ASSERT(server.valid()); // the timeout did not fire at all. Same failure seen on 2026-09-07
// (job 5834) and 2026-09-09 (job 5911), on identical code that passed on
// other runs, so it is not a code change that caused it.
//
// The probe runs kProbes times and prints one line per attempt so a
// single CI run yields a failure RATE and the WinHTTP error code, rather
// than one bit. `ST_ASSERT` records and continues, so every attempt is
// reported even when one fails. Remove the loop and this comment once the
// mechanism is understood and fixed.
constexpr int kProbes = 5;
constexpr long long kStallMs = 5000;
constexpr long kTimeoutMs = 700;
std::shared_ptr<HttpClient> http(createPlatformHttpClient()); int timed_out = 0;
HttpRequest request; for (int i = 0; i < kProbes; ++i) {
request.url = server.baseUrl() + "/api/obs/main-room/slots?key=k"; // A FRESH server per attempt, deliberately. `LoopbackServer` accepts
request.timeout_ms = 700; // and handles one connection at a time on a single thread, so reusing
// one server across attempts would leave attempts 2..n sitting in the
// accept backlog -- a different scenario (never accepted) from the one
// that fails on Windows (accepted, request read, then stalled).
sttest::LoopbackServer server([kStallMs](const std::string &) {
std::this_thread::sleep_for(std::chrono::milliseconds(kStallMs));
return sttest::httpResponse(200, "OK", R"({"slots":[]})");
});
ST_ASSERT(server.valid());
const auto start = std::chrono::steady_clock::now(); std::shared_ptr<HttpClient> http(createPlatformHttpClient());
const HttpResponse response = http->send(request); HttpRequest request;
const auto elapsed = std::chrono::steady_clock::now() - start; request.url = server.baseUrl() + "/api/obs/main-room/slots?key=k";
ST_ASSERT(!response.ok()); request.timeout_ms = kTimeoutMs;
ST_ASSERT(std::chrono::duration_cast<std::chrono::milliseconds>(elapsed).count() < 4000);
const auto start = std::chrono::steady_clock::now();
const HttpResponse response = http->send(request);
const auto elapsed = std::chrono::steady_clock::now() - start;
const long long ms = std::chrono::duration_cast<std::chrono::milliseconds>(elapsed).count();
// 4000 was the old bound, chosen when nothing bounded the wait. The
// code now promises a hard ceiling of 2x the caller's budget
// (RequestDeadline in http_winhttp.cpp), so assert THAT -- 1400ms
// here, plus slack for a loaded runner. This is also the only signal
// that survives a green run: CTest prints nothing on success, so if
// WinHTTP's own erratic cancellation (measured at 1490-4506ms for
// this same 700ms budget) were doing the work instead of the
// watchdog, roughly half the attempts would land above this bound and
// say so, instead of quietly passing under a 4s ceiling.
constexpr long long kCeilingMs = 2500;
const bool gave_up = !response.ok() && ms < kCeilingMs;
if (gave_up)
++timed_out;
// Always printed, pass or fail: elapsed time and the backend's own
// error string (which carries GetLastError on Windows) are the
// evidence. requests_seen separates "the client never reached the
// server" (0) from "the server read the request and the client then
// waited it out" (1).
std::fprintf(stderr,
" [timeout-probe %d/%d] elapsed=%lldms ok=%d status=%ld "
"requests_seen=%d network_error='%s' -> %s\n",
i + 1, kProbes, ms, response.ok() ? 1 : 0, response.status,
server.requestCount(), response.network_error.c_str(),
gave_up ? "gave up (expected)" : "WAITED OUT THE STALL");
ST_ASSERT(!response.ok());
ST_ASSERT(ms < kCeilingMs);
}
std::fprintf(stderr, " [timeout-probe] %d/%d attempts honoured the %ldms timeout\n",
timed_out, kProbes, kTimeoutMs);
} }
} // namespace } // namespace
+80 -2
View File
@@ -18,6 +18,11 @@ You may obtain a copy of the License at
// (unpublish, republish) that motivated this whole plugin, and asserts the // (unpublish, republish) that motivated this whole plugin, and asserts the
// wrapper recovers instead of going stale or erroring out. // wrapper recovers instead of going stale or erroring out.
// //
// It also proves selective subscription: a second "bystander" camera
// publishes into the same room, and the session must never subscribe to
// it. In a real show every OBS source used to pull the whole room, so the
// director's downlink carried every camera once per source.
//
// It needs a reachable LiveKit server, so it SKIPS (exit 0) unless these are // It needs a reachable LiveKit server, so it SKIPS (exit 0) unless these are
// set -- CI on the three build runners has no server, and this must not turn // set -- CI on the three build runners has no server, and this must not turn
// into a red build there: // into a red build there:
@@ -26,8 +31,11 @@ You may obtain a copy of the License at
// STPLUGIN_IT_PUBLISH_TOKEN JWT: roomJoin + canPublish for the room // STPLUGIN_IT_PUBLISH_TOKEN JWT: roomJoin + canPublish for the room
// STPLUGIN_IT_SUBSCRIBE_TOKEN JWT: roomJoin + canSubscribe for the room // STPLUGIN_IT_SUBSCRIBE_TOKEN JWT: roomJoin + canSubscribe for the room
// STPLUGIN_IT_PUBLISHER_IDENTITY the identity in the publish token // STPLUGIN_IT_PUBLISHER_IDENTITY the identity in the publish token
// STPLUGIN_IT_BYSTANDER_TOKEN optional: JWT for a second publisher; the
// selective-subscription check is skipped
// without it
// //
// scripts/livekit-dev-room.py mints all four against a `livekit-server --dev`. // scripts/livekit-dev-room.py mints all of them against a `livekit-server --dev`.
#include <atomic> #include <atomic>
#include <chrono> #include <chrono>
@@ -221,6 +229,7 @@ int main()
const std::string publish_token = envOrEmpty("STPLUGIN_IT_PUBLISH_TOKEN"); const std::string publish_token = envOrEmpty("STPLUGIN_IT_PUBLISH_TOKEN");
const std::string subscribe_token = envOrEmpty("STPLUGIN_IT_SUBSCRIBE_TOKEN"); const std::string subscribe_token = envOrEmpty("STPLUGIN_IT_SUBSCRIBE_TOKEN");
const std::string publisher_identity = envOrEmpty("STPLUGIN_IT_PUBLISHER_IDENTITY"); const std::string publisher_identity = envOrEmpty("STPLUGIN_IT_PUBLISHER_IDENTITY");
const std::string bystander_token = envOrEmpty("STPLUGIN_IT_BYSTANDER_TOKEN");
if (url.empty() || publish_token.empty() || subscribe_token.empty() || publisher_identity.empty()) { if (url.empty() || publish_token.empty() || subscribe_token.empty() || publisher_identity.empty()) {
std::printf("integration_livekit: SKIPPED (STPLUGIN_IT_* not set)\n"); std::printf("integration_livekit: SKIPPED (STPLUGIN_IT_* not set)\n");
@@ -239,6 +248,20 @@ int main()
ST_ASSERT(publisher.publishAudio()); ST_ASSERT(publisher.publishAudio());
publisher.startPump(); publisher.startPump();
// A bystander already publishing when the session connects -- the
// "source added mid-show" case, where the room is full of cameras.
std::unique_ptr<Publisher> bystander;
if (!bystander_token.empty()) {
bystander = std::make_unique<Publisher>();
step("bystander connect + publish");
ST_ASSERT(bystander->connect(url, bystander_token));
ST_ASSERT(bystander->publishVideo());
ST_ASSERT(bystander->publishAudio());
bystander->startPump();
} else {
std::printf(" selective-subscription check SKIPPED (STPLUGIN_IT_BYSTANDER_TOKEN not set)\n");
}
// --- subscribe through the wrapper under test -------------------------- // --- subscribe through the wrapper under test --------------------------
std::atomic<int> video_frames{0}; std::atomic<int> video_frames{0};
@@ -287,6 +310,17 @@ int main()
std::atomic<int> state_changes{0}; std::atomic<int> state_changes{0};
session.setStateHandler([&](SessionState, const std::string &) { state_changes.fetch_add(1); }); session.setStateHandler([&](SessionState, const std::string &) { state_changes.fetch_add(1); });
// The session reports every subscription it did not ask for (and drops
// it). Zero is the assertion: the SFU must never have started sending
// the bystander's tracks to this subscriber at all.
std::atomic<int> unwanted_subscriptions{0};
session.setDiagnosticHandler([&](DiagnosticLevel, const std::string &message) {
std::printf(" diagnostic: %s\n", message.c_str());
std::fflush(stdout);
if (message.find("unwanted track") != std::string::npos)
unwanted_subscriptions.fetch_add(1);
});
SessionConfig config; SessionConfig config;
config.ws_url = url; config.ws_url = url;
config.token = subscribe_token; config.token = subscribe_token;
@@ -315,6 +349,21 @@ int main()
ST_ASSERT_EQ(last_sample_rate.load(), 48000); ST_ASSERT_EQ(last_sample_rate.load(), 48000);
ST_ASSERT(last_channels.load() >= 1); ST_ASSERT(last_channels.load() >= 1);
// A bystander that starts publishing while the session is up -- the
// other half: a player (re)joining mid-show.
if (bystander) {
step("bystander republish mid-session");
bystander->stopPump();
ST_ASSERT(bystander->unpublishVideo());
ST_ASSERT(bystander->publishVideo());
bystander->startPump();
// Long enough for an auto-subscription to have landed if one were
// coming; the wanted feed keeps flowing meanwhile.
const int before = video_frames.load();
ST_ASSERT(waitFor([&] { return video_frames.load() >= before + 30; }, 20000));
ST_ASSERT_EQ(unwanted_subscriptions.load(), 0);
}
// --- publisher swap: the bug this plugin exists to make impossible ----- // --- publisher swap: the bug this plugin exists to make impossible -----
const int before_swap = video_frames.load(); const int before_swap = video_frames.load();
@@ -344,10 +393,38 @@ int main()
ST_ASSERT(session.state() == SessionState::Connected); ST_ASSERT(session.state() == SessionState::Connected);
ST_ASSERT_EQ(bad_frames.load(), 0); ST_ASSERT_EQ(bad_frames.load(), 0);
// --- full rejoin: the player leaves and comes back as a new session ----
//
// What a browser reload does, and what a player did twice on 2026-10-04
// to unstick their camera. A new participant session publishes brand-new
// tracks, and with auto_subscribe off nothing arrives unless the session
// asks for them again.
step("publisher leaves the room");
publisher_holder.reset();
ST_ASSERT(waitFor([&] { return !session.hasVideo(); }, 15000));
ST_ASSERT(session.state() == SessionState::Connected);
step("publisher rejoins and republishes");
publisher_holder = std::make_unique<Publisher>();
Publisher &rejoined = *publisher_holder;
ST_ASSERT(rejoined.connect(url, publish_token));
ST_ASSERT(rejoined.publishVideo());
ST_ASSERT(rejoined.publishAudio());
rejoined.startPump();
const int before_rejoin = video_frames.load();
const int audio_before_rejoin = audio_frames.load();
ST_ASSERT(waitFor([&] { return video_frames.load() >= before_rejoin + 15; }, 25000));
ST_ASSERT(waitFor([&] { return audio_frames.load() >= audio_before_rejoin + 10; }, 20000));
ST_ASSERT(session.hasVideo());
ST_ASSERT_EQ(bad_frames.load(), 0);
// --- teardown ---------------------------------------------------------- // --- teardown ----------------------------------------------------------
step("teardown"); step("teardown");
publisher.stopPump(); if (bystander)
ST_ASSERT_EQ(unwanted_subscriptions.load(), 0);
publisher_holder->stopPump();
session.disconnect(); session.disconnect();
ST_ASSERT(session.state() == SessionState::Disconnected); ST_ASSERT(session.state() == SessionState::Disconnected);
ST_ASSERT(!session.hasVideo()); ST_ASSERT(!session.hasVideo());
@@ -355,6 +432,7 @@ int main()
// The publisher's Room must be torn down while the SDK is still // The publisher's Room must be torn down while the SDK is still
// initialized, or its FFI disconnect fails on the way out. // initialized, or its FFI disconnect fails on the way out.
publisher_holder.reset(); publisher_holder.reset();
bystander.reset();
LiveKitSession::globalShutdown(); LiveKitSession::globalShutdown();
std::printf("integration_livekit: %d video frames, %d audio frames, %d state changes\n", video_frames.load(), std::printf("integration_livekit: %d video frames, %d audio frames, %d state changes\n", video_frames.load(),
+233 -2
View File
@@ -14,8 +14,10 @@ You may obtain a copy of the License at
// COVERED headlessly -- the session's own decision-making: the state // COVERED headlessly -- the session's own decision-making: the state
// machine's transitions (including the publisher-swap and reconnect paths // machine's transitions (including the publisher-swap and reconnect paths
// that motivated this plugin), track selection, frame geometry validation, // that motivated this plugin), track selection, frame geometry validation,
// and the real connect() failure paths against the real SDK (bad URL, // the stall-recovery watchdog's timing/backoff decisions (StallWatchdog,
// unreachable host, garbage token). // driven with a fake clock -- see its own section below), and the real
// connect() failure paths against the real SDK (bad URL, unreachable
// host, garbage token).
// //
// NOT COVERED here -- anything that needs a LiveKit server to answer: // NOT COVERED here -- anything that needs a LiveKit server to answer:
// a successful connect, actual subscription, and actual decoded frames // a successful connect, actual subscription, and actual decoded frames
@@ -90,6 +92,34 @@ void testTrackSelection()
ST_ASSERT(!isWantedAudioTrack(want, "other", MediaKind::Audio, MediaSource::Microphone)); ST_ASSERT(!isWantedAudioTrack(want, "other", MediaKind::Audio, MediaSource::Microphone));
} }
void testSubscriptionSelection()
{
const std::string want = "cam1";
// A camera source subscribes to exactly its slot's camera and mic.
ST_ASSERT(shouldSubscribe(want, "cam1", MediaKind::Video, MediaSource::Camera, true, true));
ST_ASSERT(shouldSubscribe(want, "cam1", MediaKind::Audio, MediaSource::Microphone, true, true));
// Every other participant's tracks are the bandwidth this exists to save:
// with N sources in one OBS, each of them used to pull the whole room.
ST_ASSERT(!shouldSubscribe(want, "cam2", MediaKind::Video, MediaSource::Camera, true, true));
ST_ASSERT(!shouldSubscribe(want, "cam2", MediaKind::Audio, MediaSource::Microphone, true, true));
// ...and so is the slot's own screenshare.
ST_ASSERT(!shouldSubscribe(want, "cam1", MediaKind::Video, MediaSource::Screenshare, true, true));
ST_ASSERT(!shouldSubscribe(want, "cam1", MediaKind::Audio, MediaSource::ScreenshareAudio, true, true));
// An audio-only source (the soundboard) never subscribes to video at all,
// rather than subscribing and then disabling it.
ST_ASSERT(!shouldSubscribe(want, "cam1", MediaKind::Video, MediaSource::Camera, false, true));
ST_ASSERT(shouldSubscribe(want, "cam1", MediaKind::Audio, MediaSource::Microphone, false, true));
// A video-only source does not pull the mic.
ST_ASSERT(!shouldSubscribe(want, "cam1", MediaKind::Audio, MediaSource::Microphone, true, false));
ST_ASSERT(shouldSubscribe(want, "cam1", MediaKind::Video, MediaSource::Camera, true, false));
// No selection subscribes to nothing.
ST_ASSERT(!shouldSubscribe("", "cam1", MediaKind::Video, MediaSource::Camera, true, true));
}
void testStateMachineHappyPath() void testStateMachineHappyPath()
{ {
SessionStateMachine m; SessionStateMachine m;
@@ -209,6 +239,201 @@ void testFailureAndRecovery()
ST_ASSERT(idle.state() == SessionState::Idle); ST_ASSERT(idle.state() == SessionState::Idle);
} }
// ---------------------------------------------------------------------------
// StallWatchdog -- the stall-recovery watchdog's pure timing/decision logic.
//
// This is deliberately driven with an explicit, fake clock (arbitrary
// steady_clock::time_points built by hand, never std::chrono::...::now())
// rather than real sleeps: every case below needs to be exact about
// "1999ms in" vs "2001ms in" and about backoff boundaries, and a test that
// actually slept for 30+ seconds to exercise the backoff ceiling would be
// exactly the kind of slow, flaky test this project's whole headless-test
// philosophy exists to avoid. See StallWatchdog's own comment
// (session_types.h) for the measured server evidence this exists to fix.
// ---------------------------------------------------------------------------
void testStallWatchdogFiresAfterThreshold()
{
const auto t0 = std::chrono::steady_clock::now();
StallWatchdog w(std::chrono::milliseconds(2000), std::chrono::milliseconds(30000));
// Nothing subscribed yet: polling is a no-op, no matter how much time
// has "passed" -- an audio-only source or a disconnected session must
// never fire.
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(10000)));
// A track becomes subscribed. Still well under the threshold: quiet.
w.setExpectingFrames(true, t0);
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(500)));
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(1999)));
// No frame ever arrived, and the threshold has now elapsed: fires
// exactly once when asked right at/after the boundary.
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(2000)));
ST_ASSERT_EQ(w.attemptsThisStall(), 1);
// A frame arriving resets the clock -- the far more common case in a
// healthy stream, where onFrameDelivered() is called every ~33ms and
// poll() (every kWatchdogPollInterval) never sees 2000ms of silence.
StallWatchdog healthy(std::chrono::milliseconds(2000), std::chrono::milliseconds(30000));
healthy.setExpectingFrames(true, t0);
for (int ms = 0; ms <= 5000; ms += 33)
healthy.onFrameDelivered(t0 + std::chrono::milliseconds(ms));
ST_ASSERT(!healthy.poll(t0 + std::chrono::milliseconds(5010)));
ST_ASSERT_EQ(healthy.attemptsThisStall(), 0);
}
void testStallWatchdogDoesNotFireWhenNotExpectingFrames()
{
const auto t0 = std::chrono::steady_clock::now();
StallWatchdog w(std::chrono::milliseconds(2000), std::chrono::milliseconds(30000));
// A muted (or disabled/unsubscribed) track is expected silence, not a
// stall -- setExpectingFrames(false, ...) is exactly what
// LiveKitSession::Impl::handleMuteChange (and detachVideo()) call in
// that case. It must not fire no matter how long it stays that way.
w.setExpectingFrames(false, t0);
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(2000)));
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(60000)));
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(600000)));
// Un-muting (setExpectingFrames(true, ...)) starts a BRAND NEW grace
// period from that moment -- it must not read "was silent for ten
// minutes" as "stalled for ten minutes" and fire immediately.
const auto unmuted_at = t0 + std::chrono::milliseconds(600000);
w.setExpectingFrames(true, unmuted_at);
ST_ASSERT(!w.poll(unmuted_at + std::chrono::milliseconds(1999)));
ST_ASSERT(w.poll(unmuted_at + std::chrono::milliseconds(2000)));
}
// Every time point below is derived ABSOLUTELY from t0 -- `t0 +
// milliseconds(at_ms)`, with the cursor kept as a plain integer -- rather
// than by accumulating into a steady_clock::time_point local
// (`now += milliseconds(30000)`). That is not a style preference; it is
// load-bearing on Windows.
//
// The MSVC 19.44 (VS 2022 BuildTools 14.44.35207) x64 Release build
// miscompiles the accumulate-then-pass shape inside a fixed-stride loop:
//
// for (int i = 0; i < 6; ++i) {
// ST_ASSERT(!w.poll(now + milliseconds(29999)));
// now += milliseconds(30000);
// ST_ASSERT(w.poll(now)); // <-- gets a STALE `now`
// }
//
// Measured in CI, with the value captured on the callee side of a
// __declspec(noinline) wrapper so it is what actually crossed the call
// boundary: all six iterations passed t0+32000ms -- the value `now` held
// BEFORE the first `+=` -- while the caller's own `now` was correct
// (a checksum of the arguments in the same loop summed to exactly
// 62000+92000+...+212000). The argument was hoisted out of the loop as if
// it were loop-invariant. Linux and macOS pass 62000, 92000, ... 212000 for
// the same source.
//
// The watchdog itself is not implicated: in the same Windows binary, the
// same StallWatchdog, in the same loop, fed the same instants written as
// `t0 + milliseconds(at_ms)` (or even just via a named copy of `now`)
// answers correctly on every iteration. Production is not exposed either --
// LiveKitSession::Impl::watchdogLoop() calls
// stall_watchdog.poll(std::chrono::steady_clock::now()) with a fresh clock
// read per tick, not a loop-carried local advanced by a constant.
//
// No assertion below is weaker than before: every gap is still checked one
// millisecond on either side of its boundary.
void testStallWatchdogBacksOffRatherThanLooping()
{
const auto t0 = std::chrono::steady_clock::now();
StallWatchdog w(std::chrono::milliseconds(2000), std::chrono::milliseconds(30000));
w.setExpectingFrames(true, t0);
// Milliseconds since t0. A plain integer cursor, advanced explicitly.
long long at_ms = 2000;
// First attempt at the threshold.
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 1);
// A genuinely gone publisher: no frame ever comes back. Immediately
// asking again (the naive "retry every poll interval forever" a
// watchdog without backoff would do) must NOT fire -- that is precisely
// the "hammered every 2 seconds forever" this backoff exists to avoid.
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 250)));
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 1999)));
// The backoff after attempt 1 is the base timeout (2000ms): the second
// attempt is allowed at +2000ms from the first, not before.
at_ms += 2000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 2);
// Backoff doubles: the third attempt needs a 4000ms gap, not 2000ms.
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 3999)));
at_ms += 4000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 3);
// ... and again to 8000ms, and again to 16000ms.
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 7999)));
at_ms += 8000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 4);
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 15999)));
at_ms += 16000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), 5);
// The backoff is capped: doubling 16000ms would be 32000ms, but it
// never exceeds max_backoff (30000ms) no matter how many attempts have
// failed, so a publisher that comes back after an hour is still
// retried at a bounded cadence, not abandoned.
for (int i = 0; i < 6; ++i) {
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 29999)));
at_ms += 30000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
}
// A frame finally arrives: the stall is over, and the NEXT one (a fresh
// stall, not a continuation) starts back at the base cadence rather
// than staying parked at the 30s ceiling forever.
w.onFrameDelivered(t0 + std::chrono::milliseconds(at_ms));
ST_ASSERT_EQ(w.attemptsThisStall(), 0);
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + 1999)));
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms + 2000)));
ST_ASSERT_EQ(w.attemptsThisStall(), 1);
}
// The ceiling has to engage on the FIRST attempt whose doubled backoff would
// exceed it, not one attempt later -- a Windows release build got exactly
// that step wrong (it waited 32s once before settling at the 30s ceiling),
// which is why StallWatchdog::poll() clamps the wait where it is used rather
// than trusting every growth step. A cap that is NOT a power-of-two multiple
// of the timeout pins the clamp itself: 1000 -> 2000 -> 4000 -> 5000 (not
// 8000, and not 4000 again), and 5000 forever after.
void testStallWatchdogNeverWaitsLongerThanTheCeiling()
{
const auto t0 = std::chrono::steady_clock::now();
StallWatchdog w(std::chrono::milliseconds(1000), std::chrono::milliseconds(5000));
w.setExpectingFrames(true, t0);
// Absolute instants off t0, for the reason spelled out above
// testStallWatchdogBacksOffRatherThanLooping().
long long at_ms = 1000;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
// Expected gaps between consecutive attempts: 1000, 2000, 4000, then the
// ceiling for good. Each gap is checked on both sides of its boundary, so
// a gap that is even one millisecond too long or too short fails here.
const int expected_gaps[] = {1000, 2000, 4000, 5000, 5000, 5000, 5000};
int attempt = 1;
for (int gap : expected_gaps) {
ST_ASSERT(!w.poll(t0 + std::chrono::milliseconds(at_ms + gap - 1)));
at_ms += gap;
ST_ASSERT(w.poll(t0 + std::chrono::milliseconds(at_ms)));
ST_ASSERT_EQ(w.attemptsThisStall(), ++attempt);
}
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Real SDK, failure paths only (no LiveKit server available headlessly) // Real SDK, failure paths only (no LiveKit server available headlessly)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -330,11 +555,17 @@ int main()
{ {
testFrameGeometry(); testFrameGeometry();
testTrackSelection(); testTrackSelection();
testSubscriptionSelection();
testStateMachineHappyPath(); testStateMachineHappyPath();
testPublisherSwapIsNotAnError(); testPublisherSwapIsNotAnError();
testReconnect(); testReconnect();
testFailureAndRecovery(); testFailureAndRecovery();
testStallWatchdogFiresAfterThreshold();
testStallWatchdogDoesNotFireWhenNotExpectingFrames();
testStallWatchdogBacksOffRatherThanLooping();
testStallWatchdogNeverWaitsLongerThanTheCeiling();
LiveKitSession::globalInitialize(); LiveKitSession::globalInitialize();
testConnectRejectsIncompleteConfig(); testConnectRejectsIncompleteConfig();
testConnectToUnreachableServerFailsCleanly(); testConnectToUnreachableServerFailsCleanly();
+7
View File
@@ -375,6 +375,13 @@ void *sourceCreate(obs_data_t *settings, obs_source_t *source)
self->session->setVideoHandler([self](const VideoFrameData &frame) { outputVideoFrame(self, frame); }); self->session->setVideoHandler([self](const VideoFrameData &frame) { outputVideoFrame(self, frame); });
self->session->setAudioHandler([self](const AudioFrameData &frame) { outputAudioFrame(self, frame); }); self->session->setAudioHandler([self](const AudioFrameData &frame) { outputAudioFrame(self, frame); });
// The stall-recovery watchdog (core/src/session.cpp) is the only thing
// that currently uses this: it logs each toggle-the-subscription
// recovery attempt, and its eventual success, so a stalled-and-fixed
// camera is diagnosable from an OBS log afterward instead of invisible.
self->session->setDiagnosticHandler([](DiagnosticLevel level, const std::string &message) {
obs_log(level == DiagnosticLevel::Warning ? LOG_WARNING : LOG_INFO, "%s", message.c_str());
});
self->session->setStateHandler([self](SessionState state, const std::string &detail) { self->session->setStateHandler([self](SessionState state, const std::string &detail) {
self->setStatus(detail.empty() ? describeSessionState(state) : detail); self->setStatus(detail.empty() ? describeSessionState(state) : detail);
self->status_is_error.store(state == SessionState::Failed); self->status_is_error.store(state == SessionState::Failed);
+8 -1
View File
@@ -6,7 +6,7 @@ Usage:
eval "$(python3 scripts/livekit-dev-room.py)" eval "$(python3 scripts/livekit-dev-room.py)"
ctest --test-dir build -R test_integration_livekit --output-on-failure ctest --test-dir build -R test_integration_livekit --output-on-failure
Prints shell `export` lines for the four STPLUGIN_IT_* variables Prints shell `export` lines for the STPLUGIN_IT_* variables
core/tests/test_integration_livekit.cpp looks for. With no arguments it uses core/tests/test_integration_livekit.cpp looks for. With no arguments it uses
`livekit-server --dev`'s built-in devkey/secret credentials. `livekit-server --dev`'s built-in devkey/secret credentials.
@@ -58,6 +58,7 @@ def main() -> None:
parser.add_argument("--room", default="obs-plugin-it") parser.add_argument("--room", default="obs-plugin-it")
parser.add_argument("--publisher-identity", default="cam-test") parser.add_argument("--publisher-identity", default="cam-test")
parser.add_argument("--subscriber-identity", default="obs:obs-plugin-it:test") parser.add_argument("--subscriber-identity", default="obs:obs-plugin-it:test")
parser.add_argument("--bystander-identity", default="cam-bystander")
args = parser.parse_args() args = parser.parse_args()
publish_token = mint(args.api_key, args.api_secret, args.publisher_identity, args.room, publish_token = mint(args.api_key, args.api_secret, args.publisher_identity, args.room,
@@ -68,10 +69,16 @@ def main() -> None:
subscribe_token = mint(args.api_key, args.api_secret, args.subscriber_identity, args.room, subscribe_token = mint(args.api_key, args.api_secret, args.subscriber_identity, args.room,
publish=False, subscribe=True) publish=False, subscribe=True)
# A second camera in the same room that the session must NOT subscribe
# to -- the selective-subscription check.
bystander_token = mint(args.api_key, args.api_secret, args.bystander_identity, args.room,
publish=True, subscribe=False)
print(f'export STPLUGIN_IT_URL="{args.url}"') print(f'export STPLUGIN_IT_URL="{args.url}"')
print(f'export STPLUGIN_IT_PUBLISH_TOKEN="{publish_token}"') print(f'export STPLUGIN_IT_PUBLISH_TOKEN="{publish_token}"')
print(f'export STPLUGIN_IT_SUBSCRIBE_TOKEN="{subscribe_token}"') print(f'export STPLUGIN_IT_SUBSCRIBE_TOKEN="{subscribe_token}"')
print(f'export STPLUGIN_IT_PUBLISHER_IDENTITY="{args.publisher_identity}"') print(f'export STPLUGIN_IT_PUBLISHER_IDENTITY="{args.publisher_identity}"')
print(f'export STPLUGIN_IT_BYSTANDER_TOKEN="{bystander_token}"')
if __name__ == "__main__": if __name__ == "__main__":