LiveKit integration: real camera feed pipeline #1

Merged
jknapp merged 17 commits from feat/livekit-integration into main 2026-09-07 06:06:50 +00:00
Showing only changes of commit 9a1a1474a9 - Show all commits
+114 -18
View File
@@ -11,10 +11,14 @@ Media-Source path for directors. Full design:
The plugin is **functionally complete on Linux and verified end to end there** The plugin is **functionally complete on Linux and verified end to end there**
(module loads into real libobs, connects to a real LiveKit server through the (module loads into real libobs, connects to a real LiveKit server through the
real streamer-tools API shape, and pushes decoded frames into real streamer-tools API shape, and pushes decoded frames into
`obs_source_output_video`/`_audio`). It has **not** been run in the OBS GUI on `obs_source_output_video`/`_audio`).
any platform, and macOS/Windows have only ever been built by CI, never loaded.
See "What is verified, and how" below for exactly what that means, and It has **not been run in the OBS GUI on any platform.** macOS builds the real
"Testing this by hand" for what a human still needs to do. module in CI but its artifact is not yet loadable (see the macOS packaging gap
under CI). Windows has not yet completed a build with the current fixes.
See "What is verified, and how" below for exactly what has and has not been
checked, and "Testing this by hand" for what a human still needs to do.
## Layout ## Layout
@@ -87,11 +91,12 @@ is waiting on; it uses a shortened 5s timeout.
## Building ## Building
Linux (the platform that is fully verified): Linux (the platform that is fully verified). `STPLUGIN_BOOTSTRAP_OBS=OFF`
skips the macOS/Windows OBS-SDK bootstrap, which Linux does not need:
``` ```
sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTPLUGIN_BOOTSTRAP_OBS=OFF
cmake --build build cmake --build build
ctest --test-dir build --output-on-failure ctest --test-dir build --output-on-failure
``` ```
@@ -165,11 +170,18 @@ livekit-server 1.13.6 in dev mode):
| **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes | | **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes |
| **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload | | **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload |
| A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash | | A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash |
| Changing the selected camera reconnects cleanly | same harness: switch to a dark slot and back. Each switch mints a fresh `obs:<room>:<nonce>` identity and reconnects; video returns; status stays `connected`; no crash, no stale frame |
| Two sources in one OBS process | same harness with a second source added: both connect with distinct nonce identities, both receive frames, both tear down cleanly |
**Not verified anywhere:** **Not verified anywhere:**
- The OBS GUI, on any platform. No human has looked at this in OBS. - The OBS GUI, on any platform. No human has looked at this in OBS.
- macOS and Windows beyond "CI compiles and the core tests pass". The WinHTTP - macOS beyond "CI builds and links the real module and the core tests pass".
backend has never run against a real streamer-tools server. Its artifact is a bare `.so` with a relative libobs install name and will
not load in OBS.app — see the macOS packaging gap under CI.
- Windows beyond "the core library and the WinHTTP backend compile and their
tests pass", from runs predating the current fixes. The WinHTTP backend has
never run against a real streamer-tools server, only against the loopback
test server in `test_api_client`.
- A/V sync and end-to-end latency against the existing egress path. - A/V sync and end-to-end latency against the existing egress path.
- Behaviour against the real production streamer-tools server (only against a - Behaviour against the real production streamer-tools server (only against a
stand-in serving the same shapes). stand-in serving the same shapes).
@@ -183,14 +195,98 @@ livekit-server 1.13.6 in dev mode):
`.gitea/workflows/build.yml` runs on every push, matrixed across the three `.gitea/workflows/build.yml` runs on every push, matrixed across the three
runners available to this repo under the `CyberCoveLLC` org. runners available to this repo under the `CyberCoveLLC` org.
| Job | `runs-on` | Runner | | Job | `runs-on` | Runner | State |
|---|---|---| |---|---|---|---|
| `linux` | `ubuntu-latest` | `gitea-runner.internal.cloud-hosting.io` (Global) | | `linux` | `ubuntu-24.04` | `localhost.localdomain` | **Green.** Builds the real adapter against Ubuntu's libobs-dev 30.0.2, runs all six test suites, uploads `build/package` as an artifact |
| `macos` | `macos-latest` | `home-mac` (Global) | | `macos` | `macos-latest` | `home-mac` (Global) | **Green.** Builds libobs 30.0.2 from source, then the real adapter; 6/6 tests; artifact uploaded. But see the macOS packaging gap below |
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) | | `windows` | `windows-latest` | `winvm-builder` (org-scoped) | **Unconfirmed** — see below |
Linux uses Ubuntu's `libobs-dev` and builds the real OBS adapter. macOS and The Linux job is pinned to `ubuntu-24.04` rather than `ubuntu-latest`: this
Windows use the `obsproject/obs-plugintemplate` buildspec bootstrap, trimmed instance's two Linux runners answer `ubuntu-latest` with different releases,
to drop `qt6` (this plugin's properties UI is plain `obs_properties_*`), with and 22.04's `libobs-dev` is OBS 27 — a different API surface, and the LiveKit
`obs-studio.version` pinned no newer than what Linux builds against — OBS SDK's own `linux-x64` asset does not even link there (hence the
rejects a module built against a newer libobs than the one running it. `ubuntu-22.04` SDK triple; see `cmake/LiveKitSDK.cmake`).
macOS and Windows use the `obsproject/obs-plugintemplate` buildspec
bootstrap, trimmed to drop `qt6` (this plugin's properties UI is plain
`obs_properties_*`), with `obs-studio.version` pinned to 30.0.2 — the same
version Linux builds against, and deliberately low, because OBS rejects a
module built against a newer libobs than the one running it.
**Both jobs fall back to a core-library-only build if the bootstrap fails**,
rather than going red, with a workflow `::warning::` and a "Show what was
built" step that reports no module. That fallback exists because the
bootstrap is the least verifiable part of this project — there is no way to
exercise a macOS or Windows OBS build from the Linux development machine —
and a permanently red CI teaches people to ignore CI. **Do not remove the
warning:** a green job that quietly stopped building the plugin is worse than
a red one.
### Where the macOS bootstrap actually got to
Six CI iterations, each fixing a real failure visible in the logs:
1. Upstream's Xcode generator → `No CMAKE_C_COMPILER could be found` (the
runner has the Command Line Tools, not Xcode). Switched to Ninja.
2. OBS's SDK version regex only matches a full-Xcode SDK path. Synthesised a
`MacOSX.platform/Developer/SDKs/MacOSX<ver>.sdk` symlink to the same SDK.
3. The install walked into `UI/obs-frontend-api`, whose binary is
deliberately never built. The install's exit code is now tolerated.
4. Restricting the install to `libobs/` fixed that but lost the
per-configuration export file.
5. `xattr -r -d com.apple.quarantine` followed the SDK symlink into the
read-only system SDK. Symlink moved to the build directory; the xattr step
is no longer fatal.
6. `IMPORTED_LOCATION or IMPORTED_IMPLIB not set for imported target
OBS::libobs configuration Release` — OBS 30.0.2 installs
`libobsTargets.cmake` without the per-config file that carries the library
path. The top-level `CMakeLists.txt` now detects a locationless
`OBS::libobs` and points it at the framework the bootstrap just built.
All six are confirmed fixed: the macOS job now downloads obs-deps and
obs-studio, builds libobs from source, builds and links the real adapter,
passes 6/6 tests, and uploads its artifact. `otool -L` on the result shows it
linked against libobs and `@rpath/liblivekit.dylib`.
### macOS packaging gap (known, unfixed)
**The macOS artifact will not load in OBS.app as it stands.** Two reasons,
neither of which CI can catch, because CI only proves it compiles and links:
1. It is a bare `streamer-tools-camera.so`. OBS on macOS loads plugins as
`<name>.plugin` bundles (`Contents/MacOS/<name>`, `Contents/Resources/`,
an `Info.plist`), which is what obs-plugintemplate's
`cmake/macos/helpers.cmake` builds and which this project deliberately did
not vendor.
2. `otool -L` shows the libobs dependency recorded as the relative path
`libobs/libobs.framework/Versions/A/libobs`, inherited from the
from-source libobs's own install name. A real plugin needs
`@rpath/libobs.framework/Versions/A/libobs` plus an `LC_RPATH` pointing at
`OBS.app/Contents/Frameworks`.
Fixing this means either vendoring the template's macOS bundle helpers or
adding an `install_name_tool` pass and a bundle layout — bounded work, but
work that has to be done and checked on an actual Mac. It is deliberately not
attempted here rather than guessed at.
### Where the Windows bootstrap got to
Windows is by far the slowest job — the `lukka/get-cmake` step alone takes
7-15 minutes on `winvm-builder`, and the runner serialises jobs, so a burst of
pushes leaves a queue that takes an hour to drain. One confirmed bug of its
own was found and fixed: upstream passes `-A x64,version=<Windows SDK>` to the
OBS sub-configure, and with a current CMake that `,version=` suffix reappears
verbatim in the sub-build's `CMAKE_VS_PLATFORM_NAME` — which obs-studio's own
dependency downloader uses as the architecture, sending it after
`windows-deps-2023-11-03-x64,version=10.0.26100.0.zip`:
```
string sub-command JSON member 'hashes windows-x64,version=10.0.26100.0' not found
Unable to download .../windows-deps-2023-11-03-x64,version=10.0.26100.0.zip
```
Plain `-A x64` now. **No Windows run has yet completed with that fix in
place**, so Windows should be treated as unverified beyond "the core library
and the WinHTTP backend compile and their tests pass", which earlier runs did
show. Expect further iterations there of the same kind the macOS bootstrap
needed.