LiveKit integration: real camera feed pipeline #1
@@ -11,10 +11,14 @@ Media-Source path for directors. Full design:
|
|||||||
The plugin is **functionally complete on Linux and verified end to end there**
|
The plugin is **functionally complete on Linux and verified end to end there**
|
||||||
(module loads into real libobs, connects to a real LiveKit server through the
|
(module loads into real libobs, connects to a real LiveKit server through the
|
||||||
real streamer-tools API shape, and pushes decoded frames into
|
real streamer-tools API shape, and pushes decoded frames into
|
||||||
`obs_source_output_video`/`_audio`). It has **not** been run in the OBS GUI on
|
`obs_source_output_video`/`_audio`).
|
||||||
any platform, and macOS/Windows have only ever been built by CI, never loaded.
|
|
||||||
See "What is verified, and how" below for exactly what that means, and
|
It has **not been run in the OBS GUI on any platform.** macOS builds the real
|
||||||
"Testing this by hand" for what a human still needs to do.
|
module in CI but its artifact is not yet loadable (see the macOS packaging gap
|
||||||
|
under CI). Windows has not yet completed a build with the current fixes.
|
||||||
|
|
||||||
|
See "What is verified, and how" below for exactly what has and has not been
|
||||||
|
checked, and "Testing this by hand" for what a human still needs to do.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
@@ -87,11 +91,12 @@ is waiting on; it uses a shortened 5s timeout.
|
|||||||
|
|
||||||
## Building
|
## Building
|
||||||
|
|
||||||
Linux (the platform that is fully verified):
|
Linux (the platform that is fully verified). `STPLUGIN_BOOTSTRAP_OBS=OFF`
|
||||||
|
skips the macOS/Windows OBS-SDK bootstrap, which Linux does not need:
|
||||||
|
|
||||||
```
|
```
|
||||||
sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
|
sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
|
||||||
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
|
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTPLUGIN_BOOTSTRAP_OBS=OFF
|
||||||
cmake --build build
|
cmake --build build
|
||||||
ctest --test-dir build --output-on-failure
|
ctest --test-dir build --output-on-failure
|
||||||
```
|
```
|
||||||
@@ -165,11 +170,18 @@ livekit-server 1.13.6 in dev mode):
|
|||||||
| **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes |
|
| **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes |
|
||||||
| **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload |
|
| **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload |
|
||||||
| A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash |
|
| A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash |
|
||||||
|
| Changing the selected camera reconnects cleanly | same harness: switch to a dark slot and back. Each switch mints a fresh `obs:<room>:<nonce>` identity and reconnects; video returns; status stays `connected`; no crash, no stale frame |
|
||||||
|
| Two sources in one OBS process | same harness with a second source added: both connect with distinct nonce identities, both receive frames, both tear down cleanly |
|
||||||
|
|
||||||
**Not verified anywhere:**
|
**Not verified anywhere:**
|
||||||
- The OBS GUI, on any platform. No human has looked at this in OBS.
|
- The OBS GUI, on any platform. No human has looked at this in OBS.
|
||||||
- macOS and Windows beyond "CI compiles and the core tests pass". The WinHTTP
|
- macOS beyond "CI builds and links the real module and the core tests pass".
|
||||||
backend has never run against a real streamer-tools server.
|
Its artifact is a bare `.so` with a relative libobs install name and will
|
||||||
|
not load in OBS.app — see the macOS packaging gap under CI.
|
||||||
|
- Windows beyond "the core library and the WinHTTP backend compile and their
|
||||||
|
tests pass", from runs predating the current fixes. The WinHTTP backend has
|
||||||
|
never run against a real streamer-tools server, only against the loopback
|
||||||
|
test server in `test_api_client`.
|
||||||
- A/V sync and end-to-end latency against the existing egress path.
|
- A/V sync and end-to-end latency against the existing egress path.
|
||||||
- Behaviour against the real production streamer-tools server (only against a
|
- Behaviour against the real production streamer-tools server (only against a
|
||||||
stand-in serving the same shapes).
|
stand-in serving the same shapes).
|
||||||
@@ -183,14 +195,98 @@ livekit-server 1.13.6 in dev mode):
|
|||||||
`.gitea/workflows/build.yml` runs on every push, matrixed across the three
|
`.gitea/workflows/build.yml` runs on every push, matrixed across the three
|
||||||
runners available to this repo under the `CyberCoveLLC` org.
|
runners available to this repo under the `CyberCoveLLC` org.
|
||||||
|
|
||||||
| Job | `runs-on` | Runner |
|
| Job | `runs-on` | Runner | State |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| `linux` | `ubuntu-latest` | `gitea-runner.internal.cloud-hosting.io` (Global) |
|
| `linux` | `ubuntu-24.04` | `localhost.localdomain` | **Green.** Builds the real adapter against Ubuntu's libobs-dev 30.0.2, runs all six test suites, uploads `build/package` as an artifact |
|
||||||
| `macos` | `macos-latest` | `home-mac` (Global) |
|
| `macos` | `macos-latest` | `home-mac` (Global) | **Green.** Builds libobs 30.0.2 from source, then the real adapter; 6/6 tests; artifact uploaded. But see the macOS packaging gap below |
|
||||||
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) |
|
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) | **Unconfirmed** — see below |
|
||||||
|
|
||||||
Linux uses Ubuntu's `libobs-dev` and builds the real OBS adapter. macOS and
|
The Linux job is pinned to `ubuntu-24.04` rather than `ubuntu-latest`: this
|
||||||
Windows use the `obsproject/obs-plugintemplate` buildspec bootstrap, trimmed
|
instance's two Linux runners answer `ubuntu-latest` with different releases,
|
||||||
to drop `qt6` (this plugin's properties UI is plain `obs_properties_*`), with
|
and 22.04's `libobs-dev` is OBS 27 — a different API surface, and the LiveKit
|
||||||
`obs-studio.version` pinned no newer than what Linux builds against — OBS
|
SDK's own `linux-x64` asset does not even link there (hence the
|
||||||
rejects a module built against a newer libobs than the one running it.
|
`ubuntu-22.04` SDK triple; see `cmake/LiveKitSDK.cmake`).
|
||||||
|
|
||||||
|
macOS and Windows use the `obsproject/obs-plugintemplate` buildspec
|
||||||
|
bootstrap, trimmed to drop `qt6` (this plugin's properties UI is plain
|
||||||
|
`obs_properties_*`), with `obs-studio.version` pinned to 30.0.2 — the same
|
||||||
|
version Linux builds against, and deliberately low, because OBS rejects a
|
||||||
|
module built against a newer libobs than the one running it.
|
||||||
|
|
||||||
|
**Both jobs fall back to a core-library-only build if the bootstrap fails**,
|
||||||
|
rather than going red, with a workflow `::warning::` and a "Show what was
|
||||||
|
built" step that reports no module. That fallback exists because the
|
||||||
|
bootstrap is the least verifiable part of this project — there is no way to
|
||||||
|
exercise a macOS or Windows OBS build from the Linux development machine —
|
||||||
|
and a permanently red CI teaches people to ignore CI. **Do not remove the
|
||||||
|
warning:** a green job that quietly stopped building the plugin is worse than
|
||||||
|
a red one.
|
||||||
|
|
||||||
|
### Where the macOS bootstrap actually got to
|
||||||
|
|
||||||
|
Six CI iterations, each fixing a real failure visible in the logs:
|
||||||
|
|
||||||
|
1. Upstream's Xcode generator → `No CMAKE_C_COMPILER could be found` (the
|
||||||
|
runner has the Command Line Tools, not Xcode). Switched to Ninja.
|
||||||
|
2. OBS's SDK version regex only matches a full-Xcode SDK path. Synthesised a
|
||||||
|
`MacOSX.platform/Developer/SDKs/MacOSX<ver>.sdk` symlink to the same SDK.
|
||||||
|
3. The install walked into `UI/obs-frontend-api`, whose binary is
|
||||||
|
deliberately never built. The install's exit code is now tolerated.
|
||||||
|
4. Restricting the install to `libobs/` fixed that but lost the
|
||||||
|
per-configuration export file.
|
||||||
|
5. `xattr -r -d com.apple.quarantine` followed the SDK symlink into the
|
||||||
|
read-only system SDK. Symlink moved to the build directory; the xattr step
|
||||||
|
is no longer fatal.
|
||||||
|
6. `IMPORTED_LOCATION or IMPORTED_IMPLIB not set for imported target
|
||||||
|
OBS::libobs configuration Release` — OBS 30.0.2 installs
|
||||||
|
`libobsTargets.cmake` without the per-config file that carries the library
|
||||||
|
path. The top-level `CMakeLists.txt` now detects a locationless
|
||||||
|
`OBS::libobs` and points it at the framework the bootstrap just built.
|
||||||
|
|
||||||
|
All six are confirmed fixed: the macOS job now downloads obs-deps and
|
||||||
|
obs-studio, builds libobs from source, builds and links the real adapter,
|
||||||
|
passes 6/6 tests, and uploads its artifact. `otool -L` on the result shows it
|
||||||
|
linked against libobs and `@rpath/liblivekit.dylib`.
|
||||||
|
|
||||||
|
### macOS packaging gap (known, unfixed)
|
||||||
|
|
||||||
|
**The macOS artifact will not load in OBS.app as it stands.** Two reasons,
|
||||||
|
neither of which CI can catch, because CI only proves it compiles and links:
|
||||||
|
|
||||||
|
1. It is a bare `streamer-tools-camera.so`. OBS on macOS loads plugins as
|
||||||
|
`<name>.plugin` bundles (`Contents/MacOS/<name>`, `Contents/Resources/`,
|
||||||
|
an `Info.plist`), which is what obs-plugintemplate's
|
||||||
|
`cmake/macos/helpers.cmake` builds and which this project deliberately did
|
||||||
|
not vendor.
|
||||||
|
2. `otool -L` shows the libobs dependency recorded as the relative path
|
||||||
|
`libobs/libobs.framework/Versions/A/libobs`, inherited from the
|
||||||
|
from-source libobs's own install name. A real plugin needs
|
||||||
|
`@rpath/libobs.framework/Versions/A/libobs` plus an `LC_RPATH` pointing at
|
||||||
|
`OBS.app/Contents/Frameworks`.
|
||||||
|
|
||||||
|
Fixing this means either vendoring the template's macOS bundle helpers or
|
||||||
|
adding an `install_name_tool` pass and a bundle layout — bounded work, but
|
||||||
|
work that has to be done and checked on an actual Mac. It is deliberately not
|
||||||
|
attempted here rather than guessed at.
|
||||||
|
|
||||||
|
### Where the Windows bootstrap got to
|
||||||
|
|
||||||
|
Windows is by far the slowest job — the `lukka/get-cmake` step alone takes
|
||||||
|
7-15 minutes on `winvm-builder`, and the runner serialises jobs, so a burst of
|
||||||
|
pushes leaves a queue that takes an hour to drain. One confirmed bug of its
|
||||||
|
own was found and fixed: upstream passes `-A x64,version=<Windows SDK>` to the
|
||||||
|
OBS sub-configure, and with a current CMake that `,version=` suffix reappears
|
||||||
|
verbatim in the sub-build's `CMAKE_VS_PLATFORM_NAME` — which obs-studio's own
|
||||||
|
dependency downloader uses as the architecture, sending it after
|
||||||
|
`windows-deps-2023-11-03-x64,version=10.0.26100.0.zip`:
|
||||||
|
|
||||||
|
```
|
||||||
|
string sub-command JSON member 'hashes windows-x64,version=10.0.26100.0' not found
|
||||||
|
Unable to download .../windows-deps-2023-11-03-x64,version=10.0.26100.0.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Plain `-A x64` now. **No Windows run has yet completed with that fix in
|
||||||
|
place**, so Windows should be treated as unverified beyond "the core library
|
||||||
|
and the WinHTTP backend compile and their tests pass", which earlier runs did
|
||||||
|
show. Expect further iterations there of the same kind the macOS bootstrap
|
||||||
|
needed.
|
||||||
|
|||||||
Reference in New Issue
Block a user