Make the OBS adapter real: properties UI, connect, and frame output

The stub source becomes an actual streamer-tools camera. On create it reads
server URL / room slug / read key / camera identity from obs_data_t, mints a
subscribe-only token through ApiClient, connects LiveKitSession, and pushes
decoded frames into obs_source_output_video / obs_source_output_audio. The
source is now OBS_SOURCE_ASYNC_VIDEO | OBS_SOURCE_AUDIO |
OBS_SOURCE_DO_NOT_DUPLICATE with an OBS_ICON_TYPE_CAMERA icon.

The file is C++ rather than C now: the core library's API is C++ and the C ABI
shim existed only to avoid that. obs-module.h already declares the module
entry points extern "C", so nothing is lost.

Properties UI: server URL, room slug, a masked read-key field (it is a
credential and is masked everywhere else in streamer-tools), a camera dropdown,
a "Refresh camera list" button, and a status line.

- The dropdown is built from a cache the worker keeps warm on every connect,
  so opening properties never blocks on the network. The button is the
  explicit way to force a round trip, with a shortened 5s timeout -- for which
  ApiClient's two calls gained a timeout_ms parameter.
- The currently-selected identity is always in the list, labelled "(not in
  this room)" if absent, so OBS cannot silently clear a working setting just
  because the room happens to be dark.
- The status line is the OBS_TEXT_INFO property's description (which is what
  OBS actually renders) and switches to the warning info type on a real error.

Threading: OBS's UI and graphics threads are never blocked on the network.
Each source owns a worker thread that mints, connects, and reconnects with
exponential backoff (1s -> 30s), waking early on any settings change via a
generation counter. Frames are pushed from LiveKitSession's reader threads
directly; obs_source_output_video/_audio are thread-safe.

Two details that matter operationally:
 - A null frame is pushed whenever the session leaves Connected, so a camera
   that stopped publishing clears instead of leaving its last frame on screen.
   Leaving stale media up is precisely the failure this plugin exists to avoid.
 - The SDK's own logging is routed into OBS's log file via
   livekit::setLogCallback, instead of stderr where a director would never
   see it. The adapter also logs the first frame and every later geometry
   change, so a log answers "did video ever arrive, and at what size".

Packaging: the build now stages a runnable layout into build/package/ --
the module (RPATH $ORIGIN / @loader_path, so it resolves the LiveKit
libraries from beside itself rather than from the build tree), liblivekit +
liblivekit_ffi, the locale data, and the licence files. third_party/livekit/
carries client-sdk-cpp's Apache-2.0 LICENSE and NOTICE from the pinned tag.

Its README records a correction to the design doc: the "bundled LICENSE.md
with ~28 third-party licence blocks" the doc expects DOES NOT EXIST at
v1.10.1 -- not in any of the five release archives (which contain only
include/, lib/, bin/ and build-info.json) and not in the repo at that tag,
which has only LICENSE and NOTICE. The aggregated third-party notice covering
the WebRTC/OpenH264 code inside liblivekit_ffi.so has not been located, and
that is flagged as an open licensing question rather than papered over.

Verified on Ubuntu 24.04 against real libobs 30.0.2, a real
livekit-server 1.13.6, and a stand-in API serving plugin.routes.ts's exact
shapes, using a headless libobs harness (obs_startup + obs_reset_audio +
obs_reset_video + obs_open_module + obs_source_create):

  registered=1  output_flags=0x87
  [streamer-tools-camera] connected to ws://127.0.0.1:7880 as
      obs:main-room:qY85r9D0PaPt, watching cam-test
  [streamer-tools-camera] video frame 640x360 I420
  camera dropdown has 3 items:
    [0] (no camera selected) =
    [1] Test Camera = cam-test
    [2] Dark Camera (offline) = other-cam
  status: connected (info_type=0)

and with a deliberately wrong read key:

  status: unknown room slug, or the read key is wrong or has been rotated
      (info_type=1)

with retry-and-backoff and no crash. ctest: 6/6 passed.

Still unverified, and the README says so plainly: the OBS GUI on any platform,
macOS/Windows beyond compiling, A/V sync, and end-to-end latency.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RL8abRmgFXkVASHkkqiJbE
This commit is contained in:
2026-09-06 21:54:11 -07:00
co-authored by Claude Sonnet 5
parent 80904a3e85
commit a484abec61
12 changed files with 1040 additions and 215 deletions
+166 -129
View File
@@ -1,159 +1,196 @@
# obs-streamer-tools-plugin
Native OBS Studio source plugin that will pull streamer-tools camera feeds
directly from LiveKit over WebRTC (via LiveKit's `livekit-ffi`), replacing
the current SRT/RTSP-via-VLC-or-Media-Source path for directors. Full
design: `docs/superpowers/specs/2026-09-06-obs-camera-plugin-design.md` in
the `streamer-tools` repo (as of this writing, that doc lives on the
`worktree-obs-plugin-server-api` branch there, not yet merged to `main`).
Native OBS Studio source plugin that pulls streamer-tools camera feeds
directly from LiveKit over WebRTC, replacing the current SRT/RTSP-via-VLC-or-
Media-Source path for directors. Full design:
`docs/superpowers/specs/2026-09-06-obs-camera-plugin-design.md` in the
`streamer-tools` repo.
## Status: scaffolding only
## Status
**This repository does not talk to LiveKit or streamer-tools yet.** This
first pass exists to prove the CMake toolchain, the core-library/OBS-adapter
split, and the three-platform Gitea Actions CI pipeline all actually work,
so the next phase (real `livekit-ffi` integration) can be planned against
verified facts instead of assumptions. See the design doc's "Components"
and "CI / build pipeline" sections for the target architecture this scaffold
is standing up.
The plugin is **functionally complete on Linux and verified end to end there**
(module loads into real libobs, connects to a real LiveKit server through the
real streamer-tools API shape, and pushes decoded frames into
`obs_source_output_video`/`_audio`). It has **not** been run in the OBS GUI on
any platform, and macOS/Windows have only ever been built by CI, never loaded.
See "What is verified, and how" below for exactly what that means, and
"Testing this by hand" for what a human still needs to do.
## Layout
```
core/ - core library (C++17, no OBS dependency, headless-testable)
cmake/LiveKitSDK.cmake - downloads + unpacks the pinned client-sdk-cpp release
core/ - core library (C++17, no OBS dependency, headless-testable)
include/stplugin/
core.h C++ API (ConnectionConfig, core_version())
core_c.h C ABI wrapper the OBS adapter calls into
src/core.cpp
tests/ dependency-free CTest unit tests
obs-adapter/ - thin OBS glue (C, adapted from obsproject/obs-plugintemplate)
src/plugin-main.c obs_module_load/unload + a stub source registration
src/plugin-support.{h,c.in}
core.h version + ConnectionConfig
json.h small strict JSON reader
http.h injectable HTTP client interface
api_client.h the two /api/obs/:slug/* calls
session_types.h media/state types + the pure session logic
session.h LiveKitSession, the livekit::Room wrapper
src/
http_curl.cpp libcurl backend (Linux/macOS)
http_winhttp.cpp WinHTTP backend (Windows)
tests/ dependency-free CTest suites
obs-adapter/ - thin OBS glue (C++)
src/plugin-main.cpp obs_source_info, properties UI, frame output
data/locale/en-US.ini
.gitea/workflows/build.yml - 3-platform CI matrix (see below)
scripts/livekit-dev-room.py - mints tokens for the integration test
third_party/livekit/ - redistribution notices for the LiveKit binaries
.gitea/workflows/build.yml - 3-platform CI matrix
```
Everything in `core/` is real, working, unit-tested code -- it just doesn't
do anything useful yet (a version string, a config struct with non-empty
validation). Everything in `obs-adapter/` is real OBS module code -- it
registers an actual `obs_source_info` and builds as a real, dynamically
loadable OBS module (see "Verified" below) -- but the source is a stub:
`create`/`destroy` allocate/free a dummy blob, there is no properties UI,
and no frames are ever pushed. That's the boundary this task was scoped to.
## How it works
## What's real vs. deliberately deferred
1. The operator fills in the streamer-tools server URL, room slug and read key,
and picks a camera from the dropdown.
2. The source's own worker thread calls `POST /api/obs/:slug/token?key=…` to
mint a hidden, subscribe-only LiveKit token
(identity `obs:<slug>:<nonce>` — a fresh nonce per mint, so two OBS
installs watching the same room can never kick each other).
3. `LiveKitSession` connects `livekit::Room` to the returned `wsUrl`, waits for
the chosen participant's `Source.Camera` video track (and their microphone),
and reads decoded frames off `VideoStream`/`AudioStream`.
4. The adapter hands those straight to `obs_source_output_video` /
`obs_source_output_audio`.
Deferred, per the task that produced this scaffold (out of scope for this
pass, in scope for the next one):
Nothing on the OBS UI thread ever blocks on the network. The one deliberate
exception is the "Refresh camera list" button, which the operator pressed and
is waiting on; it uses a shortened 5s timeout.
- No `livekit-ffi` linkage of any kind.
- No streamer-tools API client (auth, slot-listing, token minting).
- No properties UI (server URL / room slug / read key / camera dropdown).
- No frame output (`obs_source_output_video`/`_audio`).
- No packaging/release step (the design doc's "on a version tag" job).
### Design decisions worth knowing before changing this
## Toolchain notes (verified on this machine: Ubuntu 24.04 / Linux)
- **Frames come from `VideoStream::fromTrack` with our own reader threads, not
from `Room::setOnVideoFrameCallback`.** The dispatcher API is keyed by
(participant identity, track *name*), which is only knowable once the track
is published — and disassembly of `liblivekit.so` 1.10.1 confirms that
neither `Room::setOnVideoFrameCallback` nor the dispatcher's own version
starts a reader for an already-subscribed track; they only record the
registration. Registering at the only moment the name exists would therefore
have silently produced no video.
- **Every stream operation runs on one owned worker thread**, never on a
LiveKit room event thread: `Room::disconnect()` from inside a delegate
callback is documented to deadlock.
- **`VideoStream::Options::capacity` is 3**, making the SDK queue a
drop-oldest ring buffer. A stalled consumer can only fall three frames
behind and then sees the *newest* frame, not a backlog — the structural
answer to the stale-media bug that motivated this plugin.
- **Video and audio are both timestamped with `os_gettime_ns()` at arrival.**
The SDK gives video a WebRTC capture timestamp and audio none; mixing two
epochs inside one OBS source would guarantee A/V drift. This relies on the
SDK's jitter buffering having already aligned them — the assumption the
design doc flags for verification on real hardware. **Still unverified.**
- **WebRTC changes resolution mid-stream.** Observed directly in the
integration test: the first frames after (re)subscribing arrive at a
downscaled spatial layer before ramping to the published size. The adapter
passes each frame's own geometry through, and logs geometry changes.
- **CMake 3.28.3**, **Ninja 1.11.1**, GCC 13.3.0 -- all installed via
`apt-get install cmake ninja-build`. Top-level `CMakeLists.txt` requires
CMake >= 3.16 (deliberately lower than the official
obsproject/obs-plugintemplate's `3.28...3.30` floor -- see below).
- **OBS plugin template used as reference**: obsproject/obs-plugintemplate,
commit `3e7d7ac3b5342cd7d9b88890b9c70b472d1520fc` (2025-12-09, "Fix typo
of Visual Studio in README"), fetched fresh from GitHub. `src/plugin-main.c`,
`src/plugin-support.{h,c.in}`, and the empty `data/locale/en-US.ini` in
`obs-adapter/` are adapted directly from it.
- **Deliberate deviation from the template's own build system**: the
official template's `CMakeLists.txt` chains into
`cmake/common/bootstrap.cmake`, which in turn reads `buildspec.json` and
*downloads full OBS source archives (pinned to OBS 31.1.1) plus prebuilt
dependency bundles* for macOS and Windows. That machinery is real,
actively maintained, and probably the right long-term answer for
cross-platform reproducible builds -- but it's heavy (multi-hundred-MB
downloads, a whole `cmake/{macos,windows,common}` support tree, Qt6,
code-signing hooks) and out of scope to stand up and debug in one pass.
This scaffold instead uses a much simpler hand-written top-level
`CMakeLists.txt` that calls `find_package(libobs)` directly.
- **On Linux, this actually works far better than expected**: Ubuntu ships
a real `libobs-dev` package (`30.0.2+dfsg-3build1` on 24.04, i.e. **not**
the 31.1.1 the template's buildspec.json pins -- worth reconciling before
the next phase if API surface matters) with genuine CMake package config
files (`/usr/lib/x86_64-linux-gnu/cmake/libobs/libobsConfig.cmake`,
`libobsTargets.cmake`) that export an `OBS::libobs` imported target --
the exact target name the official template expects. `find_package(libobs
QUIET)` finds it with zero extra plumbing. This means the OBS adapter in
this repo links against **real OBS headers and a real `libobs.so`**, not
a stub -- confirmed by `ldd` showing `libobs.so.0` and `nm -D` showing
real `obs_module_*` exports (see "Verified" below). Install via
`apt-get install libobs-dev` (pulls in Qt6 as a dependency chain, ~seconds
on a fast mirror).
- **macOS/Windows have no equivalent system package** (there's no Homebrew
formula or winget package that ships `libobsConfig.cmake` the way Ubuntu's
`libobs-dev` does). For those platforms the choices are: (a) adopt the
template's full buildspec-driven source/prebuilt-deps download, or (b)
find/produce a lighter prebuilt SDK bundle. **This is now a concrete,
scoped decision for the next phase**, not a guess -- the CI workflow in
this repo currently takes option (c) for this pass only: skip building
the OBS adapter on macOS/Windows and build+test just the core library,
via the same `find_package(libobs QUIET)` fallback the top-level
`CMakeLists.txt` already has for exactly this situation.
- **`ENABLE_QT`/`ENABLE_FRONTEND_API` template options were not carried
over** -- this scaffold's properties-UI-free stub doesn't need Qt yet;
the real adapter will need to revisit this once the properties UI
(server URL / room slug / read key / camera dropdown) is built.
## Building
## What actually builds, and how it was verified
Linux (the platform that is fully verified):
```
$ cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
-- libobs found (/usr/lib/x86_64-linux-gnu/cmake/libobs) -- building OBS adapter module
-- Configuring done
-- Generating done
$ cmake --build build
[1/7] Building C object obs-adapter/CMakeFiles/streamer-tools-camera.dir/plugin-support.c.o
[2/7] Building C object obs-adapter/CMakeFiles/streamer-tools-camera.dir/src/plugin-main.c.o
[3/7] Building CXX object core/CMakeFiles/stplugin_core.dir/src/core.cpp.o
[4/7] Linking CXX static library core/libstplugin_core.a
[5/7] Building CXX object core/tests/CMakeFiles/stplugin_core_tests.dir/test_core.cpp.o
[6/7] Linking CXX shared module obs-adapter/streamer-tools-camera.so
[7/7] Linking CXX executable core/tests/stplugin_core_tests
$ ctest --test-dir build --output-on-failure
1/1 Test #1: stplugin_core_tests .............. Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
$ ldd build/obs-adapter/streamer-tools-camera.so | grep obs
libobs.so.0 => /lib/x86_64-linux-gnu/libobs.so.0 (...)
$ nm -D build/obs-adapter/streamer-tools-camera.so | grep obs_module
0000000000001430 T obs_module_free_locale
0000000000001450 T obs_module_load
...
0000000000001490 T obs_module_unload
sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure
```
This is a genuine, dynamically-linked OBS module -- not the "standalone
shared library without linking OBS" fallback the scaffolding task's scope
explicitly allowed as an acceptable compromise. That fallback path is still
exercised (and needed) on macOS/Windows CI for now; see above.
The configure step downloads the pinned `client-sdk-cpp` release (~13 MB) into
`build/_deps/livekit-sdk`. Point `-DSTPLUGIN_LIVEKIT_SDK_DIR=<path>` at a
persistent directory to cache it across builds; `-DSTPLUGIN_LIVEKIT_SDK_VERSION`
and `-DSTPLUGIN_LIVEKIT_SDK_TRIPLE` override the pin and the release triple.
The build stages a runnable layout into `build/package/`:
```
build/package/bin/streamer-tools-camera.so (RPATH=$ORIGIN)
build/package/bin/liblivekit.so
build/package/bin/liblivekit_ffi.so
build/package/data/locale/en-US.ini
build/package/licenses/...
```
`build/package/bin` is what gets installed — the module resolves the LiveKit
libraries from `$ORIGIN` / `@loader_path`, not from the build tree.
## Testing this by hand
**Nobody has yet run this in the OBS GUI. That test is still outstanding on
all three platforms.** To do it on Linux:
```
mkdir -p ~/.config/obs-studio/plugins/streamer-tools-camera
cp -r build/package/bin build/package/data \
~/.config/obs-studio/plugins/streamer-tools-camera/
obs
```
Then: Sources → `+` → "streamer-tools Camera" → fill in the server URL, room
slug and read key from the room's settings page → "Refresh camera list" → pick
a camera. Check `~/.config/obs-studio/logs/` for
`[streamer-tools-camera] connected to …` and `[streamer-tools-camera] video
frame WxH I420`.
What to look for that automated testing could not answer:
- Does video actually *appear*, right way up, right colours?
- Is A/V sync acceptable? (see the timestamp caveat above)
- What is the end-to-end latency versus the existing egress path?
- Does a publisher restarting mid-show recover cleanly on screen?
### Running the LiveKit integration test
`core/tests/test_integration_livekit` publishes a synthetic camera into a real
room and subscribes to it through the wrapper. It skips unless
`STPLUGIN_IT_*` is set:
```
livekit-server --dev --bind 127.0.0.1 &
eval "$(python3 scripts/livekit-dev-room.py)"
ctest --test-dir build -R test_integration_livekit --output-on-failure
```
## What is verified, and how
Verified on Ubuntu 24.04 (libobs 30.0.2, client-sdk-cpp 1.10.1,
livekit-server 1.13.6 in dev mode):
| Claim | How it was checked |
|---|---|
| The pinned LiveKit SDK links and is callable | `test_livekit_smoke`: `initialize()`/`shutdown()` round-trip, header version asserted equal to the CMake pin |
| The JSON reader handles real and hostile input | `test_json`, 158 checks, including truncated bodies, HTML error pages, binary garbage, lone surrogates, and a depth-limit case |
| The API client parses the real response shapes and every error branch | `test_api_client`, 121 checks, against a fake HTTP client **and** a real loopback HTTP server driving the actual platform backend |
| A dead/stalled/garbage server cannot hang or crash the plugin | loopback cases: truncated JSON, connection closed with no reply, non-HTTP bytes, dead port, stalled server cut off by the client timeout |
| Session state transitions, track selection, frame geometry | `test_session`, 81 checks, plus real `connect()` failures against the real SDK |
| **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes |
| **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload |
| A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash |
**Not verified anywhere:**
- The OBS GUI, on any platform. No human has looked at this in OBS.
- macOS and Windows beyond "CI compiles and the core tests pass". The WinHTTP
backend has never run against a real streamer-tools server.
- A/V sync and end-to-end latency against the existing egress path.
- Behaviour against the real production streamer-tools server (only against a
stand-in serving the same shapes).
- Token expiry after an hour. Expiry is handled *reactively*: a fatal
disconnect makes the worker mint a fresh token and reconnect. The design
doc's "proactively refreshed before expiry" is **not** implemented —
`client-sdk-cpp` 1.10.1 exposes no way to hand a live `Room` a new token.
## CI
`.gitea/workflows/build.yml` runs on every push/PR, matrixed across the
three runners confirmed available to this repo by living under the
`CyberCoveLLC` org (see the design doc's "CI / build pipeline" section):
`.gitea/workflows/build.yml` runs on every push, matrixed across the three
runners available to this repo under the `CyberCoveLLC` org.
| Job | `runs-on` | Runner |
|---|---|---|
| `linux` | `ubuntu-latest` | `gitea-runner.internal.cloud-hosting.io` (Global) or `localhost.localdomain` (org-scoped; **note:** now online with `ubuntu-latest`/`ubuntu-24.04`/`ubuntu-22.04` labels -- the design doc recorded it as offline, that's since changed) |
| `linux` | `ubuntu-latest` | `gitea-runner.internal.cloud-hosting.io` (Global) |
| `macos` | `macos-latest` | `home-mac` (Global) |
| `windows` | `windows-latest` | `winvm-builder` (org-scoped to `CyberCoveLLC`) |
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) |
The Linux job installs `libobs-dev` and builds the real OBS adapter module
plus the core library, then runs `ctest`. The macOS/Windows jobs build and
test only the core library for now (see toolchain notes above for why).
No packaging/release step yet -- out of scope for this pass.
Linux uses Ubuntu's `libobs-dev` and builds the real OBS adapter. macOS and
Windows use the `obsproject/obs-plugintemplate` buildspec bootstrap, trimmed
to drop `qt6` (this plugin's properties UI is plain `obs_properties_*`), with
`obs-studio.version` pinned no newer than what Linux builds against — OBS
rejects a module built against a newer libobs than the one running it.