Files

T

shadowdaoandClaude Sonnet 5 a484abec61 Make the OBS adapter real: properties UI, connect, and frame output

The stub source becomes an actual streamer-tools camera. On create it reads
server URL / room slug / read key / camera identity from obs_data_t, mints a
subscribe-only token through ApiClient, connects LiveKitSession, and pushes
decoded frames into obs_source_output_video / obs_source_output_audio. The
source is now OBS_SOURCE_ASYNC_VIDEO | OBS_SOURCE_AUDIO |
OBS_SOURCE_DO_NOT_DUPLICATE with an OBS_ICON_TYPE_CAMERA icon.

The file is C++ rather than C now: the core library's API is C++ and the C ABI
shim existed only to avoid that. obs-module.h already declares the module
entry points extern "C", so nothing is lost.

Properties UI: server URL, room slug, a masked read-key field (it is a
credential and is masked everywhere else in streamer-tools), a camera dropdown,
a "Refresh camera list" button, and a status line.

- The dropdown is built from a cache the worker keeps warm on every connect,
  so opening properties never blocks on the network. The button is the
  explicit way to force a round trip, with a shortened 5s timeout -- for which
  ApiClient's two calls gained a timeout_ms parameter.
- The currently-selected identity is always in the list, labelled "(not in
  this room)" if absent, so OBS cannot silently clear a working setting just
  because the room happens to be dark.
- The status line is the OBS_TEXT_INFO property's description (which is what
  OBS actually renders) and switches to the warning info type on a real error.

Threading: OBS's UI and graphics threads are never blocked on the network.
Each source owns a worker thread that mints, connects, and reconnects with
exponential backoff (1s -> 30s), waking early on any settings change via a
generation counter. Frames are pushed from LiveKitSession's reader threads
directly; obs_source_output_video/_audio are thread-safe.

Two details that matter operationally:
 - A null frame is pushed whenever the session leaves Connected, so a camera
   that stopped publishing clears instead of leaving its last frame on screen.
   Leaving stale media up is precisely the failure this plugin exists to avoid.
 - The SDK's own logging is routed into OBS's log file via
   livekit::setLogCallback, instead of stderr where a director would never
   see it. The adapter also logs the first frame and every later geometry
   change, so a log answers "did video ever arrive, and at what size".

Packaging: the build now stages a runnable layout into build/package/ --
the module (RPATH $ORIGIN / @loader_path, so it resolves the LiveKit
libraries from beside itself rather than from the build tree), liblivekit +
liblivekit_ffi, the locale data, and the licence files. third_party/livekit/
carries client-sdk-cpp's Apache-2.0 LICENSE and NOTICE from the pinned tag.

Its README records a correction to the design doc: the "bundled LICENSE.md
with ~28 third-party licence blocks" the doc expects DOES NOT EXIST at
v1.10.1 -- not in any of the five release archives (which contain only
include/, lib/, bin/ and build-info.json) and not in the repo at that tag,
which has only LICENSE and NOTICE. The aggregated third-party notice covering
the WebRTC/OpenH264 code inside liblivekit_ffi.so has not been located, and
that is flagged as an open licensing question rather than papered over.

Verified on Ubuntu 24.04 against real libobs 30.0.2, a real
livekit-server 1.13.6, and a stand-in API serving plugin.routes.ts's exact
shapes, using a headless libobs harness (obs_startup + obs_reset_audio +
obs_reset_video + obs_open_module + obs_source_create):

  registered=1  output_flags=0x87
  [streamer-tools-camera] connected to ws://127.0.0.1:7880 as
      obs:main-room:qY85r9D0PaPt, watching cam-test
  [streamer-tools-camera] video frame 640x360 I420
  camera dropdown has 3 items:
    [0] (no camera selected) =
    [1] Test Camera = cam-test
    [2] Dark Camera (offline) = other-cam
  status: connected (info_type=0)

and with a deliberately wrong read key:

  status: unknown room slug, or the read key is wrong or has been rotated
      (info_type=1)

with retry-and-backoff and no crash. ctest: 6/6 passed.

Still unverified, and the README says so plainly: the OBS GUI on any platform,
macOS/Windows beyond compiling, A/V sync, and end-to-end latency.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RL8abRmgFXkVASHkkqiJbE

2026-09-06 21:54:11 -07:00

10 KiB

Raw Blame History

obs-streamer-tools-plugin

Native OBS Studio source plugin that pulls streamer-tools camera feeds directly from LiveKit over WebRTC, replacing the current SRT/RTSP-via-VLC-or- Media-Source path for directors. Full design: docs/superpowers/specs/2026-09-06-obs-camera-plugin-design.md in the streamer-tools repo.

Status

The plugin is functionally complete on Linux and verified end to end there (module loads into real libobs, connects to a real LiveKit server through the real streamer-tools API shape, and pushes decoded frames into obs_source_output_video/_audio). It has not been run in the OBS GUI on any platform, and macOS/Windows have only ever been built by CI, never loaded. See "What is verified, and how" below for exactly what that means, and "Testing this by hand" for what a human still needs to do.

Layout

cmake/LiveKitSDK.cmake   - downloads + unpacks the pinned client-sdk-cpp release
core/                    - core library (C++17, no OBS dependency, headless-testable)
  include/stplugin/
    core.h                 version + ConnectionConfig
    json.h                 small strict JSON reader
    http.h                 injectable HTTP client interface
    api_client.h           the two /api/obs/:slug/* calls
    session_types.h        media/state types + the pure session logic
    session.h              LiveKitSession, the livekit::Room wrapper
  src/
    http_curl.cpp          libcurl backend (Linux/macOS)
    http_winhttp.cpp       WinHTTP backend (Windows)
  tests/                   dependency-free CTest suites
obs-adapter/             - thin OBS glue (C++)
  src/plugin-main.cpp      obs_source_info, properties UI, frame output
  data/locale/en-US.ini
scripts/livekit-dev-room.py - mints tokens for the integration test
third_party/livekit/     - redistribution notices for the LiveKit binaries
.gitea/workflows/build.yml  - 3-platform CI matrix

How it works

The operator fills in the streamer-tools server URL, room slug and read key, and picks a camera from the dropdown.
The source's own worker thread calls POST /api/obs/:slug/token?key=… to mint a hidden, subscribe-only LiveKit token (identity obs:<slug>:<nonce> — a fresh nonce per mint, so two OBS installs watching the same room can never kick each other).
LiveKitSession connects livekit::Room to the returned wsUrl, waits for the chosen participant's Source.Camera video track (and their microphone), and reads decoded frames off VideoStream/AudioStream.
The adapter hands those straight to obs_source_output_video / obs_source_output_audio.

Nothing on the OBS UI thread ever blocks on the network. The one deliberate exception is the "Refresh camera list" button, which the operator pressed and is waiting on; it uses a shortened 5s timeout.

Design decisions worth knowing before changing this

Frames come from VideoStream::fromTrack with our own reader threads, not from Room::setOnVideoFrameCallback. The dispatcher API is keyed by (participant identity, track name), which is only knowable once the track is published — and disassembly of liblivekit.so 1.10.1 confirms that neither Room::setOnVideoFrameCallback nor the dispatcher's own version starts a reader for an already-subscribed track; they only record the registration. Registering at the only moment the name exists would therefore have silently produced no video.
Every stream operation runs on one owned worker thread, never on a LiveKit room event thread: Room::disconnect() from inside a delegate callback is documented to deadlock.
VideoStream::Options::capacity is 3, making the SDK queue a drop-oldest ring buffer. A stalled consumer can only fall three frames behind and then sees the newest frame, not a backlog — the structural answer to the stale-media bug that motivated this plugin.
Video and audio are both timestamped with os_gettime_ns() at arrival. The SDK gives video a WebRTC capture timestamp and audio none; mixing two epochs inside one OBS source would guarantee A/V drift. This relies on the SDK's jitter buffering having already aligned them — the assumption the design doc flags for verification on real hardware. Still unverified.
WebRTC changes resolution mid-stream. Observed directly in the integration test: the first frames after (re)subscribing arrive at a downscaled spatial layer before ramping to the published size. The adapter passes each frame's own geometry through, and logs geometry changes.

Building

Linux (the platform that is fully verified):

sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

The configure step downloads the pinned client-sdk-cpp release (~13 MB) into build/_deps/livekit-sdk. Point -DSTPLUGIN_LIVEKIT_SDK_DIR=<path> at a persistent directory to cache it across builds; -DSTPLUGIN_LIVEKIT_SDK_VERSION and -DSTPLUGIN_LIVEKIT_SDK_TRIPLE override the pin and the release triple.

The build stages a runnable layout into build/package/:

build/package/bin/streamer-tools-camera.so   (RPATH=$ORIGIN)
build/package/bin/liblivekit.so
build/package/bin/liblivekit_ffi.so
build/package/data/locale/en-US.ini
build/package/licenses/...

build/package/bin is what gets installed — the module resolves the LiveKit libraries from $ORIGIN / @loader_path, not from the build tree.

Testing this by hand

Nobody has yet run this in the OBS GUI. That test is still outstanding on all three platforms. To do it on Linux:

mkdir -p ~/.config/obs-studio/plugins/streamer-tools-camera
cp -r build/package/bin build/package/data \
      ~/.config/obs-studio/plugins/streamer-tools-camera/
obs

Then: Sources → + → "streamer-tools Camera" → fill in the server URL, room slug and read key from the room's settings page → "Refresh camera list" → pick a camera. Check ~/.config/obs-studio/logs/ for [streamer-tools-camera] connected to … and [streamer-tools-camera] video frame WxH I420.

What to look for that automated testing could not answer:

Does video actually appear, right way up, right colours?
Is A/V sync acceptable? (see the timestamp caveat above)
What is the end-to-end latency versus the existing egress path?
Does a publisher restarting mid-show recover cleanly on screen?

Running the LiveKit integration test

core/tests/test_integration_livekit publishes a synthetic camera into a real room and subscribes to it through the wrapper. It skips unless STPLUGIN_IT_* is set:

livekit-server --dev --bind 127.0.0.1 &
eval "$(python3 scripts/livekit-dev-room.py)"
ctest --test-dir build -R test_integration_livekit --output-on-failure

What is verified, and how

Verified on Ubuntu 24.04 (libobs 30.0.2, client-sdk-cpp 1.10.1, livekit-server 1.13.6 in dev mode):

Claim	How it was checked
The pinned LiveKit SDK links and is callable	`test_livekit_smoke`: `initialize()`/`shutdown()` round-trip, header version asserted equal to the CMake pin
The JSON reader handles real and hostile input	`test_json`, 158 checks, including truncated bodies, HTML error pages, binary garbage, lone surrogates, and a depth-limit case
The API client parses the real response shapes and every error branch	`test_api_client`, 121 checks, against a fake HTTP client and a real loopback HTTP server driving the actual platform backend
A dead/stalled/garbage server cannot hang or crash the plugin	loopback cases: truncated JSON, connection closed with no reply, non-HTTP bytes, dead port, stalled server cut off by the client timeout
Session state transitions, track selection, frame geometry	`test_session`, 81 checks, plus real `connect()` failures against the real SDK
Media actually flows	`test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with no further frames from the dead publisher, republish → video resumes
The module loads into real libobs and pushes frames	a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload
A wrong read key is reported, not silently swallowed	same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash

Not verified anywhere:

The OBS GUI, on any platform. No human has looked at this in OBS.
macOS and Windows beyond "CI compiles and the core tests pass". The WinHTTP backend has never run against a real streamer-tools server.
A/V sync and end-to-end latency against the existing egress path.
Behaviour against the real production streamer-tools server (only against a stand-in serving the same shapes).
Token expiry after an hour. Expiry is handled reactively: a fatal disconnect makes the worker mint a fresh token and reconnect. The design doc's "proactively refreshed before expiry" is not implemented — client-sdk-cpp 1.10.1 exposes no way to hand a live Room a new token.

CI

.gitea/workflows/build.yml runs on every push, matrixed across the three runners available to this repo under the CyberCoveLLC org.

Job	`runs-on`	Runner
`linux`	`ubuntu-latest`	`gitea-runner.internal.cloud-hosting.io` (Global)
`macos`	`macos-latest`	`home-mac` (Global)
`windows`	`windows-latest`	`winvm-builder` (org-scoped)

Linux uses Ubuntu's libobs-dev and builds the real OBS adapter. macOS and Windows use the obsproject/obs-plugintemplate buildspec bootstrap, trimmed to drop qt6 (this plugin's properties UI is plain obs_properties_*), with obs-studio.version pinned no newer than what Linux builds against — OBS rejects a module built against a newer libobs than the one running it.

10 KiB Raw Blame History