shadowdaoandClaude Sonnet 5
a484abec61
The stub source becomes an actual streamer-tools camera. On create it reads
server URL / room slug / read key / camera identity from obs_data_t, mints a
subscribe-only token through ApiClient, connects LiveKitSession, and pushes
decoded frames into obs_source_output_video / obs_source_output_audio. The
source is now OBS_SOURCE_ASYNC_VIDEO | OBS_SOURCE_AUDIO |
OBS_SOURCE_DO_NOT_DUPLICATE with an OBS_ICON_TYPE_CAMERA icon.
The file is C++ rather than C now: the core library's API is C++ and the C ABI
shim existed only to avoid that. obs-module.h already declares the module
entry points extern "C", so nothing is lost.
Properties UI: server URL, room slug, a masked read-key field (it is a
credential and is masked everywhere else in streamer-tools), a camera dropdown,
a "Refresh camera list" button, and a status line.
- The dropdown is built from a cache the worker keeps warm on every connect,
so opening properties never blocks on the network. The button is the
explicit way to force a round trip, with a shortened 5s timeout -- for which
ApiClient's two calls gained a timeout_ms parameter.
- The currently-selected identity is always in the list, labelled "(not in
this room)" if absent, so OBS cannot silently clear a working setting just
because the room happens to be dark.
- The status line is the OBS_TEXT_INFO property's description (which is what
OBS actually renders) and switches to the warning info type on a real error.
Threading: OBS's UI and graphics threads are never blocked on the network.
Each source owns a worker thread that mints, connects, and reconnects with
exponential backoff (1s -> 30s), waking early on any settings change via a
generation counter. Frames are pushed from LiveKitSession's reader threads
directly; obs_source_output_video/_audio are thread-safe.
Two details that matter operationally:
- A null frame is pushed whenever the session leaves Connected, so a camera
that stopped publishing clears instead of leaving its last frame on screen.
Leaving stale media up is precisely the failure this plugin exists to avoid.
- The SDK's own logging is routed into OBS's log file via
livekit::setLogCallback, instead of stderr where a director would never
see it. The adapter also logs the first frame and every later geometry
change, so a log answers "did video ever arrive, and at what size".
Packaging: the build now stages a runnable layout into build/package/ --
the module (RPATH $ORIGIN / @loader_path, so it resolves the LiveKit
libraries from beside itself rather than from the build tree), liblivekit +
liblivekit_ffi, the locale data, and the licence files. third_party/livekit/
carries client-sdk-cpp's Apache-2.0 LICENSE and NOTICE from the pinned tag.
Its README records a correction to the design doc: the "bundled LICENSE.md
with ~28 third-party licence blocks" the doc expects DOES NOT EXIST at
v1.10.1 -- not in any of the five release archives (which contain only
include/, lib/, bin/ and build-info.json) and not in the repo at that tag,
which has only LICENSE and NOTICE. The aggregated third-party notice covering
the WebRTC/OpenH264 code inside liblivekit_ffi.so has not been located, and
that is flagged as an open licensing question rather than papered over.
Verified on Ubuntu 24.04 against real libobs 30.0.2, a real
livekit-server 1.13.6, and a stand-in API serving plugin.routes.ts's exact
shapes, using a headless libobs harness (obs_startup + obs_reset_audio +
obs_reset_video + obs_open_module + obs_source_create):
registered=1 output_flags=0x87
[streamer-tools-camera] connected to ws://127.0.0.1:7880 as
obs:main-room:qY85r9D0PaPt, watching cam-test
[streamer-tools-camera] video frame 640x360 I420
camera dropdown has 3 items:
[0] (no camera selected) =
[1] Test Camera = cam-test
[2] Dark Camera (offline) = other-cam
status: connected (info_type=0)
and with a deliberately wrong read key:
status: unknown room slug, or the read key is wrong or has been rotated
(info_type=1)
with retry-and-backoff and no crash. ctest: 6/6 passed.
Still unverified, and the README says so plainly: the OBS GUI on any platform,
macOS/Windows beyond compiling, A/V sync, and end-to-end latency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RL8abRmgFXkVASHkkqiJbE
197 lines
10 KiB
Markdown
197 lines
10 KiB
Markdown
# obs-streamer-tools-plugin
|
|
|
|
Native OBS Studio source plugin that pulls streamer-tools camera feeds
|
|
directly from LiveKit over WebRTC, replacing the current SRT/RTSP-via-VLC-or-
|
|
Media-Source path for directors. Full design:
|
|
`docs/superpowers/specs/2026-09-06-obs-camera-plugin-design.md` in the
|
|
`streamer-tools` repo.
|
|
|
|
## Status
|
|
|
|
The plugin is **functionally complete on Linux and verified end to end there**
|
|
(module loads into real libobs, connects to a real LiveKit server through the
|
|
real streamer-tools API shape, and pushes decoded frames into
|
|
`obs_source_output_video`/`_audio`). It has **not** been run in the OBS GUI on
|
|
any platform, and macOS/Windows have only ever been built by CI, never loaded.
|
|
See "What is verified, and how" below for exactly what that means, and
|
|
"Testing this by hand" for what a human still needs to do.
|
|
|
|
## Layout
|
|
|
|
```
|
|
cmake/LiveKitSDK.cmake - downloads + unpacks the pinned client-sdk-cpp release
|
|
core/ - core library (C++17, no OBS dependency, headless-testable)
|
|
include/stplugin/
|
|
core.h version + ConnectionConfig
|
|
json.h small strict JSON reader
|
|
http.h injectable HTTP client interface
|
|
api_client.h the two /api/obs/:slug/* calls
|
|
session_types.h media/state types + the pure session logic
|
|
session.h LiveKitSession, the livekit::Room wrapper
|
|
src/
|
|
http_curl.cpp libcurl backend (Linux/macOS)
|
|
http_winhttp.cpp WinHTTP backend (Windows)
|
|
tests/ dependency-free CTest suites
|
|
obs-adapter/ - thin OBS glue (C++)
|
|
src/plugin-main.cpp obs_source_info, properties UI, frame output
|
|
data/locale/en-US.ini
|
|
scripts/livekit-dev-room.py - mints tokens for the integration test
|
|
third_party/livekit/ - redistribution notices for the LiveKit binaries
|
|
.gitea/workflows/build.yml - 3-platform CI matrix
|
|
```
|
|
|
|
## How it works
|
|
|
|
1. The operator fills in the streamer-tools server URL, room slug and read key,
|
|
and picks a camera from the dropdown.
|
|
2. The source's own worker thread calls `POST /api/obs/:slug/token?key=…` to
|
|
mint a hidden, subscribe-only LiveKit token
|
|
(identity `obs:<slug>:<nonce>` — a fresh nonce per mint, so two OBS
|
|
installs watching the same room can never kick each other).
|
|
3. `LiveKitSession` connects `livekit::Room` to the returned `wsUrl`, waits for
|
|
the chosen participant's `Source.Camera` video track (and their microphone),
|
|
and reads decoded frames off `VideoStream`/`AudioStream`.
|
|
4. The adapter hands those straight to `obs_source_output_video` /
|
|
`obs_source_output_audio`.
|
|
|
|
Nothing on the OBS UI thread ever blocks on the network. The one deliberate
|
|
exception is the "Refresh camera list" button, which the operator pressed and
|
|
is waiting on; it uses a shortened 5s timeout.
|
|
|
|
### Design decisions worth knowing before changing this
|
|
|
|
- **Frames come from `VideoStream::fromTrack` with our own reader threads, not
|
|
from `Room::setOnVideoFrameCallback`.** The dispatcher API is keyed by
|
|
(participant identity, track *name*), which is only knowable once the track
|
|
is published — and disassembly of `liblivekit.so` 1.10.1 confirms that
|
|
neither `Room::setOnVideoFrameCallback` nor the dispatcher's own version
|
|
starts a reader for an already-subscribed track; they only record the
|
|
registration. Registering at the only moment the name exists would therefore
|
|
have silently produced no video.
|
|
- **Every stream operation runs on one owned worker thread**, never on a
|
|
LiveKit room event thread: `Room::disconnect()` from inside a delegate
|
|
callback is documented to deadlock.
|
|
- **`VideoStream::Options::capacity` is 3**, making the SDK queue a
|
|
drop-oldest ring buffer. A stalled consumer can only fall three frames
|
|
behind and then sees the *newest* frame, not a backlog — the structural
|
|
answer to the stale-media bug that motivated this plugin.
|
|
- **Video and audio are both timestamped with `os_gettime_ns()` at arrival.**
|
|
The SDK gives video a WebRTC capture timestamp and audio none; mixing two
|
|
epochs inside one OBS source would guarantee A/V drift. This relies on the
|
|
SDK's jitter buffering having already aligned them — the assumption the
|
|
design doc flags for verification on real hardware. **Still unverified.**
|
|
- **WebRTC changes resolution mid-stream.** Observed directly in the
|
|
integration test: the first frames after (re)subscribing arrive at a
|
|
downscaled spatial layer before ramping to the published size. The adapter
|
|
passes each frame's own geometry through, and logs geometry changes.
|
|
|
|
## Building
|
|
|
|
Linux (the platform that is fully verified):
|
|
|
|
```
|
|
sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
|
|
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
|
|
cmake --build build
|
|
ctest --test-dir build --output-on-failure
|
|
```
|
|
|
|
The configure step downloads the pinned `client-sdk-cpp` release (~13 MB) into
|
|
`build/_deps/livekit-sdk`. Point `-DSTPLUGIN_LIVEKIT_SDK_DIR=<path>` at a
|
|
persistent directory to cache it across builds; `-DSTPLUGIN_LIVEKIT_SDK_VERSION`
|
|
and `-DSTPLUGIN_LIVEKIT_SDK_TRIPLE` override the pin and the release triple.
|
|
|
|
The build stages a runnable layout into `build/package/`:
|
|
|
|
```
|
|
build/package/bin/streamer-tools-camera.so (RPATH=$ORIGIN)
|
|
build/package/bin/liblivekit.so
|
|
build/package/bin/liblivekit_ffi.so
|
|
build/package/data/locale/en-US.ini
|
|
build/package/licenses/...
|
|
```
|
|
|
|
`build/package/bin` is what gets installed — the module resolves the LiveKit
|
|
libraries from `$ORIGIN` / `@loader_path`, not from the build tree.
|
|
|
|
## Testing this by hand
|
|
|
|
**Nobody has yet run this in the OBS GUI. That test is still outstanding on
|
|
all three platforms.** To do it on Linux:
|
|
|
|
```
|
|
mkdir -p ~/.config/obs-studio/plugins/streamer-tools-camera
|
|
cp -r build/package/bin build/package/data \
|
|
~/.config/obs-studio/plugins/streamer-tools-camera/
|
|
obs
|
|
```
|
|
|
|
Then: Sources → `+` → "streamer-tools Camera" → fill in the server URL, room
|
|
slug and read key from the room's settings page → "Refresh camera list" → pick
|
|
a camera. Check `~/.config/obs-studio/logs/` for
|
|
`[streamer-tools-camera] connected to …` and `[streamer-tools-camera] video
|
|
frame WxH I420`.
|
|
|
|
What to look for that automated testing could not answer:
|
|
- Does video actually *appear*, right way up, right colours?
|
|
- Is A/V sync acceptable? (see the timestamp caveat above)
|
|
- What is the end-to-end latency versus the existing egress path?
|
|
- Does a publisher restarting mid-show recover cleanly on screen?
|
|
|
|
### Running the LiveKit integration test
|
|
|
|
`core/tests/test_integration_livekit` publishes a synthetic camera into a real
|
|
room and subscribes to it through the wrapper. It skips unless
|
|
`STPLUGIN_IT_*` is set:
|
|
|
|
```
|
|
livekit-server --dev --bind 127.0.0.1 &
|
|
eval "$(python3 scripts/livekit-dev-room.py)"
|
|
ctest --test-dir build -R test_integration_livekit --output-on-failure
|
|
```
|
|
|
|
## What is verified, and how
|
|
|
|
Verified on Ubuntu 24.04 (libobs 30.0.2, client-sdk-cpp 1.10.1,
|
|
livekit-server 1.13.6 in dev mode):
|
|
|
|
| Claim | How it was checked |
|
|
|---|---|
|
|
| The pinned LiveKit SDK links and is callable | `test_livekit_smoke`: `initialize()`/`shutdown()` round-trip, header version asserted equal to the CMake pin |
|
|
| The JSON reader handles real and hostile input | `test_json`, 158 checks, including truncated bodies, HTML error pages, binary garbage, lone surrogates, and a depth-limit case |
|
|
| The API client parses the real response shapes and every error branch | `test_api_client`, 121 checks, against a fake HTTP client **and** a real loopback HTTP server driving the actual platform backend |
|
|
| A dead/stalled/garbage server cannot hang or crash the plugin | loopback cases: truncated JSON, connection closed with no reply, non-HTTP bytes, dead port, stalled server cut off by the client timeout |
|
|
| Session state transitions, track selection, frame geometry | `test_session`, 81 checks, plus real `connect()` failures against the real SDK |
|
|
| **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes |
|
|
| **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload |
|
|
| A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash |
|
|
|
|
**Not verified anywhere:**
|
|
- The OBS GUI, on any platform. No human has looked at this in OBS.
|
|
- macOS and Windows beyond "CI compiles and the core tests pass". The WinHTTP
|
|
backend has never run against a real streamer-tools server.
|
|
- A/V sync and end-to-end latency against the existing egress path.
|
|
- Behaviour against the real production streamer-tools server (only against a
|
|
stand-in serving the same shapes).
|
|
- Token expiry after an hour. Expiry is handled *reactively*: a fatal
|
|
disconnect makes the worker mint a fresh token and reconnect. The design
|
|
doc's "proactively refreshed before expiry" is **not** implemented —
|
|
`client-sdk-cpp` 1.10.1 exposes no way to hand a live `Room` a new token.
|
|
|
|
## CI
|
|
|
|
`.gitea/workflows/build.yml` runs on every push, matrixed across the three
|
|
runners available to this repo under the `CyberCoveLLC` org.
|
|
|
|
| Job | `runs-on` | Runner |
|
|
|---|---|---|
|
|
| `linux` | `ubuntu-latest` | `gitea-runner.internal.cloud-hosting.io` (Global) |
|
|
| `macos` | `macos-latest` | `home-mac` (Global) |
|
|
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) |
|
|
|
|
Linux uses Ubuntu's `libobs-dev` and builds the real OBS adapter. macOS and
|
|
Windows use the `obsproject/obs-plugintemplate` buildspec bootstrap, trimmed
|
|
to drop `qt6` (this plugin's properties UI is plain `obs_properties_*`), with
|
|
`obs-studio.version` pinned no newer than what Linux builds against — OBS
|
|
rejects a module built against a newer libobs than the one running it.
|