shadowdaoandClaude Sonnet 5
6829dd545a
The staged package put the module in build/package/bin. OBS does not look
there. From AddExtraModulePaths() in obs-studio's UI/window-basic-main.cpp,
the per-user plugin layout on Linux and Windows is:
<config>/obs-studio/plugins/<name>/bin/64bit/<name>.{so,dll}
<config>/obs-studio/plugins/<name>/data/
so build/package/ now uses bin/64bit and is a straight drop-in. Re-verified in
the headless libobs harness from the new path: the module loads, both sources
connect, frames arrive, the camera switch round-trips, and `ldd` on the staged
copy resolves liblivekit and liblivekit_ffi from bin/64bit via $ORIGIN.
macOS deliberately keeps the flat bin/ -- there OBS looks for a
<name>.plugin/Contents/MacOS bundle, which this build does not produce. That
gap is documented in README.md rather than papered over with a directory name
that would only look right.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RL8abRmgFXkVASHkkqiJbE
302 lines
16 KiB
Markdown
302 lines
16 KiB
Markdown
# obs-streamer-tools-plugin
|
|
|
|
Native OBS Studio source plugin that pulls streamer-tools camera feeds
|
|
directly from LiveKit over WebRTC, replacing the current SRT/RTSP-via-VLC-or-
|
|
Media-Source path for directors. Full design:
|
|
`docs/superpowers/specs/2026-09-06-obs-camera-plugin-design.md` in the
|
|
`streamer-tools` repo.
|
|
|
|
## Status
|
|
|
|
The plugin is **functionally complete on Linux and verified end to end there**
|
|
(module loads into real libobs, connects to a real LiveKit server through the
|
|
real streamer-tools API shape, and pushes decoded frames into
|
|
`obs_source_output_video`/`_audio`).
|
|
|
|
It has **not been run in the OBS GUI on any platform.** macOS builds the real
|
|
module in CI but its artifact is not yet loadable (see the macOS packaging gap
|
|
under CI). Windows has not yet completed a build with the current fixes.
|
|
|
|
See "What is verified, and how" below for exactly what has and has not been
|
|
checked, and "Testing this by hand" for what a human still needs to do.
|
|
|
|
## Layout
|
|
|
|
```
|
|
cmake/LiveKitSDK.cmake - downloads + unpacks the pinned client-sdk-cpp release
|
|
core/ - core library (C++17, no OBS dependency, headless-testable)
|
|
include/stplugin/
|
|
core.h version + ConnectionConfig
|
|
json.h small strict JSON reader
|
|
http.h injectable HTTP client interface
|
|
api_client.h the two /api/obs/:slug/* calls
|
|
session_types.h media/state types + the pure session logic
|
|
session.h LiveKitSession, the livekit::Room wrapper
|
|
src/
|
|
http_curl.cpp libcurl backend (Linux/macOS)
|
|
http_winhttp.cpp WinHTTP backend (Windows)
|
|
tests/ dependency-free CTest suites
|
|
obs-adapter/ - thin OBS glue (C++)
|
|
src/plugin-main.cpp obs_source_info, properties UI, frame output
|
|
data/locale/en-US.ini
|
|
scripts/livekit-dev-room.py - mints tokens for the integration test
|
|
third_party/livekit/ - redistribution notices for the LiveKit binaries
|
|
.gitea/workflows/build.yml - 3-platform CI matrix
|
|
```
|
|
|
|
## How it works
|
|
|
|
1. The operator fills in the streamer-tools server URL, room slug and read key,
|
|
and picks a camera from the dropdown.
|
|
2. The source's own worker thread calls `POST /api/obs/:slug/token?key=…` to
|
|
mint a hidden, subscribe-only LiveKit token
|
|
(identity `obs:<slug>:<nonce>` — a fresh nonce per mint, so two OBS
|
|
installs watching the same room can never kick each other).
|
|
3. `LiveKitSession` connects `livekit::Room` to the returned `wsUrl`, waits for
|
|
the chosen participant's `Source.Camera` video track (and their microphone),
|
|
and reads decoded frames off `VideoStream`/`AudioStream`.
|
|
4. The adapter hands those straight to `obs_source_output_video` /
|
|
`obs_source_output_audio`.
|
|
|
|
Nothing on the OBS UI thread ever blocks on the network. The one deliberate
|
|
exception is the "Refresh camera list" button, which the operator pressed and
|
|
is waiting on; it uses a shortened 5s timeout.
|
|
|
|
### Design decisions worth knowing before changing this
|
|
|
|
- **Frames come from `VideoStream::fromTrack` with our own reader threads, not
|
|
from `Room::setOnVideoFrameCallback`.** The dispatcher API is keyed by
|
|
(participant identity, track *name*), which is only knowable once the track
|
|
is published — and disassembly of `liblivekit.so` 1.10.1 confirms that
|
|
neither `Room::setOnVideoFrameCallback` nor the dispatcher's own version
|
|
starts a reader for an already-subscribed track; they only record the
|
|
registration. Registering at the only moment the name exists would therefore
|
|
have silently produced no video.
|
|
- **Every stream operation runs on one owned worker thread**, never on a
|
|
LiveKit room event thread: `Room::disconnect()` from inside a delegate
|
|
callback is documented to deadlock.
|
|
- **`VideoStream::Options::capacity` is 3**, making the SDK queue a
|
|
drop-oldest ring buffer. A stalled consumer can only fall three frames
|
|
behind and then sees the *newest* frame, not a backlog — the structural
|
|
answer to the stale-media bug that motivated this plugin.
|
|
- **Video and audio are both timestamped with `os_gettime_ns()` at arrival.**
|
|
The SDK gives video a WebRTC capture timestamp and audio none; mixing two
|
|
epochs inside one OBS source would guarantee A/V drift. This relies on the
|
|
SDK's jitter buffering having already aligned them — the assumption the
|
|
design doc flags for verification on real hardware. **Still unverified.**
|
|
- **WebRTC changes resolution mid-stream.** Observed directly in the
|
|
integration test: the first frames after (re)subscribing arrive at a
|
|
downscaled spatial layer before ramping to the published size. The adapter
|
|
passes each frame's own geometry through, and logs geometry changes.
|
|
|
|
## Building
|
|
|
|
Linux (the platform that is fully verified). `STPLUGIN_BOOTSTRAP_OBS=OFF`
|
|
skips the macOS/Windows OBS-SDK bootstrap, which Linux does not need:
|
|
|
|
```
|
|
sudo apt-get install -y cmake ninja-build libobs-dev libcurl4-openssl-dev
|
|
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTPLUGIN_BOOTSTRAP_OBS=OFF
|
|
cmake --build build
|
|
ctest --test-dir build --output-on-failure
|
|
```
|
|
|
|
The configure step downloads the pinned `client-sdk-cpp` release (~13 MB) into
|
|
`build/_deps/livekit-sdk`. Point `-DSTPLUGIN_LIVEKIT_SDK_DIR=<path>` at a
|
|
persistent directory to cache it across builds; `-DSTPLUGIN_LIVEKIT_SDK_VERSION`
|
|
and `-DSTPLUGIN_LIVEKIT_SDK_TRIPLE` override the pin and the release triple.
|
|
|
|
The build stages a runnable layout into `build/package/`:
|
|
|
|
```
|
|
build/package/bin/64bit/streamer-tools-camera.so (RPATH=$ORIGIN)
|
|
build/package/bin/64bit/liblivekit.so
|
|
build/package/bin/64bit/liblivekit_ffi.so
|
|
build/package/data/locale/en-US.ini
|
|
build/package/licenses/...
|
|
```
|
|
|
|
That is exactly the layout OBS searches on Linux and Windows —
|
|
`<config>/obs-studio/plugins/<name>/bin/64bit` plus a sibling `data/`, per
|
|
`AddExtraModulePaths()` in obs-studio's `UI/window-basic-main.cpp` — so
|
|
`build/package/` is a straight drop-in. The module resolves the LiveKit
|
|
libraries from `$ORIGIN` (verified: `ldd` on the staged copy resolves both
|
|
to `bin/64bit/`), not from the build tree. macOS is not this shape; see the
|
|
macOS packaging gap under CI.
|
|
|
|
## Testing this by hand
|
|
|
|
**Nobody has yet run this in the OBS GUI. That test is still outstanding on
|
|
all three platforms.** To do it on Linux:
|
|
|
|
```
|
|
mkdir -p ~/.config/obs-studio/plugins/streamer-tools-camera
|
|
cp -r build/package/bin build/package/data \
|
|
~/.config/obs-studio/plugins/streamer-tools-camera/
|
|
obs
|
|
```
|
|
|
|
(That yields `.../streamer-tools-camera/bin/64bit/streamer-tools-camera.so`
|
|
and `.../streamer-tools-camera/data/locale/en-US.ini`, which is what OBS
|
|
looks for.)
|
|
|
|
Then: Sources → `+` → "streamer-tools Camera" → fill in the server URL, room
|
|
slug and read key from the room's settings page → "Refresh camera list" → pick
|
|
a camera. Check `~/.config/obs-studio/logs/` for
|
|
`[streamer-tools-camera] connected to …` and `[streamer-tools-camera] video
|
|
frame WxH I420`.
|
|
|
|
What to look for that automated testing could not answer:
|
|
- Does video actually *appear*, right way up, right colours?
|
|
- Is A/V sync acceptable? (see the timestamp caveat above)
|
|
- What is the end-to-end latency versus the existing egress path?
|
|
- Does a publisher restarting mid-show recover cleanly on screen?
|
|
|
|
### Running the LiveKit integration test
|
|
|
|
`core/tests/test_integration_livekit` publishes a synthetic camera into a real
|
|
room and subscribes to it through the wrapper. It skips unless
|
|
`STPLUGIN_IT_*` is set:
|
|
|
|
```
|
|
livekit-server --dev --bind 127.0.0.1 &
|
|
eval "$(python3 scripts/livekit-dev-room.py)"
|
|
ctest --test-dir build -R test_integration_livekit --output-on-failure
|
|
```
|
|
|
|
## What is verified, and how
|
|
|
|
Verified on Ubuntu 24.04 (libobs 30.0.2, client-sdk-cpp 1.10.1,
|
|
livekit-server 1.13.6 in dev mode):
|
|
|
|
| Claim | How it was checked |
|
|
|---|---|
|
|
| The pinned LiveKit SDK links and is callable | `test_livekit_smoke`: `initialize()`/`shutdown()` round-trip, header version asserted equal to the CMake pin |
|
|
| The JSON reader handles real and hostile input | `test_json`, 158 checks, including truncated bodies, HTML error pages, binary garbage, lone surrogates, and a depth-limit case |
|
|
| The API client parses the real response shapes and every error branch | `test_api_client`, 121 checks, against a fake HTTP client **and** a real loopback HTTP server driving the actual platform backend |
|
|
| A dead/stalled/garbage server cannot hang or crash the plugin | loopback cases: truncated JSON, connection closed with no reply, non-HTTP bytes, dead port, stalled server cut off by the client timeout |
|
|
| Session state transitions, track selection, frame geometry | `test_session`, 81 checks, plus real `connect()` failures against the real SDK |
|
|
| **Media actually flows** | `test_integration_livekit` against a real LiveKit server: 36 video frames + 323 audio frames, correct I420 geometry and plane pointers, publisher unpublish → `hasVideo()` false with **no further frames from the dead publisher**, republish → video resumes |
|
|
| **The module loads into real libobs and pushes frames** | a headless libobs harness (`obs_startup` + `obs_reset_audio`/`obs_reset_video` + `obs_open_module`) driving the built module against a stand-in streamer-tools API in front of a real LiveKit server. Log: `connected to ws://… watching cam-test` then `video frame 640x360 I420`; the camera dropdown populated as `Test Camera` / `Dark Camera (offline)`; status `connected`; clean destroy and unload |
|
|
| A wrong read key is reported, not silently swallowed | same harness with a bad key: status `unknown room slug, or the read key is wrong or has been rotated`, warning info type, retry with backoff, no crash |
|
|
| Changing the selected camera reconnects cleanly | same harness: switch to a dark slot and back. Each switch mints a fresh `obs:<room>:<nonce>` identity and reconnects; video returns; status stays `connected`; no crash, no stale frame |
|
|
| Two sources in one OBS process | same harness with a second source added: both connect with distinct nonce identities, both receive frames, both tear down cleanly |
|
|
|
|
**Not verified anywhere:**
|
|
- The OBS GUI, on any platform. No human has looked at this in OBS.
|
|
- macOS beyond "CI builds and links the real module and the core tests pass".
|
|
Its artifact is a bare `.so` with a relative libobs install name and will
|
|
not load in OBS.app — see the macOS packaging gap under CI.
|
|
- Windows beyond "the core library and the WinHTTP backend compile and their
|
|
tests pass", from runs predating the current fixes. The WinHTTP backend has
|
|
never run against a real streamer-tools server, only against the loopback
|
|
test server in `test_api_client`.
|
|
- A/V sync and end-to-end latency against the existing egress path.
|
|
- Behaviour against the real production streamer-tools server (only against a
|
|
stand-in serving the same shapes).
|
|
- Token expiry after an hour. Expiry is handled *reactively*: a fatal
|
|
disconnect makes the worker mint a fresh token and reconnect. The design
|
|
doc's "proactively refreshed before expiry" is **not** implemented —
|
|
`client-sdk-cpp` 1.10.1 exposes no way to hand a live `Room` a new token.
|
|
|
|
## CI
|
|
|
|
`.gitea/workflows/build.yml` runs on every push, matrixed across the three
|
|
runners available to this repo under the `CyberCoveLLC` org.
|
|
|
|
| Job | `runs-on` | Runner | State |
|
|
|---|---|---|---|
|
|
| `linux` | `ubuntu-24.04` | `localhost.localdomain` | **Green.** Builds the real adapter against Ubuntu's libobs-dev 30.0.2, runs all six test suites, uploads `build/package` as an artifact |
|
|
| `macos` | `macos-latest` | `home-mac` (Global) | **Green.** Builds libobs 30.0.2 from source, then the real adapter; 6/6 tests; artifact uploaded. But see the macOS packaging gap below |
|
|
| `windows` | `windows-latest` | `winvm-builder` (org-scoped) | **Unconfirmed** — see below |
|
|
|
|
The Linux job is pinned to `ubuntu-24.04` rather than `ubuntu-latest`: this
|
|
instance's two Linux runners answer `ubuntu-latest` with different releases,
|
|
and 22.04's `libobs-dev` is OBS 27 — a different API surface, and the LiveKit
|
|
SDK's own `linux-x64` asset does not even link there (hence the
|
|
`ubuntu-22.04` SDK triple; see `cmake/LiveKitSDK.cmake`).
|
|
|
|
macOS and Windows use the `obsproject/obs-plugintemplate` buildspec
|
|
bootstrap, trimmed to drop `qt6` (this plugin's properties UI is plain
|
|
`obs_properties_*`), with `obs-studio.version` pinned to 30.0.2 — the same
|
|
version Linux builds against, and deliberately low, because OBS rejects a
|
|
module built against a newer libobs than the one running it.
|
|
|
|
**Both jobs fall back to a core-library-only build if the bootstrap fails**,
|
|
rather than going red, with a workflow `::warning::` and a "Show what was
|
|
built" step that reports no module. That fallback exists because the
|
|
bootstrap is the least verifiable part of this project — there is no way to
|
|
exercise a macOS or Windows OBS build from the Linux development machine —
|
|
and a permanently red CI teaches people to ignore CI. **Do not remove the
|
|
warning:** a green job that quietly stopped building the plugin is worse than
|
|
a red one.
|
|
|
|
### Where the macOS bootstrap actually got to
|
|
|
|
Six CI iterations, each fixing a real failure visible in the logs:
|
|
|
|
1. Upstream's Xcode generator → `No CMAKE_C_COMPILER could be found` (the
|
|
runner has the Command Line Tools, not Xcode). Switched to Ninja.
|
|
2. OBS's SDK version regex only matches a full-Xcode SDK path. Synthesised a
|
|
`MacOSX.platform/Developer/SDKs/MacOSX<ver>.sdk` symlink to the same SDK.
|
|
3. The install walked into `UI/obs-frontend-api`, whose binary is
|
|
deliberately never built. The install's exit code is now tolerated.
|
|
4. Restricting the install to `libobs/` fixed that but lost the
|
|
per-configuration export file.
|
|
5. `xattr -r -d com.apple.quarantine` followed the SDK symlink into the
|
|
read-only system SDK. Symlink moved to the build directory; the xattr step
|
|
is no longer fatal.
|
|
6. `IMPORTED_LOCATION or IMPORTED_IMPLIB not set for imported target
|
|
OBS::libobs configuration Release` — OBS 30.0.2 installs
|
|
`libobsTargets.cmake` without the per-config file that carries the library
|
|
path. The top-level `CMakeLists.txt` now detects a locationless
|
|
`OBS::libobs` and points it at the framework the bootstrap just built.
|
|
|
|
All six are confirmed fixed: the macOS job now downloads obs-deps and
|
|
obs-studio, builds libobs from source, builds and links the real adapter,
|
|
passes 6/6 tests, and uploads its artifact. `otool -L` on the result shows it
|
|
linked against libobs and `@rpath/liblivekit.dylib`.
|
|
|
|
### macOS packaging gap (known, unfixed)
|
|
|
|
**The macOS artifact will not load in OBS.app as it stands.** Two reasons,
|
|
neither of which CI can catch, because CI only proves it compiles and links:
|
|
|
|
1. It is a bare `streamer-tools-camera.so`. OBS on macOS loads plugins as
|
|
`<name>.plugin` bundles (`Contents/MacOS/<name>`, `Contents/Resources/`,
|
|
an `Info.plist`), which is what obs-plugintemplate's
|
|
`cmake/macos/helpers.cmake` builds and which this project deliberately did
|
|
not vendor.
|
|
2. `otool -L` shows the libobs dependency recorded as the relative path
|
|
`libobs/libobs.framework/Versions/A/libobs`, inherited from the
|
|
from-source libobs's own install name. A real plugin needs
|
|
`@rpath/libobs.framework/Versions/A/libobs` plus an `LC_RPATH` pointing at
|
|
`OBS.app/Contents/Frameworks`.
|
|
|
|
Fixing this means either vendoring the template's macOS bundle helpers or
|
|
adding an `install_name_tool` pass and a bundle layout — bounded work, but
|
|
work that has to be done and checked on an actual Mac. It is deliberately not
|
|
attempted here rather than guessed at.
|
|
|
|
### Where the Windows bootstrap got to
|
|
|
|
Windows is by far the slowest job — the `lukka/get-cmake` step alone takes
|
|
7-15 minutes on `winvm-builder`, and the runner serialises jobs, so a burst of
|
|
pushes leaves a queue that takes an hour to drain. One confirmed bug of its
|
|
own was found and fixed: upstream passes `-A x64,version=<Windows SDK>` to the
|
|
OBS sub-configure, and with a current CMake that `,version=` suffix reappears
|
|
verbatim in the sub-build's `CMAKE_VS_PLATFORM_NAME` — which obs-studio's own
|
|
dependency downloader uses as the architecture, sending it after
|
|
`windows-deps-2023-11-03-x64,version=10.0.26100.0.zip`:
|
|
|
|
```
|
|
string sub-command JSON member 'hashes windows-x64,version=10.0.26100.0' not found
|
|
Unable to download .../windows-deps-2023-11-03-x64,version=10.0.26100.0.zip
|
|
```
|
|
|
|
Plain `-A x64` now. **No Windows run has yet completed with that fix in
|
|
place**, so Windows should be treated as unverified beyond "the core library
|
|
and the WinHTTP backend compile and their tests pass", which earlier runs did
|
|
show. Expect further iterations there of the same kind the macOS bootstrap
|
|
needed.
|