Add the LiveKit session wrapper, verified end-to-end against a real room
Build / macOS (macos-latest) (push) Successful in 14s
Build / Linux (ubuntu-latest) (push) Successful in 41s
Build / Windows (windows-latest) (push) Failing after 8m18s

stplugin::LiveKitSession wraps livekit::Room for exactly one subscribed slot:
connect with the wsUrl/lkToken the API client minted, find the chosen
participant's camera (and microphone), and hand decoded frames to
callback-shaped handlers the OBS adapter can consume directly.

Two architectural decisions worth recording, both forced by reading the SDK
rather than guessed:

1. Frames come from VideoStream/AudioStream::fromTrack with our own reader
   threads, NOT from Room::setOnVideoFrameCallback. The dispatcher API is
   keyed by (participant identity, track NAME), which we cannot know before
   the track is published -- and disassembling liblivekit.so confirms that
   both Room::setOnVideoFrameCallback and the dispatcher's own
   setOnVideoFrameCallback merely record the registration: neither starts a
   reader for a track that is already subscribed. Registering after the
   subscription event, which is the only time the track name exists, would
   therefore have silently produced no video. Taking the shared_ptr<Track>
   straight off the TrackSubscribedEvent sidesteps the name entirely, and
   lets us pick the camera by TrackSource (streamer-tools publishes cameras
   as Source.Camera and screenshares separately -- apps/web/src/avatar/
   publish.ts), which is what we actually mean.

2. Every stream operation runs on one owned worker thread, never on a room
   event thread. The SDK documents that Room::disconnect() from inside a
   delegate callback deadlocks, and Room's own event dispatch holds a mutex,
   so delegate callbacks only ever enqueue a command here.

VideoStream::Options::capacity is set (3 frames) so the SDK's queue is a
drop-oldest ring buffer: a stalled consumer can only fall three frames
behind, and what it then sees is the newest frame rather than a backlog.
That is the structural answer to the stale-media bug that motivated this
plugin.

The pure decision-making -- the state machine, track selection, frame
geometry validation -- lives in session_types.h/.cpp with no LiveKit or OBS
types, so it is unit-testable headlessly (81 checks in test_session,
including the publisher-swap and reconnect transitions, plus the real
connect() failure paths against the real SDK: unreachable host, garbage
token, incomplete config, and destruction mid-connect).

test_integration_livekit is the test that proves media actually flows. It
publishes a synthetic camera and microphone into a real LiveKit room using
the same SDK, subscribes through LiveKitSession, and asserts on the exact
fields the OBS adapter will dereference. It skips (exit 0) unless
STPLUGIN_IT_* is set, so the three build runners stay green;
scripts/livekit-dev-room.py mints the tokens for a local `livekit-server
--dev`.

Verified locally against livekit-server 1.13.6 in dev mode:

  integration_livekit: 36 video frames, 323 audio frames, 10 state changes
  integration_livekit: 32 checks passed

covering: connect; subscribe to the named participant's camera; 320x240
I420 frames with three planes, non-null plane pointers and strides >= the
frame's own width; 48kHz audio; unpublish -> hasVideo() false, state stays
Connected (a dark camera is the placeholder state, never an error) and NO
further frames arrive from the dead publisher; republish -> video resumes;
clean disconnect.

One real finding from that run, now handled: WebRTC ramps a new subscription
up from a downscaled spatial layer, so the first frames after (re)subscribing
legitimately arrive smaller than what is being published. The OBS adapter
must cope with a mid-stream resolution change; the test asserts per-frame
geometry rather than the publisher's, and separately asserts the stream does
reach full size.

Full suite: ctest -> 6/6 passed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RL8abRmgFXkVASHkkqiJbE
This commit is contained in:
2026-09-06 21:39:49 -07:00
co-authored by Claude Sonnet 5
parent bf33966a4e
commit 80904a3e85
9 changed files with 2036 additions and 0 deletions
+118
View File
@@ -0,0 +1,118 @@
/*
streamer-tools OBS Camera Plugin - LiveKit session wrapper
Copyright (C) 2026 CyberCoveLLC <jknapp85@gmail.com>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program. If not, see <https://www.gnu.org/licenses/>
*/
#pragma once
#include <memory>
#include <string>
#include "stplugin/session_types.h"
namespace stplugin {
struct SessionConfig {
/// LiveKit websocket URL, from POST /api/obs/:slug/token.
std::string ws_url;
/// LiveKit JWT, from the same call.
std::string token;
/// The slot's participant identity to subscribe to.
std::string participant_identity;
/// Pixel format requested from the SDK. I420 costs no conversion on
/// either side.
PixelFormat video_format = PixelFormat::I420;
/// Ring-buffer depth for decoded video. Non-zero means the SDK drops the
/// OLDEST frame when the queue is full, which is the structural answer to
/// the stale-frame-after-publisher-swap bug that motivated this plugin
/// (see the design doc's Approach section): a stalled consumer can only
/// ever fall this far behind, and what it then sees is the newest frame,
/// not a backlog.
std::size_t video_queue_capacity = 3;
/// Same for audio. A little deeper because audio frames are 10ms each.
std::size_t audio_queue_capacity = 20;
bool subscribe_audio = true;
/// How long connect() waits for the room to come up before giving up.
int connect_timeout_ms = 15000;
};
/// Wraps livekit::Room for exactly one subscribed slot.
///
/// Threading contract, which the OBS adapter depends on:
/// - connect() and disconnect() are blocking and must be called from an
/// ordinary thread. They must NOT be called from inside a handler this
/// class invokes: the SDK documents that Room::disconnect() deadlocks if
/// called from a room event callback, and Room's own callback registration
/// is not re-entrant either.
/// - The video and audio handlers are invoked on dedicated reader threads,
/// one per track. Frame pointers are valid only for the duration of the
/// call.
/// - The state handler is invoked from whichever thread observed the
/// change. It must not block and must not call back into this object.
/// - All handlers must be installed before connect(); they are not
/// synchronised against a running session.
class LiveKitSession {
public:
LiveKitSession();
~LiveKitSession();
LiveKitSession(const LiveKitSession &) = delete;
LiveKitSession &operator=(const LiveKitSession &) = delete;
void setVideoHandler(VideoFrameHandler handler);
void setAudioHandler(AudioFrameHandler handler);
void setStateHandler(SessionStateHandler handler);
/// Connect and start subscribing. Returns true once the room is up; the
/// selected slot's tracks may still arrive later (or not at all, if the
/// camera is dark), which is reported through hasVideo()/the state
/// handler rather than as a connect failure.
bool connect(const SessionConfig &config);
/// Tear everything down. Safe to call when never connected, and safe to
/// call twice.
void disconnect();
SessionState state() const;
std::string stateDetail() const;
bool hasVideo() const;
bool hasAudio() const;
/// True when connected but the slot is not publishing: the source should
/// show its placeholder, not an error.
bool waitingForCamera() const;
/// Monotonic counters, for logging and for the adapter to tell "connected
/// but silent" from "never started".
std::uint64_t videoFrameCount() const;
std::uint64_t audioFrameCount() const;
/// Process-wide SDK init/teardown. Reference-counted, so several sources
/// can each hold one. The OBS adapter calls these from obs_module_load /
/// obs_module_unload.
static void globalInitialize();
static void globalShutdown();
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace stplugin
+187
View File
@@ -0,0 +1,187 @@
/*
streamer-tools OBS Camera Plugin - session types and pure session logic
Copyright (C) 2026 CyberCoveLLC <jknapp85@gmail.com>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program. If not, see <https://www.gnu.org/licenses/>
*/
#pragma once
// Everything in this header is deliberately free of both LiveKit and OBS
// types. That is what makes the session's decision-making unit-testable
// headlessly: the parts of LiveKitSession that can go wrong without a server
// -- which state a sequence of room events leaves us in, whether a given
// published track is the one we want, and whether a frame's geometry is
// self-consistent -- all live here, and LiveKitSession is the (much thinner)
// piece that wires real SDK callbacks into them.
#include <cstddef>
#include <cstdint>
#include <functional>
#include <string>
namespace stplugin {
// ---------------------------------------------------------------------------
// Media description (mirrors of the LiveKit enums, converted at the boundary)
// ---------------------------------------------------------------------------
enum class MediaKind { Unknown, Audio, Video };
enum class MediaSource { Unknown, Camera, Microphone, Screenshare, ScreenshareAudio, Other };
/// Pixel formats this plugin is willing to receive. I420 is the default
/// because it is what a WebRTC decoder produces and what OBS accepts
/// natively, so neither side pays for a conversion.
enum class PixelFormat { I420, NV12, BGRA };
const char *describePixelFormat(PixelFormat format);
/// Number of planes a format uses (3 for I420, 2 for NV12, 1 for BGRA).
int planeCount(PixelFormat format);
/// Total bytes a tightly-packed frame of this format and geometry occupies.
/// Returns 0 for non-positive dimensions. Used to reject a frame whose
/// buffer does not match its claimed size before its pointers reach OBS.
std::size_t expectedFrameBytes(PixelFormat format, int width, int height);
// ---------------------------------------------------------------------------
// Track selection
// ---------------------------------------------------------------------------
/// Does this published track belong to the slot we were asked to show, and is
/// it the camera (never the screenshare)?
///
/// streamer-tools publishes cameras as Track.Source.Camera and screenshares
/// as Track.Source.ScreenShare (apps/web/src/avatar/publish.ts), so the
/// source is the reliable discriminator. A track that reports no source at
/// all is accepted on kind alone rather than dropped, since an unknown source
/// on a video track from the right participant is far more likely to be a
/// camera than anything else.
bool isWantedVideoTrack(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
MediaSource source);
/// Same, for the slot's microphone.
bool isWantedAudioTrack(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
MediaSource source);
// ---------------------------------------------------------------------------
// Session state
// ---------------------------------------------------------------------------
enum class SessionState {
/// Never asked to connect, or fully torn down.
Idle,
/// connect() is in flight.
Connecting,
/// Signalling is up. Says nothing about whether video is arriving --
/// that is what hasVideo() is for.
Connected,
/// The SDK is re-establishing the connection on its own.
Reconnecting,
/// The room ended: either we disconnected, or the server did.
Disconnected,
/// connect() failed, or the session died in a way retrying will not fix
/// (a rejected token, a duplicate identity).
Failed,
};
const char *describeSessionState(SessionState state);
/// The pure state machine behind LiveKitSession. Not thread-safe on its own;
/// LiveKitSession owns the lock.
///
/// It exists separately so the transitions that matter operationally -- a
/// publisher swap must not read as a failure, a reconnect must not read as a
/// fresh connection, a failure detail must survive until the next connect --
/// can be tested without a LiveKit server.
class SessionStateMachine {
public:
SessionState state() const { return state_; }
/// Human-readable reason for the current state. Empty when there is
/// nothing to say. Never contains a token or read key.
const std::string &detail() const { return detail_; }
/// Whether a video track is currently attached and delivering.
bool hasVideo() const { return has_video_; }
bool hasAudio() const { return has_audio_; }
/// True when the source should be showing its "waiting for camera"
/// placeholder rather than an error: we are up, the slot just isn't live.
bool waitingForCamera() const;
void onConnectRequested();
void onConnectSucceeded();
void onConnectFailed(const std::string &reason);
void onReconnecting();
void onReconnected();
/// The server (or the SDK) ended the room. `fatal` distinguishes a reason
/// that retrying cannot fix from an ordinary drop.
void onRoomEnded(const std::string &reason, bool fatal);
void onLocalDisconnect();
void onVideoAttached();
void onVideoDetached();
void onAudioAttached();
void onAudioDetached();
private:
SessionState state_ = SessionState::Idle;
std::string detail_;
bool has_video_ = false;
bool has_audio_ = false;
};
// ---------------------------------------------------------------------------
// Frames handed to the OBS adapter
// ---------------------------------------------------------------------------
struct VideoPlane {
const std::uint8_t *data = nullptr;
std::uint32_t stride = 0;
std::uint32_t size = 0;
};
/// A decoded video frame. All pointers are owned by the SDK and are valid
/// only for the duration of the callback -- copy or consume synchronously.
struct VideoFrameData {
int width = 0;
int height = 0;
PixelFormat format = PixelFormat::I420;
const std::uint8_t *data = nullptr;
std::size_t size = 0;
VideoPlane planes[4];
int plane_count = 0;
/// WebRTC capture-time timestamp, microseconds.
std::int64_t timestamp_us = 0;
};
/// Interleaved int16 PCM. client-sdk-cpp's AudioFrameCallback carries no
/// timestamp, so the adapter stamps arrival time itself -- see the design
/// doc's "Audio/video sync verification" note, which flags that as an
/// assumption to check on real hardware rather than a guarantee.
struct AudioFrameData {
const std::int16_t *samples = nullptr;
std::size_t sample_count = 0;
int sample_rate = 0;
int channels = 0;
int samples_per_channel = 0;
};
using VideoFrameHandler = std::function<void(const VideoFrameData &)>;
using AudioFrameHandler = std::function<void(const AudioFrameData &)>;
using SessionStateHandler = std::function<void(SessionState state, const std::string &detail)>;
} // namespace stplugin