Add the LiveKit session wrapper, verified end-to-end against a real room
stplugin::LiveKitSession wraps livekit::Room for exactly one subscribed slot: connect with the wsUrl/lkToken the API client minted, find the chosen participant's camera (and microphone), and hand decoded frames to callback-shaped handlers the OBS adapter can consume directly. Two architectural decisions worth recording, both forced by reading the SDK rather than guessed: 1. Frames come from VideoStream/AudioStream::fromTrack with our own reader threads, NOT from Room::setOnVideoFrameCallback. The dispatcher API is keyed by (participant identity, track NAME), which we cannot know before the track is published -- and disassembling liblivekit.so confirms that both Room::setOnVideoFrameCallback and the dispatcher's own setOnVideoFrameCallback merely record the registration: neither starts a reader for a track that is already subscribed. Registering after the subscription event, which is the only time the track name exists, would therefore have silently produced no video. Taking the shared_ptr<Track> straight off the TrackSubscribedEvent sidesteps the name entirely, and lets us pick the camera by TrackSource (streamer-tools publishes cameras as Source.Camera and screenshares separately -- apps/web/src/avatar/ publish.ts), which is what we actually mean. 2. Every stream operation runs on one owned worker thread, never on a room event thread. The SDK documents that Room::disconnect() from inside a delegate callback deadlocks, and Room's own event dispatch holds a mutex, so delegate callbacks only ever enqueue a command here. VideoStream::Options::capacity is set (3 frames) so the SDK's queue is a drop-oldest ring buffer: a stalled consumer can only fall three frames behind, and what it then sees is the newest frame rather than a backlog. That is the structural answer to the stale-media bug that motivated this plugin. The pure decision-making -- the state machine, track selection, frame geometry validation -- lives in session_types.h/.cpp with no LiveKit or OBS types, so it is unit-testable headlessly (81 checks in test_session, including the publisher-swap and reconnect transitions, plus the real connect() failure paths against the real SDK: unreachable host, garbage token, incomplete config, and destruction mid-connect). test_integration_livekit is the test that proves media actually flows. It publishes a synthetic camera and microphone into a real LiveKit room using the same SDK, subscribes through LiveKitSession, and asserts on the exact fields the OBS adapter will dereference. It skips (exit 0) unless STPLUGIN_IT_* is set, so the three build runners stay green; scripts/livekit-dev-room.py mints the tokens for a local `livekit-server --dev`. Verified locally against livekit-server 1.13.6 in dev mode: integration_livekit: 36 video frames, 323 audio frames, 10 state changes integration_livekit: 32 checks passed covering: connect; subscribe to the named participant's camera; 320x240 I420 frames with three planes, non-null plane pointers and strides >= the frame's own width; 48kHz audio; unpublish -> hasVideo() false, state stays Connected (a dark camera is the placeholder state, never an error) and NO further frames arrive from the dead publisher; republish -> video resumes; clean disconnect. One real finding from that run, now handled: WebRTC ramps a new subscription up from a downscaled spatial layer, so the first frames after (re)subscribing legitimately arrive smaller than what is being published. The OBS adapter must cope with a mid-stream resolution change; the test asserts per-frame geometry rather than the publisher's, and separately asserts the stream does reach full size. Full suite: ctest -> 6/6 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RL8abRmgFXkVASHkkqiJbE
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
/*
|
||||
streamer-tools OBS Camera Plugin - LiveKit session wrapper
|
||||
Copyright (C) 2026 CyberCoveLLC <jknapp85@gmail.com>
|
||||
|
||||
This program is free software; you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation; either version 2 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License along
|
||||
with this program. If not, see <https://www.gnu.org/licenses/>
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "stplugin/session_types.h"
|
||||
|
||||
namespace stplugin {
|
||||
|
||||
struct SessionConfig {
|
||||
/// LiveKit websocket URL, from POST /api/obs/:slug/token.
|
||||
std::string ws_url;
|
||||
/// LiveKit JWT, from the same call.
|
||||
std::string token;
|
||||
/// The slot's participant identity to subscribe to.
|
||||
std::string participant_identity;
|
||||
|
||||
/// Pixel format requested from the SDK. I420 costs no conversion on
|
||||
/// either side.
|
||||
PixelFormat video_format = PixelFormat::I420;
|
||||
|
||||
/// Ring-buffer depth for decoded video. Non-zero means the SDK drops the
|
||||
/// OLDEST frame when the queue is full, which is the structural answer to
|
||||
/// the stale-frame-after-publisher-swap bug that motivated this plugin
|
||||
/// (see the design doc's Approach section): a stalled consumer can only
|
||||
/// ever fall this far behind, and what it then sees is the newest frame,
|
||||
/// not a backlog.
|
||||
std::size_t video_queue_capacity = 3;
|
||||
|
||||
/// Same for audio. A little deeper because audio frames are 10ms each.
|
||||
std::size_t audio_queue_capacity = 20;
|
||||
|
||||
bool subscribe_audio = true;
|
||||
|
||||
/// How long connect() waits for the room to come up before giving up.
|
||||
int connect_timeout_ms = 15000;
|
||||
};
|
||||
|
||||
/// Wraps livekit::Room for exactly one subscribed slot.
|
||||
///
|
||||
/// Threading contract, which the OBS adapter depends on:
|
||||
/// - connect() and disconnect() are blocking and must be called from an
|
||||
/// ordinary thread. They must NOT be called from inside a handler this
|
||||
/// class invokes: the SDK documents that Room::disconnect() deadlocks if
|
||||
/// called from a room event callback, and Room's own callback registration
|
||||
/// is not re-entrant either.
|
||||
/// - The video and audio handlers are invoked on dedicated reader threads,
|
||||
/// one per track. Frame pointers are valid only for the duration of the
|
||||
/// call.
|
||||
/// - The state handler is invoked from whichever thread observed the
|
||||
/// change. It must not block and must not call back into this object.
|
||||
/// - All handlers must be installed before connect(); they are not
|
||||
/// synchronised against a running session.
|
||||
class LiveKitSession {
|
||||
public:
|
||||
LiveKitSession();
|
||||
~LiveKitSession();
|
||||
|
||||
LiveKitSession(const LiveKitSession &) = delete;
|
||||
LiveKitSession &operator=(const LiveKitSession &) = delete;
|
||||
|
||||
void setVideoHandler(VideoFrameHandler handler);
|
||||
void setAudioHandler(AudioFrameHandler handler);
|
||||
void setStateHandler(SessionStateHandler handler);
|
||||
|
||||
/// Connect and start subscribing. Returns true once the room is up; the
|
||||
/// selected slot's tracks may still arrive later (or not at all, if the
|
||||
/// camera is dark), which is reported through hasVideo()/the state
|
||||
/// handler rather than as a connect failure.
|
||||
bool connect(const SessionConfig &config);
|
||||
|
||||
/// Tear everything down. Safe to call when never connected, and safe to
|
||||
/// call twice.
|
||||
void disconnect();
|
||||
|
||||
SessionState state() const;
|
||||
std::string stateDetail() const;
|
||||
bool hasVideo() const;
|
||||
bool hasAudio() const;
|
||||
/// True when connected but the slot is not publishing: the source should
|
||||
/// show its placeholder, not an error.
|
||||
bool waitingForCamera() const;
|
||||
|
||||
/// Monotonic counters, for logging and for the adapter to tell "connected
|
||||
/// but silent" from "never started".
|
||||
std::uint64_t videoFrameCount() const;
|
||||
std::uint64_t audioFrameCount() const;
|
||||
|
||||
/// Process-wide SDK init/teardown. Reference-counted, so several sources
|
||||
/// can each hold one. The OBS adapter calls these from obs_module_load /
|
||||
/// obs_module_unload.
|
||||
static void globalInitialize();
|
||||
static void globalShutdown();
|
||||
|
||||
private:
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace stplugin
|
||||
@@ -0,0 +1,187 @@
|
||||
/*
|
||||
streamer-tools OBS Camera Plugin - session types and pure session logic
|
||||
Copyright (C) 2026 CyberCoveLLC <jknapp85@gmail.com>
|
||||
|
||||
This program is free software; you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation; either version 2 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License along
|
||||
with this program. If not, see <https://www.gnu.org/licenses/>
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
// Everything in this header is deliberately free of both LiveKit and OBS
|
||||
// types. That is what makes the session's decision-making unit-testable
|
||||
// headlessly: the parts of LiveKitSession that can go wrong without a server
|
||||
// -- which state a sequence of room events leaves us in, whether a given
|
||||
// published track is the one we want, and whether a frame's geometry is
|
||||
// self-consistent -- all live here, and LiveKitSession is the (much thinner)
|
||||
// piece that wires real SDK callbacks into them.
|
||||
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <string>
|
||||
|
||||
namespace stplugin {
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Media description (mirrors of the LiveKit enums, converted at the boundary)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
enum class MediaKind { Unknown, Audio, Video };
|
||||
|
||||
enum class MediaSource { Unknown, Camera, Microphone, Screenshare, ScreenshareAudio, Other };
|
||||
|
||||
/// Pixel formats this plugin is willing to receive. I420 is the default
|
||||
/// because it is what a WebRTC decoder produces and what OBS accepts
|
||||
/// natively, so neither side pays for a conversion.
|
||||
enum class PixelFormat { I420, NV12, BGRA };
|
||||
|
||||
const char *describePixelFormat(PixelFormat format);
|
||||
|
||||
/// Number of planes a format uses (3 for I420, 2 for NV12, 1 for BGRA).
|
||||
int planeCount(PixelFormat format);
|
||||
|
||||
/// Total bytes a tightly-packed frame of this format and geometry occupies.
|
||||
/// Returns 0 for non-positive dimensions. Used to reject a frame whose
|
||||
/// buffer does not match its claimed size before its pointers reach OBS.
|
||||
std::size_t expectedFrameBytes(PixelFormat format, int width, int height);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Track selection
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Does this published track belong to the slot we were asked to show, and is
|
||||
/// it the camera (never the screenshare)?
|
||||
///
|
||||
/// streamer-tools publishes cameras as Track.Source.Camera and screenshares
|
||||
/// as Track.Source.ScreenShare (apps/web/src/avatar/publish.ts), so the
|
||||
/// source is the reliable discriminator. A track that reports no source at
|
||||
/// all is accepted on kind alone rather than dropped, since an unknown source
|
||||
/// on a video track from the right participant is far more likely to be a
|
||||
/// camera than anything else.
|
||||
bool isWantedVideoTrack(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
|
||||
MediaSource source);
|
||||
|
||||
/// Same, for the slot's microphone.
|
||||
bool isWantedAudioTrack(const std::string &wanted_identity, const std::string &track_identity, MediaKind kind,
|
||||
MediaSource source);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Session state
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
enum class SessionState {
|
||||
/// Never asked to connect, or fully torn down.
|
||||
Idle,
|
||||
/// connect() is in flight.
|
||||
Connecting,
|
||||
/// Signalling is up. Says nothing about whether video is arriving --
|
||||
/// that is what hasVideo() is for.
|
||||
Connected,
|
||||
/// The SDK is re-establishing the connection on its own.
|
||||
Reconnecting,
|
||||
/// The room ended: either we disconnected, or the server did.
|
||||
Disconnected,
|
||||
/// connect() failed, or the session died in a way retrying will not fix
|
||||
/// (a rejected token, a duplicate identity).
|
||||
Failed,
|
||||
};
|
||||
|
||||
const char *describeSessionState(SessionState state);
|
||||
|
||||
/// The pure state machine behind LiveKitSession. Not thread-safe on its own;
|
||||
/// LiveKitSession owns the lock.
|
||||
///
|
||||
/// It exists separately so the transitions that matter operationally -- a
|
||||
/// publisher swap must not read as a failure, a reconnect must not read as a
|
||||
/// fresh connection, a failure detail must survive until the next connect --
|
||||
/// can be tested without a LiveKit server.
|
||||
class SessionStateMachine {
|
||||
public:
|
||||
SessionState state() const { return state_; }
|
||||
|
||||
/// Human-readable reason for the current state. Empty when there is
|
||||
/// nothing to say. Never contains a token or read key.
|
||||
const std::string &detail() const { return detail_; }
|
||||
|
||||
/// Whether a video track is currently attached and delivering.
|
||||
bool hasVideo() const { return has_video_; }
|
||||
bool hasAudio() const { return has_audio_; }
|
||||
|
||||
/// True when the source should be showing its "waiting for camera"
|
||||
/// placeholder rather than an error: we are up, the slot just isn't live.
|
||||
bool waitingForCamera() const;
|
||||
|
||||
void onConnectRequested();
|
||||
void onConnectSucceeded();
|
||||
void onConnectFailed(const std::string &reason);
|
||||
void onReconnecting();
|
||||
void onReconnected();
|
||||
/// The server (or the SDK) ended the room. `fatal` distinguishes a reason
|
||||
/// that retrying cannot fix from an ordinary drop.
|
||||
void onRoomEnded(const std::string &reason, bool fatal);
|
||||
void onLocalDisconnect();
|
||||
|
||||
void onVideoAttached();
|
||||
void onVideoDetached();
|
||||
void onAudioAttached();
|
||||
void onAudioDetached();
|
||||
|
||||
private:
|
||||
SessionState state_ = SessionState::Idle;
|
||||
std::string detail_;
|
||||
bool has_video_ = false;
|
||||
bool has_audio_ = false;
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Frames handed to the OBS adapter
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
struct VideoPlane {
|
||||
const std::uint8_t *data = nullptr;
|
||||
std::uint32_t stride = 0;
|
||||
std::uint32_t size = 0;
|
||||
};
|
||||
|
||||
/// A decoded video frame. All pointers are owned by the SDK and are valid
|
||||
/// only for the duration of the callback -- copy or consume synchronously.
|
||||
struct VideoFrameData {
|
||||
int width = 0;
|
||||
int height = 0;
|
||||
PixelFormat format = PixelFormat::I420;
|
||||
const std::uint8_t *data = nullptr;
|
||||
std::size_t size = 0;
|
||||
VideoPlane planes[4];
|
||||
int plane_count = 0;
|
||||
/// WebRTC capture-time timestamp, microseconds.
|
||||
std::int64_t timestamp_us = 0;
|
||||
};
|
||||
|
||||
/// Interleaved int16 PCM. client-sdk-cpp's AudioFrameCallback carries no
|
||||
/// timestamp, so the adapter stamps arrival time itself -- see the design
|
||||
/// doc's "Audio/video sync verification" note, which flags that as an
|
||||
/// assumption to check on real hardware rather than a guarantee.
|
||||
struct AudioFrameData {
|
||||
const std::int16_t *samples = nullptr;
|
||||
std::size_t sample_count = 0;
|
||||
int sample_rate = 0;
|
||||
int channels = 0;
|
||||
int samples_per_channel = 0;
|
||||
};
|
||||
|
||||
using VideoFrameHandler = std::function<void(const VideoFrameData &)>;
|
||||
using AudioFrameHandler = std::function<void(const AudioFrameData &)>;
|
||||
using SessionStateHandler = std::function<void(SessionState state, const std::string &detail)>;
|
||||
|
||||
} // namespace stplugin
|
||||
Reference in New Issue
Block a user