Best Practices

The key things to get right for a smooth, low-latency avatar pipeline with OjinSTVClient, whether you're prototyping straight to a browser, relaying media over WebRTC, or publishing directly into a Li

Essentials (every setup)

Run it server-side. The SDK connects to Ojin over a server-to-server WebSocket. Run it in a backend process, never on an end-user device, and keep your API key off the client. In production, run it in US East, close to Ojin's inference, for the lowest latency.
Feed audio as your TTS produces it. Call start_turn() once per utterance, then send_tts_audio() for each chunk as it arrives, even tiny 40 ms fragments. The SDK handles batching. Don't collect several seconds or the whole utterance yourself. See Optimizing Performance for the recommended initial audio amounts and tuning.
Present frames as they arrive. Don't re-sync. output_stream() is already paced to realtime 25 fps, with audio and video in sync and a small jitter buffer. Forward each frame the moment it arrives; never hold audio to wait for a video frame, and don't add your own A/V sync layer.
Handle the lifecycle. Wait for SESSION_READY before relying on the stream; use BOT_STARTED_SPEAKING / BOT_STOPPED_SPEAKING for UI state; on ERROR, retry with backoff (e.g. NO_BACKEND_SERVER_AVAILABLE).
Barge-in with interrupt(). Stop the current TTS producer and call interrupt() when the user starts talking. Call start_turn() before feeding the next utterance.

On Pipecat? pipecat-ojin's OjinVideoService already wires all of this into a pipeline and pushes frames into your transport, start there instead of hand-rolling the loop.

Setup A. Local, output to a browser (prototyping)

Run the SDK in a local Python process and stream frames to a browser you control (e.g. over a WebSocket to a <canvas>). Great for demos and development; not for production over real networks.

Forward the JPEG straight to the browser. Keep the default decoder and read the raw JPEG from STVVideoFrame.source_bytes, then draw it to a <canvas>/<img> as it arrives. It is empty (b"") on a held tick — the tick where no new server frame was ready — so skip those and keep showing the last image. Do not use a PassthroughDecoder to skip the decode: it makes the client emit no video frames at all, so the loop below never runs. (Need raw RGB in Python instead? The default decoder gives you frame.rgb as well.)
Play audio at the rate you fed. Run the browser AudioContext at the rate you pass to send_tts_audio(). STVAudioFrame.sample_rate reports it per frame, but frames emitted before your first TTS audio default to 16 kHz — don't configure the context from the first frame you see. You can feed higher-quality TTS (e.g. 24 kHz) for better sound; the SDK plays back your original audio while lip-sync uses a 16 kHz copy.
Present as they come. Draw each video frame and queue each audio chunk the instant it arrives, no re-sync; the stream is already aligned.

client = OjinSTVClient(
    api_key=..., config_id=...,
)
async for frame in client.output_stream():
    if isinstance(frame, STVVideoFrame):
        if frame.source_bytes:                         # empty on a held tick
            await ws.send_bytes(frame.source_bytes)    # raw JPEG -> draw on a canvas
    elif isinstance(frame, STVAudioFrame):
        await ws.send_bytes(frame.pcm)                 # play at frame.sample_rate

Setup B. Backend, relay to a WebRTC service (production)

Run the SDK on a backend and relay the media to your users over a WebRTC service (LiveKit, Daily, mediasoup, …), which absorbs packet loss, jitter, and varying networks that a raw WebSocket can't.

On LiveKit or Daily? Skip the relay and use Setup C below: Ojin publishes the avatar straight into your room, for lower latency and no media through your backend. Use Setup B when you need the frames in your process or use another WebRTC service.

Deploy in US East, close to Ojin's inference.
Push frames into the transport with their pts. Each STVAudioFrame / STVVideoFrame carries a pts timestamp in monotonic nanoseconds, so convert for your transport (frame.pts // 1000 for a microsecond API, frame.pts / 1e9 for seconds). Hand it to your WebRTC tracks and let the transport sync them. Don't re-sync yourself.
Push, don't poll, when you can. Inject a custom STVOutput sink to write frames straight into your transport instead of draining output_stream(), for lower latency and less glue. (This is exactly what pipecat-ojin does.)
Match the video track to the model's frame size: read STVVideoFrame.width / height and configure your outgoing track to match.
Deliver to end users over WebRTC, not the raw WebSocket. Keep the Ojin WebSocket strictly server-to-server.

async for frame in client.output_stream():
    if isinstance(frame, STVVideoFrame):
        video_track.push(frame.rgb, pts=frame.pts)     # feed your WebRTC track
    elif isinstance(frame, STVAudioFrame):
        audio_track.push(frame.pcm, pts=frame.pts)     # transport handles A/V sync

Setup C. Direct WebRTC into your LiveKit or Daily room (recommended for rooms)

If your viewers are in a LiveKit or Daily room, let Ojin publish the avatar straight into it. This gives viewers the lowest latency, and your backend relays no media: it only sends TTS audio and handles events.

Pass webrtc=WebRTCSettings(...) with the room URL and a token for the ojin-avatar participant. Mint the token on your backend and keep it there.
Set audio_sample_rate to your TTS output rate so audio reaches the room without resampling.
Give the join room to breathe. Set webrtc_join_timeout_s to about 90 s in production; it covers the room join and the model's cold start, which alone can take more than 30 s.
Drive UI state from events, not frames. output_stream() receives no frames in this mode; use WEBRTC_CONNECTED, BOT_STARTED_SPEAKING, and BOT_STOPPED_SPEAKING.
Treat ERROR with fatal=True as the end of the session. There is no fallback to the WebSocket; start a new session.
Keep your own bot from hearing the avatar. If your agent is also in the room, unsubscribe it from the avatar's microphone.

client = OjinSTVClient(
    api_key=..., config_id=...,
    webrtc=WebRTCSettings(provider="livekit", room_url=LIVEKIT_URL, token=avatar_token,
                          audio_sample_rate=24000, webrtc_join_timeout_s=90.0),
)

See Direct WebRTC (LiveKit & Daily) for credentials, events, errors, and Pipecat.

Essentials (every setup)

Setup A. Local, output to a browser (prototyping)

Setup B. Backend, relay to a WebRTC service (production)

Setup C. Direct WebRTC into your LiveKit or Daily room (recommended for rooms)

See also