Best Practices

The key things to get right for a smooth, low-latency avatar pipeline with OjinSTVClient — whether you're prototyping straight to a browser or running a backend that relays media over WebRTC.

Essentials (every setup)

Run it server-side. The SDK connects to Ojin over a server-to-server WebSocket. Run it in a backend process — never on an end-user device — and keep your API key off the client. In production, run it in US East, close to Ojin's inference, for the lowest latency.
Feed audio as your TTS produces it. Call start_turn() once per utterance, then send_tts_audio() for each chunk as it arrives — even tiny 40 ms fragments. The SDK shapes the feed (primes a ~1 s lead, then coalesces to ≥400 ms) so the model never starves. Don't batch the whole utterance yourself.
Present frames as they arrive — don't re-sync. output_stream() is already paced to realtime 25 fps, with audio and video in sync and a small jitter buffer. Forward each frame the moment it arrives; never hold audio to wait for a video frame, and don't add your own A/V sync layer.
Handle the lifecycle. Wait for SESSION_READY before relying on the stream; use BOT_STARTED_SPEAKING / BOT_STOPPED_SPEAKING for UI state; on ERROR, retry with backoff (e.g. NO_BACKEND_SERVER_AVAILABLE).
Barge-in with interrupt(). It fades the current audio and cancels the turn server-side — just call it when the user starts talking.

On Pipecat? pipecat-ojin's OjinVideoService already wires all of this into a pipeline and pushes frames into your transport — start there instead of hand-rolling the loop.

Setup A — Local, output to a browser (prototyping)

Run the SDK in a local Python process and stream frames to a browser you control (e.g. over a WebSocket to a <canvas>). Great for demos and development; not for production over real networks.

Forward the JPEG straight to the browser. Use a PassthroughDecoder so the SDK skips decoding — read the raw JPEG from STVVideoFrame.source_bytes (~60 KB) and draw it to a <canvas>/<img> as it arrives. (Need raw RGB in Python instead? Keep the default decoder and read frame.rgb.)
Play audio at the rate you fed. Run the browser AudioContext at the same sample rate as the audio you sent (STVAudioFrame.sample_rate). You can feed higher-quality TTS (e.g. 24 kHz) for better sound — the SDK plays back your original audio while lip-sync uses a 16 kHz copy.
Present as they come. Draw each video frame and queue each audio chunk the instant it arrives — no re-sync; the stream is already aligned.

client = OjinSTVClient(
    api_key=..., config_id=...,
    decoder=PassthroughDecoder(),     # forward JPEG; the browser decodes it
)
async for frame in client.output_stream():
    if isinstance(frame, STVVideoFrame):
        await ws.send_bytes(frame.source_bytes)        # raw JPEG -> draw on a canvas
    elif isinstance(frame, STVAudioFrame):
        await ws.send_bytes(frame.pcm)                 # play at frame.sample_rate

Setup B — Backend, relay to a WebRTC service (production)

Run the SDK on a backend and relay the media to your users over a WebRTC service (LiveKit, Daily, mediasoup, …), which absorbs packet loss, jitter, and varying networks that a raw WebSocket can't.

Deploy in US East, close to Ojin's inference.
Push frames into the transport with their pts. Each STVAudioFrame / STVVideoFrame carries a pts timestamp — hand it to your WebRTC audio/video tracks and let the transport sync them for the end user. Don't re-sync yourself.
Push, don't poll, when you can. Inject a custom STVOutput sink to write frames straight into your transport instead of draining output_stream() — lower latency and less glue. (This is exactly what pipecat-ojin does.)
Match the video track to the model's frame size — read STVVideoFrame.width / height and configure your outgoing track to match.
Deliver to end users over WebRTC, not the raw WebSocket. Keep the Ojin WebSocket strictly server-to-server.

async for frame in client.output_stream():
    if isinstance(frame, STVVideoFrame):
        video_track.push(frame.rgb, pts=frame.pts)     # feed your WebRTC track
    elif isinstance(frame, STVAudioFrame):
        audio_track.push(frame.pcm, pts=frame.pts)     # transport handles A/V sync

Essentials (every setup)

Setup A — Local, output to a browser (prototyping)

Setup B — Backend, relay to a WebRTC service (production)

See also