Ctrlk
Dashboard
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Optimizing Performance

How to feed audio and play back frames for stable, low-latency lip-sync, and how the Python SDK and Pipecat handle most of it for you.

Using the Python SDK or Pipecat? They already implement everything on this page, audio feeding, the playback clock, buffering, and audio/video sync. Read on if you want to tune them, or if you're building directly on the raw WebSocket API.

Each section below is tagged 🟢 SDK-handled (automatic with the SDK or Pipecat, only your concern on the raw WebSocket API) or 🟠 Your setup (you own it regardless of integration, deployment, network, credentials).

Feed audio for stable, low-latency lip-sync

🟢 SDK-handled. OjinSTVClient shapes the feed automatically; the contract below is only for the raw WebSocket API.

Start with at least 320 ms of audio for Presence or 200 ms for Portrait, then keep sending audio as your TTS produces it.

The recommended streaming pattern is:

  1. Start each turn with 500 ms for Presence or 400 ms for Portrait, then stream the rest in real time as it arrives.

  2. Then send the largest chunks you can: coalesce queued fragments into ~400 ms sends (staying under the 512 KB message cap).

  3. Stay truly realtime: never wait for the whole utterance before you start.

OjinSTVClient shapes the input for you. Feed audio as your TTS produces it. To use the recommended initial amounts, set STVConfig.server_feed_initial_chunk_ms to 500 for Presence or 400 for Portrait. The SDK handles batching automatically. A pause in your TTS or network can still interrupt playback. See the server_feed_* fields on STVConfig in the ojin-client README for tuning. The contract above only matters if you drive the raw WebSocket API yourself.

# With the SDK, feed audio as it arrives; shaping is automatic.
await client.start_turn()
async for chunk in tts_stream:          # e.g. 40 ms ElevenLabs / Cartesia chunks
    await client.send_tts_audio(chunk.pcm, sample_rate=chunk.rate, num_channels=1)

Play back at realtime

🟢 SDK-handled: output_stream() is paced and buffered for you.

The model delivers frames at realtime 25 fps, so your buffer doesn't grow on its own. You just keep a small buffer to absorb network jitter (a few frames). How you consume them depends on your integration:

  • SDK / Pipecat: output_stream() is already paced to realtime 25 fps with audio and video in sync, and the client maintains the small jitter buffer for you (initial_buffer_frames). You don't manage buffering at all.

  • Raw WebSocket: keep a small jitter buffer of a few frames before starting playback. If frames ever back up (e.g. a brief network stall, then a burst), trim idle frames (frame_type == 0) to recover, never drop speech (1), start-of-speech (3), or fade-out (2) frames. Drop a frame only when both the frame you just presented and the frame you are about to drop are idle, and stop as soon as the backlog is gone: trimming across a transition into speech leaves a visible gap. See Buffer Management and Audio and Video Synchronization in the API Reference.

Don't re-sync the SDK's output yourself. output_stream() is already paced to 25 fps with audio and video aligned, so just present each frame as it arrives, never hold audio to wait for a video frame:

  • Playing directly (e.g. in a browser): play audio and video frames as they come. No audio-clock or A/V re-sync logic needed.

  • Forwarding to a media transport (e.g. WebRTC): each STVAudioFrame and STVVideoFrame carries a pts timestamp, pass it through and let the transport use it for sync. Don't build your own sync layer on top.

Building your own loop on the raw WebSocket instead? Every frame arrives with its own audio, so play that audio and render that image together. The server keeps them aligned, so you do not need an audio master clock there either. See Audio and Video Synchronization in the API Reference.

Where to run

🟠 Your setup: deployment is yours to get right, with the SDK or the raw API.

The realtime API is a WebSocket built for server-to-server delivery over a stable connection. It is not meant to run on an end-user device on flaky Wi‑Fi or mobile networks.

  • Run the client on a backend server, not in the browser or on the user's device. This also keeps your API key off the client.

  • Deploy in US East, where Ojin's realtime endpoint runs, for the lowest round-trip latency.

  • Deliver the final media to end users over a realtime transport built for varying network conditions (typically WebRTC) rather than exposing the raw WebSocket to them.

  • If your users are already on WebRTC, the SDK can have Ojin publish audio and video straight into your room (OjinSTVWebRTCClient with WebRTCSettings), instead of you draining frames and relaying them yourself.

Publish directly into LiveKit or Daily

🟠 Your setup: choose it when your viewers are in a LiveKit or Daily room.

If your viewers are in a LiveKit or Daily room, use direct WebRTC. Ojin publishes the avatar straight into the room, which removes the relay hop through your backend and gives viewers the lowest latency.

  • Match audio_sample_rate to your TTS output rate (for example 24000) so audio isn't resampled on the way to the room.

  • Set a generous webrtc_join_timeout_s, about 90 s in production. It covers the room join and the model's cold start, which alone can take more than 30 s.

Play back higher-quality audio

🟢 SDK-handled: the SDK plays back your original audio automatically.

On the WebSocket transport, OjinSTVClient plays back the original audio you fed it. It only resamples a separate 16 kHz copy to send to the model for lip-sync. Your listeners hear your own audio rather than a 16 kHz round-trip. The client may trim or pad a few frames at a turn boundary to hold lip-sync, and fades the tail when you call interrupt().

So you can feed higher-quality TTS (for example 24 kHz instead of 16 kHz) for better sound while lip-sync still works on the 16 kHz copy. This is an SDK-side capability: on the raw WebSocket API the audio you send must be 16 kHz mono PCM int16. Just make sure your player runs at the rate you fed — STVAudioFrame.sample_rate reports it per frame, but frames emitted before your first send_tts_audio() default to 16 kHz, so don't configure a player from the very first frame.

With direct WebRTC there is no local playback: set audio_sample_rate to your TTS rate and the room receives that audio without resampling.

Relay frames without re-encoding

🟢 SDK-handled: the SDK exposes the raw JPEG (source_bytes) for you.

If you forward video frames to a browser or media transport that decodes JPEG itself, skip the decode:

  • The default decoder populates STVVideoFrame.rgb (decoded RGB) and also keeps the raw JPEG in STVVideoFrame.source_bytes.

  • source_bytes is empty (b"") on a held tick (no new server frame that tick), so skip those rather than relaying a zero-byte payload. rgb still carries the previous image if you need something on every tick.

Do not reach for PassthroughDecoder to skip the decode. It makes the client emit no video frames at all, so your relay loop silently never runs. Its decode() returns None for every frame, and the client only ever emits a video frame once it has an RGB image to attach, so it drops every tick instead. Keep the default decoder and read source_bytes.

Keep latency low

🟢 SDK-handled · 🟠 Your setup (deployment)

Keep your buffers as small as your network allows; large buffers add latency. On the raw API, tune your target buffer size by watching it during playback: low enough to minimise latency, high enough to absorb jitter without starving playback. With the SDK, the relevant knobs live on STVConfig (initial_buffer_frames, max_buffered_video_frames) — see the ojin-client README for the full list.

Next steps

Troubleshooting

Symptoms, causes, and fixes.

Python SDK Best Practices

Working pipelines for browser output and WebRTC backends.

PreviousDirect WebRTC (LiveKit & Daily)
NextTroubleshooting

Last updated

Was this helpful?