Optimizing Performance
How to feed audio and play back frames for stable, low-latency lip-sync — and how the Python SDK and Pipecat handle most of it for you.
Using the Python SDK or Pipecat? They already implement everything on this page — audio feeding, the playback clock, buffering, and audio/video sync. Read on if you want to tune them, or if you're building directly on the raw WebSocket API.
Each section below is tagged 🟢 SDK-handled (automatic with the SDK or Pipecat — only your concern on the raw WebSocket API) or 🟠 Your setup (you own it regardless of integration — deployment, network, credentials).
Feed audio for stable, low-latency lip-sync
🟢 SDK-handled — OjinSTVClient shapes the feed automatically; the contract below is only for the raw WebSocket API.
The model runs a fixed 25 fps virtual timeline and needs your audio input to stay slightly ahead of that timeline to generate speech frames continuously. If you forward tiny TTS fragments one at a time — for example the ~40 ms chunks many TTS providers stream — your input rate matches the output rate, the model's lead never builds, and it starves: it falls back to idle frames in between, and lip-sync skips or drifts.
The rule is realtime cadence with a cushion, not per-fragment forwarding:
Prime ~1 second of audio at the start of each turn before you rely on realtime delivery.
Then send the largest chunks you can — coalesce queued fragments into ~400 ms sends (staying under the 512 KB message cap).
Stay truly realtime: never wait for the whole utterance before you start.
OjinSTVClient shapes the input for you. The SDK accumulates a ~1 s initial chunk after each start_turn() to establish the lead, then coalesces your TTS into ≥400 ms sends automatically — so the inference head never starves. You just feed audio as your TTS produces it; the SDK takes care of the input shape to optimize latency and stability (tunable via the server_feed_* fields on STVConfig). The contract above only matters if you drive the raw WebSocket API yourself.
# With the SDK, feed audio as it arrives — shaping is automatic.
await client.start_turn()
async for chunk in tts_stream: # e.g. 40 ms ElevenLabs / Cartesia chunks
await client.send_tts_audio(chunk.pcm, sample_rate=chunk.rate, num_channels=1)Play back at realtime
🟢 SDK-handled — output_stream() is paced and buffered for you.
The model delivers frames at realtime 25 fps, so your buffer doesn't grow on its own — you just keep a small buffer to absorb network jitter (a few frames). How you consume them depends on your integration:
SDK / Pipecat —
output_stream()is already paced to realtime 25 fps with audio and video in sync, and the client maintains the small jitter buffer for you (initial_buffer_frames). You don't manage buffering at all.Raw WebSocket — keep a small jitter buffer of a few frames before starting playback. If frames ever back up (e.g. a brief network stall, then a burst), trim idle frames (
frame_type == 0) to recover — never drop speech (1), start-of-speech (3), or fade-out (2) frames. See Buffer Management and Audio and Video Synchronization in the API Reference.
Don't re-sync the SDK's output yourself. output_stream() is already paced to 25 fps with audio and video aligned, so just present each frame as it arrives — never hold audio to wait for a video frame:
Playing directly (e.g. in a browser): play audio and video frames as they come. No audio-clock or A/V re-sync logic needed.
Forwarding to a media transport (e.g. WebRTC): each
STVAudioFrameandSTVVideoFramecarries aptstimestamp — pass it through and let the transport use it for sync. Don't build your own sync layer on top.
Building your own loop on the raw WebSocket instead? That's where the audio-as-clock technique applies — see Audio and Video Synchronization in the API Reference.
Where to run
🟠 Your setup — deployment is yours to get right, with the SDK or the raw API.
The realtime API is a WebSocket built for server-to-server delivery over a stable connection — it is not meant to run on an end-user device on flaky Wi‑Fi or mobile networks.
Run the client on a backend server, not in the browser or on the user's device. This also keeps your API key off the client.
Deploy in US East, close to Ojin's inference, for the lowest round-trip latency.
Deliver the final media to end users over a realtime transport built for varying network conditions — typically WebRTC — rather than exposing the raw WebSocket to them.
Play back higher-quality audio
🟢 SDK-handled — the SDK plays back your original audio automatically.
OjinSTVClient plays back the original audio you fed it — it only resamples a separate 16 kHz copy to send to the model for lip-sync. Your listeners hear exactly the audio you sent.
So you can feed higher-quality TTS (for example 24 kHz instead of 16 kHz) for better sound while lip-sync still works on the 16 kHz copy. Just make sure your player runs at the same sample rate you fed — STVAudioFrame.sample_rate reports it per frame.
Relay frames without re-encoding
🟢 SDK-handled — the SDK exposes the raw JPEG (source_bytes) for you.
If you forward video frames to a browser or media transport that decodes JPEG itself, skip the decode:
The default decoder populates
STVVideoFrame.rgb(decoded RGB) and also keeps the raw JPEG inSTVVideoFrame.source_bytes.A
PassthroughDecoderskips the decode entirely:rgbisNoneandoutput_stream()won't decode for you — read the raw JPEG fromsource_bytesand relay it directly, no re-encode.
Keep latency low
🟢 SDK-handled · 🟠 Your setup (deployment)
Keep your buffers as small as your network allows — large buffers add latency. On the raw API, tune your target buffer size by watching it during playback: low enough to minimise latency, high enough to absorb jitter without starving playback. With the SDK, the relevant knobs live on STVConfig (initial_buffer_frames, max_buffered_video_frames).
Next steps
Troubleshooting
Symptoms, causes, and fixes
Python SDK Best Practices
Working pipelines for browser output and WebRTC backends
Last updated
Was this helpful?