> For the complete documentation index, see [llms.txt](https://docs.ojin.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ojin.ai/guides/optimizing-performance.md).
# Optimizing Performance
How to feed audio and play back frames for stable, low-latency lip-sync, and how the Python SDK and Pipecat handle most of it for you.
{% hint style="success" %}
**Using the** [**Python SDK**](/models/build-with-python-sdk.md) **or** [**Pipecat**](/models/introduction/integrations.md)**?** They already implement everything on this page, audio feeding, the playback clock, buffering, and audio/video sync. Read on if you want to tune them, or if you're building directly on the [raw WebSocket API](/models/introduction/api.md).
{% endhint %}
Each section below is tagged **๐ข SDK-handled** (automatic with the SDK or Pipecat, only your concern on the raw WebSocket API) or **๐ Your setup** (you own it regardless of integration, deployment, network, credentials).
## Feed audio for stable, low-latency lip-sync
๐ข **SDK-handled**. `OjinSTVClient` shapes the feed automatically; the contract below is only for the raw WebSocket API.
The model runs a fixed **25 fps** virtual timeline and needs your audio **input to stay slightly ahead** of that timeline to generate speech frames continuously. If you forward tiny TTS fragments one at a time (for example the \~40 ms chunks many TTS providers stream) your input rate matches the output rate, the model's lead never builds, and it **starves**: it falls back to idle frames in between, and lip-sync skips or drifts.
The rule is **lead, then realtime cadence**, not per-fragment forwarding:
1. **Lead with 2-3 seconds of audio** at the start of each turn, then stream the rest in real time as it arrives. The server needs about **1.3 seconds of audio before it can start generating**, and the lead is what keeps it ahead while your TTS keeps producing. It does not make running short impossible: a pause long enough to drain the lead still leaves the server waiting. If it runs short for more than about **280 ms** the server treats the turn as finished, and the avatar can visibly jump when your speech resumes. Keep the lead in the 2-3 s range rather than letting it grow: audio you have already sent is discarded on a barge-in.
2. Then send the **largest chunks you can**: coalesce queued fragments into **\~400 ms** sends (staying under the 512 KB message cap).
3. Stay truly realtime: never wait for the whole utterance before you start.
{% hint style="success" %}
**`OjinSTVClient` shapes the input for you.** The SDK accumulates an initial chunk after each `start_turn()` to establish the lead, then coalesces your TTS into โฅ400 ms sends automatically, so the inference head stays ahead for as long as your TTS keeps producing. It cannot cover a gap in your own audio: a pause long enough to drain the lead still runs the server short. You just feed audio as your TTS produces it; the SDK takes care of the input shape to optimize latency and stability (tunable via the `server_feed_*` fields on `STVConfig`, listed in the [`ojin-client` README](https://github.com/ojinai/python-sdk)). The contract above only matters if you drive the raw WebSocket API yourself.
{% endhint %}
```python
# With the SDK, feed audio as it arrives; shaping is automatic.
await client.start_turn()
async for chunk in tts_stream: # e.g. 40 ms ElevenLabs / Cartesia chunks
await client.send_tts_audio(chunk.pcm, sample_rate=chunk.rate, num_channels=1)
```
## Play back at realtime
๐ข **SDK-handled**: `output_stream()` is paced and buffered for you.
The model delivers frames at **realtime 25 fps**, so your buffer doesn't grow on its own. You just keep a **small buffer to absorb network jitter** (a few frames). How you consume them depends on your integration:
* **SDK / Pipecat**: `output_stream()` is already paced to realtime **25 fps with audio and video in sync**, and the client maintains the small jitter buffer for you (`initial_buffer_frames`). You don't manage buffering at all.
* **Raw WebSocket**: keep a small jitter buffer of a few frames before starting playback. If frames ever back up (e.g. a brief network stall, then a burst), trim idle frames (`frame_type == 0`) to recover, never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames. Drop a frame only when both the frame you just presented and the frame you are about to drop are idle, and stop as soon as the backlog is gone: trimming across a transition into speech leaves a visible gap. See [Buffer Management](/models/introduction/api.md#buffer-management) and [Audio and Video Synchronization](/models/introduction/api.md#audio-and-video-synchronization) in the API Reference.
{% hint style="info" %}
**Don't re-sync the SDK's output yourself.** `output_stream()` is already paced to 25 fps with audio and video aligned, so just present each frame **as it arrives**, never hold audio to wait for a video frame:
* **Playing directly (e.g. in a browser):** play audio and video frames as they come. No audio-clock or A/V re-sync logic needed.
* **Forwarding to a media transport (e.g. WebRTC):** each `STVAudioFrame` and `STVVideoFrame` carries a `pts` timestamp, pass it through and let the transport use it for sync. Don't build your own sync layer on top.
Building your own loop on the **raw WebSocket** instead? Every frame arrives with its own audio, so play that audio and render that image together. The server keeps them aligned, so you do not need an audio master clock there either. See [Audio and Video Synchronization](/models/introduction/api.md#audio-and-video-synchronization) in the API Reference.
{% endhint %}
## Where to run
๐ **Your setup**: deployment is yours to get right, with the SDK or the raw API.
The realtime API is a **WebSocket built for server-to-server delivery over a stable connection**. It is not meant to run on an end-user device on flaky WiโFi or mobile networks.
* **Run the client on a backend server**, not in the browser or on the user's device. This also keeps your API key off the client.
* **Deploy in US East**, where Ojin's realtime endpoint runs, for the lowest round-trip latency.
* **Deliver the final media to end users over a realtime transport** built for varying network conditions (typically **WebRTC**) rather than exposing the raw WebSocket to them.
* If your users are already on WebRTC, the SDK can have Ojin publish audio and video straight into your room (`OjinSTVWebRTCClient` with `WebRTCSettings`), instead of you draining frames and relaying them yourself.
## Publish directly into LiveKit or Daily
๐ **Your setup**: choose it when your viewers are in a LiveKit or Daily room.
If your viewers are in a **LiveKit** or **Daily** room, use [direct WebRTC](/models/build-with-python-sdk/python-sdk-webrtc.md). Ojin publishes the avatar straight into the room, which removes the relay hop through your backend and gives viewers the lowest latency.
* **Match `audio_sample_rate` to your TTS output rate** (for example 24000) so audio isn't resampled on the way to the room.
* **Set a generous `webrtc_join_timeout_s`**, about 30 s in production. It covers the room join and the model's cold start.
## Play back higher-quality audio
๐ข **SDK-handled**: the SDK plays back your original audio automatically.
On the WebSocket transport, `OjinSTVClient` plays back the **original audio you fed it**. It only resamples a separate 16 kHz copy to send to the model for lip-sync. Your listeners hear your own audio rather than a 16 kHz round-trip. The client may trim or pad a few frames at a turn boundary to hold lip-sync, and fades the tail when you call `interrupt()`.
So you can feed **higher-quality TTS** (for example 24 kHz instead of 16 kHz) for better sound while lip-sync still works on the 16 kHz copy. This is an SDK-side capability: on the raw WebSocket API the audio you send must be 16 kHz mono PCM int16. Just make sure your player runs at the rate you fed โ `STVAudioFrame.sample_rate` reports it per frame, but frames emitted before your first `send_tts_audio()` default to 16 kHz, so don't configure a player from the very first frame.
With [direct WebRTC](/models/build-with-python-sdk/python-sdk-webrtc.md) there is no local playback: set `audio_sample_rate` to your TTS rate and the room receives that audio without resampling.
## Relay frames without re-encoding
๐ข **SDK-handled**: the SDK exposes the raw JPEG (`source_bytes`) for you.
If you forward video frames to a browser or media transport that decodes JPEG itself, skip the decode:
* The **default decoder** populates `STVVideoFrame.rgb` (decoded RGB) and also keeps the raw JPEG in `STVVideoFrame.source_bytes`.
* `source_bytes` is **empty (`b""`) on a held tick** (no new server frame that tick), so skip those rather than relaying a zero-byte payload. `rgb` still carries the previous image if you need something on every tick.
{% hint style="warning" %}
**Do not reach for `PassthroughDecoder` to skip the decode.** It makes the client emit no video frames at all, so your relay loop silently never runs. Its `decode()` returns `None` for every frame, and the client only ever emits a video frame once it has an RGB image to attach, so it drops every tick instead. Keep the default decoder and read `source_bytes`.
{% endhint %}
## Keep latency low
๐ข **SDK-handled** ยท ๐ **Your setup** (deployment)
Keep your buffers as small as your network allows; large buffers add latency. On the raw API, tune your target buffer size by watching it during playback: low enough to minimise latency, high enough to absorb jitter without starving playback. With the SDK, the relevant knobs live on `STVConfig` (`initial_buffer_frames`, `max_buffered_video_frames`) โ see the [`ojin-client` README](https://github.com/ojinai/python-sdk) for the full list.
## Next steps
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.ojin.ai/guides/optimizing-performance.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.