> For the complete documentation index, see [llms.txt](https://docs.ojin.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ojin.ai/guides/optimizing-performance.md).

# Optimizing Performance

> How to feed audio and play back frames for stable, low-latency lip-sync — and how the Python SDK and Pipecat handle most of it for you.

{% hint style="success" %}
**Using the** [**Python SDK**](/models/build-with-python-sdk.md) **or** [**Pipecat**](/models/introduction/integrations.md)**?** They already implement everything on this page — audio feeding, the playback clock, buffering, and audio/video sync. Read on if you want to tune them, or if you're building directly on the [raw WebSocket API](/models/introduction/api.md).
{% endhint %}

Each section below is tagged **🟢 SDK-handled** (automatic with the SDK or Pipecat — only your concern on the raw WebSocket API) or **🟠 Your setup** (you own it regardless of integration — deployment, network, credentials).

## Feed audio for stable, low-latency lip-sync

🟢 **SDK-handled** — `OjinSTVClient` shapes the feed automatically; the contract below is only for the raw WebSocket API.

The model runs a fixed **25 fps** virtual timeline and needs your audio **input to stay slightly ahead** of that timeline to generate speech frames continuously. If you forward tiny TTS fragments one at a time — for example the \~40 ms chunks many TTS providers stream — your input rate matches the output rate, the model's lead never builds, and it **starves**: it falls back to idle frames in between, and lip-sync skips or drifts.

The rule is **realtime cadence with a cushion**, not per-fragment forwarding:

1. **Prime \~1 second of audio** at the start of each turn before you rely on realtime delivery.
2. Then send the **largest chunks you can** — coalesce queued fragments into **\~400 ms** sends (staying under the 512 KB message cap).
3. Stay truly realtime: never wait for the whole utterance before you start.

{% hint style="success" %}
**`OjinSTVClient` shapes the input for you.** The SDK accumulates a \~1 s initial chunk after each `start_turn()` to establish the lead, then coalesces your TTS into ≥400 ms sends automatically — so the inference head never starves. You just feed audio as your TTS produces it; the SDK takes care of the input shape to optimize latency and stability (tunable via the `server_feed_*` fields on `STVConfig`). The contract above only matters if you drive the raw WebSocket API yourself.
{% endhint %}

```python
# With the SDK, feed audio as it arrives — shaping is automatic.
await client.start_turn()
async for chunk in tts_stream:          # e.g. 40 ms ElevenLabs / Cartesia chunks
    await client.send_tts_audio(chunk.pcm, sample_rate=chunk.rate, num_channels=1)
```

## Play back at realtime

🟢 **SDK-handled** — `output_stream()` is paced and buffered for you.

The model delivers frames at **realtime 25 fps**, so your buffer doesn't grow on its own — you just keep a **small buffer to absorb network jitter** (a few frames). How you consume them depends on your integration:

* **SDK / Pipecat** — `output_stream()` is already paced to realtime **25 fps with audio and video in sync**, and the client maintains the small jitter buffer for you (`initial_buffer_frames`). You don't manage buffering at all.
* **Raw WebSocket** — keep a small jitter buffer of a few frames before starting playback. If frames ever back up (e.g. a brief network stall, then a burst), trim idle frames (`frame_type == 0`) to recover — never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames. See [Buffer Management](/models/introduction/api.md#buffer-management) and [Audio and Video Synchronization](/models/introduction/api.md#audio-and-video-synchronization) in the API Reference.

{% hint style="info" %}
**Don't re-sync the SDK's output yourself.** `output_stream()` is already paced to 25 fps with audio and video aligned, so just present each frame **as it arrives** — never hold audio to wait for a video frame:

* **Playing directly (e.g. in a browser):** play audio and video frames as they come. No audio-clock or A/V re-sync logic needed.
* **Forwarding to a media transport (e.g. WebRTC):** each `STVAudioFrame` and `STVVideoFrame` carries a `pts` timestamp — pass it through and let the transport use it for sync. Don't build your own sync layer on top.

Building your own loop on the **raw WebSocket** instead? That's where the audio-as-clock technique applies — see [Audio and Video Synchronization](/models/introduction/api.md#audio-and-video-synchronization) in the API Reference.
{% endhint %}

## Where to run

🟠 **Your setup** — deployment is yours to get right, with the SDK or the raw API.

The realtime API is a **WebSocket built for server-to-server delivery over a stable connection** — it is not meant to run on an end-user device on flaky Wi‑Fi or mobile networks.

* **Run the client on a backend server**, not in the browser or on the user's device. This also keeps your API key off the client.
* **Deploy in US East**, close to Ojin's inference, for the lowest round-trip latency.
* **Deliver the final media to end users over a realtime transport** built for varying network conditions — typically **WebRTC** — rather than exposing the raw WebSocket to them.

## Play back higher-quality audio

🟢 **SDK-handled** — the SDK plays back your original audio automatically.

`OjinSTVClient` plays back the **original audio you fed it** — it only resamples a separate 16 kHz copy to send to the model for lip-sync. Your listeners hear exactly the audio you sent.

So you can feed **higher-quality TTS** (for example 24 kHz instead of 16 kHz) for better sound while lip-sync still works on the 16 kHz copy. Just make sure your player runs at the **same sample rate you fed** — `STVAudioFrame.sample_rate` reports it per frame.

## Relay frames without re-encoding

🟢 **SDK-handled** — the SDK exposes the raw JPEG (`source_bytes`) for you.

If you forward video frames to a browser or media transport that decodes JPEG itself, skip the decode:

* The **default decoder** populates `STVVideoFrame.rgb` (decoded RGB) and also keeps the raw JPEG in `STVVideoFrame.source_bytes`.
* A **`PassthroughDecoder`** skips the decode entirely: `rgb` is `None` and `output_stream()` won't decode for you — read the raw JPEG from `source_bytes` and relay it directly, no re-encode.

## Keep latency low

🟢 **SDK-handled** · 🟠 **Your setup** (deployment)

Keep your buffers as small as your network allows — large buffers add latency. On the raw API, tune your target buffer size by watching it during playback: low enough to minimise latency, high enough to absorb jitter without starving playback. With the SDK, the relevant knobs live on [`STVConfig`](/models/build-with-python-sdk.md) (`initial_buffer_frames`, `max_buffered_video_frames`).

## Next steps

### Troubleshooting

Symptoms, causes, and fixes

[Troubleshooting guide →](/guides/troubleshooting.md)

### Python SDK Best Practices

Working pipelines for browser output and WebRTC backends

[Best practices →](/models/build-with-python-sdk/python-sdk-best-practices.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.ojin.ai/guides/optimizing-performance.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.