> For the complete documentation index, see [llms.txt](https://docs.ojin.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ojin.ai/models/introduction/api.md).

# API reference

{% hint style="info" %}
Most builders should use the [**Python SDK**](/models/build-with-python-sdk.md) or [**Pipecat**](/models/introduction/integrations.md). They implement this protocol, buffering, and audio/video sync for you. Read on only if you need low-level WebSocket control.
{% endhint %}

## Overview

Real-time talking head synthesis API. Send speech audio, receive synchronized video and audio frames.

After connecting and receiving `SessionReady`, the server immediately begins streaming video and audio frames at 25fps. It does not wait for you to send anything first. When no speech audio has been sent, the server generates **silence frames** (persona at rest with idle animation). When you send speech audio, the server generates **speech frames** with lip-synced animation synchronized to your audio.

You only need to send speech audio. No silence, padding, or keep-alive messages are required.

{% hint style="info" %}
**Production deployments:** This WebSocket API is intended for **server-to-server** use over a stable connection. Connect from a backend server rather than a front-end client, to keep your API key secure and because the raw WebSocket isn't built for flaky end-user networks. Run the backend in **US East**, close to Ojin's inference, for the lowest latency, and deliver the final media stream to end users over a transport built for varying network conditions, typically **WebRTC**, for smooth, reliable, low-latency playback. If your viewers are in a LiveKit or Daily room, request [direct WebRTC](#direct-webrtc) and Ojin publishes the media straight into that room.
{% endhint %}

***

## How It Works

1. **Connect** to the WebSocket endpoint with your API key and config ID
2. **Receive `SessionReady`**: the server has allocated inference resources for your session
3. **The server starts streaming frames immediately**: silence frames with idle animation, no request needed
4. **Send speech audio** whenever it becomes available, for example TTS output from your language model
5. **Receive speech frames**: the server transitions to lip-synced animation and returns to silence frames when audio runs out
6. **Render video frames** at 25fps, keeping a small jitter buffer (trim idle frames only if they back up after a stall)
7. **Play the audio payload that arrives with each frame** alongside that frame's image. The server returns them already aligned, so there is no audio clock to run

### Frame Types

Every frame arrives as a binary `InteractionResponse` containing both a JPEG image and a PCM audio chunk. Frames are always delivered in order. The `frame_type` field classifies each frame:

| `frame_type` | Description                                                                                                                                        |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`          | **Idle**, persona at rest with idle animation. Generated automatically when no speech audio is queued                                              |
| `1`          | **Speech**, lip-synced animation generated from your audio input                                                                                   |
| `2`          | **Fade-out**, post-cancel ramp back toward idle after an interruption                                                                              |
| `3`          | **Start of speech**, first speech frame of a turn **resuming after an interruption/cancel**. Natural (uninterrupted) turns start with `1`, not `3` |

### Buffering for Network Jitter

The server delivers frames at **realtime 25 fps**, so your buffer does not grow on its own. Keep a **small client-side buffer** (a few frames) to absorb network jitter and prevent stuttering during speech, and start playback once it's filled.

The right buffer size depends on your network conditions and latency requirements: keep it as low as possible to minimize latency, but high enough to absorb jitter without starving playback.

If frames ever back up, for example a brief network stall followed by a burst of queued frames arriving at once. You can recover by trimming **idle** frames (`frame_type == 0`). Only idle frames are safe to drop, never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames.

```python
# When consuming frames from the buffer:
frame = buffer.popleft()

# If frames backed up after a stall, recover by skipping every other idle frame.
# Test the frame you are about to drop, not the one you just took: an idle
# frame is often followed by speech, and dropping that leaves a media gap.
if len(buffer) > target_buffer_size and buffer[0].frame_type == 0:
    skip_counter += 1
    if skip_counter % 2 == 0:
        buffer.popleft()  # drop one idle frame
```

***

## Connection Flow

```mermaid
sequenceDiagram
    participant Client
    participant Server

    Note over Client,Server: Connection
    Client->>Server: WebSocket Connect
    Server->>Client: SessionReady (JSON)

    Note over Client,Server: Server Streams Immediately
    Server->>Client: Frame (idle, frame_type=0)
    Server->>Client: Frame (idle, frame_type=0)
    Server->>Client: Frame (idle, frame_type=0)

    Note over Client,Server: Client Sends Speech Audio
    Client->>Server: InteractionInput (TTS audio chunk 1)
    Client->>Server: InteractionInput (TTS audio chunk 2)

    Note over Client,Server: Server Transitions to Speech
    Server->>Client: Frame (start-of-speech, frame_type=3)
    Server->>Client: Frame (speech, frame_type=1)
    Server->>Client: Frame (speech, frame_type=1)
    Note right of Server: Delivered at realtime 25 fps

    Note over Client,Server: Audio Runs Out → Back to Idle
    Server->>Client: Frame (idle, frame_type=0)
    Server->>Client: Frame (idle, frame_type=0)
    Note right of Client: Client keeps a small jitter buffer
```

### Direct WebRTC

If you requested direct WebRTC in the [handshake](#websocket-handshake), the flow is the same with three differences:

1. **Check the outcome in `SessionReady`.** `payload.parameters.webrtc` reports whether the avatar joined your room:

   | `webrtc` value                                                                        | Meaning                                                                  |
   | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
   | `{"version": 2, "status": "connected", "provider": "daily", "participant_id": "..."}` | The avatar is in the room and publishes its audio and video there        |
   | `{"status": "failed", "error": {"code": "AUTH", "message": "..."}}`                   | The room join failed. `code` is `AUTH`, `NETWORK`, or `INVALID_SETTINGS` |
   | key absent                                                                            | This session doesn't support direct WebRTC                               |
2. **Send audio at your declared rate.** Once connected, send `InteractionInput` audio as mono int16 PCM at `webrtc_audio_sample_rate`, not 16 kHz.
3. **Frames carry metadata only.** Every `InteractionResponse` keeps the same header, including the trailing frame type, but has no image or audio payload. Use them for speaking state and timing; the media is in the room.

After connecting, the server may send a [`WebrtcStatus`](#webrtcstatus-server-client-json) message if the avatar loses the room.

***

## WebSocket Handshake

```
GET wss://models.ojin.ai/realtime?config_id=<your-config-id>
Authorization: <your-api-key>
```

Provide your API key in the `Authorization` header and the persona's `config_id` as a query parameter. The server upgrades the connection, sends `SessionReady`, and begins streaming frames immediately.

| Parameter                  | In     | Required | Description                                                                                           |
| -------------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------- |
| `Authorization`            | header | yes      | Your raw API key. No `Bearer` prefix                                                                  |
| `config_id`                | query  | yes      | Configuration ID for the persona. Create one in the dashboard, under Human Presence or Human Portrait |
| `webrtc_version`           | query  | no       | Direct WebRTC only. Always `2`                                                                        |
| `webrtc_provider`          | query  | no       | Direct WebRTC only. `livekit` or `daily`                                                              |
| `webrtc_room_url`          | query  | no       | Direct WebRTC only. URL-encoded LiveKit server URL (`wss://…`) or Daily room URL                      |
| `webrtc_audio_sample_rate` | query  | no       | Direct WebRTC only. Sample rate of the audio you send, e.g. `24000` (8000 to 48000, divisible by 25)  |
| `X-Ojin-Webrtc-Token`      | header | no       | Direct WebRTC only. The room token for the avatar participant. Send it as a header, never in the URL  |

{% hint style="info" %}
**Direct WebRTC.** Add the `webrtc_*` parameters and the `X-Ojin-Webrtc-Token` header to have Ojin publish the avatar straight into your LiveKit or Daily room instead of streaming media over this WebSocket. See [Direct WebRTC](#direct-webrtc) below. Using the Python SDK? Pass `webrtc=` instead; see [Direct WebRTC (LiveKit & Daily)](/models/build-with-python-sdk/python-sdk-webrtc.md).
{% endhint %}

| Status | Meaning                                                                                                              |
| ------ | -------------------------------------------------------------------------------------------------------------------- |
| `101`  | Upgrade successful. The server sends a `SessionReady` JSON message, then streams binary `InteractionResponse` frames |
| `403`  | No API key was supplied, or `config_id` is missing. The body is empty                                                |

An **invalid** key does not fail the handshake. The upgrade succeeds with `101`, and the rejection arrives on the open socket as an `errorResponse` with code `AUTH_FAILED`, followed by close code `1008`. Handle authentication failures on the message stream, not on the HTTP status.

Recommended client settings: `ping_interval` 30 seconds, `ping_timeout` 10 seconds.

***

## Message Format

{% hint style="info" %}
**Mixed message types:** Both JSON (text) and binary messages are exchanged on the same WebSocket connection. Your client must check the WebSocket frame type to distinguish them:

* **Text frames (JSON):** `SessionReady`, `ErrorResponse`, `WebrtcStatus` (server → client), `CancelInteraction` (client → server)
* **Binary frames:** `InteractionResponse` (server → client), `InteractionInput` (client → server)
  {% endhint %}

{% hint style="info" %}
**Byte order:** All multi-byte integer fields in binary messages use **network byte order (big-endian)**.
{% endhint %}

***

## Messages Reference

### Server → Client Messages

| Message                                                            | Frame  | Description                                                                                                                                    |
| ------------------------------------------------------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `SessionReady`                                                     | JSON   | Sent once after the connection is established and inference resources are allocated. The server begins streaming frames immediately afterwards |
| [`InteractionResponse`](#interactionresponse-server-client-binary) | binary | A video frame and its synchronized audio chunk. Streamed continuously: idle frames when at rest, speech frames when processing your audio      |
| [`ErrorResponse`](#errorresponse-server-client-json)               | JSON   | Sent when an error occurs. In some conditions, such as no backend server being available, the connection may close without one                 |
| [`WebrtcStatus`](#webrtcstatus-server-client-json)                 | JSON   | Direct WebRTC only. Sent when the avatar's room connection changes after `SessionReady`                                                        |

### Client → Server Messages

| Message                                                      | Frame  | Description                                                                                                                                                       |
| ------------------------------------------------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`InteractionInput`](#interactioninput-client-server-binary) | binary | Speech audio. Only send speech audio, never silence or padding                                                                                                    |
| [`CancelInteraction`](#cancelinteraction)                    | JSON   | Abort the current turn. Queued speech frames are discarded, then the server sends a `2` fade-out frame and continues streaming idle frames. Use for interruptions |

Field-level layouts for the binary messages are in [Message Details](#message-details) below.

***

## Message Details

### InteractionInput (Client → Server, Binary)

Binary message for sending speech audio to the server. **Only send speech audio**. Do not send silence or padding.

**Binary structure:**

```
[1 byte ]  Payload type       : uint8, always 1 for audio
[8 bytes]  Timestamp           : uint64, milliseconds since Unix epoch
[4 bytes]  Params size         : uint32, byte length of the JSON params block (0 if no params)
[N bytes]  Params JSON         : UTF-8 encoded JSON (only present if params size > 0)
[M bytes]  Audio payload       : raw PCM int16 speech audio data
```

**Header fields** use **big-endian** byte order. The PCM audio samples in the payload use **little-endian** (standard for PCM int16). In Python: `struct.pack('!BQI', payload_type, timestamp, params_size)`.

**Audio requirements:**

The server discards any payload whose bytes are all zero. It is dropped before ingest, produces no frames, and is not counted, so zero padding cannot be used to keep a stream alive.

| Property         | Value                                                                                                                                                                                                                                                       |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Format           | PCM signed 16-bit integers (little-endian samples)                                                                                                                                                                                                          |
| Sample rate      | 16,000 Hz                                                                                                                                                                                                                                                   |
| Channels         | 1 (mono)                                                                                                                                                                                                                                                    |
| Max message size | 512 KB (entire binary message including header). Do not rely on a particular failure mode for an oversized message: depending on where it is rejected you may get a `FRAME_SIZE_EXCEEDED` error or the connection may close without one. Stay under the cap |

**Recommended streaming pattern:**

The server needs about **1.3 seconds of audio before it can start generating**, and it runs a 25 fps virtual timeline that your input has to stay ahead of after that. Forwarding tiny TTS fragments one at a time (e.g. 40 ms chunks) makes input rate match output rate, so the server runs short and emits idle frames between speech, lip-sync skips or drifts. Instead, lead each turn, **not** per-fragment:

1. **Lead with 2-3 seconds of audio** at the start of the turn. This keeps the server from ever running short of its 1.3 s requirement mid-turn. If it does run short for more than about **280 ms**, it treats the turn as finished and the avatar can visibly jump when your speech resumes.
2. Then send the **largest chunks you can**, coalesce queued fragments into **\~400 ms** sends (under the 512 KB cap).
3. Stay realtime: never wait for the whole utterance before sending.

{% hint style="success" %}
The [**Python SDK**](/models/build-with-python-sdk.md) and [**Pipecat**](/models/introduction/integrations.md) shape the input for you. `OjinSTVClient` primes the lead and coalesces your TTS into large chunks automatically, so you just feed audio as it arrives. This pattern only applies if you drive this WebSocket API directly. See [Optimizing Performance](/guides/optimizing-performance.md).
{% endhint %}

```python
import struct, json, time

def build_audio_message(audio_bytes):
    header = struct.pack('!BQI',
        1,                         # payload type: audio
        int(time.time() * 1000),   # timestamp ms
        0,                         # params size (no per-chunk params are read today)
    )
    return header + audio_bytes
```

***

### InteractionResponse (Server → Client, Binary)

Binary message containing a video frame and synchronized audio. The server streams these continuously after `SessionReady`. **Frames always arrive in order.**

**Binary structure:**

```
[1 byte  ]  Is final flag      : uint8, always 0 on the realtime stream; reserved
[16 bytes]  Interaction ID      : UUID bytes (big-endian)
[8 bytes ]  Timestamp           : uint64, milliseconds since Unix epoch
[4 bytes ]  Usage               : uint32, usage metric for this response
[4 bytes ]  Index               : uint32, legacy frame classifier, superseded by the trailing frame-type byte; ignore
[4 bytes ]  Num payloads        : uint32, number of payload entries that follow

For each payload entry:
  [4 bytes]  Data size           : uint32, byte length of the payload data only
  [1 byte ]  Payload type        : uint8, 1 = audio, 2 = image
  [N bytes]  Payload data        : raw payload bytes

[1 byte ]  Frame type          : uint8, appended after all payload entries: 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
```

All multi-byte integers are **big-endian**. In Python: `struct.unpack('!B16sQIII', header_bytes)` for the main header, `struct.unpack('!IB', entry_bytes)` for each payload entry. The `Frame type` byte is a single trailing `uint8` after the last payload entry and is the authoritative frame classifier.

Do not use `is_final` as a turn or interaction boundary: the server sends `0` on every frame.

There is no definitive end-of-turn marker on the wire either. An idle frame (`0`) tells you the persona is currently at rest, not that the turn is over: a server that has run short of audio mid-utterance emits idle frames and then resumes speech. Treat idle as a state, and if you track turns, wait out a run of idle frames rather than ending the turn on the first one. A fade-out frame (`2`) is specific: it acknowledges a `CancelInteraction` you sent.

**Frame type:**

| Frame type | Meaning                                                                                                                |
| ---------- | ---------------------------------------------------------------------------------------------------------------------- |
| `0`        | **Idle**, persona at rest with idle animation                                                                          |
| `1`        | **Speech**, lip-synced animation from your audio                                                                       |
| `2`        | **Fade-out**, post-cancel ramp toward idle                                                                             |
| `3`        | **Start of speech**, first speech frame of a turn resuming after an interruption/cancel (natural turns start with `1`) |

**Payload types:**

| Type      | Format                | Typical size per frame                                                                                                                      |
| --------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 (audio) | PCM int16, 16kHz mono | **1,280 bytes** (640 samples = 40ms at 25fps)                                                                                               |
| 2 (image) | JPEG-encoded image    | Square, and fixed for the life of a session. Human Portrait delivers 1024x1024. Read the size off the first frame rather than hardcoding it |

**Parsing example:**

```python
import struct, uuid

HEADER_FMT = '!B16sQIII'
HEADER_SIZE = struct.calcsize(HEADER_FMT)   # 37 bytes
ENTRY_FMT = '!IB'
ENTRY_SIZE = struct.calcsize(ENTRY_FMT)     # 5 bytes

def parse_response(data):
    # The 5th header field is the legacy index; ignore it.
    is_final, uuid_bytes, timestamp, usage, _index, num_payloads = \
        struct.unpack(HEADER_FMT, data[:HEADER_SIZE])

    offset = HEADER_SIZE
    image = audio = None

    for _ in range(num_payloads):
        size, ptype = struct.unpack(ENTRY_FMT, data[offset:offset + ENTRY_SIZE])
        offset += ENTRY_SIZE
        payload = data[offset:offset + size]
        offset += size

        if ptype == 2:
            image = payload   # JPEG bytes
        elif ptype == 1:
            audio = payload   # PCM int16 bytes

    # Trailing frame type byte, appended after the payload entries.
    frame_type = data[offset]

    return {
        'is_final': bool(is_final),
        'frame_type': frame_type,    # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
        'image': image,
        'audio': audio,
    }
```

***

### CancelInteraction

Sent as a **text frame**:

```json
{"type": "cancelInteraction", "payload": {"timestamp": 1723567892000}}
```

| Message             | Purpose                | Server behavior                                                                                                                                                                                                       | Use case          |
| ------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `CancelInteraction` | Abort the current turn | Discards queued and in-flight speech frames. **The stream does not stop**: the server keeps delivering at 25 fps, sends one `frame_type` `2` (fade-out) frame to acknowledge the cut, then returns to `0` idle frames | User interruption |

A second `CancelInteraction` sent before any new audio has been accepted is ignored, so it cannot restart the fade. Wait for the acknowledging `2` frame before treating the cut as done.

***

### ErrorResponse (Server → Client, JSON)

{% hint style="warning" %}
**Plain text errors:** In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON `ErrorResponse`. Your client should handle non-JSON text messages gracefully.
{% endhint %}

**Error codes:**

| Code                  | Description                              |
| --------------------- | ---------------------------------------- |
| `AUTH_FAILED`         | Invalid API key                          |
| `UNAUTHORIZED`        | Caller lacks permission                  |
| `MISSING_CONFIG_ID`   | `config_id` query parameter not provided |
| `INVALID_MESSAGE`     | Malformed or unsupported message payload |
| `INVALID_HEADERS`     | Missing or invalid headers               |
| `MODEL_NOT_FOUND`     | Config ID not found or invalid           |
| `BACKEND_UNAVAILABLE` | No healthy inference backend available   |
| `RATE_LIMITED`        | Too many requests                        |
| `TIMEOUT`             | Operation exceeded processing time       |
| `CANCELLED`           | Interaction cancelled by client          |
| `INTERNAL_ERROR`      | Unexpected server error                  |
| `FRAME_SIZE_EXCEEDED` | Message exceeded 512KB limit             |

***

### WebrtcStatus (Server → Client, JSON)

Direct WebRTC only. Sent after `SessionReady` when the avatar loses your room:

```json
{
  "type": "webrtcStatus",
  "payload": {
    "status": "failed",
    "provider": "daily",
    "error": {"code": "REJOIN_FAILED", "message": "..."}
  }
}
```

| `status`       | Meaning                                                                                                                                |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `disconnected` | The avatar dropped out of the room and Ojin is trying to rejoin. Treat it as a warning                                                 |
| `failed`       | The rejoin failed (`error.code` is `REJOIN_FAILED`). The avatar is no longer in the room; close the connection and start a new session |

Ignore `status` values you don't recognize.

***

## Rate Limits & Constraints

| Constraint       | Value                      |
| ---------------- | -------------------------- |
| Rate limit       | 6 requests per second      |
| Max message size | 512 KB per message         |
| Video output     | 25 fps (realtime delivery) |

Exceeding limits results in an `ErrorResponse` with code `RATE_LIMITED`.

***

## Best Practices

### Audio Input

* **Send speech audio only.** The server starts streaming idle frames on its own, so there is no handshake frame to send first and no need to pad the gaps between utterances
* **Lead with 2-3 s of audio, then send the largest chunks you can (\~400 ms).** The server needs about 1.3 s of audio before it can start generating, and needs your input to stay ahead of its 25 fps timeline after that. Forwarding tiny fragments one at a time leaves it short and produces idle frames between speech, so coalesce what you send. See the **InteractionInput** message details above and [Optimizing Performance](/guides/optimizing-performance.md).
* The [Python SDK](/models/build-with-python-sdk.md) and [Pipecat](/models/introduction/integrations.md) handle this input shaping for you. This only matters when driving the WebSocket API directly.

### Buffer Management

* Play frames at **25 fps** (40ms per frame)
* **Play each frame's audio payload with that frame's image.** The server returns them aligned, so you do not need an audio master clock
* The server delivers at realtime 25 fps. Keep a **small jitter buffer** (a few frames); it won't grow on its own
* **Recovery:** if frames back up after a network stall, drop a frame only when both the frame you just presented and the frame you are about to drop are idle (`frame_type == 0`), and stop as soon as the backlog clears. That keeps every drop inside an idle stretch; trimming across a transition into speech leaves a visible gap
* **When recovering from jitter, only idle (`0`) frames are safe to drop.** Never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames. Clearing the buffer on a deliberate interruption is the exception, see [Interruption Handling](#interruption-handling)
* Tune your target buffer size based on your network conditions. Keep it as low as possible for minimal latency

### Audio and Video Synchronization

Each `InteractionResponse` carries a JPEG image and the audio for that same frame, 40 ms of PCM, and frames always arrive in order at realtime 25 fps. The server aligns the two for you. You do not need an audio master clock, and you do not need to gate playback on `frame_type`:

1. **Render each frame's image and play that frame's audio payload together**, in arrival order
2. Keep a **small jitter buffer** (a few frames) to absorb network jitter
3. **Render video from every frame**, whatever its `frame_type`

Speech frames carry back your own 16 kHz audio, sliced per frame. It round-trips through the model's float format, so individual samples can differ by one least-significant bit. That is inaudible, but it is not a bit-exact copy of your bytes. Idle and fade-out frames carry silence. Playing every frame's audio keeps the mouth and the voice locked together with no gating logic.

One trade to know about. When the server has no generated frame ready at a 40 ms tick, because your audio ran dry or generation fell behind, it emits a filler frame carrying silence to hold the 25 fps cadence. Playing per-frame audio reproduces that filler as a short gap in your speech. That is the running-short case covered in [Troubleshooting](/guides/troubleshooting.md), and a 2-3 second lead is what prevents it.

```python
# When TTS audio arrives from your speech service:
await ojin.send_audio(tts_audio_chunk)            # send to Ojin for lip-sync

# In your playback loop, present what arrived together:
frame = buffer.popleft()
render_video(frame.image)                         # always render the video
play_audio(frame.audio)                           # the matching 40 ms of audio
```

If you play your own copy of the TTS audio instead, you take the alignment back from the server: the voice is never interrupted, but nothing keeps the mouth on it. In that case you buffer your TTS locally, start playback on the first speech frame (`frame_type` `1` or `3`), and keep it running at a steady rate. Stop it when your own audio source ends, not on the first idle frame: an idle frame only means the persona is at rest right now, and one can appear mid-utterance when the server runs short, so stopping there cuts off speech that is still coming. A fade-out frame (`2`) is different: it acknowledges a `CancelInteraction` you sent, so it is a real stop signal.

### Error Handling

* Handle both JSON `ErrorResponse` messages and plain text error strings
* Implement exponential backoff for reconnection
* Monitor server `load` in the `SessionReady` message

### Interruption Handling

* Use `CancelInteraction` for immediate stops (e.g., user interrupts the bot)
* Then pick a strategy for what you have already buffered:
  * **Instant cut:** drop everything buffered and resume rendering from the next frame that arrives. Lowest latency, and the pose can visibly jump.
  * **Smooth:** stop audio playback immediately but keep rendering the frames you hold, including the `2` fade-out frame the server sends. The avatar eases back to idle instead of snapping, at the cost of a few frames of latency.
* This is the one case where dropping non-idle frames is intended. The never-drop rule under Buffer Management applies to jitter recovery, not to a deliberate barge-in.

***

## Complete Example

```python
import asyncio
import json
import struct
import time
from collections import deque
import numpy as np
import websockets
from dotenv import load_dotenv
import os

load_dotenv()

API_KEY = os.getenv("OJIN_API_KEY", "")
CONFIG_ID = os.getenv("OJIN_CONFIG_ID", "")
URL = f"wss://models.ojin.ai/realtime?config_id={CONFIG_ID}"

SAMPLE_RATE = 16000
FPS = 25
TARGET_BUFFER = 10  # Tune based on your network conditions

def build_audio_message(audio_bytes):
    """Build a binary InteractionInput message."""
    header = struct.pack('!BQI', 1, int(time.time() * 1000), 0)
    return header + audio_bytes

def parse_response(data):
    """Parse a binary InteractionResponse message."""
    fmt = '!B16sQIII'
    hdr_size = struct.calcsize(fmt)
    # The 5th header field is reserved; ignore it.
    is_final, uid_bytes, ts, usage, _index, n_payloads = struct.unpack(fmt, data[:hdr_size])

    offset = hdr_size
    image = audio = None
    for _ in range(n_payloads):
        size, ptype = struct.unpack('!IB', data[offset:offset+5])
        offset += 5
        if ptype == 2:
            image = data[offset:offset+size]
        elif ptype == 1:
            audio = data[offset:offset+size]
        offset += size

    # Trailing frame type byte, appended after the payload entries.
    frame_type = data[offset]

    return {
        'is_final': bool(is_final),
        'frame_type': frame_type,    # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
        'image': image,
        'audio': audio,
    }

async def main():
    headers = {"Authorization": API_KEY}
    # For older websockets versions, use extra_headers instead.
    async with websockets.connect(URL, additional_headers=headers, ping_interval=30) as ws:
        # 1. Wait for SessionReady; the server starts streaming frames immediately after
        msg = json.loads(await ws.recv())
        assert msg["type"] == "sessionReady"
        print(f"Session ready: {msg['payload']}")

        buffer = deque()
        skip_counter = 0
        playback_started = False
        frame_count = 0
        audio_sent = False

        # 2. Receive and process frames. The server streams idle frames
        #    immediately after SessionReady; nothing needs to be sent first.
        async for data in ws:
            if isinstance(data, str):
                msg = json.loads(data)
                if msg.get("type") == "errorResponse":
                    print(f"Error: {msg['payload']}")
                    break
                continue

            frame = parse_response(data)
            buffer.append(frame)
            frame_count += 1

            # Wait for initial buffer before playback
            if not playback_started:
                if len(buffer) >= TARGET_BUFFER:
                    playback_started = True
                    print(f"Buffer filled ({TARGET_BUFFER} frames), starting playback")
                continue

            # Consume one frame
            if buffer:
                play_frame = buffer.popleft()

                # Drop excess idle frames when the buffer is too large. This example
                # consumes one frame per arrival, so a backlog never builds here; the branch
                # matters once you drain on a 40 ms timer instead.
                # Only idle (frame_type 0) is safe to drop; keep speech (1),
                # start-of-speech (3), and fade-out (2). Both the frame just
                # presented and the one about to be dropped must be idle, so the
                # drop always happens inside an idle stretch and never across a
                # transition into speech, which would leave a visible gap.
                if (len(buffer) > TARGET_BUFFER
                        and play_frame['frame_type'] == 0
                        and buffer[0]['frame_type'] == 0):
                    skip_counter += 1
                    if skip_counter % 2 == 0:
                        buffer.popleft()  # drop one idle frame

                kind = {0: "idle", 1: "speech", 2: "fade-out", 3: "start-of-speech"}.get(
                    play_frame['frame_type'], "?"
                )
                print(f"[{kind}] frame #{frame_count}, buffer={len(buffer)}")

                # In a real app: render play_frame['image'] and play play_frame['audio']

            # Demo: send speech audio after receiving some silence frames
            if frame_count == 50 and not audio_sent:
                # Synthetic one-shot test buffer, not the recommended live pattern:
                # with real TTS you send as audio arrives rather than withholding 3 s.
                t = np.linspace(0, 3.0, SAMPLE_RATE * 3, endpoint=False)
                audio_data = (32767 * 0.5 * np.sin(2 * np.pi * 440 * t)).astype(np.int16)
                chunk_size = SAMPLE_RATE * 3  # ~3s lead, 96 KB, under the 512 KB cap
                for i in range(0, len(audio_data), chunk_size):
                    chunk = audio_data[i:i + chunk_size]
                    await ws.send(build_audio_message(chunk.tobytes()))
                audio_sent = True
                print("Sent 3 seconds of speech audio")

            if frame_count > 200:
                break

asyncio.run(main())
```

***

## Troubleshooting

Symptoms, causes, and fixes, connection and auth, a starved/idle mouth, choppy playback, latency, and frame lag, live in the global [Troubleshooting guide](/guides/troubleshooting.md), where each entry is tagged 🟢 SDK-handled or 🟠 your setup.

***

## Example Implementation

A complete working Python example integrating Ojin Human Portrait with a speech-to-speech service (Hume EVI) is available here:

[**github.com/journee-live/speech-to-video-samples/tree/main/samples**](https://github.com/journee-live/speech-to-video-samples/tree/main/samples) (includes Hume STS → Human Portrait walkthroughs)

The repository demonstrates the full integration pattern: microphone capture → STS service → TTS audio → Ojin lip-sync → synchronized video and audio playback at 25fps. It includes the buffer management and frame handling approach described in [Best Practices](#best-practices) above.

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.ojin.ai/models/introduction/api.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.