API reference

Most builders should use the Python SDK or Pipecat. They implement this protocol, buffering, and audio/video sync for you. Read on only if you need low-level WebSocket control.

Overview

Real-time talking head synthesis API. Send speech audio, receive synchronized video and audio frames.

After connecting and receiving SessionReady, the server immediately begins streaming video and audio frames at 25fps. It does not wait for you to send anything first. When no speech audio has been sent, the server generates silence frames (persona at rest with idle animation). When you send speech audio, the server generates speech frames with lip-synced animation synchronized to your audio.

Send your TTS audio as it arrives, including its natural quiet or silent pauses. No extra silent padding or keep-alive messages are required.

Production deployments: This WebSocket API is intended for server-to-server use over a stable connection. Connect from a backend server rather than a front-end client, to keep your API key secure and because the raw WebSocket isn't built for flaky end-user networks. Run the backend in US East, close to Ojin's inference, for the lowest latency, and deliver the final media stream to end users over a transport built for varying network conditions, typically WebRTC, for smooth, reliable, low-latency playback. If your viewers are in a LiveKit or Daily room, request direct WebRTC and Ojin publishes the media straight into that room.

---

How It Works

1. Connect to the WebSocket endpoint with your API key and config ID

2. Receive SessionReady: the server has allocated inference resources for your session

3. The server starts streaming frames immediately: silence frames with idle animation, no request needed

4. Send speech audio whenever it becomes available, for example TTS output from your language model

5. Receive speech frames: the server transitions to lip-synced animation and returns to silence frames when audio runs out

6. Render video frames at 25fps, keeping a small jitter buffer (trim idle frames only if they back up after a stall)

7. Play the audio payload that arrives with each frame alongside that frame's image. The server returns them already aligned, so there is no audio clock to run

Frame Types

Every frame arrives as a binary InteractionResponse containing both a JPEG image and a PCM audio chunk. Frames are always delivered in order. The frame_type field classifies each frame:

frame_type	Description
0	Idle, persona at rest with idle animation. Generated automatically when no speech audio is queued
1	Speech, animation generated from your audio input, including quiet pauses within the turn
2	Fade-out, post-cancel ramp back toward idle after an interruption
3	Start of speech, first speech frame of a turn resuming after an interruption/cancel. Natural (uninterrupted) turns start with 1, not 3

A speech frame can contain silent audio from your input. Its type does not indicate whether the samples are audible.

Buffering for Network Jitter

The server delivers frames at realtime 25 fps, so your buffer does not grow on its own. Keep a small client-side buffer (a few frames) to absorb network jitter and prevent stuttering during speech, and start playback once it's filled.

The right buffer size depends on your network conditions and latency requirements: keep it as low as possible to minimize latency, but high enough to absorb jitter without starving playback.

If frames ever back up, for example a brief network stall followed by a burst of queued frames arriving at once. You can recover by trimming idle frames (frame_type == 0). Only idle frames are safe to drop, never drop speech (1), start-of-speech (3), or fade-out (2) frames.

# When consuming frames from the buffer:
frame = buffer.popleft()

# If frames backed up after a stall, recover by skipping every other idle frame.
# Test the frame you are about to drop, not the one you just took: an idle
# frame is often followed by speech, and dropping that leaves a media gap.
if len(buffer) > target_buffer_size and buffer[0].frame_type == 0:
    skip_counter += 1
    if skip_counter % 2 == 0:
        buffer.popleft()  # drop one idle frame

---

Connection Flow

Direct WebRTC

If you requested direct WebRTC in the handshake, the flow is the same with three differences:

1. Check the outcome in SessionReady. payload.parameters.webrtc reports whether the avatar joined your room:

   webrtc value	Meaning
   {"version": 2, "status": "connected", "provider": "daily", "participant_id": "..."}	The avatar is in the room and publishes its audio and video there
   {"status": "failed", "error": {"code": "AUTH", "message": "..."}}	The room join failed. code is AUTH, NETWORK, or INVALID_SETTINGS
   key absent	This session doesn't support direct WebRTC

2. Send audio at your declared rate. Once connected, send InteractionInput audio as mono int16 PCM at webrtc_audio_sample_rate, not 16 kHz.

3. Frames carry metadata only. Every InteractionResponse keeps the same header, including the trailing frame type, but has no image or audio payload. Use them for speaking state and timing; the media is in the room.

After connecting, the server may send a WebrtcStatus message if the avatar loses the room.

---

WebSocket Handshake

GET wss://models.ojin.ai/realtime?config_id=<your-config-id>
Authorization: <your-api-key>

Provide your API key in the Authorization header and the persona's config_id as a query parameter. The server upgrades the connection, sends SessionReady, and begins streaming frames immediately.

Parameter	In	Required	Description
Authorization	header	yes	Your raw API key. No Bearer prefix
config_id	query	yes	Configuration ID for the persona. Create one in the dashboard, under Human Presence or Human Portrait
webrtc_version	query	no	Direct WebRTC only. Always 2
webrtc_provider	query	no	Direct WebRTC only. livekit or daily
webrtc_room_url	query	no	Direct WebRTC only. URL-encoded LiveKit server URL (wss://…) or Daily room URL
webrtc_audio_sample_rate	query	no	Direct WebRTC only. Sample rate of the audio you send, e.g. 24000 (8000 to 48000, divisible by 25)
X-Ojin-Webrtc-Token	header	no	Direct WebRTC only. The room token for the avatar participant. Send it as a header, never in the URL

Direct WebRTC. Add the webrtc_* parameters and the X-Ojin-Webrtc-Token header to have Ojin publish the avatar straight into your LiveKit or Daily room instead of streaming media over this WebSocket. See Direct WebRTC below. Using the Python SDK? Pass webrtc= instead; see Direct WebRTC (LiveKit & Daily).

Status	Meaning
101	Upgrade successful. The server sends a SessionReady JSON message, then streams binary InteractionResponse frames
403	No API key was supplied, or config_id is missing. The body is empty

An invalid key does not fail the handshake. The upgrade succeeds with 101, and the rejection arrives on the open socket as an errorResponse with code AUTH_FAILED, followed by close code 1008. Handle authentication failures on the message stream, not on the HTTP status.

Recommended client settings: ping_interval 30 seconds, ping_timeout 10 seconds.

---

Message Format

Mixed message types: Both JSON (text) and binary messages are exchanged on the same WebSocket connection. Your client must check the WebSocket frame type to distinguish them:

• Text frames (JSON): SessionReady, ErrorResponse, WebrtcStatus (server → client), CancelInteraction (client → server)

• Binary frames: InteractionResponse (server → client), InteractionInput (client → server)

Byte order: All multi-byte integer fields in binary messages use network byte order (big-endian).

---

Messages Reference

Server → Client Messages

Message	Frame	Description
SessionReady	JSON	Sent once after the connection is established and inference resources are allocated. The server begins streaming frames immediately afterwards
InteractionResponse	binary	A video frame and its synchronized audio chunk. Streamed continuously: idle frames when at rest, speech frames when processing your audio
ErrorResponse	JSON	Sent when an error occurs. In some conditions, such as no backend server being available, the connection may close without one
WebrtcStatus	JSON	Direct WebRTC only. Sent when the avatar's room connection changes after SessionReady

Client → Server Messages

Message	Frame	Description
InteractionInput	binary	Your source audio, including its natural pauses. No extra silent padding is required
CancelInteraction	JSON	Abort the current turn. Queued speech frames are discarded, then the server sends a 2 fade-out frame and continues streaming idle frames. Use for interruptions

Field-level layouts for the binary messages are in Message Details below.

---

Message Details

InteractionInput (Client → Server, Binary)

Binary message for sending your source audio to the server. Include natural pauses in your TTS audio, even when the PCM samples are all zero. The server generates idle frames when it has no source audio to process, so no extra silent padding is required.

Binary structure:

[1 byte ]  Payload type       : uint8, always 1 for audio
[8 bytes]  Timestamp           : uint64, milliseconds since Unix epoch
[4 bytes]  Params size         : uint32, byte length of the JSON params block (0 if no params)
[N bytes]  Params JSON         : UTF-8 encoded JSON (only present if params size > 0)
[M bytes]  Audio payload       : raw PCM int16 speech audio data

Header fields use big-endian byte order. The PCM audio samples in the payload use little-endian (standard for PCM int16). In Python: struct.pack('!BQI', payload_type, timestamp, params_size).

Audio requirements:

All-zero payloads are accepted and counted as input audio. Presence with steady small chunks preserves quiet source audio from the start of a turn, without an extra frame holdback. Other serving modes may trim a short, bounded quiet prefix. Quiet source audio within a turn retains its duration and speech-frame classification. All-zero input does not start the time-to-first-audible-frame measurement.

Property	Value
Format	PCM signed 16-bit integers (little-endian samples)
Sample rate	16,000 Hz
Channels	1 (mono)
Max message size	512 KB (entire binary message including header). Do not rely on a particular failure mode for an oversized message: depending on where it is rejected you may get a FRAME_SIZE_EXCEEDED error or the connection may close without one. Stay under the cap

Recommended streaming pattern:

The minimum initial audio is 320 ms for Presence or 200 ms for Portrait. Smaller messages are accepted.

1. Start each turn with 500 ms for Presence or 400 ms for Portrait, then stream the rest as it arrives.

2. Then send the largest chunks you can, coalesce queued fragments into ~400 ms sends (under the 512 KB cap).

3. Stay realtime: never wait for the whole utterance before sending.

The Python SDK and Pipecat shape the input for you. OjinSTVClient batches your TTS audio automatically, so you just feed audio as it arrives. This pattern only applies if you drive this WebSocket API directly. See Optimizing Performance.

import struct, json, time

def build_audio_message(audio_bytes):
    header = struct.pack('!BQI',
        1,                         # payload type: audio
        int(time.time() * 1000),   # timestamp ms
        0,                         # params size (no per-chunk params are read today)
    )
    return header + audio_bytes

---

InteractionResponse (Server → Client, Binary)

Binary message containing a video frame and synchronized audio. The server streams these continuously after SessionReady. Frames always arrive in order.

Binary structure:

[1 byte  ]  Is final flag      : uint8, always 0 on the realtime stream; reserved
[16 bytes]  Interaction ID      : UUID bytes (big-endian)
[8 bytes ]  Timestamp           : uint64, milliseconds since Unix epoch
[4 bytes ]  Usage               : uint32, usage metric for this response
[4 bytes ]  Index               : uint32, legacy frame classifier, superseded by the trailing frame-type byte; ignore
[4 bytes ]  Num payloads        : uint32, number of payload entries that follow

For each payload entry:
  [4 bytes]  Data size           : uint32, byte length of the payload data only
  [1 byte ]  Payload type        : uint8, 1 = audio, 2 = image
  [N bytes]  Payload data        : raw payload bytes

[1 byte ]  Frame type          : uint8, appended after all payload entries: 0=idle, 1=speech, 2=fade-out, 3=start-of-speech

All multi-byte integers are big-endian. In Python: struct.unpack('!B16sQIII', header_bytes) for the main header, struct.unpack('!IB', entry_bytes) for each payload entry. The Frame type byte is a single trailing uint8 after the last payload entry and is the authoritative frame classifier.

Do not use is_final as a turn or interaction boundary: the server sends 0 on every frame.

There is no definitive end-of-turn marker on the wire either. An idle frame (0) tells you the persona is currently at rest, not that the turn is over: a server that has run short of audio mid-utterance emits idle frames and then resumes speech. Treat idle as a state, and if you track turns, wait out a run of idle frames rather than ending the turn on the first one. A fade-out frame (2) is specific: it acknowledges a CancelInteraction you sent.

Frame type:

Frame type	Meaning
0	Idle, persona at rest with idle animation
1	Speech, animation from your audio, including quiet pauses within the turn
2	Fade-out, post-cancel ramp toward idle
3	Start of speech, first speech frame of a turn resuming after an interruption/cancel (natural turns start with 1)

Payload types:

Type	Format	Typical size per frame
1 (audio)	PCM int16, 16kHz mono	1,280 bytes (640 samples = 40ms at 25fps)
2 (image)	JPEG-encoded image	Square, and fixed for the life of a session. Human Portrait delivers 1024x1024. Read the size off the first frame rather than hardcoding it

Parsing example:

import struct, uuid

HEADER_FMT = '!B16sQIII'
HEADER_SIZE = struct.calcsize(HEADER_FMT)   # 37 bytes
ENTRY_FMT = '!IB'
ENTRY_SIZE = struct.calcsize(ENTRY_FMT)     # 5 bytes

def parse_response(data):
    # The 5th header field is the legacy index; ignore it.
    is_final, uuid_bytes, timestamp, usage, _index, num_payloads = \
        struct.unpack(HEADER_FMT, data[:HEADER_SIZE])

    offset = HEADER_SIZE
    image = audio = None

    for _ in range(num_payloads):
        size, ptype = struct.unpack(ENTRY_FMT, data[offset:offset + ENTRY_SIZE])
        offset += ENTRY_SIZE
        payload = data[offset:offset + size]
        offset += size

        if ptype == 2:
            image = payload   # JPEG bytes
        elif ptype == 1:
            audio = payload   # PCM int16 bytes

    # Trailing frame type byte, appended after the payload entries.
    frame_type = data[offset]

    return {
        'is_final': bool(is_final),
        'frame_type': frame_type,    # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
        'image': image,
        'audio': audio,
    }

---

CancelInteraction

Sent as a text frame:

{"type": "cancelInteraction", "payload": {"timestamp": 1723567892000}}

Message	Purpose	Server behavior	Use case
CancelInteraction	Abort the current turn	Discards queued and in-flight speech frames. The stream does not stop: the server keeps delivering at 25 fps, sends one frame_type 2 (fade-out) frame to acknowledge the cut, then returns to 0 idle frames	User interruption

A second CancelInteraction sent before any new audio has been accepted is ignored, so it cannot restart the fade. Wait for the acknowledging 2 frame before treating the cut as done.

---

ErrorResponse (Server → Client, JSON)

Plain text errors: In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON ErrorResponse. Your client should handle non-JSON text messages gracefully.

Error codes:

Code	Description
AUTH_FAILED	Invalid API key
UNAUTHORIZED	Caller lacks permission
MISSING_CONFIG_ID	config_id query parameter not provided
INVALID_MESSAGE	Malformed or unsupported message payload
INVALID_HEADERS	Missing or invalid headers
MODEL_NOT_FOUND	Config ID not found or invalid
BACKEND_UNAVAILABLE	No healthy inference backend available
RATE_LIMITED	Too many requests
TIMEOUT	Operation exceeded processing time
CANCELLED	Interaction cancelled by client
INTERNAL_ERROR	Unexpected server error
FRAME_SIZE_EXCEEDED	Message exceeded 512KB limit

---

WebrtcStatus (Server → Client, JSON)

Direct WebRTC only. Sent after SessionReady when the avatar loses your room:

{
  "type": "webrtcStatus",
  "payload": {
    "status": "failed",
    "provider": "daily",
    "error": {"code": "REJOIN_FAILED", "message": "..."}
  }
}

status	Meaning
disconnected	The avatar dropped out of the room and Ojin is trying to rejoin. Treat it as a warning
failed	The rejoin failed (error.code is REJOIN_FAILED). The avatar is no longer in the room; close the connection and start a new session

Ignore status values you don't recognize.

---

Rate Limits & Constraints

Constraint	Value
Rate limit	6 requests per second
Max message size	512 KB per message
Video output	25 fps (realtime delivery)

Exceeding limits results in an ErrorResponse with code RATE_LIMITED.

---

Best Practices

Audio Input

• Send your TTS audio with its natural pauses. The server starts streaming idle frames on its own, so there is no handshake frame to send first and no need to add silent padding between utterances

• Start with 500 ms of audio for Presence or 400 ms for Portrait, then send the largest chunks you can (~400 ms). The minimum initial audio is 320 ms for Presence or 200 ms for Portrait. See the InteractionInput message details above and Optimizing Performance.

• The Python SDK and Pipecat handle this input shaping for you. This only matters when driving the WebSocket API directly.

Buffer Management

• Play frames at 25 fps (40ms per frame)

• Play each frame's audio payload with that frame's image. The server returns them aligned, so you do not need an audio master clock

• The server delivers at realtime 25 fps. Keep a small jitter buffer (a few frames); it won't grow on its own

• Recovery: if frames back up after a network stall, drop a frame only when both the frame you just presented and the frame you are about to drop are idle (frame_type == 0), and stop as soon as the backlog clears. That keeps every drop inside an idle stretch; trimming across a transition into speech leaves a visible gap

• When recovering from jitter, only idle (0) frames are safe to drop. Never drop speech (1), start-of-speech (3), or fade-out (2) frames. Clearing the buffer on a deliberate interruption is the exception, see Interruption Handling

• Tune your target buffer size based on your network conditions. Keep it as low as possible for minimal latency

Audio and Video Synchronization

Each InteractionResponse carries a JPEG image and the audio for that same frame, 40 ms of PCM, and frames always arrive in order at realtime 25 fps. The server aligns the two for you. You do not need an audio master clock, and you do not need to gate playback on frame_type:

1. Render each frame's image and play that frame's audio payload together, in arrival order

2. Keep a small jitter buffer (a few frames) to absorb network jitter

3. Render video from every frame, whatever its frame_type

Speech frames carry back your own 16 kHz audio, sliced per frame. It round-trips through the model's float format, so individual samples can differ by one least-significant bit. That is inaudible, but it is not a bit-exact copy of your bytes. Idle and fade-out frames carry silence. Playing every frame's audio keeps the mouth and the voice locked together with no gating logic.

Playback may pause or include silence if audio delivery stalls. Use the recommended initial amount and keep subsequent audio flowing. See Troubleshooting.

# When TTS audio arrives from your speech service:
await ojin.send_audio(tts_audio_chunk)            # send to Ojin for lip-sync

# In your playback loop, present what arrived together:
frame = buffer.popleft()
render_video(frame.image)                         # always render the video
play_audio(frame.audio)                           # the matching 40 ms of audio

If you play your own copy of the TTS audio instead, you take the alignment back from the server: the voice is never interrupted, but nothing keeps the mouth on it. In that case you buffer your TTS locally, start playback on the first speech frame (frame_type 1 or 3), and keep it running at a steady rate. Stop it when your own audio source ends, not on the first idle frame: an idle frame only means the persona is at rest right now, and one can appear mid-utterance when the server runs short, so stopping there cuts off speech that is still coming. A fade-out frame (2) is different: it acknowledges a CancelInteraction you sent, so it is a real stop signal.

Error Handling

• Handle both JSON ErrorResponse messages and plain text error strings

• Implement exponential backoff for reconnection

• Monitor server load in the SessionReady message

Interruption Handling

• Stop old-turn audio sends and use CancelInteraction when the user interrupts. Wait for the acknowledging 2 frame before sending the next turn's audio.

• Then pick a strategy for what you have already buffered:

  • Instant cut: drop everything buffered and resume rendering from the next frame that arrives. Lowest latency, and the pose can visibly jump.

  • Smooth: stop audio playback immediately but keep rendering the frames you hold, including the 2 fade-out frame the server sends. The avatar eases back to idle instead of snapping, at the cost of a few frames of latency.

• This is the one case where dropping non-idle frames is intended. The never-drop rule under Buffer Management applies to jitter recovery, not to a deliberate barge-in.

---

Complete Example

import asyncio
import json
import struct
import time
from collections import deque
import numpy as np
import websockets
from dotenv import load_dotenv
import os

load_dotenv()

API_KEY = os.getenv("OJIN_API_KEY", "")
CONFIG_ID = os.getenv("OJIN_CONFIG_ID", "")
URL = f"wss://models.ojin.ai/realtime?config_id={CONFIG_ID}"

SAMPLE_RATE = 16000
FPS = 25
TARGET_BUFFER = 10  # Tune based on your network conditions

def build_audio_message(audio_bytes):
    """Build a binary InteractionInput message."""
    header = struct.pack('!BQI', 1, int(time.time() * 1000), 0)
    return header + audio_bytes

def parse_response(data):
    """Parse a binary InteractionResponse message."""
    fmt = '!B16sQIII'
    hdr_size = struct.calcsize(fmt)
    # The 5th header field is reserved; ignore it.
    is_final, uid_bytes, ts, usage, _index, n_payloads = struct.unpack(fmt, data[:hdr_size])

    offset = hdr_size
    image = audio = None
    for _ in range(n_payloads):
        size, ptype = struct.unpack('!IB', data[offset:offset+5])
        offset += 5
        if ptype == 2:
            image = data[offset:offset+size]
        elif ptype == 1:
            audio = data[offset:offset+size]
        offset += size

    # Trailing frame type byte, appended after the payload entries.
    frame_type = data[offset]

    return {
        'is_final': bool(is_final),
        'frame_type': frame_type,    # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
        'image': image,
        'audio': audio,
    }

async def main():
    headers = {"Authorization": API_KEY}
    # For older websockets versions, use extra_headers instead.
    async with websockets.connect(URL, additional_headers=headers, ping_interval=30) as ws:
        # 1. Wait for SessionReady; the server starts streaming frames immediately after
        msg = json.loads(await ws.recv())
        assert msg["type"] == "sessionReady"
        print(f"Session ready: {msg['payload']}")

        buffer = deque()
        skip_counter = 0
        playback_started = False
        frame_count = 0
        audio_sent = False

        # 2. Receive and process frames. The server streams idle frames
        #    immediately after SessionReady; nothing needs to be sent first.
        async for data in ws:
            if isinstance(data, str):
                msg = json.loads(data)
                if msg.get("type") == "errorResponse":
                    print(f"Error: {msg['payload']}")
                    break
                continue

            frame = parse_response(data)
            buffer.append(frame)
            frame_count += 1

            # Wait for initial buffer before playback
            if not playback_started:
                if len(buffer) >= TARGET_BUFFER:
                    playback_started = True
                    print(f"Buffer filled ({TARGET_BUFFER} frames), starting playback")
                continue

            # Consume one frame
            if buffer:
                play_frame = buffer.popleft()

                # Drop excess idle frames when the buffer is too large. This example
                # consumes one frame per arrival, so a backlog never builds here; the branch
                # matters once you drain on a 40 ms timer instead.
                # Only idle (frame_type 0) is safe to drop; keep speech (1),
                # start-of-speech (3), and fade-out (2). Both the frame just
                # presented and the one about to be dropped must be idle, so the
                # drop always happens inside an idle stretch and never across a
                # transition into speech, which would leave a visible gap.
                if (len(buffer) > TARGET_BUFFER
                        and play_frame['frame_type'] == 0
                        and buffer[0]['frame_type'] == 0):
                    skip_counter += 1
                    if skip_counter % 2 == 0:
                        buffer.popleft()  # drop one idle frame

                kind = {0: "idle", 1: "speech", 2: "fade-out", 3: "start-of-speech"}.get(
                    play_frame['frame_type'], "?"
                )
                print(f"[{kind}] frame #{frame_count}, buffer={len(buffer)}")

                # In a real app: render play_frame['image'] and play play_frame['audio']

            # Demo: send test audio after receiving some silence frames
            if frame_count == 50 and not audio_sent:
                # Synthetic one-shot test buffer, not the recommended live pattern:
                # with real TTS, send audio as it arrives.
                duration_ms = 500  # Presence; use 400 for Portrait
                samples = SAMPLE_RATE * duration_ms // 1000
                t = np.arange(samples) / SAMPLE_RATE
                audio_data = (32767 * 0.5 * np.sin(2 * np.pi * 440 * t)).astype(np.int16)
                await ws.send(build_audio_message(audio_data.tobytes()))
                audio_sent = True
                print(f"Sent {duration_ms} ms of test audio")

            if frame_count > 200:
                break

asyncio.run(main())

---

Troubleshooting

Symptoms, causes, and fixes, connection and auth, a starved/idle mouth, choppy playback, latency, and frame lag, live in the global Troubleshooting guide, where each entry is tagged 🟢 SDK-handled or 🟠 your setup.

---

Example Implementation

A complete working Python example integrating Ojin Human Portrait with a speech-to-speech service (Hume EVI) is available here:

github.com/journee-live/speech-to-video-samples/tree/main/samples (includes Hume STS → Human Portrait walkthroughs)

The repository demonstrates the full integration pattern: microphone capture → STS service → TTS audio → Ojin lip-sync → synchronized video and audio playback at 25fps. It includes the buffer management and frame handling approach described in Best Practices above.

---