API reference

Most builders should use the Python SDK or Pipecat — they implement this protocol, buffering, and audio/video sync for you. Read on only if you need low-level WebSocket control.

Overview

Real-time talking head synthesis API. Send speech audio, receive synchronized video and audio frames.

After connecting and receiving SessionReady, you should send one initial audio message with silence (one frame worth of silent audio) to start the interaction on the server. The server then immediately begins streaming video and audio frames at 25fps. When no speech audio has been sent, the server generates silence frames (persona at rest with idle animation). When you send speech audio, the server generates speech frames with lip-synced animation synchronized to your audio.

After the initial silence frame, you only need to send speech audio — no additional silence, padding, or keep-alive messages are required.

Production deployments: This WebSocket API is intended for server-to-server use over a stable connection. Connect from a backend server rather than a front-end client — to keep your API key secure and because the raw WebSocket isn't built for flaky end-user networks. Run the backend in US East, close to Ojin's inference, for the lowest latency, and deliver the final media stream to end users over a transport built for varying network conditions — typically WebRTC — for smooth, reliable, low-latency playback.

---

How It Works

1. Connect to the WebSocket endpoint with your API key and config ID

2. Receive SessionReady — the server has allocated inference resources for your session

3. Send initial audio message with silence for one frame.

4. The server starts streaming frames immediately — silence frames with idle animation

5. Send speech audio whenever it becomes available (e.g., TTS output from your language model) — also buffer it locally for playback

6. Receive speech frames — the server transitions to lip-synced animation and returns to silence frames when audio runs out

7. Render video frames at 25fps, keeping a small jitter buffer (trim idle frames only if they back up after a stall)

8. Start playing your buffered TTS audio when the first speech frame (frame_type 1 or 3) arrives from Ojin — stop when speech ends

Frame Types

Every frame arrives as a binary InteractionResponse containing both a JPEG image and a PCM audio chunk. Frames are always delivered in order. The frame_type field classifies each frame:

frame_type	Description
0	Idle — persona at rest with idle animation. Generated automatically when no speech audio is queued
1	Speech — lip-synced animation generated from your audio input
2	Fade-out — post-cancel ramp back toward idle after an interruption
3	Start of speech — first speech frame of a turn resuming after an interruption/cancel. Natural (uninterrupted) turns start with 1, not 3

Buffering for Network Jitter

The server delivers frames at realtime 25 fps, so your buffer does not grow on its own. Keep a small client-side buffer — a few frames — to absorb network jitter and prevent stuttering during speech, and start playback once it's filled.

The right buffer size depends on your network conditions and latency requirements: keep it as low as possible to minimize latency, but high enough to absorb jitter without starving playback.

If frames ever back up — for example a brief network stall followed by a burst of queued frames arriving at once — you can recover by trimming idle frames (frame_type == 0). Only idle frames are safe to drop — never drop speech (1), start-of-speech (3), or fade-out (2) frames.

# When consuming frames from the buffer:
frame = buffer.popleft()

# If frames backed up after a stall, recover by skipping every other idle frame
if len(buffer) > target_buffer_size and frame.frame_type == 0:
    skip_counter += 1
    if skip_counter % 2 == 0 and buffer:
        buffer.popleft()  # drop one idle frame

---

Connection Flow

---

WebSocket Handshake

Open WebSocket connection

get

Connect to the WebSocket endpoint providing an API key in the Authorization header and a config_id query parameter. The server upgrades the connection to WebSocket and immediately begins streaming frames after sending SessionReady.

Recommended WebSocket settings:

• ping_interval: 30 seconds

• ping_timeout: 10 seconds

Authorizations

ApiKeyAuth

AuthorizationstringRequired

Raw API key (no Bearer prefix).

Query parameters

config_idstringRequired

Configuration ID for the persona, created in the Oris Portrait tab of the dashboard.

Header parameters

AuthorizationstringRequired

Your raw API key. No Bearer prefix.

Example: your-api-key

Responses

101

WebSocket upgrade successful. After the upgrade, the server sends a SessionReady JSON message and begins streaming binary InteractionResponse frames immediately.

application/json

Sent once by the server after the WebSocket connection is established and inference resources are allocated. The server begins streaming frames immediately after this message.

Format: JSON text frame.

typestring · enumRequiredPossible values:

401

Unauthorized — invalid or missing API key.

application/json

get/

HTTPcURLJavaScriptPython

GET /realtime/?config_id=text HTTP/1.1
Authorization: your-api-key
Accept: */*

{
  "type": "sessionReady",
  "payload": {
    "trace_id": "123e4567-e89b-12d3-a456-426614174000",
    "status": "success",
    "load": 0.42,
    "timestamp": 1723567893000,
    "parameters": {
      "speech_filter_amount": 5,
      "idle_filter_amount": 1000,
      "idle_mouth_opening_scale": 0,
      "speech_mouth_opening_scale": 1
    }
  }
}

---

Message Format

Mixed message types: Both JSON (text) and binary messages are exchanged on the same WebSocket connection. Your client must check the WebSocket frame type to distinguish them:

• Text frames (JSON): SessionReady, ErrorResponse (server → client), CancelInteraction (client → server)

• Binary frames: InteractionResponse (server → client), InteractionInput (client → server)

Byte order: All multi-byte integer fields in binary messages use network byte order (big-endian).

---

Messages Reference

Server → Client Messages

Client → Server Messages

---

Message Details

InteractionInput (Client → Server, Binary)

Binary message for sending speech audio to the server. Only send speech audio — do not send silence or padding.

Binary structure:

[1 byte ]  Payload type      — uint8, always 1 for audio
[8 bytes]  Timestamp          — uint64, milliseconds since Unix epoch
[4 bytes]  Params size        — uint32, byte length of the JSON params block (0 if no params)
[N bytes]  Params JSON        — UTF-8 encoded JSON (only present if params size > 0)
[M bytes]  Audio payload      — raw PCM int16 speech audio data

Header fields use big-endian byte order. The PCM audio samples in the payload use little-endian (standard for PCM int16). In Python: struct.pack('!BQI', payload_type, timestamp, params_size).

Audio requirements:

Property	Value
Format	PCM signed 16-bit integers (little-endian samples)
Sample rate	16,000 Hz
Channels	1 (mono)
Max message size	512 KB (entire binary message including header)

Recommended streaming pattern:

The server runs a 25 fps virtual timeline and needs your audio input to stay slightly ahead of it. Forwarding tiny TTS fragments one at a time (e.g. 40 ms chunks) makes input rate match output rate, so the server starves and emits idle frames between speech — lip-sync skips or drifts. Instead, feed audio with a cushion, not per-fragment:

1. Prime ~1 second of speech audio at the start of a turn.

2. Then send the largest chunks you can — coalesce queued fragments into ~400 ms sends (under the 512 KB cap).

3. Stay realtime: never wait for the whole utterance before sending.

The Python SDK and Pipecat shape the input for you — OjinSTVClient primes the lead and coalesces your TTS into large chunks automatically, so you just feed audio as it arrives. This pattern only applies if you drive this WebSocket API directly. See Optimizing Performance.

import struct, json, time

def build_audio_message(audio_bytes):
    header = struct.pack('!BQI',
        1,                         # payload type: audio
        int(time.time() * 1000),   # timestamp ms
        0,                         # params size (unused for oris-portrait)
    )
    return header + audio_bytes

---

InteractionResponse (Server → Client, Binary)

Binary message containing a video frame and synchronized audio. The server streams these continuously after SessionReady. Frames always arrive in order.

Binary structure:

[1 byte  ]  Is final flag     — uint8, 1 = last frame for this interaction, 0 = more coming
[16 bytes]  Interaction ID     — UUID bytes (big-endian)
[8 bytes ]  Timestamp          — uint64, milliseconds since Unix epoch
[4 bytes ]  Usage              — uint32, usage metric for this response
[4 bytes ]  Reserved           — uint32, reserved; ignore
[4 bytes ]  Num payloads       — uint32, number of payload entries that follow

For each payload entry:
  [4 bytes]  Data size          — uint32, byte length of the payload data only
  [1 byte ]  Payload type       — uint8, 1 = audio, 2 = image
  [N bytes]  Payload data       — raw payload bytes

[1 byte ]  Frame type         — uint8, appended after all payload entries: 0=idle, 1=speech, 2=fade-out, 3=start-of-speech

All multi-byte integers are big-endian. In Python: struct.unpack('!B16sQIII', header_bytes) for the main header, struct.unpack('!IB', entry_bytes) for each payload entry. The Frame type byte is a single trailing uint8 after the last payload entry and is the authoritative frame classifier.

Frame type:

Frame type	Meaning
0	Idle — persona at rest with idle animation
1	Speech — lip-synced animation from your audio
2	Fade-out — post-cancel ramp toward idle
3	Start of speech — first speech frame of a turn resuming after an interruption/cancel (natural turns start with 1)

Payload types:

Type	Format	Typical size per frame
1 (audio)	PCM int16, 16kHz mono	1,280 bytes (640 samples = 40ms at 25fps)
2 (image)	JPEG-encoded image	Variable (resolution depends on config, e.g. 1280×720)

Parsing example:

import struct, uuid

HEADER_FMT = '!B16sQIII'
HEADER_SIZE = struct.calcsize(HEADER_FMT)   # 37 bytes
ENTRY_FMT = '!IB'
ENTRY_SIZE = struct.calcsize(ENTRY_FMT)     # 5 bytes

def parse_response(data):
    # The 5th header field is reserved; ignore it.
    is_final, uuid_bytes, timestamp, usage, _reserved, num_payloads = \
        struct.unpack(HEADER_FMT, data[:HEADER_SIZE])

    offset = HEADER_SIZE
    image = audio = None

    for _ in range(num_payloads):
        size, ptype = struct.unpack(ENTRY_FMT, data[offset:offset + ENTRY_SIZE])
        offset += ENTRY_SIZE
        payload = data[offset:offset + size]
        offset += size

        if ptype == 2:
            image = payload   # JPEG bytes
        elif ptype == 1:
            audio = payload   # PCM int16 bytes

    # Trailing frame type byte, appended after the payload entries.
    frame_type = data[offset]

    return {
        'is_final': bool(is_final),
        'frame_type': frame_type,    # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
        'image': image,
        'audio': audio,
    }

---

CancelInteraction

Message	Purpose	Server behavior	Use case
CancelInteraction	Immediate stop	Stops processing, discards remaining frames	User interruption

---

ErrorResponse (Server → Client, JSON)

Plain text errors: In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON ErrorResponse. Your client should handle non-JSON text messages gracefully.

Error codes:

Code	Description
AUTH_FAILED	Invalid API key
UNAUTHORIZED	Caller lacks permission
MISSING_CONFIG_ID	config_id query parameter not provided
INVALID_MESSAGE	Malformed or unsupported message payload
INVALID_HEADERS	Missing or invalid headers
MODEL_NOT_FOUND	Config ID not found or invalid
BACKEND_UNAVAILABLE	No healthy inference backend available
RATE_LIMITED	Too many requests
TIMEOUT	Operation exceeded processing time
CANCELLED	Interaction cancelled by client
INTERNAL_ERROR	Unexpected server error
FRAME_SIZE_EXCEEDED	Message exceeded 512KB limit

---

Rate Limits & Constraints

Constraint	Value
Rate limit	6 requests per second
Max message size	512 KB per message
Video output	25 fps (realtime delivery)

Exceeding limits results in an ErrorResponse with code RATE_LIMITED.

---

Best Practices

Audio Input

• Send one silence frame first to start the conversation, then send speech audio when available

• Prime ~1 s of audio, then send the largest chunks you can (~400 ms). The server needs your input to stay slightly ahead of its 25 fps timeline. Forwarding tiny fragments one at a time starves it and produces idle frames between speech — buffer locally for playback, but coalesce what you send. See the InteractionInput message details above and Optimizing Performance.

• The Python SDK and Pipecat handle this input shaping for you — this only matters when driving the WebSocket API directly.

Buffer Management

• Play frames at 25 fps (40ms per frame)

• The server delivers at realtime 25 fps — keep a small jitter buffer (a few frames); it won't grow on its own

• Recovery: if frames back up after a network stall, skip 1 out of every 2 idle frames (frame_type == 0) until the buffer shrinks back

• Never drop speech (1), start-of-speech (3), or fade-out (2) frames

• Tune your target buffer size based on your network conditions — keep it as low as possible for minimal latency

Audio and Video Synchronization

Each InteractionResponse contains both a JPEG image and a PCM audio chunk. However, the recommended approach for audio playback is:

1. Buffer your TTS/source audio locally as it arrives from your speech service (for later playback only. You do not need to buffer it to send it to Ojin)

2. Forward TTS audio to Ojin immediately as it arrives from your speech service

3. Wait for the first speech frame (frame_type 1 or 3) to arrive from Ojin

4. Start playing your buffered TTS audio at that moment

5. Stop audio playback when speech ends — the first idle or fade-out frame (frame_type 0 or 2) after speech

6. Render video from every frame regardless of type

Sending is immediate, but playback is gated on the first speech frame. This ensures audio and video stay in sync.

# When TTS audio arrives from your speech service:
speech_audio_buffer.extend(tts_audio_chunk)      # buffer locally for playback
await ojin.send_audio(tts_audio_chunk)            # send to Ojin for lip-sync

# In your video playback loop:
frame = buffer.popleft()

# Speech frames are frame_type 1 (speech) and 3 (start-of-speech).
if frame.frame_type in (1, 3) and not audio_playing:
    start_audio_playback(speech_audio_buffer)     # begin draining the buffer
    audio_playing = True

# Non-speech frames are frame_type 0 (idle) and 2 (fade-out).
if frame.frame_type in (0, 2) and audio_playing:
    stop_audio_playback()
    audio_playing = False

render_video(frame.image)                         # always render the video

Error Handling

• Handle both JSON ErrorResponse messages and plain text error strings

• Implement exponential backoff for reconnection

• Monitor server load in the SessionReady message

Interruption Handling

• Use CancelInteraction for immediate stops (e.g., user interrupts the bot)

• Clear your frame buffer on interruption

---

Complete Example

import asyncio
import json
import struct
import time
from collections import deque
import numpy as np
import websockets
from dotenv import load_dotenv
import os

load_dotenv()

API_KEY = os.getenv("OJIN_API_KEY", "")
CONFIG_ID = os.getenv("OJIN_CONFIG_ID", "")
URL = f"wss://models.ojin.ai/realtime?config_id={CONFIG_ID}"

SAMPLE_RATE = 16000
FPS = 25
TARGET_BUFFER = 10  # Tune based on your network conditions

def build_audio_message(audio_bytes):
    """Build a binary InteractionInput message."""
    header = struct.pack('!BQI', 1, int(time.time() * 1000), 0)
    return header + audio_bytes

def parse_response(data):
    """Parse a binary InteractionResponse message."""
    fmt = '!B16sQIII'
    hdr_size = struct.calcsize(fmt)
    # The 5th header field is reserved; ignore it.
    is_final, uid_bytes, ts, usage, _reserved, n_payloads = struct.unpack(fmt, data[:hdr_size])

    offset = hdr_size
    image = audio = None
    for _ in range(n_payloads):
        size, ptype = struct.unpack('!IB', data[offset:offset+5])
        offset += 5
        if ptype == 2:
            image = data[offset:offset+size]
        elif ptype == 1:
            audio = data[offset:offset+size]
        offset += size

    # Trailing frame type byte, appended after the payload entries.
    frame_type = data[offset]

    return {
        'is_final': bool(is_final),
        'frame_type': frame_type,    # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech
        'image': image,
        'audio': audio,
    }

async def main():
    headers = {"Authorization": API_KEY}
    # For older websockets versions, use extra_headers instead.
    async with websockets.connect(URL, additional_headers=headers, ping_interval=30) as ws:
        # 1. Wait for SessionReady — server starts streaming frames immediately after
        msg = json.loads(await ws.recv())
        assert msg["type"] == "sessionReady"
        print(f"Session ready: {msg['payload']}")

        buffer = deque()
        skip_counter = 0
        playback_started = False
        frame_count = 0
        audio_sent = False

        # 2. Send one silence frame to start processing
        audio_data = b"\00\00" * (SAMPLE_RATE // 25)
        await ws.send(build_audio_message(audio_data))

        # 3. Receive and process frames
        async for data in ws:
            if isinstance(data, str):
                msg = json.loads(data)
                if msg.get("type") == "errorResponse":
                    print(f"Error: {msg['payload']}")
                    break
                continue

            frame = parse_response(data)
            buffer.append(frame)
            frame_count += 1

            # Wait for initial buffer before playback
            if not playback_started:
                if len(buffer) >= TARGET_BUFFER:
                    playback_started = True
                    print(f"Buffer filled ({TARGET_BUFFER} frames), starting playback")
                continue

            # Consume one frame
            if buffer:
                play_frame = buffer.popleft()

                # Drop excess idle frames: skip 1 out of 2 when buffer is too large.
                # Only idle (frame_type 0) is safe to drop — keep speech (1),
                # start-of-speech (3), and fade-out (2).
                if len(buffer) > TARGET_BUFFER and play_frame['frame_type'] == 0:
                    skip_counter += 1
                    if skip_counter % 2 == 0 and len(buffer) > 0:
                        buffer.popleft()  # drop one idle frame

                kind = {0: "idle", 1: "speech", 2: "fade-out", 3: "start-of-speech"}.get(
                    play_frame['frame_type'], "?"
                )
                print(f"[{kind}] frame #{frame_count}, buffer={len(buffer)}")

                # In a real app: render play_frame['image'] and play play_frame['audio']

            # Demo: send speech audio after receiving some silence frames
            if frame_count == 50 and not audio_sent:
                t = np.linspace(0, 1.0, SAMPLE_RATE, endpoint=False)
                audio_data = (32767 * 0.5 * np.sin(2 * np.pi * 440 * t)).astype(np.int16)
                chunk_size = SAMPLE_RATE # 1s chunks
                for i in range(0, len(audio_data), chunk_size):
                    chunk = audio_data[i:i + chunk_size]
                    await ws.send(build_audio_message(chunk.tobytes()))
                audio_sent = True
                print("Sent 1 second of speech audio")

            if frame_count > 200:
                break

asyncio.run(main())

---

Troubleshooting

Symptoms, causes, and fixes — connection and auth, a starved/idle mouth, choppy playback, latency, and frame lag (speech_filter_amount) — live in the global Troubleshooting guide, where each entry is tagged 🟢 SDK-handled or 🟠 your setup.

---

Example Implementation

A complete working Python example integrating Ojin Oris Portrait with a speech-to-speech service (Hume EVI) is available here:

github.com/journee-live/speech-to-video-samples/tree/main/samples (includes Hume STS → Oris Portrait walkthroughs)

The repository demonstrates the full integration pattern: microphone capture → STS service → TTS audio → Ojin lip-sync → synchronized video and audio playback at 25fps. It includes the buffer management and frame handling approach described in Best Practices above.

---