> For the complete documentation index, see [llms.txt](https://docs.ojin.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ojin.ai/models/introduction/api.md). # API reference {% hint style="info" %} Most builders should use the [**Python SDK**](/models/build-with-python-sdk.md) or [**Pipecat**](/models/introduction/integrations.md) — they implement this protocol, buffering, and audio/video sync for you. Read on only if you need low-level WebSocket control. {% endhint %} ## Overview Real-time talking head synthesis API. Send speech audio, receive synchronized video and audio frames. After connecting and receiving `SessionReady`, you should send one initial audio message with silence (one frame worth of silent audio) to start the interaction on the server. The server then immediately begins streaming video and audio frames at 25fps. When no speech audio has been sent, the server generates **silence frames** (persona at rest with idle animation). When you send speech audio, the server generates **speech frames** with lip-synced animation synchronized to your audio. After the initial silence frame, you only need to send speech audio — no additional silence, padding, or keep-alive messages are required. {% hint style="info" %} **Production deployments:** This WebSocket API is intended for **server-to-server** use over a stable connection. Connect from a backend server rather than a front-end client — to keep your API key secure and because the raw WebSocket isn't built for flaky end-user networks. Run the backend in **US East**, close to Ojin's inference, for the lowest latency, and deliver the final media stream to end users over a transport built for varying network conditions — typically **WebRTC** — for smooth, reliable, low-latency playback. {% endhint %} *** ## How It Works 1. **Connect** to the WebSocket endpoint with your API key and config ID 2. **Receive `SessionReady`** — the server has allocated inference resources for your session 3. **Send** initial audio message with silence for one frame. 4. **The server starts streaming frames immediately** — silence frames with idle animation 5. **Send speech audio** whenever it becomes available (e.g., TTS output from your language model) — also buffer it locally for playback 6. **Receive speech frames** — the server transitions to lip-synced animation and returns to silence frames when audio runs out 7. **Render video frames** at 25fps, keeping a small jitter buffer (trim idle frames only if they back up after a stall) 8. **Start playing your buffered TTS audio** when the first speech frame (`frame_type` `1` or `3`) arrives from Ojin — stop when speech ends ### Frame Types Every frame arrives as a binary `InteractionResponse` containing both a JPEG image and a PCM audio chunk. Frames are always delivered in order. The `frame_type` field classifies each frame: | `frame_type` | Description | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `0` | **Idle** — persona at rest with idle animation. Generated automatically when no speech audio is queued | | `1` | **Speech** — lip-synced animation generated from your audio input | | `2` | **Fade-out** — post-cancel ramp back toward idle after an interruption | | `3` | **Start of speech** — first speech frame of a turn **resuming after an interruption/cancel**. Natural (uninterrupted) turns start with `1`, not `3` | ### Buffering for Network Jitter The server delivers frames at **realtime 25 fps**, so your buffer does not grow on its own. Keep a **small client-side buffer** — a few frames — to absorb network jitter and prevent stuttering during speech, and start playback once it's filled. The right buffer size depends on your network conditions and latency requirements: keep it as low as possible to minimize latency, but high enough to absorb jitter without starving playback. If frames ever back up — for example a brief network stall followed by a burst of queued frames arriving at once — you can recover by trimming **idle** frames (`frame_type == 0`). Only idle frames are safe to drop — never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames. ```python # When consuming frames from the buffer: frame = buffer.popleft() # If frames backed up after a stall, recover by skipping every other idle frame if len(buffer) > target_buffer_size and frame.frame_type == 0: skip_counter += 1 if skip_counter % 2 == 0 and buffer: buffer.popleft() # drop one idle frame ``` *** ## Connection Flow ```mermaid sequenceDiagram participant Client participant Server Note over Client,Server: Connection Client->>Server: WebSocket Connect Server->>Client: SessionReady (JSON) Client->>Server: InteractionInput (silence audio for one frame) Note over Client,Server: Server Streams Immediately Server->>Client: Frame (idle, frame_type=0) Server->>Client: Frame (idle, frame_type=0) Server->>Client: Frame (idle, frame_type=0) Note over Client,Server: Client Sends Speech Audio Client->>Server: InteractionInput (TTS audio chunk 1) Client->>Server: InteractionInput (TTS audio chunk 2) Note over Client,Server: Server Transitions to Speech Server->>Client: Frame (start-of-speech, frame_type=3) Server->>Client: Frame (speech, frame_type=1) Server->>Client: Frame (speech, frame_type=1) Note right of Server: Delivered at realtime 25 fps Note over Client,Server: Audio Runs Out → Back to Idle Server->>Client: Frame (idle, frame_type=0) Server->>Client: Frame (idle, frame_type=0) Note right of Client: Client keeps a small jitter buffer ``` *** ## WebSocket Handshake ## Open WebSocket connection > Connect to the WebSocket endpoint providing an API key in the \`Authorization\` header and a \`config\_id\` query parameter. The server upgrades the connection to WebSocket and immediately begins streaming frames after sending \`SessionReady\`.\ > \ > \*\*Recommended WebSocket settings:\*\*\ > \- \`ping\_interval\`: 30 seconds\ > \- \`ping\_timeout\`: 10 seconds ```json {"openapi":"3.0.3","info":{"title":"Ojin Oris Portrait Realtime API","version":"1.0.0"},"servers":[{"url":"wss://models.ojin.ai/realtime","description":"WebSocket endpoint"}],"security":[{"ApiKeyAuth":[]}],"components":{"securitySchemes":{"ApiKeyAuth":{"type":"apiKey","in":"header","name":"Authorization","description":"Raw API key (no `Bearer` prefix)."}},"schemas":{"SessionReadyMessage":{"type":"object","description":"Sent once by the server after the WebSocket connection is established and inference resources are allocated. **The server begins streaming frames immediately after this message.**\n\n**Format:** JSON text frame.","required":["type","payload"],"properties":{"type":{"type":"string","enum":["sessionReady"]},"payload":{"type":"object","required":["trace_id","status","load"],"properties":{"trace_id":{"type":"string","format":"uuid","description":"Unique session identifier assigned by the server."},"status":{"type":"string","enum":["success"],"description":"Always `success`."},"load":{"type":"number","format":"float","minimum":0,"maximum":1,"description":"Current load of the inference server (0.0–1.0)."},"timestamp":{"type":"integer","format":"int64","description":"Server timestamp in milliseconds since Unix epoch."},"parameters":{"type":"object","additionalProperties":true,"nullable":true,"description":"Optional model-specific session parameters returned by the server."}}}}},"ErrorResponseMessage":{"type":"object","description":"Sent by the server when an error occurs.\n\n**Format:** JSON text frame.\n\n> **Note:** In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON `ErrorResponse`. Your client should handle non-JSON text messages gracefully.","required":["type","payload"],"properties":{"type":{"type":"string","enum":["errorResponse"]},"payload":{"type":"object","required":["code","message","timestamp"],"properties":{"code":{"type":"string","description":"Machine-readable error code.","enum":["AUTH_FAILED","UNAUTHORIZED","MISSING_CONFIG_ID","INVALID_MESSAGE","INVALID_HEADERS","MODEL_NOT_FOUND","BACKEND_UNAVAILABLE","RATE_LIMITED","TIMEOUT","CANCELLED","INTERNAL_ERROR","FRAME_SIZE_EXCEEDED"]},"message":{"type":"string","description":"Human-readable description of the error."},"interaction_id":{"type":"string","nullable":true,"description":"The interaction ID related to the error, if applicable."},"details":{"type":"object","additionalProperties":true,"nullable":true,"description":"Optional additional structured details about the error."},"timestamp":{"type":"integer","format":"int64","description":"Milliseconds since Unix epoch when the error was sent."}}}}}}},"paths":{"/":{"get":{"summary":"Open WebSocket connection","description":"Connect to the WebSocket endpoint providing an API key in the `Authorization` header and a `config_id` query parameter. The server upgrades the connection to WebSocket and immediately begins streaming frames after sending `SessionReady`.\n\n**Recommended WebSocket settings:**\n- `ping_interval`: 30 seconds\n- `ping_timeout`: 10 seconds","operationId":"wsHandshake","parameters":[{"in":"query","name":"config_id","required":true,"schema":{"type":"string"},"description":"Configuration ID for the persona, created in the Oris Portrait tab of the dashboard."},{"in":"header","name":"Authorization","required":true,"schema":{"type":"string"},"description":"Your raw API key. No `Bearer` prefix."}],"responses":{"101":{"description":"WebSocket upgrade successful. After the upgrade, the server sends a `SessionReady` JSON message and begins streaming binary `InteractionResponse` frames immediately.","content":{"application/json":{"schema":{"$ref":"#/components/schemas/SessionReadyMessage"}}}},"401":{"description":"Unauthorized — invalid or missing API key.","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponseMessage"}}}}}}}}} ``` *** ## Message Format {% hint style="info" %} **Mixed message types:** Both JSON (text) and binary messages are exchanged on the same WebSocket connection. Your client must check the WebSocket frame type to distinguish them: * **Text frames (JSON):** `SessionReady`, `ErrorResponse` (server → client), `CancelInteraction` (client → server) * **Binary frames:** `InteractionResponse` (server → client), `InteractionInput` (client → server) {% endhint %} {% hint style="info" %} **Byte order:** All multi-byte integer fields in binary messages use **network byte order (big-endian)**. {% endhint %} *** ## Messages Reference ### Server → Client Messages ## The SessionReadyMessage object ```json {"openapi":"3.0.3","info":{"title":"Ojin Oris Portrait Realtime API","version":"1.0.0"},"components":{"schemas":{"SessionReadyMessage":{"type":"object","description":"Sent once by the server after the WebSocket connection is established and inference resources are allocated. **The server begins streaming frames immediately after this message.**\n\n**Format:** JSON text frame.","required":["type","payload"],"properties":{"type":{"type":"string","enum":["sessionReady"]},"payload":{"type":"object","required":["trace_id","status","load"],"properties":{"trace_id":{"type":"string","format":"uuid","description":"Unique session identifier assigned by the server."},"status":{"type":"string","enum":["success"],"description":"Always `success`."},"load":{"type":"number","format":"float","minimum":0,"maximum":1,"description":"Current load of the inference server (0.0–1.0)."},"timestamp":{"type":"integer","format":"int64","description":"Server timestamp in milliseconds since Unix epoch."},"parameters":{"type":"object","additionalProperties":true,"nullable":true,"description":"Optional model-specific session parameters returned by the server."}}}}}}}} ``` ## The InteractionResponseMessage object ````json {"openapi":"3.0.3","info":{"title":"Ojin Oris Portrait Realtime API","version":"1.0.0"},"components":{"schemas":{"InteractionResponseMessage":{"type":"object","description":"Binary message containing a video frame and synchronized audio chunk. The server streams these continuously after `SessionReady` — silence frames when idle, speech frames when processing your audio.\n\n**Format:** Binary frame.\n\n**Binary structure (big-endian):**\n```\n[1 byte ] Is final flag — uint8, 1 = last frame, 0 = more coming\n[16 bytes] Interaction ID — UUID bytes\n[8 bytes ] Timestamp — uint64, milliseconds since Unix epoch\n[4 bytes ] Usage — uint32, usage metric\n[4 bytes ] Frame index — uint32, 0 = silence, 1 = speech\n[4 bytes ] Num payloads — uint32, number of payload entries\n\nFor each payload entry:\n [4 bytes] Data size — uint32, byte length of payload data\n [1 byte ] Payload type — uint8, 1 = audio, 2 = image\n [N bytes] Payload data — raw payload bytes\n```\n\nPython unpack: `struct.unpack('!B16sQIII', header)` for the main header, `struct.unpack('!IB', entry)` for each payload entry.","required":["is_final","interaction_id","timestamp","usage","index","payloads"],"properties":{"is_final":{"type":"boolean","description":"`true` if this is the last frame for the current interaction. `false` if more frames are coming."},"interaction_id":{"type":"string","format":"uuid","description":"UUID identifying this response. Use to correlate frames across a single interaction."},"timestamp":{"type":"integer","format":"int64","description":"Milliseconds since Unix epoch when the frame was sent."},"usage":{"type":"integer","format":"int32","description":"Usage metric for this response."},"index":{"type":"integer","format":"int32","enum":[0,1],"description":"Frame type. `0` = silence frame (idle animation, no speech input). `1` = speech frame (lip-synced to your audio). **Drop silence frames (`0`) to manage buffer size. Never drop speech frames (`1`).**"},"payloads":{"type":"array","description":"List of payload entries in this frame. Each frame typically contains one audio entry and one image entry.","items":{"type":"object","required":["payload_type","data_size","data"],"properties":{"payload_type":{"type":"integer","enum":[1,2],"description":"`1` = audio (PCM int16, 16kHz mono, 1,280 bytes = 640 samples = 40ms). `2` = image (JPEG-encoded, resolution depends on config e.g. 1280×720)."},"data_size":{"type":"integer","format":"int32","description":"Byte length of the payload data."},"data":{"type":"string","format":"binary","description":"Raw payload bytes. For audio: PCM int16 bytes. For image: JPEG bytes."}}}}}}}}} ```` ## The ErrorResponseMessage object ```json {"openapi":"3.0.3","info":{"title":"Ojin Oris Portrait Realtime API","version":"1.0.0"},"components":{"schemas":{"ErrorResponseMessage":{"type":"object","description":"Sent by the server when an error occurs.\n\n**Format:** JSON text frame.\n\n> **Note:** In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON `ErrorResponse`. Your client should handle non-JSON text messages gracefully.","required":["type","payload"],"properties":{"type":{"type":"string","enum":["errorResponse"]},"payload":{"type":"object","required":["code","message","timestamp"],"properties":{"code":{"type":"string","description":"Machine-readable error code.","enum":["AUTH_FAILED","UNAUTHORIZED","MISSING_CONFIG_ID","INVALID_MESSAGE","INVALID_HEADERS","MODEL_NOT_FOUND","BACKEND_UNAVAILABLE","RATE_LIMITED","TIMEOUT","CANCELLED","INTERNAL_ERROR","FRAME_SIZE_EXCEEDED"]},"message":{"type":"string","description":"Human-readable description of the error."},"interaction_id":{"type":"string","nullable":true,"description":"The interaction ID related to the error, if applicable."},"details":{"type":"object","additionalProperties":true,"nullable":true,"description":"Optional additional structured details about the error."},"timestamp":{"type":"integer","format":"int64","description":"Milliseconds since Unix epoch when the error was sent."}}}}}}}} ``` ### Client → Server Messages ## The InteractionInputMessage object ````json {"openapi":"3.0.3","info":{"title":"Ojin Oris Portrait Realtime API","version":"1.0.0"},"components":{"schemas":{"InteractionInputMessage":{"type":"object","description":"Binary message for sending speech audio to the server. **Only send speech audio** — do not send silence or padding. The server generates silence frames automatically.\n\n**Format:** Binary frame.\n\n**Binary structure (big-endian):**\n```\n[1 byte ] Payload type — uint8, always 1 for audio\n[8 bytes] Timestamp — uint64, milliseconds since Unix epoch\n[4 bytes] Params size — uint32, byte length of JSON params (0 if none)\n[N bytes] Params JSON — UTF-8 JSON (only present if params size > 0)\n[M bytes] Audio payload — raw PCM int16 speech audio\n```\n\nPython pack: `struct.pack('!BQI', payload_type, timestamp, params_size)`","required":["payload_type","timestamp","params_size","audio_payload"],"properties":{"payload_type":{"type":"integer","enum":[1],"description":"Always `1` for audio."},"timestamp":{"type":"integer","format":"int64","description":"Milliseconds since Unix epoch when the message was sent."},"params_size":{"type":"integer","format":"int32","minimum":0,"description":"Byte length of the JSON params block. `0` if no params."},"params":{"type":"object","nullable":true,"description":"Optional per-chunk parameters. Overrides session defaults for this audio chunk.","properties":{"speech_filter_amount":{"type":"number","format":"float","default":5,"description":"Smoothing for speech animation. Higher = smoother, less responsive."},"idle_filter_amount":{"type":"number","format":"float","default":1000,"description":"Smoothing for idle animation."},"idle_mouth_opening_scale":{"type":"number","format":"float","default":0,"description":"Mouth movement scale during idle. `0.0` = closed."},"speech_mouth_opening_scale":{"type":"number","format":"float","default":1,"description":"Mouth movement scale during speech. `1.0` = full movement."},"client_frame_index":{"type":"integer","format":"int32","default":0,"description":"Frame index the client is currently displaying. Helps the server manage silence-to-speech transitions smoothly."}}},"audio_payload":{"type":"string","format":"binary","description":"Raw PCM int16 speech audio. Requirements: 16,000 Hz sample rate, mono (1 channel), little-endian int16 samples. Entire message must be under 512 KB. Recommended chunk size: 400ms = 6,400 samples = 12,800 bytes."}}}}}} ```` ## The CancelInteractionMessage object ```json {"openapi":"3.0.3","info":{"title":"Ojin Oris Portrait Realtime API","version":"1.0.0"},"components":{"schemas":{"CancelInteractionMessage":{"type":"object","description":"Immediately stop processing and discard all remaining frames. No final frame is sent. Use for interruptions (e.g., user starts speaking while the persona is talking).\n\n**Format:** JSON text frame.","required":["type","payload"],"properties":{"type":{"type":"string","enum":["cancelInteraction"]},"payload":{"type":"object","properties":{"timestamp":{"type":"integer","format":"int64","nullable":true,"description":"Optional. Milliseconds since Unix epoch when the message was sent."}}}}}}}} ``` *** ## Message Details ### InteractionInput (Client → Server, Binary) Binary message for sending speech audio to the server. **Only send speech audio** — do not send silence or padding. **Binary structure:** ``` [1 byte ] Payload type — uint8, always 1 for audio [8 bytes] Timestamp — uint64, milliseconds since Unix epoch [4 bytes] Params size — uint32, byte length of the JSON params block (0 if no params) [N bytes] Params JSON — UTF-8 encoded JSON (only present if params size > 0) [M bytes] Audio payload — raw PCM int16 speech audio data ``` **Header fields** use **big-endian** byte order. The PCM audio samples in the payload use **little-endian** (standard for PCM int16). In Python: `struct.pack('!BQI', payload_type, timestamp, params_size)`. **Audio requirements:** | Property | Value | | ---------------- | -------------------------------------------------- | | Format | PCM signed 16-bit integers (little-endian samples) | | Sample rate | 16,000 Hz | | Channels | 1 (mono) | | Max message size | 512 KB (entire binary message including header) | **Recommended streaming pattern:** The server runs a 25 fps virtual timeline and needs your audio **input to stay slightly ahead** of it. Forwarding tiny TTS fragments one at a time (e.g. 40 ms chunks) makes input rate match output rate, so the server starves and emits idle frames between speech — lip-sync skips or drifts. Instead, feed audio with a cushion, **not** per-fragment: 1. **Prime \~1 second** of speech audio at the start of a turn. 2. Then send the **largest chunks you can** — coalesce queued fragments into **\~400 ms** sends (under the 512 KB cap). 3. Stay realtime: never wait for the whole utterance before sending. {% hint style="success" %} The [**Python SDK**](/models/build-with-python-sdk.md) and [**Pipecat**](/models/introduction/integrations.md) shape the input for you — `OjinSTVClient` primes the lead and coalesces your TTS into large chunks automatically, so you just feed audio as it arrives. This pattern only applies if you drive this WebSocket API directly. See [Optimizing Performance](/guides/optimizing-performance.md). {% endhint %} ```python import struct, json, time def build_audio_message(audio_bytes): header = struct.pack('!BQI', 1, # payload type: audio int(time.time() * 1000), # timestamp ms 0, # params size (unused for oris-portrait) ) return header + audio_bytes ``` *** ### InteractionResponse (Server → Client, Binary) Binary message containing a video frame and synchronized audio. The server streams these continuously after `SessionReady`. **Frames always arrive in order.** **Binary structure:** ``` [1 byte ] Is final flag — uint8, 1 = last frame for this interaction, 0 = more coming [16 bytes] Interaction ID — UUID bytes (big-endian) [8 bytes ] Timestamp — uint64, milliseconds since Unix epoch [4 bytes ] Usage — uint32, usage metric for this response [4 bytes ] Reserved — uint32, reserved; ignore [4 bytes ] Num payloads — uint32, number of payload entries that follow For each payload entry: [4 bytes] Data size — uint32, byte length of the payload data only [1 byte ] Payload type — uint8, 1 = audio, 2 = image [N bytes] Payload data — raw payload bytes [1 byte ] Frame type — uint8, appended after all payload entries: 0=idle, 1=speech, 2=fade-out, 3=start-of-speech ``` All multi-byte integers are **big-endian**. In Python: `struct.unpack('!B16sQIII', header_bytes)` for the main header, `struct.unpack('!IB', entry_bytes)` for each payload entry. The `Frame type` byte is a single trailing `uint8` after the last payload entry and is the authoritative frame classifier. **Frame type:** | Frame type | Meaning | | ---------- | ----------------------------------------------------------------------------------------------------------------------- | | `0` | **Idle** — persona at rest with idle animation | | `1` | **Speech** — lip-synced animation from your audio | | `2` | **Fade-out** — post-cancel ramp toward idle | | `3` | **Start of speech** — first speech frame of a turn resuming after an interruption/cancel (natural turns start with `1`) | **Payload types:** | Type | Format | Typical size per frame | | --------- | --------------------- | ------------------------------------------------------ | | 1 (audio) | PCM int16, 16kHz mono | **1,280 bytes** (640 samples = 40ms at 25fps) | | 2 (image) | JPEG-encoded image | Variable (resolution depends on config, e.g. 1280×720) | **Parsing example:** ```python import struct, uuid HEADER_FMT = '!B16sQIII' HEADER_SIZE = struct.calcsize(HEADER_FMT) # 37 bytes ENTRY_FMT = '!IB' ENTRY_SIZE = struct.calcsize(ENTRY_FMT) # 5 bytes def parse_response(data): # The 5th header field is reserved; ignore it. is_final, uuid_bytes, timestamp, usage, _reserved, num_payloads = \ struct.unpack(HEADER_FMT, data[:HEADER_SIZE]) offset = HEADER_SIZE image = audio = None for _ in range(num_payloads): size, ptype = struct.unpack(ENTRY_FMT, data[offset:offset + ENTRY_SIZE]) offset += ENTRY_SIZE payload = data[offset:offset + size] offset += size if ptype == 2: image = payload # JPEG bytes elif ptype == 1: audio = payload # PCM int16 bytes # Trailing frame type byte, appended after the payload entries. frame_type = data[offset] return { 'is_final': bool(is_final), 'frame_type': frame_type, # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech 'image': image, 'audio': audio, } ``` *** ### CancelInteraction | Message | Purpose | Server behavior | Use case | | ------------------- | -------------- | ------------------------------------------- | ----------------- | | `CancelInteraction` | Immediate stop | Stops processing, discards remaining frames | User interruption | *** ### ErrorResponse (Server → Client, JSON) {% hint style="warning" %} **Plain text errors:** In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON `ErrorResponse`. Your client should handle non-JSON text messages gracefully. {% endhint %} **Error codes:** | Code | Description | | --------------------- | ---------------------------------------- | | `AUTH_FAILED` | Invalid API key | | `UNAUTHORIZED` | Caller lacks permission | | `MISSING_CONFIG_ID` | `config_id` query parameter not provided | | `INVALID_MESSAGE` | Malformed or unsupported message payload | | `INVALID_HEADERS` | Missing or invalid headers | | `MODEL_NOT_FOUND` | Config ID not found or invalid | | `BACKEND_UNAVAILABLE` | No healthy inference backend available | | `RATE_LIMITED` | Too many requests | | `TIMEOUT` | Operation exceeded processing time | | `CANCELLED` | Interaction cancelled by client | | `INTERNAL_ERROR` | Unexpected server error | | `FRAME_SIZE_EXCEEDED` | Message exceeded 512KB limit | *** ## Rate Limits & Constraints | Constraint | Value | | ---------------- | -------------------------- | | Rate limit | 6 requests per second | | Max message size | 512 KB per message | | Video output | 25 fps (realtime delivery) | Exceeding limits results in an `ErrorResponse` with code `RATE_LIMITED`. *** ## Best Practices ### Audio Input * **Send one silence frame first** to start the conversation, then send speech audio when available * **Prime \~1 s of audio, then send the largest chunks you can (\~400 ms).** The server needs your input to stay slightly ahead of its 25 fps timeline. Forwarding tiny fragments one at a time starves it and produces idle frames between speech — buffer locally for playback, but coalesce what you send. See the **InteractionInput** message details above and [Optimizing Performance](/guides/optimizing-performance.md). * The [Python SDK](/models/build-with-python-sdk.md) and [Pipecat](/models/introduction/integrations.md) handle this input shaping for you — this only matters when driving the WebSocket API directly. ### Buffer Management * Play frames at **25 fps** (40ms per frame) * The server delivers at realtime 25 fps — keep a **small jitter buffer** (a few frames); it won't grow on its own * **Recovery:** if frames back up after a network stall, skip 1 out of every 2 idle frames (`frame_type == 0`) until the buffer shrinks back * **Never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames** * Tune your target buffer size based on your network conditions — keep it as low as possible for minimal latency ### Audio and Video Synchronization Each `InteractionResponse` contains both a JPEG image and a PCM audio chunk. However, the **recommended approach for audio playback** is: 1. **Buffer your TTS/source audio locally** as it arrives from your speech service (for later playback only. You do not need to buffer it to send it to Ojin) 2. **Forward TTS audio to Ojin immediately** as it arrives from your speech service 3. **Wait for the first speech frame** (`frame_type` `1` or `3`) to arrive from Ojin 4. **Start playing your buffered TTS audio** at that moment 5. **Stop audio playback** when speech ends — the first idle or fade-out frame (`frame_type` `0` or `2`) after speech 6. **Render video** from every frame regardless of type Sending is immediate, but playback is gated on the first speech frame. This ensures audio and video stay in sync. ```python # When TTS audio arrives from your speech service: speech_audio_buffer.extend(tts_audio_chunk) # buffer locally for playback await ojin.send_audio(tts_audio_chunk) # send to Ojin for lip-sync # In your video playback loop: frame = buffer.popleft() # Speech frames are frame_type 1 (speech) and 3 (start-of-speech). if frame.frame_type in (1, 3) and not audio_playing: start_audio_playback(speech_audio_buffer) # begin draining the buffer audio_playing = True # Non-speech frames are frame_type 0 (idle) and 2 (fade-out). if frame.frame_type in (0, 2) and audio_playing: stop_audio_playback() audio_playing = False render_video(frame.image) # always render the video ``` ### Error Handling * Handle both JSON `ErrorResponse` messages and plain text error strings * Implement exponential backoff for reconnection * Monitor server `load` in the `SessionReady` message ### Interruption Handling * Use `CancelInteraction` for immediate stops (e.g., user interrupts the bot) * Clear your frame buffer on interruption *** ## Complete Example ```python import asyncio import json import struct import time from collections import deque import numpy as np import websockets from dotenv import load_dotenv import os load_dotenv() API_KEY = os.getenv("OJIN_API_KEY", "") CONFIG_ID = os.getenv("OJIN_CONFIG_ID", "") URL = f"wss://models.ojin.ai/realtime?config_id={CONFIG_ID}" SAMPLE_RATE = 16000 FPS = 25 TARGET_BUFFER = 10 # Tune based on your network conditions def build_audio_message(audio_bytes): """Build a binary InteractionInput message.""" header = struct.pack('!BQI', 1, int(time.time() * 1000), 0) return header + audio_bytes def parse_response(data): """Parse a binary InteractionResponse message.""" fmt = '!B16sQIII' hdr_size = struct.calcsize(fmt) # The 5th header field is reserved; ignore it. is_final, uid_bytes, ts, usage, _reserved, n_payloads = struct.unpack(fmt, data[:hdr_size]) offset = hdr_size image = audio = None for _ in range(n_payloads): size, ptype = struct.unpack('!IB', data[offset:offset+5]) offset += 5 if ptype == 2: image = data[offset:offset+size] elif ptype == 1: audio = data[offset:offset+size] offset += size # Trailing frame type byte, appended after the payload entries. frame_type = data[offset] return { 'is_final': bool(is_final), 'frame_type': frame_type, # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech 'image': image, 'audio': audio, } async def main(): headers = {"Authorization": API_KEY} # For older websockets versions, use extra_headers instead. async with websockets.connect(URL, additional_headers=headers, ping_interval=30) as ws: # 1. Wait for SessionReady — server starts streaming frames immediately after msg = json.loads(await ws.recv()) assert msg["type"] == "sessionReady" print(f"Session ready: {msg['payload']}") buffer = deque() skip_counter = 0 playback_started = False frame_count = 0 audio_sent = False # 2. Send one silence frame to start processing audio_data = b"\00\00" * (SAMPLE_RATE // 25) await ws.send(build_audio_message(audio_data)) # 3. Receive and process frames async for data in ws: if isinstance(data, str): msg = json.loads(data) if msg.get("type") == "errorResponse": print(f"Error: {msg['payload']}") break continue frame = parse_response(data) buffer.append(frame) frame_count += 1 # Wait for initial buffer before playback if not playback_started: if len(buffer) >= TARGET_BUFFER: playback_started = True print(f"Buffer filled ({TARGET_BUFFER} frames), starting playback") continue # Consume one frame if buffer: play_frame = buffer.popleft() # Drop excess idle frames: skip 1 out of 2 when buffer is too large. # Only idle (frame_type 0) is safe to drop — keep speech (1), # start-of-speech (3), and fade-out (2). if len(buffer) > TARGET_BUFFER and play_frame['frame_type'] == 0: skip_counter += 1 if skip_counter % 2 == 0 and len(buffer) > 0: buffer.popleft() # drop one idle frame kind = {0: "idle", 1: "speech", 2: "fade-out", 3: "start-of-speech"}.get( play_frame['frame_type'], "?" ) print(f"[{kind}] frame #{frame_count}, buffer={len(buffer)}") # In a real app: render play_frame['image'] and play play_frame['audio'] # Demo: send speech audio after receiving some silence frames if frame_count == 50 and not audio_sent: t = np.linspace(0, 1.0, SAMPLE_RATE, endpoint=False) audio_data = (32767 * 0.5 * np.sin(2 * np.pi * 440 * t)).astype(np.int16) chunk_size = SAMPLE_RATE # 1s chunks for i in range(0, len(audio_data), chunk_size): chunk = audio_data[i:i + chunk_size] await ws.send(build_audio_message(chunk.tobytes())) audio_sent = True print("Sent 1 second of speech audio") if frame_count > 200: break asyncio.run(main()) ``` *** ## Troubleshooting Symptoms, causes, and fixes — connection and auth, a starved/idle mouth, choppy playback, latency, and frame lag (`speech_filter_amount`) — live in the global [Troubleshooting guide](/guides/troubleshooting.md), where each entry is tagged 🟢 SDK-handled or 🟠 your setup. *** ## Example Implementation A complete working Python example integrating Ojin Oris Portrait with a speech-to-speech service (Hume EVI) is available here: [**github.com/journee-live/speech-to-video-samples/tree/main/samples**](https://github.com/journee-live/speech-to-video-samples/tree/main/samples) (includes Hume STS → Oris Portrait walkthroughs) The repository demonstrates the full integration pattern: microphone capture → STS service → TTS audio → Ojin lip-sync → synchronized video and audio playback at 25fps. It includes the buffer management and frame handling approach described in [Best Practices](#best-practices) above. *** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.ojin.ai/models/introduction/api.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.