# Welcome Build AI Agents That Feel Human. In Minutes.

Live in real time. Speech in, synchronized video out.

*** ## Get started in 2 minutes The fastest way to deploy a conversational AI agent with a lifelike avatar: **1. Get your API key** → [Create one here](/getting-started/authentication) **2. Create an agent** in the [dashboard](https://ojin.ai/dashboard). Pick a face, voice, and personality. **3. Embed on your site:** {% tabs %} {% tab title="Plain HTML" %} ```html ``` {% endtab %} {% tab title="React" %} Vite or Create React App. Load the script once, then drop in the tag. Vite uses the project-root `index.html`. Create React App uses `public/index.html`. ```html ``` ```tsx declare module "react" { namespace JSX { interface IntrinsicElements { "ojin-agent": React.DetailedHTMLProps< React.HTMLAttributes, HTMLElement > & { "agent-id"?: string; "core-api-endpoint"?: string; }; } } } export function AgentEmbed() { return ( ); } ``` {% endtab %} {% tab title="Next.js" %} App Router. Load the script on the client only. The widget uses `window` at mount time: ```tsx import Script from "next/script"; declare module "react" { namespace JSX { interface IntrinsicElements { "ojin-agent": React.DetailedHTMLProps< React.HTMLAttributes, HTMLElement > & { "agent-id"?: string; "core-api-endpoint"?: string; }; } } } export default function RootLayout({ children, }: { children: React.ReactNode; }) { return ( {children} ``` {% endtab %} {% tab title="React / Next.js" %} The widget is a web component, so it needs to mount on the client. It reads `window` at mount time, which is why the App Router example loads the script with `strategy="afterInteractive"` rather than during server rendering. Declaring the element in `IntrinsicElements` is what stops TypeScript rejecting the tag. ```tsx // app/layout.tsx (Next.js App Router) import Script from "next/script"; declare module "react" { namespace JSX { interface IntrinsicElements { "ojin-agent": React.DetailedHTMLProps< React.HTMLAttributes, HTMLElement > & { "agent-id"?: string; "core-api-endpoint"?: string; collapsable?: string; "default-collapsed"?: string; "enable-video"?: string; "preview-url"?: string; "widget-resizable"?: string; }; } } } export default function RootLayout({ children, }: { children: React.ReactNode; }) { return ( {children} ``` {% endtab %} {% tab title="Vue" %} Load the bundle once in `index.html`, then use the tag in any template. ```html ``` Vue 3's template compiler treats an unknown tag as a component, so tell it `ojin-agent` is a custom element. Without this you get a `Failed to resolve component: ojin-agent` warning: ```js // vite.config.js import vue from "@vitejs/plugin-vue"; export default { plugins: [ vue({ template: { compilerOptions: { isCustomElement: (tag) => tag.startsWith("ojin-"), }, }, }), ], }; ``` ```vue ``` {% endtab %} {% tab title="REST + Daily" %} Only reach for this if you are building your own UI. You give up everything the widget handles: the avatar layout, audio controls, captions, reconnection and session teardown. Call [`POST /v1/public/agents/connect`](/apps/overview/api-reference) to get a Daily room and a token, then join the room yourself. ```ts // Install: npm i @daily-co/daily-js import DailyIframe from "@daily-co/daily-js"; const AGENT_ID = "your-agent-id"; const CONNECT_URL = "https://api.ojin.ai/v1/public/agents/connect"; async function startSession(clientUserRef?: string) { const res = await fetch(CONNECT_URL, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ agent_id: AGENT_ID, client_user_ref: clientUserRef }), }); if (!res.ok) throw new Error(`connect failed: ${res.status}`); const { room_url, token } = await res.json(); const call = DailyIframe.createCallObject(); await call.join({ url: room_url, token }); return call; } ``` The response also carries `session_id` and `agent_id`. End the session with the cancel beacon described in the [Session API Reference](/apps/overview/api-reference). {% endtab %} {% endtabs %} {% hint style="info" %} The same script URL, `core-api-endpoint` and your agent id are in the embed snippet on your agent's settings page in the [dashboard](https://ojin.ai/dashboard). Copy that snippet rather than retyping the id. {% endhint %} ## Embed Attributes | Attribute | Default | Description | | ------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `agent-id` | required | The agent to connect to. Copy it from the agent settings page | | `core-api-endpoint` | required | `https://api.ojin.ai` | | `enable-video` | `true` | Set `"false"` for an audio-only agent | | `collapsable` | `true` when floating | Set `"false"` to pin the widget open. `widget-placement="inline"` is an alias for this | | `default-collapsed` | `false` | Set `"true"` to load collapsed. Only the value at mount matters; use `collapse()` and `expand()` afterwards | | `widget-placement` | `floating` | `inline` embeds the widget in the page flow instead of floating over it | | `widget-resizable` | `false` | Set `"true"` so visitors can scale the floating launcher from the corners. Ignored when inline. After a visitor resize, the Big/Small preset no longer applies. | `preview-url` appears in the TypeScript declaration above because the widget *writes* it, exposing the resolved preview image. Do not set it yourself. ## Appearance & Layout The widget renders as a floating overlay on your page. It includes the avatar video feed, audio controls, and session management: all self-contained. The agent owner can choose `off`, `conversation`, `tiktok`, `youtube`, `arthouse`, or `journalist` in the dashboard. They can also choose dark or light text, an accent colour, text size, and no background or a solid background. New agents start with captions off. Caption settings come from the agent configuration. The embed has no caption attributes or viewer controls. Captions hide while the viewer opens the chat panel and return when it closes. {% hint style="info" %} Widget styling and positioning options are configured in your agent's settings in the [dashboard](https://ojin.ai/dashboard). Check your agent's widget configuration for available customization options. {% endhint %} ## Finding Your Agent ID After [creating and configuring your agent](/apps/overview/configure), copy the agent ID from the agent settings page in the [Ojin dashboard](https://ojin.ai/dashboard). ## Controlling Access Agents are **private by default**: connections from any website are rejected until you configure access. ### Hostname Allowlist In the [dashboard](https://ojin.ai/dashboard), add the domains that are allowed to connect to your agent: | Configuration | Behaviour | | -------------------------------- | --------------------------------------------------- | | No hostnames configured | **Private**: all connections rejected | | `myapp.com`, `staging.myapp.com` | Only requests from those exact domains are accepted | | `*` | **Public**: connections accepted from any domain | {% hint style="warning" %} Using `*` (wildcard) allows anyone to connect to your agent. This is useful for demos but not recommended for production. Use specific hostnames to restrict access to your own domains. {% endhint %} {% hint style="info" %} No API key is needed in the widget. Access is controlled entirely by the hostname allowlist: the server checks the request's origin against your configured domains. {% endhint %} ## How It Works When the widget loads on your page: 1. The widget calls the Session API (`POST /v1/public/agents/connect`) with your agent ID 2. The server validates the request origin against the hostname allowlist 3. On success, the server returns a WebRTC room URL and token 4. The widget connects to the WebRTC room directly 5. Audio, video, and signaling all flow over a single WebRTC connection The entire handshake takes a few seconds. After that, your user is in a live conversation with the agent. ## Network Requirements If your site sets a Content Security Policy, the widget needs these sources allowed. Blocking the widget script stops it loading at all; blocking anything else fails quietly, leaving the widget running with the affected capability gone. The "If blocked" column says which is which. | Source | Directive | Purpose | If blocked | | ------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `https://cdn.jsdelivr.net` | `script-src` | The widget bundle itself | The widget does not load | | `https://api.ojin.ai` | `connect-src` | Session API: starting and monitoring a session | Sessions cannot start | | `https://*.daily.co` | `connect-src` | WebRTC signalling, and fetching the call engine the widget joins the room with | No audio or video | | `https://*.pluot.blue`, `wss:` | `connect-src` | Daily's media servers and their signalling sockets | No audio or video | | `'unsafe-eval'`, `blob:` | `script-src` | The call engine is fetched at runtime and evaluated in the page | No audio or video | | `'self'`, `blob:` | `worker-src` | Daily runs noise cancellation in a worker started from a blob URL | No audio or video | | `https://*.amazonaws.com` | `media-src`, `img-src` | The agent's idle preview video and poster image: the Session API returns them as presigned S3 URLs | The idle preview is blank; sessions still work | | `https://fonts.googleapis.com` | `style-src` | The Manrope webfont stylesheet | The widget renders in the fallback system font stack; everything else works | | `https://fonts.gstatic.com` | `font-src` | The Manrope font files | Same as above | | `'unsafe-inline'` | `style-src` | The widget's components inject their own styles at runtime | The widget renders unstyled | | `https://*.ingest.us.sentry.io` | `connect-src` | Error and reliability reporting | Reporting stops; conversations are unaffected | A policy that covers all of it: ``` default-src 'self'; script-src 'self' https://cdn.jsdelivr.net 'unsafe-eval' blob:; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data: blob: https://*.amazonaws.com; media-src 'self' blob: https://*.amazonaws.com; connect-src 'self' https://api.ojin.ai https://*.daily.co https://*.pluot.blue https://*.ingest.us.sentry.io wss:; worker-src 'self' blob:; frame-src 'self'; ``` {% hint style="info" %} `'unsafe-eval'` is needed because the WebRTC SDK downloads its call engine at connect time and evaluates it in the page. If your policy cannot allow it, get in touch: the SDK has a script-loading mode that trades `'unsafe-eval'` for `script-src https://*.daily.co`. {% endhint %} The widget also needs `microphone` permission via `Permissions-Policy` if you serve one, and WebRTC must not be disabled by a browser extension or corporate proxy. Note the widget has always contacted a Sentry host: the WebRTC library it embeds reports its own errors there. Ojin's own reporting uses the same host. ### What the widget reports The widget records how each connection attempt ended (succeeded, the microphone was refused, the agent was unavailable, the connection dropped) so we can find and fix problems that only appear on real devices. It does **not** send conversation audio, video, transcripts, or anything your user types. The conversation itself is never recorded. It does not set cookies for this, and does not send an identifier that follows a visitor between sessions. What it sends is the outcome, a coarse attribution of where the fault lay, a coarse browser family (for example "Safari"), whether the page is inside an in-app browser, the microphone permission state, the HTTP status when the connect was rejected, the widget version, the avatar model variant that served the attempt, and, when an attempt timed out without reporting, how far it had got. ## Reacting to the Agent The widget emits DOM events on `window` while a session runs (speaking state, live transcripts, and client-delivered tool calls) so your page can build custom UI or trigger actions. See [**Tools & Events**](/apps/overview/tools) for the full event reference and for handling function-calling tools. ## Custom Integrations If you need more control than the widget provides, for example building a native mobile app or a custom web experience, you can call the Session API directly from your backend and connect to the WebRTC room yourself. The **REST + Daily** tab in [Quick Start](#quick-start) shows the browser version of this. See the [Session API Reference](/apps/overview/api-reference) for details. ## Troubleshooting ### Widget not loading * Use `https://cdn.jsdelivr.net/npm/@ojin/agent@latest/dist/ojin-agent.js`. Do not use `https://widget.ojin.ai/ojin-agent.js`: that host has no DNS record, so the script never loads and `` stays an empty tag. * Confirm the `agent-id` attribute matches your agent's ID in the dashboard * Set `core-api-endpoint` to the value in the dashboard snippet (`https://api.ojin.ai` in production) ### "agent\_offline" error * Check that your agent's status is set to **online** in the [dashboard](https://ojin.ai/dashboard) * Ensure all required fields are filled: the dashboard won't let you go online with missing configuration ### "auth\_failed" error * Verify your domain is in the agent's hostname allowlist in the [dashboard](https://ojin.ai/dashboard) * Check that the `Origin` header sent by the browser matches one of your configured hostnames exactly * If testing locally, add `localhost` or `127.0.0.1` to the allowlist ### "concurrency\_limit" error * Your agent's maximum concurrent sessions are in use. Wait for an active session to end or increase the **max concurrency** setting in the dashboard ### No audio or video * Ensure the user's browser has granted microphone and camera permissions * Check that the WebRTC connection is not blocked by a firewall or corporate proxy # Session API Reference Start agent sessions programmatically. The widget uses this API automatically, call it directly for custom integrations. ## Base URL ``` https://api.ojin.ai ``` There is one endpoint. It is not regional, so there is no region to choose. Every path below already includes the `/v1` prefix, so append it to the base URL as written. {% hint style="info" %} **This is not the same endpoint as the one the "deploy in US East" guidance refers to.** They are two different things: * **`api.ojin.ai` (this API)** is the agent control plane. You call it to start and cancel sessions. One global endpoint, no region in the hostname. * **The models WebSocket** is where audio and video stream. If you drive the face models yourself with the [Python SDK](/models/build-with-python-sdk) or the [realtime API](/models/introduction/api), run **your own backend** in US East, close to Ojin's inference. See [Optimizing Performance](/guides/optimizing-performance). If you only use the Human Agent widget or this Session API, the US East guidance does not apply to you. {% endhint %} ## Start a Session ### `POST /v1/public/agents/connect` Start a new session for a given agent. Returns connection details for joining the WebRTC room. ### Request **Headers:** | Header | Required | Description | | -------------- | -------- | ------------------ | | `Content-Type` | Yes | `application/json` | **Body:** ```json { "agent_id": "your-agent-id", "client_user_ref": "optional-user-identifier" } ``` | Field | Type | Required | Description | | ----------------- | ------ | -------- | --------------------------------------------------------------------------------------------------------------- | | `agent_id` | string | Yes | The ID of the agent to start a session with | | `client_user_ref` | string | No | An opaque identifier for your end user. Appears in call history for analytics and tracking. Max 256 characters. | ### Response (200 OK) ```json { "session_id": "550e8400-e29b-41d4-a716-446655440000", "room_url": "https://example.daily.co/room-name", "token": "eyJhbGciOiJIUzI1NiIs..." } ``` | Field | Type | Description | | ------------ | ------ | ----------------------------------------- | | `session_id` | string | Unique identifier for this session | | `room_url` | string | WebRTC room URL to connect to | | `token` | string | Authentication token for joining the room | Use `room_url` and `token` to connect to the WebRTC room from your client. If you're building a web application, you can use the [Daily JavaScript SDK](https://docs.daily.co/reference/daily-js) to join the room. ### Error Responses All errors return a JSON body with `error_code` and `message`: ```json { "error_code": "agent_unpublished", "message": "Agent exists but is currently unpublished" } ``` For `concurrency_limit` errors, the response includes an additional `retry_after_seconds` field indicating how long to wait before retrying: ```json { "error_code": "concurrency_limit", "message": "Max concurrency reached", "retry_after_seconds": 5 } ``` | Error Code | HTTP Status | Description | | ---------------------- | ----------- | ---------------------------------------------------------------- | | `agent_not_found` | 404 | Agent ID does not exist or is not visible to the caller | | `agent_unpublished` | 403 | Agent exists but status is unpublished | | `auth_failed` | 403 | Hostname allowlist validation failed | | `concurrency_limit` | 429 | Max concurrency reached. Response includes `retry_after_seconds` | | `room_creation_failed` | 502 | WebRTC room could not be created | | `orchestrator_error` | 502 | Agent orchestrator is unreachable or returned an error | | `internal_error` | 500 | Unexpected server error | ### Authentication Currently, the public Session API uses **hostname allowlist** authentication. When a browser request arrives, the server checks the `Origin` header against the domains configured on the agent: * If the origin matches an allowed hostname, the request proceeds * If no hostnames are configured, all requests are rejected (agent is private) * If `*` is configured, all origins are accepted No API key or token is needed from the client side when using hostname allowlist. {% hint style="info" %} Server-side calls without a browser `Origin` header only work for agents configured with `*` in their allowed hostnames. Production server-to-server authentication is planned separately. {% endhint %} ## Cancel a Session by Room ### `POST /v1/public/agents/connect/{room}` Cancel an active public session after a room has been allocated. This is mainly used by browser clients during unload, tab-hide, or abort cleanup. `{room}` is the Daily room name from the `room_url` returned by `POST /v1/public/agents/connect`. ### Cancel Response (200 OK) ```json { "released": 1 } ``` | Field | Type | Description | | ---------- | ------ | ----------------------------------------------------------------------------------- | | `released` | number | Number of active sessions released. `0` means the room was already gone or unknown. | ## Examples {% tabs %} {% tab title="curl" %} ```bash # Works only for wildcard/demo agents because curl does not attach a browser Origin. curl -X POST https://api.ojin.ai/v1/public/agents/connect \ -H "Content-Type: application/json" \ -d '{"agent_id": "your-agent-id"}' ``` {% endtab %} {% tab title="Python" %} ```python import requests # Works only for wildcard/demo agents because server-side HTTP clients do not # attach a browser Origin. response = requests.post( "https://api.ojin.ai/v1/public/agents/connect", json={"agent_id": "your-agent-id"} ) data = response.json() print(f"Session: {data['session_id']}") print(f"Room: {data['room_url']}") ``` {% endtab %} {% tab title="JavaScript" %} ```javascript const response = await fetch( "https://api.ojin.ai/v1/public/agents/connect", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ agent_id: "your-agent-id" }) } ); const { session_id, room_url, token } = await response.json(); ``` Release the session when the user leaves the page: ```javascript const room = new URL(room_url).pathname.split("/").filter(Boolean).pop(); if (room) { navigator.sendBeacon( `https://api.ojin.ai/v1/public/agents/connect/${encodeURIComponent(room)}`, "" ); } ``` {% endtab %} {% endtabs %} ## Connecting to the Room After receiving the `room_url` and `token` from the Session API, connect to the WebRTC room using the [Daily JavaScript SDK](https://docs.daily.co/reference/daily-js): ```bash npm install @daily-co/daily-js ``` ```javascript import DailyIframe from '@daily-co/daily-js'; // Create a call frame (attaches to DOM automatically) const callFrame = DailyIframe.createFrame(); // Join the room with the token from the Session API await callFrame.join({ url: room_url, token: token }); // The agent's audio and video tracks are now available callFrame.on('track-started', (event) => { if (event.participant && !event.participant.local) { // Remote participant = the agent // event.track contains the audio or video MediaStreamTrack } }); ``` {% hint style="info" %} This is only needed for custom integrations. If you're using the [widget](/apps/overview/widget-integration), it handles the room connection automatically. {% endhint %} # Introduction Ojin offers two real-time face models. [**Human Portrait**](/models/human-portrait) and [**Human Presence**](/models/human-presence). Both turn speech audio into a synchronized talking-avatar video in real time, and both are driven the same way: pick the model with its `config_id` and the integration code is identical. ## Presence vs Portrait, which face model? {% columns %} {% column %}

Human Presence, expression and movement, including the hands.

{% endcolumn %} {% column %}

Human Portrait, a focused talking face at the best price.

{% endcolumn %} {% endcolumns %} *Two personas, one rendered by each model. Presence moves through the whole frame; Portrait concentrates on the face.* | | **Human Presence** | **Human Portrait** | | -------- | ----------------------------------------------------- | -------------------------------------- | | Quality | Flagship, the most expressive and lifelike | Great, focused on the talking face | | Motion | Generative expressions **and** movement (incl. hands) | Lip-sync with facial expression | | Best for | When quality and expressiveness matter most | A solid talking face at the best price | | Cost | Higher | Lower | Both run in real time, and both are driven by the same [Python SDK](/models/build-with-python-sdk) and [Pipecat](/models/introduction/integrations) integration. You select the model with its `config_id`. To build, head to [**Get started**](/models/introduction/integrations). # Get started Build a real-time **Ojin** avatar in minutes, the same flow drives both [Human Portrait](/models/human-portrait) and [Human Presence](/models/human-presence). Pick the integration that fits your stack. We recommend the **Python SDK** or **Pipecat**. The raw WebSocket API is there when you need low-level control. ## Prerequisites 1. An Ojin account with an active API key, if you don't have one, [get your API key](/getting-started/authentication). 2. [Create a custom persona](/models/introduction/creating-persona) or use a [Persona Template](/models/introduction/using-persona-template). 3. A `config_id` (your Persona Configuration ID) from the [dashboard](https://ojin.ai/dashboard). {% hint style="info" %} **Production deployments:** the realtime transport is a WebSocket for **server-to-server** use over a stable connection. Drive it from a backend server (ideally in **US East**, near Ojin's inference) to keep your API key secure and latency low, then deliver the final media to end users over a transport built for real-time video (typically WebRTC). The Python SDK and Pipecat integration are designed to run server-side. {% endhint %} {% tabs %} {% tab title="Python SDK" %} ### Python SDK The [`ojin-client`](https://pypi.org/project/ojin-client/) SDK is the fastest way to drive Ojin from your own code. It streams a synchronized talking-avatar video from the TTS audio you feed it. ```bash pip install "ojin-client[stv]" ``` ```python import os from ojin.stv import OjinSTVClient client = OjinSTVClient( api_key=os.environ["OJIN_API_KEY"], config_id=os.environ["OJIN_CONFIG_ID"], # your Human Portrait or Human Presence config ) # Feed TTS audio with client.say(...), or start_turn() + send_tts_audio() to stream, # and consume the synced 25 fps audio + video from client.output_stream(). ``` **Full walkthrough →** [**Build with the Python SDK**](/models/build-with-python-sdk)**.** {% endtab %} {% tab title="Pipecat" %} ### Pipecat Drop an Ojin face into a [Pipecat](https://github.com/pipecat-ai/pipecat) voice agent with the [`pipecat-ojin`](https://pypi.org/project/pipecat-ojin/) package. `OjinVideoService` is the one stage that turns a voice agent into a video-call avatar. It sits right after your TTS and lip-syncs to it: ```mermaid flowchart LR A["transport.input()"] --> B[STT] --> C[LLM] --> D[TTS] D --> E["OjinVideoService"] E --> F["transport.output()"] style E fill:#6366f1,stroke:#4338ca,color:#ffffff ``` `OjinVideoService` is the only stage you add. Everything upstream of it is your existing voice pipeline, unchanged. ```bash pip install pipecat-ojin ``` ```python from pipecat_ojin import OjinVideoService, OjinVideoSettings avatar = OjinVideoService( OjinVideoSettings( api_key="OJIN_API_KEY", config_id="OJIN_CONFIG_ID", # your Human Portrait or Human Presence config image_size=(512, 512), # must match the transport's video_out size ) ) ``` #### How it works 1. The microphone captures speech input 2. Voice Activity Detection identifies speech segments 3. Your **STT** service transcribes user audio to text 4. Your **LLM** generates the assistant's reply 5. Your **TTS** service synthesizes the reply to audio 6. `OjinVideoService` animates your persona from that TTS audio 7. Synchronized video and audio frames stream back to the client in real time A complete, runnable voice + avatar agent (browser WebRTC or Daily) lives in [`pipecat-ojin/examples/ojin-bot`](https://github.com/ojinai/pipecat-ojin/tree/main/examples/ojin-bot). You bring the STT / LLM / TTS services for the pipeline. {% endtab %} {% tab title="WebSocket (advanced)" %} ### WebSocket (advanced) For low-level control, connect directly to the real-time WebSocket API and implement the binary protocol yourself. See the [API Reference (advanced) →](/models/introduction/api). {% endtab %} {% endtabs %} ## Next Steps * [**Build with the Python SDK →**](/models/build-with-python-sdk): install, auth, events, and the full quickstart * [**API Reference (advanced) →**](/models/introduction/api): the raw WebSocket protocol for custom integrations # API reference {% hint style="info" %} Most builders should use the [**Python SDK**](/models/build-with-python-sdk) or [**Pipecat**](/models/introduction/integrations). They implement this protocol, buffering, and audio/video sync for you. Read on only if you need low-level WebSocket control. {% endhint %} ## Overview Real-time talking head synthesis API. Send speech audio, receive synchronized video and audio frames. After connecting and receiving `SessionReady`, the server immediately begins streaming video and audio frames at 25fps. It does not wait for you to send anything first. When no speech audio has been sent, the server generates **silence frames** (persona at rest with idle animation). When you send speech audio, the server generates **speech frames** with lip-synced animation synchronized to your audio. You only need to send speech audio. No silence, padding, or keep-alive messages are required. {% hint style="info" %} **Production deployments:** This WebSocket API is intended for **server-to-server** use over a stable connection. Connect from a backend server rather than a front-end client, to keep your API key secure and because the raw WebSocket isn't built for flaky end-user networks. Run the backend in **US East**, close to Ojin's inference, for the lowest latency, and deliver the final media stream to end users over a transport built for varying network conditions, typically **WebRTC**, for smooth, reliable, low-latency playback. {% endhint %} *** ## How It Works 1. **Connect** to the WebSocket endpoint with your API key and config ID 2. **Receive `SessionReady`**: the server has allocated inference resources for your session 3. **The server starts streaming frames immediately**: silence frames with idle animation, no request needed 4. **Send speech audio** whenever it becomes available (e.g., TTS output from your language model), also buffer it locally for playback 5. **Receive speech frames**: the server transitions to lip-synced animation and returns to silence frames when audio runs out 6. **Render video frames** at 25fps, keeping a small jitter buffer (trim idle frames only if they back up after a stall) 7. **Start playing your buffered TTS audio** when the first speech frame (`frame_type` `1` or `3`) arrives from Ojin, stop when speech ends ### Frame Types Every frame arrives as a binary `InteractionResponse` containing both a JPEG image and a PCM audio chunk. Frames are always delivered in order. The `frame_type` field classifies each frame: | `frame_type` | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `0` | **Idle**, persona at rest with idle animation. Generated automatically when no speech audio is queued | | `1` | **Speech**, lip-synced animation generated from your audio input | | `2` | **Fade-out**, post-cancel ramp back toward idle after an interruption | | `3` | **Start of speech**, first speech frame of a turn **resuming after an interruption/cancel**. Natural (uninterrupted) turns start with `1`, not `3` | ### Buffering for Network Jitter The server delivers frames at **realtime 25 fps**, so your buffer does not grow on its own. Keep a **small client-side buffer** (a few frames) to absorb network jitter and prevent stuttering during speech, and start playback once it's filled. The right buffer size depends on your network conditions and latency requirements: keep it as low as possible to minimize latency, but high enough to absorb jitter without starving playback. If frames ever back up, for example a brief network stall followed by a burst of queued frames arriving at once. You can recover by trimming **idle** frames (`frame_type == 0`). Only idle frames are safe to drop, never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames. ```python # When consuming frames from the buffer: frame = buffer.popleft() # If frames backed up after a stall, recover by skipping every other idle frame. # Test the frame you are about to drop, not the one you just took: an idle # frame is often followed by speech, and dropping that leaves a media gap. if len(buffer) > target_buffer_size and buffer[0].frame_type == 0: skip_counter += 1 if skip_counter % 2 == 0: buffer.popleft() # drop one idle frame ``` *** ## Connection Flow ```mermaid sequenceDiagram participant Client participant Server Note over Client,Server: Connection Client->>Server: WebSocket Connect Server->>Client: SessionReady (JSON) Note over Client,Server: Server Streams Immediately Server->>Client: Frame (idle, frame_type=0) Server->>Client: Frame (idle, frame_type=0) Server->>Client: Frame (idle, frame_type=0) Note over Client,Server: Client Sends Speech Audio Client->>Server: InteractionInput (TTS audio chunk 1) Client->>Server: InteractionInput (TTS audio chunk 2) Note over Client,Server: Server Transitions to Speech Server->>Client: Frame (start-of-speech, frame_type=3) Server->>Client: Frame (speech, frame_type=1) Server->>Client: Frame (speech, frame_type=1) Note right of Server: Delivered at realtime 25 fps Note over Client,Server: Audio Runs Out → Back to Idle Server->>Client: Frame (idle, frame_type=0) Server->>Client: Frame (idle, frame_type=0) Note right of Client: Client keeps a small jitter buffer ``` *** ## WebSocket Handshake ``` GET wss://models.ojin.ai/realtime?config_id= Authorization: ``` Provide your API key in the `Authorization` header and the persona's `config_id` as a query parameter. The server upgrades the connection, sends `SessionReady`, and begins streaming frames immediately. | Parameter | In | Required | Description | | --------------- | ------ | -------- | ------------------------------------------------------------------------------------ | | `Authorization` | header | yes | Your raw API key. No `Bearer` prefix | | `config_id` | query | yes | Configuration ID for the persona, created in the Human Portrait tab of the dashboard | | Status | Meaning | | ------ | -------------------------------------------------------------------------------------------------------------------- | | `101` | Upgrade successful. The server sends a `SessionReady` JSON message, then streams binary `InteractionResponse` frames | | `401` | Invalid or missing API key | Recommended client settings: `ping_interval` 30 seconds, `ping_timeout` 10 seconds. *** ## Message Format {% hint style="info" %} **Mixed message types:** Both JSON (text) and binary messages are exchanged on the same WebSocket connection. Your client must check the WebSocket frame type to distinguish them: * **Text frames (JSON):** `SessionReady`, `ErrorResponse` (server → client), `CancelInteraction` (client → server) * **Binary frames:** `InteractionResponse` (server → client), `InteractionInput` (client → server) {% endhint %} {% hint style="info" %} **Byte order:** All multi-byte integer fields in binary messages use **network byte order (big-endian)**. {% endhint %} *** ## Messages Reference ### Server → Client Messages | Message | Frame | Description | | ------------------------------------------------------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `SessionReady` | JSON | Sent once after the connection is established and inference resources are allocated. The server begins streaming frames immediately afterwards | | [`InteractionResponse`](#interactionresponse-server-client-binary) | binary | A video frame and its synchronized audio chunk. Streamed continuously: idle frames when at rest, speech frames when processing your audio | | [`ErrorResponse`](#errorresponse-server-client-json) | JSON | Sent when an error occurs. In some conditions, such as no backend server being available, the connection may close without one | ### Client → Server Messages | Message | Frame | Description | | ------------------------------------------------------------ | ------ | ------------------------------------------------------------------------------------------------------- | | [`InteractionInput`](#interactioninput-client-server-binary) | binary | Speech audio. Only send speech audio, never silence or padding | | [`CancelInteraction`](#cancelinteraction) | JSON | Stop processing and discard remaining frames immediately. No final frame is sent. Use for interruptions | Field-level layouts for the binary messages are in [Message Details](#message-details) below. *** ## Message Details ### InteractionInput (Client → Server, Binary) Binary message for sending speech audio to the server. **Only send speech audio**. Do not send silence or padding. **Binary structure:** ``` [1 byte ] Payload type : uint8, always 1 for audio [8 bytes] Timestamp : uint64, milliseconds since Unix epoch [4 bytes] Params size : uint32, byte length of the JSON params block (0 if no params) [N bytes] Params JSON : UTF-8 encoded JSON (only present if params size > 0) [M bytes] Audio payload : raw PCM int16 speech audio data ``` **Header fields** use **big-endian** byte order. The PCM audio samples in the payload use **little-endian** (standard for PCM int16). In Python: `struct.pack('!BQI', payload_type, timestamp, params_size)`. **Audio requirements:** | Property | Value | | ---------------- | -------------------------------------------------- | | Format | PCM signed 16-bit integers (little-endian samples) | | Sample rate | 16,000 Hz | | Channels | 1 (mono) | | Max message size | 512 KB (entire binary message including header) | **Recommended streaming pattern:** The server runs a 25 fps virtual timeline and needs your audio **input to stay slightly ahead** of it. Forwarding tiny TTS fragments one at a time (e.g. 40 ms chunks) makes input rate match output rate, so the server starves and emits idle frames between speech, lip-sync skips or drifts. Instead, feed audio with a cushion, **not** per-fragment: 1. **Prime \~1 second** of speech audio at the start of a turn. 2. Then send the **largest chunks you can**, coalesce queued fragments into **\~400 ms** sends (under the 512 KB cap). 3. Stay realtime: never wait for the whole utterance before sending. {% hint style="success" %} The [**Python SDK**](/models/build-with-python-sdk) and [**Pipecat**](/models/introduction/integrations) shape the input for you. `OjinSTVClient` primes the lead and coalesces your TTS into large chunks automatically, so you just feed audio as it arrives. This pattern only applies if you drive this WebSocket API directly. See [Optimizing Performance](/guides/optimizing-performance). {% endhint %} ```python import struct, json, time def build_audio_message(audio_bytes): header = struct.pack('!BQI', 1, # payload type: audio int(time.time() * 1000), # timestamp ms 0, # params size (unused for human-portrait) ) return header + audio_bytes ``` *** ### InteractionResponse (Server → Client, Binary) Binary message containing a video frame and synchronized audio. The server streams these continuously after `SessionReady`. **Frames always arrive in order.** **Binary structure:** ``` [1 byte ] Is final flag : uint8, 1 = last frame for this interaction, 0 = more coming [16 bytes] Interaction ID : UUID bytes (big-endian) [8 bytes ] Timestamp : uint64, milliseconds since Unix epoch [4 bytes ] Usage : uint32, usage metric for this response [4 bytes ] Reserved : uint32, reserved; ignore [4 bytes ] Num payloads : uint32, number of payload entries that follow For each payload entry: [4 bytes] Data size : uint32, byte length of the payload data only [1 byte ] Payload type : uint8, 1 = audio, 2 = image [N bytes] Payload data : raw payload bytes [1 byte ] Frame type : uint8, appended after all payload entries: 0=idle, 1=speech, 2=fade-out, 3=start-of-speech ``` All multi-byte integers are **big-endian**. In Python: `struct.unpack('!B16sQIII', header_bytes)` for the main header, `struct.unpack('!IB', entry_bytes)` for each payload entry. The `Frame type` byte is a single trailing `uint8` after the last payload entry and is the authoritative frame classifier. **Frame type:** | Frame type | Meaning | | ---------- | ---------------------------------------------------------------------------------------------------------------------- | | `0` | **Idle**, persona at rest with idle animation | | `1` | **Speech**, lip-synced animation from your audio | | `2` | **Fade-out**, post-cancel ramp toward idle | | `3` | **Start of speech**, first speech frame of a turn resuming after an interruption/cancel (natural turns start with `1`) | **Payload types:** | Type | Format | Typical size per frame | | --------- | --------------------- | ------------------------------------------------------ | | 1 (audio) | PCM int16, 16kHz mono | **1,280 bytes** (640 samples = 40ms at 25fps) | | 2 (image) | JPEG-encoded image | Variable (resolution depends on config, e.g. 1280×720) | **Parsing example:** ```python import struct, uuid HEADER_FMT = '!B16sQIII' HEADER_SIZE = struct.calcsize(HEADER_FMT) # 37 bytes ENTRY_FMT = '!IB' ENTRY_SIZE = struct.calcsize(ENTRY_FMT) # 5 bytes def parse_response(data): # The 5th header field is reserved; ignore it. is_final, uuid_bytes, timestamp, usage, _reserved, num_payloads = \ struct.unpack(HEADER_FMT, data[:HEADER_SIZE]) offset = HEADER_SIZE image = audio = None for _ in range(num_payloads): size, ptype = struct.unpack(ENTRY_FMT, data[offset:offset + ENTRY_SIZE]) offset += ENTRY_SIZE payload = data[offset:offset + size] offset += size if ptype == 2: image = payload # JPEG bytes elif ptype == 1: audio = payload # PCM int16 bytes # Trailing frame type byte, appended after the payload entries. frame_type = data[offset] return { 'is_final': bool(is_final), 'frame_type': frame_type, # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech 'image': image, 'audio': audio, } ``` *** ### CancelInteraction | Message | Purpose | Server behavior | Use case | | ------------------- | -------------- | ------------------------------------------- | ----------------- | | `CancelInteraction` | Immediate stop | Stops processing, discards remaining frames | User interruption | *** ### ErrorResponse (Server → Client, JSON) {% hint style="warning" %} **Plain text errors:** In some error conditions (e.g., no backend servers available), the server may send a plain text message instead of a structured JSON `ErrorResponse`. Your client should handle non-JSON text messages gracefully. {% endhint %} **Error codes:** | Code | Description | | --------------------- | ---------------------------------------- | | `AUTH_FAILED` | Invalid API key | | `UNAUTHORIZED` | Caller lacks permission | | `MISSING_CONFIG_ID` | `config_id` query parameter not provided | | `INVALID_MESSAGE` | Malformed or unsupported message payload | | `INVALID_HEADERS` | Missing or invalid headers | | `MODEL_NOT_FOUND` | Config ID not found or invalid | | `BACKEND_UNAVAILABLE` | No healthy inference backend available | | `RATE_LIMITED` | Too many requests | | `TIMEOUT` | Operation exceeded processing time | | `CANCELLED` | Interaction cancelled by client | | `INTERNAL_ERROR` | Unexpected server error | | `FRAME_SIZE_EXCEEDED` | Message exceeded 512KB limit | *** ## Rate Limits & Constraints | Constraint | Value | | ---------------- | -------------------------- | | Rate limit | 6 requests per second | | Max message size | 512 KB per message | | Video output | 25 fps (realtime delivery) | Exceeding limits results in an `ErrorResponse` with code `RATE_LIMITED`. *** ## Best Practices ### Audio Input * **Send speech audio only.** The server starts streaming idle frames on its own, so there is no handshake frame to send first and no need to pad the gaps between utterances * **Prime \~1 s of audio, then send the largest chunks you can (\~400 ms).** The server needs your input to stay slightly ahead of its 25 fps timeline. Forwarding tiny fragments one at a time starves it and produces idle frames between speech, buffer locally for playback, but coalesce what you send. See the **InteractionInput** message details above and [Optimizing Performance](/guides/optimizing-performance). * The [Python SDK](/models/build-with-python-sdk) and [Pipecat](/models/introduction/integrations) handle this input shaping for you. This only matters when driving the WebSocket API directly. ### Buffer Management * Play frames at **25 fps** (40ms per frame) * The server delivers at realtime 25 fps. Keep a **small jitter buffer** (a few frames); it won't grow on its own * **Recovery:** if frames back up after a network stall, skip 1 out of every 2 idle frames (`frame_type == 0`) until the buffer shrinks back * **Never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames** * Tune your target buffer size based on your network conditions. Keep it as low as possible for minimal latency ### Audio and Video Synchronization Each `InteractionResponse` contains both a JPEG image and a PCM audio chunk. However, the **recommended approach for audio playback** is: 1. **Buffer your TTS/source audio locally** as it arrives from your speech service (for later playback only. You do not need to buffer it to send it to Ojin) 2. **Forward TTS audio to Ojin immediately** as it arrives from your speech service 3. **Wait for the first speech frame** (`frame_type` `1` or `3`) to arrive from Ojin 4. **Start playing your buffered TTS audio** at that moment 5. **Stop audio playback** when speech ends, the first idle or fade-out frame (`frame_type` `0` or `2`) after speech 6. **Render video** from every frame regardless of type Sending is immediate, but playback is gated on the first speech frame. This ensures audio and video stay in sync. ```python # When TTS audio arrives from your speech service: speech_audio_buffer.extend(tts_audio_chunk) # buffer locally for playback await ojin.send_audio(tts_audio_chunk) # send to Ojin for lip-sync # In your video playback loop: frame = buffer.popleft() # Speech frames are frame_type 1 (speech) and 3 (start-of-speech). if frame.frame_type in (1, 3) and not audio_playing: start_audio_playback(speech_audio_buffer) # begin draining the buffer audio_playing = True # Non-speech frames are frame_type 0 (idle) and 2 (fade-out). if frame.frame_type in (0, 2) and audio_playing: stop_audio_playback() audio_playing = False render_video(frame.image) # always render the video ``` ### Error Handling * Handle both JSON `ErrorResponse` messages and plain text error strings * Implement exponential backoff for reconnection * Monitor server `load` in the `SessionReady` message ### Interruption Handling * Use `CancelInteraction` for immediate stops (e.g., user interrupts the bot) * Clear your frame buffer on interruption *** ## Complete Example ```python import asyncio import json import struct import time from collections import deque import numpy as np import websockets from dotenv import load_dotenv import os load_dotenv() API_KEY = os.getenv("OJIN_API_KEY", "") CONFIG_ID = os.getenv("OJIN_CONFIG_ID", "") URL = f"wss://models.ojin.ai/realtime?config_id={CONFIG_ID}" SAMPLE_RATE = 16000 FPS = 25 TARGET_BUFFER = 10 # Tune based on your network conditions def build_audio_message(audio_bytes): """Build a binary InteractionInput message.""" header = struct.pack('!BQI', 1, int(time.time() * 1000), 0) return header + audio_bytes def parse_response(data): """Parse a binary InteractionResponse message.""" fmt = '!B16sQIII' hdr_size = struct.calcsize(fmt) # The 5th header field is reserved; ignore it. is_final, uid_bytes, ts, usage, _reserved, n_payloads = struct.unpack(fmt, data[:hdr_size]) offset = hdr_size image = audio = None for _ in range(n_payloads): size, ptype = struct.unpack('!IB', data[offset:offset+5]) offset += 5 if ptype == 2: image = data[offset:offset+size] elif ptype == 1: audio = data[offset:offset+size] offset += size # Trailing frame type byte, appended after the payload entries. frame_type = data[offset] return { 'is_final': bool(is_final), 'frame_type': frame_type, # 0=idle, 1=speech, 2=fade-out, 3=start-of-speech 'image': image, 'audio': audio, } async def main(): headers = {"Authorization": API_KEY} # For older websockets versions, use extra_headers instead. async with websockets.connect(URL, additional_headers=headers, ping_interval=30) as ws: # 1. Wait for SessionReady; the server starts streaming frames immediately after msg = json.loads(await ws.recv()) assert msg["type"] == "sessionReady" print(f"Session ready: {msg['payload']}") buffer = deque() skip_counter = 0 playback_started = False frame_count = 0 audio_sent = False # 2. Receive and process frames. The server streams idle frames # immediately after SessionReady; nothing needs to be sent first. async for data in ws: if isinstance(data, str): msg = json.loads(data) if msg.get("type") == "errorResponse": print(f"Error: {msg['payload']}") break continue frame = parse_response(data) buffer.append(frame) frame_count += 1 # Wait for initial buffer before playback if not playback_started: if len(buffer) >= TARGET_BUFFER: playback_started = True print(f"Buffer filled ({TARGET_BUFFER} frames), starting playback") continue # Consume one frame if buffer: play_frame = buffer.popleft() # Drop excess idle frames: skip 1 out of 2 when buffer is too large. # Only idle (frame_type 0) is safe to drop; keep speech (1), # start-of-speech (3), and fade-out (2). Check the frame you are # about to drop, not the one you just played: an idle frame is # often followed by speech, and dropping that leaves a media gap. if len(buffer) > TARGET_BUFFER and buffer[0]['frame_type'] == 0: skip_counter += 1 if skip_counter % 2 == 0: buffer.popleft() # drop one idle frame kind = {0: "idle", 1: "speech", 2: "fade-out", 3: "start-of-speech"}.get( play_frame['frame_type'], "?" ) print(f"[{kind}] frame #{frame_count}, buffer={len(buffer)}") # In a real app: render play_frame['image'] and play play_frame['audio'] # Demo: send speech audio after receiving some silence frames if frame_count == 50 and not audio_sent: t = np.linspace(0, 1.0, SAMPLE_RATE, endpoint=False) audio_data = (32767 * 0.5 * np.sin(2 * np.pi * 440 * t)).astype(np.int16) chunk_size = SAMPLE_RATE # 1s chunks for i in range(0, len(audio_data), chunk_size): chunk = audio_data[i:i + chunk_size] await ws.send(build_audio_message(chunk.tobytes())) audio_sent = True print("Sent 1 second of speech audio") if frame_count > 200: break asyncio.run(main()) ``` *** ## Troubleshooting Symptoms, causes, and fixes, connection and auth, a starved/idle mouth, choppy playback, latency, and frame lag (`speech_filter_amount`), live in the global [Troubleshooting guide](/guides/troubleshooting), where each entry is tagged 🟢 SDK-handled or 🟠 your setup. *** ## Example Implementation A complete working Python example integrating Ojin Human Portrait with a speech-to-speech service (Hume EVI) is available here: [**github.com/journee-live/speech-to-video-samples/tree/main/samples**](https://github.com/journee-live/speech-to-video-samples/tree/main/samples) (includes Hume STS → Human Portrait walkthroughs) The repository demonstrates the full integration pattern: microphone capture → STS service → TTS audio → Ojin lip-sync → synchronized video and audio playback at 25fps. It includes the buffer management and frame handling approach described in [Best Practices](#best-practices) above. *** # Using a Persona Template Learn how to create a persona out of a template to get started ASAP with your application. Templates work for both [Human Portrait](/models/human-portrait) and [Human Presence](/models/human-presence). ## Prerequisites * An Ojin account with an active API key ## Creating a Persona through the Dashboard The simplest way to create a persona is through the Ojin Dashboard: 1. Log in to the [Ojin Dashboard](https://ojin.ai) 2. Navigate to the [**Human Portrait**](https://ojin.ai/models/ojin/human-portrait) or [**Human Presence**](https://ojin.ai/models/ojin/human-presence) section 3. Navigate to the **Configs** sub-section 4. Select a persona template and press **Copy Template** 5. Open the newly created model configuration and save the **Model Config ID** parameter which will be used by your application 6. You can now integrate it through the [model API endpoints](/models/introduction/api) ## Next Steps Once your persona is ready, you can:
Integration GuideIntegrate your persona using Pipecat or the WebSocket API.Get started
API ReferenceExplore the complete API documentation.API reference
# Creating a custom persona Create a persona by uploading a reference image when you add a configuration in the [Ojin Dashboard](https://ojin.ai/dashboard). The flow is the same for both face models. ## Creating a configuration through the Dashboard 1. Log in to the [Ojin Dashboard](https://ojin.ai/dashboard) 2. Select your model: [**Human Portrait**](/models/human-portrait) or [**Human Presence**](/models/human-presence) 3. Navigate to the **Configs** sub-section 4. Press **New Configuration** to create a new configuration 5. Fill in the required fields and upload a reference image; follow your model's reference-image best practices for the best result 6. Click **Create Configuration** 7. Open your new configuration and copy the **Model Config ID** for your application {% hint style="info" %} Reference-image guidance is specific to each model. See **Best practices** for [Human Portrait](/models/human-portrait/best-practices) or [Human Presence](/models/human-presence/best-practices). {% endhint %} ## Next Steps * [**Get started**](/models/introduction/integrations): integrate with the Python SDK, Pipecat, or WebSocket * [**API Reference (advanced)**](/models/introduction/api): the raw WebSocket protocol # Human Portrait A cost-effective, lifelike persona model that transforms reference images into natural animated personas ## Overview The ojin/human-portrait model creates realistic, expressive digital humans from a single reference image. It excels at producing natural facial animations, lip-syncing, and emotional expressions that bring your persona to life with synchronized speech, at the best price for a real-time talking face. {% columns %} {% column %}

Hana, virtual assistant

{% endcolumn %} {% column %}

Marcus, tutor

{% endcolumn %} {% column %}

Asha, health assistant

{% endcolumn %} {% endcolumns %} *One model, any persona. Each generated from a single reference image.* Want the most expressive, lifelike result? [Human Presence](/models/human-presence) is our flagship face model, with generative expressions and natural movement. Both models are driven the same way. See [Build with the Python SDK](/models/build-with-python-sdk). ## Key Features * **Full persona look control** - Generate a persona based on any image reference, the persona will behave exactly the same * **No training required** - You don't need to wait for your persona to be ready, as soon as the reference image is uploaded, you can start using it * **Natural Lip-Syncing** - Precise lip movements synchronized with speech audio * **Emotional Expressions** - Support for multiple emotional states and expressions * **Real-Time** - Streams synchronized audio and video at a fixed 25 fps, for live conversation * **High Resolution** - Support for up to 720p output resolution ## Quick Start Getting started with ojin/human-portrait is simple: 1. [**Create an API key**](/getting-started/authentication) - Set up authentication for the Ojin platform 2. [**Use a persona template**](/models/introduction/using-persona-template) - Use a persona template to generate your persona in seconds 3. [**Integrate with your application**](/models/introduction/integrations) - Recommended: the [Python SDK](/models/build-with-python-sdk) or Pipecat ## Use Cases * **Virtual Assistants** - Create responsive customer service personas * **Educational Content** - Develop engaging tutors and instructors * **Entertainment** - Produce animated characters for games and media * **Presentations** - Transform static slides into dynamic video presentations * **Healthcare** - Build empathetic virtual health assistants # Best practices Follow the guidance below for the best results. Your reference image is the foundation of the persona, so a clean, well-framed image gives the most natural output. ## Reference image best practices * **Image content**: * Mouth should be closed, but smiles or subtle expressions are fine * The eyes should be looking directly into the camera * Keep the expression balanced, the image is used as the base for both the idle loop and speech, so avoid extreme poses * **Format**: JPEG, PNG, or WebP * **Lighting**: Even lighting with no harsh shadows * **Face Position**: The face should be clearly visible and centered * **Background**: Simple backgrounds work best * **Accessories**: Avoid sunglasses or items that obscure facial features # Human Presence Our flagship face model, a fully expressive, generative presence that goes beyond lip-sync. ## Overview **Human Presence** is Ojin's highest-quality real-time face model. It generates a fully expressive *presence*, rich facial expressions and natural movement, including the hands, for the most lifelike, dynamic result. {% columns %} {% column %}

Seraphina, game character

{% endcolumn %} {% column %}

Mathieu, chef

{% endcolumn %} {% column %}

Dr. Robert, clinician

{% endcolumn %} {% column %}

Olivia, brand spokesperson

{% endcolumn %} {% endcolumns %} *Generative expression and movement (including the hands) across four personas, photoreal and stylised alike.* Feed it speech audio and it streams back a synchronized talking-avatar video in real time. Pick the model with its `config_id`. New here? See the [Introduction](/models/introduction) to compare Ojin's face models and choose the right one. ## Key features * **Fully expressive presence**: generative facial expressions and natural movement, beyond lip-sync * **Real-time**: conversation-ready, streamed as synchronized audio + video * **One config, any integration**: drive it from the Python SDK, Pipecat, or the raw WebSocket API * **Model slug**: `ojin/human-presence` ## Quick Start The fastest way to build with Presence is the **Python SDK** or **Pipecat**: 1. [**Create an API key**](/getting-started/authentication): set up authentication for the Ojin platform 2. **Get a Human Presence `config_id`** from the [dashboard](https://ojin.ai/dashboard). See [Best practices](/models/human-presence/best-practices) for reference-image guidance 3. [**Integrate with your application**](/models/introduction/integrations): recommended: Python SDK or Pipecat ## Use Cases * **Brand & spokesperson**: a premium, lifelike presence that represents your brand * **High-touch support & sales**: expressive agents for moments that matter * **Education & training**: engaging tutors with natural expression and movement * **Interactive experiences**: installations, kiosks, and showcases that feel alive # Best practices Follow the guidance below for the best results. The higher the quality and the more detail your reference image has, the richer and more lifelike the generated video. ## Reference image best practices * **Resolution**: Use a **768 × 1280** reference image for the best results. * **Framing**: A **medium shot** (more of the body and hands in frame, best for rich, expressive motion) through to a **close-up** (best for fine facial detail) is the recommended range for this model. Choose where you sit in that range depending on whether you want more motion or more detail. * **Avoid full-body shots**: At full-body distance the face and hands become too small and the result loses detail, stay within the medium-shot-to-close-up range. * **Hands and body**: Show the hands in a passive, resting state and the body in a relaxed, resting position. The model generates natural body and hand movement from there. * **Backgrounds can animate**: A good, detailed background can come to life too. You are not limited to a plain backdrop. Try to avoid other visible faces close to your persona's face. * **Keep the face clearly visible**: A clearly visible, reasonably centered face is the main guideline; beyond that you have freedom over styling, pose, and background. * **Format**: JPEG, PNG, or WebP. # Python SDK Drive Ojin's face models (Human Presence and Human Portrait) straight from Python. Feed in your TTS audio; get back a synchronized, lip-synced talking-avatar stream. The [`ojin-client`](https://pypi.org/project/ojin-client/) SDK is the fastest way to add a real-time talking avatar to a Python app or voice agent. You bring the words (any TTS); the SDK streams back a face that speaks them (perfectly in sync) handling buffering, the playback clock, resampling, and barge-in for you. {% hint style="info" %} **Just want a complete agent?** If you'd rather not assemble a pipeline, the [Human Agent](/apps/overview) gives you a full speech-in, speech-out avatar with a single line of embed code. Already on [Pipecat](https://github.com/pipecat-ai/pipecat)? Jump to the [Pipecat integration](/models/introduction/integrations). {% endhint %} ## Requirements * **Python 3.10+** * An **Ojin API key**, [get yours here](/getting-started/authentication) * A face-model **config ID**, a Presence or Portrait configuration from your [dashboard](https://ojin.ai/dashboard) New accounts start with **$10 in free credits**. ## Install With [uv](https://docs.astral.sh/uv/) (recommended): ```bash uv add "ojin-client[stv]" ``` Or with pip: ```bash pip install "ojin-client[stv]" ``` The `[stv]` extra pulls in the speech-to-video helpers (`numpy`, `opencv-python-headless`, `soxr`) that handle resampling, decoding, and the sync math for you. ## Authenticate The client needs your API key and a configuration ID. Keep both out of source. Read them from the environment: ```bash export OJIN_API_KEY="…" # from your Ojin account export OJIN_CONFIG_ID="…" # a Presence or Portrait configuration to drive ``` ```python import os from ojin.stv import OjinSTVClient client = OjinSTVClient( api_key=os.environ["OJIN_API_KEY"], config_id=os.environ["OJIN_CONFIG_ID"], ) ``` Or let the SDK load and validate them for you. `resolve_credentials()` reads an optional `.env` file and raises a clear error if either value is missing: ```python from ojin import resolve_credentials from ojin.stv import OjinSTVClient creds = resolve_credentials() # reads OJIN_API_KEY + OJIN_CONFIG_ID client = OjinSTVClient(api_key=creds.api_key, config_id=creds.config_id) ``` By default the client connects to `wss://models.ojin.ai/realtime`. {% hint style="warning" %} Never hardcode your API key in source or commit it to version control. {% endhint %} ## Pick your model The SDK is **identical for both face models**; the `config_id` you pass selects which one runs. | A `config_id` from… | You get | Choose it when | | --------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------- | | a **Human Presence** config | the flagship, fully expressive generative presence, rich expressions and natural movement | quality and expressiveness matter most | | a **Human Portrait** config | a cost-effective, real-time lip-synced talking face | you want a great talking face at the best price | Create configurations in the [dashboard](https://ojin.ai/dashboard). The code below is the same either way; only the `config_id` changes. ## Two clients, one package | | `OjinSTVClient` (high-level) | `OjinClient` (low-level) | | ----------- | ---------------------------------------------------------- | ----------------------------------------- | | Import | `from ojin.stv import OjinSTVClient` | `from ojin.ojin_client import OjinClient` | | You give it | TTS audio at **any** sample rate | 16 kHz mono `int16` PCM | | You get | synced `STVAudioFrame` + `STVVideoFrame` on a 25 fps clock | raw protocol messages (JPEG + PCM) | | It handles | buffering, playback clock, resampling, barge-in re-sync | just the WebSocket + wire protocol | Most applications want **`OjinSTVClient`**. Reach for `OjinClient` only when you're writing your own playback/sync layer. ## Quickstart Feed the avatar one utterance of TTS audio and consume the synchronized stream it speaks back. ```python import asyncio import os import wave from ojin.stv import OjinSTVClient, STVEvent, STVAudioFrame, STVVideoFrame async def main() -> None: ready = asyncio.Event() done = asyncio.Event() async with OjinSTVClient( api_key=os.environ["OJIN_API_KEY"], config_id=os.environ["OJIN_CONFIG_ID"], ) as client: # Events may be sync or async; register with a decorator or add_listener(). client.add_listener(STVEvent.SESSION_READY, lambda **_: ready.set()) client.add_listener(STVEvent.BOT_STOPPED_SPEAKING, lambda **_: done.set()) @client.on(STVEvent.ERROR) def _on_error(message: str, **_): print("ojin error:", message) # Wait until the avatar session is live (race-free: skip if already up). if not client.is_connected: await ready.wait() # Send one utterance of mono int16 PCM TTS audio at ANY sample rate: # the SDK resamples the copy it sends to the server for lip-sync. with wave.open("hello.wav", "rb") as wav: # a mono WAV await client.say( wav.readframes(wav.getnframes()), sample_rate=wav.getframerate(), num_channels=1, ) # Consume the synchronized 25 fps audio + video stream. async def consume() -> None: async for frame in client.output_stream(): if isinstance(frame, STVVideoFrame) and frame.rgb is not None: ... # frame.rgb = width*height*3 RGB bytes, render it elif isinstance(frame, STVAudioFrame): ... # frame.pcm = int16 PCM @ frame.sample_rate, play it consumer = asyncio.create_task(consume()) await done.wait() # the avatar finished the turn await asyncio.sleep(0.5) # let the tail frames drain consumer.cancel() asyncio.run(main()) ``` `say()` is a one-shot helper for `start_turn()` + `send_tts_audio()`. For a **live agent**, call `start_turn()` once per utterance and stream chunks with `send_tts_audio()` as your TTS produces them, the user hears the exact audio you sent, lip-synced to the avatar. {% hint style="success" %} **The SDK shapes the input for you.** Feed audio at whatever cadence your TTS produces, even tiny 40 ms fragments. `OjinSTVClient` primes a \~1 s lead after each `start_turn()` and coalesces your audio into large chunks before forwarding it, so the inference head never starves and lip-sync stays stable. It also paces `output_stream()` to realtime 25 fps with audio and video already in sync. You don't manage input buffering, the playback clock, or frame-dropping yourself. See [Optimizing Performance](/guides/optimizing-performance) for how it works and how to tune it. {% endhint %} ## Events you'll use Register handlers with `@client.on(STVEvent.X)` or `client.add_listener(STVEvent.X, cb)`. Handlers may be sync or async; a failing handler never breaks the playback loop. | Event | Fires when | | ---------------------- | ------------------------------------------ | | `SESSION_READY` | the session is live and ready for audio | | `BOT_STARTED_SPEAKING` | the first speech frame of a turn is played | | `BOT_STOPPED_SPEAKING` | a turn has finished playing | | `INTERRUPTED` | a barge-in was accepted | | `ERROR` | a transport or server error occurred | | `CLOSED` | the session has been torn down | ## Where to go next
Runnable examplesRealtime streaming and render-to-MP4, ready to clone.https://github.com/ojinai/python-sdk/tree/main/examples
Build a voice agentDrop the avatar into a Pipecat pipeline.Get started
* **Best practices**: [working-pipeline patterns](/models/build-with-python-sdk/python-sdk-best-practices) for browser output and WebRTC backends * **Choose a face model**: [Human Presence](/models/human-presence) (flagship) or [Human Portrait](/models/human-portrait) (cost-effective) * **Tune latency & stability**: [Optimizing Performance](/guides/optimizing-performance) and the [Troubleshooting guide](/guides/troubleshooting) * **Full API surface**: the complete client reference lives in the [`ojin-client` README](https://github.com/ojinai/python-sdk) and on [PyPI](https://pypi.org/project/ojin-client/) * **Starter examples**: `02-realtime-speech-to-video` (live streaming) and `01-speech-to-video-mp4-generator` (render to file) # Best Practices The key things to get right for a smooth, low-latency avatar pipeline with OjinSTVClient, whether you're prototyping straight to a browser or running a backend that relays media over WebRTC. ## Essentials (every setup) * **Run it server-side.** The SDK connects to Ojin over a **server-to-server** WebSocket. Run it in a backend process, never on an end-user device, and keep your API key off the client. In production, run it in **US East**, close to Ojin's inference, for the lowest latency. * **Feed audio as your TTS produces it.** Call `start_turn()` once per utterance, then `send_tts_audio()` for each chunk as it arrives, even tiny 40 ms fragments. The SDK shapes the feed (primes a \~1 s lead, then coalesces to ≥400 ms) so the model never starves. Don't batch the whole utterance yourself. * **Present frames as they arrive. Don't re-sync.** `output_stream()` is already paced to realtime **25 fps**, with audio and video in sync and a small jitter buffer. Forward each frame the moment it arrives; never hold audio to wait for a video frame, and don't add your own A/V sync layer. * **Handle the lifecycle.** Wait for `SESSION_READY` before relying on the stream; use `BOT_STARTED_SPEAKING` / `BOT_STOPPED_SPEAKING` for UI state; on `ERROR`, retry with backoff (e.g. `NO_BACKEND_SERVER_AVAILABLE`). * **Barge-in with `interrupt()`.** It fades the current audio and cancels the turn server-side, just call it when the user starts talking. {% hint style="info" %} On [Pipecat](/models/introduction/integrations)? `pipecat-ojin`'s `OjinVideoService` already wires all of this into a pipeline and pushes frames into your transport, start there instead of hand-rolling the loop. {% endhint %} ## Setup A. Local, output to a browser (prototyping) Run the SDK in a local Python process and stream frames to a browser you control (e.g. over a WebSocket to a ``). Great for demos and development; **not** for production over real networks. * **Forward the JPEG straight to the browser.** Use a `PassthroughDecoder` so the SDK skips decoding. Read the raw JPEG from `STVVideoFrame.source_bytes` (\~60 KB) and draw it to a ``/`` as it arrives. (Need raw RGB in Python instead? Keep the default decoder and read `frame.rgb`.) * **Play audio at the rate you fed.** Run the browser `AudioContext` at the **same sample rate** as the audio you sent (`STVAudioFrame.sample_rate`). You can feed higher-quality TTS (e.g. 24 kHz) for better sound, the SDK plays back your original audio while lip-sync uses a 16 kHz copy. * **Present as they come.** Draw each video frame and queue each audio chunk the instant it arrives, no re-sync; the stream is already aligned. ```python client = OjinSTVClient( api_key=..., config_id=..., decoder=PassthroughDecoder(), # forward JPEG; the browser decodes it ) async for frame in client.output_stream(): if isinstance(frame, STVVideoFrame): await ws.send_bytes(frame.source_bytes) # raw JPEG -> draw on a canvas elif isinstance(frame, STVAudioFrame): await ws.send_bytes(frame.pcm) # play at frame.sample_rate ``` ## Setup B. Backend, relay to a WebRTC service (production) Run the SDK on a backend and relay the media to your users over a **WebRTC** service (LiveKit, Daily, mediasoup, …), which absorbs packet loss, jitter, and varying networks that a raw WebSocket can't. * **Deploy in US East**, close to Ojin's inference. * **Push frames into the transport with their `pts`.** Each `STVAudioFrame` / `STVVideoFrame` carries a `pts` timestamp, hand it to your WebRTC audio/video tracks and let the transport sync them for the end user. Don't re-sync yourself. * **Push, don't poll, when you can.** Inject a custom `STVOutput` sink to write frames straight into your transport instead of draining `output_stream()`, for lower latency and less glue. (This is exactly what `pipecat-ojin` does.) * **Match the video track to the model's frame size**: read `STVVideoFrame.width` / `height` and configure your outgoing track to match. * **Deliver to end users over WebRTC, not the raw WebSocket.** Keep the Ojin WebSocket strictly server-to-server. ```python async for frame in client.output_stream(): if isinstance(frame, STVVideoFrame): video_track.push(frame.rgb, pts=frame.pts) # feed your WebRTC track elif isinstance(frame, STVAudioFrame): audio_track.push(frame.pcm, pts=frame.pts) # transport handles A/V sync ``` ## See also * [Build with the Python SDK](/models/build-with-python-sdk), install, auth, events, full quickstart * [Optimizing Performance](/guides/optimizing-performance), the audio-feeding contract and tuning * [Troubleshooting](/guides/troubleshooting), symptoms, causes, and fixes * [API Reference (advanced)](/models/introduction/api), the raw WebSocket protocol # Optimizing Performance How to feed audio and play back frames for stable, low-latency lip-sync, and how the Python SDK and Pipecat handle most of it for you. {% hint style="success" %} **Using the** [**Python SDK**](/models/build-with-python-sdk) **or** [**Pipecat**](/models/introduction/integrations)**?** They already implement everything on this page, audio feeding, the playback clock, buffering, and audio/video sync. Read on if you want to tune them, or if you're building directly on the [raw WebSocket API](/models/introduction/api). {% endhint %} Each section below is tagged **🟢 SDK-handled** (automatic with the SDK or Pipecat, only your concern on the raw WebSocket API) or **🟠 Your setup** (you own it regardless of integration, deployment, network, credentials). ## Feed audio for stable, low-latency lip-sync 🟢 **SDK-handled**. `OjinSTVClient` shapes the feed automatically; the contract below is only for the raw WebSocket API. The model runs a fixed **25 fps** virtual timeline and needs your audio **input to stay slightly ahead** of that timeline to generate speech frames continuously. If you forward tiny TTS fragments one at a time (for example the \~40 ms chunks many TTS providers stream) your input rate matches the output rate, the model's lead never builds, and it **starves**: it falls back to idle frames in between, and lip-sync skips or drifts. The rule is **realtime cadence with a cushion**, not per-fragment forwarding: 1. **Prime \~1 second of audio** at the start of each turn before you rely on realtime delivery. 2. Then send the **largest chunks you can**: coalesce queued fragments into **\~400 ms** sends (staying under the 512 KB message cap). 3. Stay truly realtime: never wait for the whole utterance before you start. {% hint style="success" %} **`OjinSTVClient` shapes the input for you.** The SDK accumulates a \~1 s initial chunk after each `start_turn()` to establish the lead, then coalesces your TTS into ≥400 ms sends automatically, so the inference head never starves. You just feed audio as your TTS produces it; the SDK takes care of the input shape to optimize latency and stability (tunable via the `server_feed_*` fields on `STVConfig`). The contract above only matters if you drive the raw WebSocket API yourself. {% endhint %} ```python # With the SDK, feed audio as it arrives; shaping is automatic. await client.start_turn() async for chunk in tts_stream: # e.g. 40 ms ElevenLabs / Cartesia chunks await client.send_tts_audio(chunk.pcm, sample_rate=chunk.rate, num_channels=1) ``` ## Play back at realtime 🟢 **SDK-handled**: `output_stream()` is paced and buffered for you. The model delivers frames at **realtime 25 fps**, so your buffer doesn't grow on its own. You just keep a **small buffer to absorb network jitter** (a few frames). How you consume them depends on your integration: * **SDK / Pipecat**: `output_stream()` is already paced to realtime **25 fps with audio and video in sync**, and the client maintains the small jitter buffer for you (`initial_buffer_frames`). You don't manage buffering at all. * **Raw WebSocket**: keep a small jitter buffer of a few frames before starting playback. If frames ever back up (e.g. a brief network stall, then a burst), trim idle frames (`frame_type == 0`) to recover, never drop speech (`1`), start-of-speech (`3`), or fade-out (`2`) frames. See [Buffer Management](/models/introduction/api#buffer-management) and [Audio and Video Synchronization](/models/introduction/api#audio-and-video-synchronization) in the API Reference. {% hint style="info" %} **Don't re-sync the SDK's output yourself.** `output_stream()` is already paced to 25 fps with audio and video aligned, so just present each frame **as it arrives**, never hold audio to wait for a video frame: * **Playing directly (e.g. in a browser):** play audio and video frames as they come. No audio-clock or A/V re-sync logic needed. * **Forwarding to a media transport (e.g. WebRTC):** each `STVAudioFrame` and `STVVideoFrame` carries a `pts` timestamp, pass it through and let the transport use it for sync. Don't build your own sync layer on top. Building your own loop on the **raw WebSocket** instead? That's where the audio-as-clock technique applies. See [Audio and Video Synchronization](/models/introduction/api#audio-and-video-synchronization) in the API Reference. {% endhint %} ## Where to run 🟠 **Your setup**: deployment is yours to get right, with the SDK or the raw API. The realtime API is a **WebSocket built for server-to-server delivery over a stable connection**. It is not meant to run on an end-user device on flaky Wi‑Fi or mobile networks. * **Run the client on a backend server**, not in the browser or on the user's device. This also keeps your API key off the client. * **Deploy in US East**, close to Ojin's inference, for the lowest round-trip latency. * **Deliver the final media to end users over a realtime transport** built for varying network conditions (typically **WebRTC**) rather than exposing the raw WebSocket to them. ## Play back higher-quality audio 🟢 **SDK-handled**: the SDK plays back your original audio automatically. `OjinSTVClient` plays back the **original audio you fed it**. It only resamples a separate 16 kHz copy to send to the model for lip-sync. Your listeners hear exactly the audio you sent. So you can feed **higher-quality TTS** (for example 24 kHz instead of 16 kHz) for better sound while lip-sync still works on the 16 kHz copy. Just make sure your player runs at the **same sample rate you fed**. `STVAudioFrame.sample_rate` reports it per frame. ## Relay frames without re-encoding 🟢 **SDK-handled**: the SDK exposes the raw JPEG (`source_bytes`) for you. If you forward video frames to a browser or media transport that decodes JPEG itself, skip the decode: * The **default decoder** populates `STVVideoFrame.rgb` (decoded RGB) and also keeps the raw JPEG in `STVVideoFrame.source_bytes`. * A **`PassthroughDecoder`** skips the decode entirely: `rgb` is `None` and `output_stream()` won't decode for you. Read the raw JPEG from `source_bytes` and relay it directly, no re-encode. ## Keep latency low 🟢 **SDK-handled** · 🟠 **Your setup** (deployment) Keep your buffers as small as your network allows; large buffers add latency. On the raw API, tune your target buffer size by watching it during playback: low enough to minimise latency, high enough to absorb jitter without starving playback. With the SDK, the relevant knobs live on [`STVConfig`](/models/build-with-python-sdk) (`initial_buffer_frames`, `max_buffered_video_frames`). ## Next steps
TroubleshootingSymptoms, causes, and fixes.Troubleshooting
Python SDK Best PracticesWorking pipelines for browser output and WebRTC backends.Best Practices
# Troubleshooting Common symptoms when building real-time avatars, with their cause and fix. {% hint style="info" %} **🟢 SDK-handled**: the [Python SDK](/models/build-with-python-sdk) and [Pipecat](/models/introduction/integrations) take care of this for you; it surfaces only on the raw WebSocket API (or if you turn a feature off). The simplest fix is usually "use the SDK." **🟠 Your setup**: you own this regardless of integration: deployment, network, credentials, capacity, or raw-API tuning. {% endhint %} ## Jump to a symptom * [Mouth barely moves or only idle animation](#mouth-barely-moves-or-only-idle-animation) * [Lip-sync drifts out of sync with the audio](#lip-sync-drifts-out-of-sync-with-the-audio) * [Choppy or crackling playback](#choppy-or-crackling-playback) * [No video frames at all](#no-video-frames-at-all) * [Growing latency over time](#growing-latency-over-time) * [Frame lag during speech on the raw API](#frame-lag-during-speech-on-the-raw-api) * [No backend servers available](#no-backend-servers-available) * [Connection or authentication fails](#connection-or-authentication-fails) * [Interruptions look abrupt or the avatar keeps speaking after a barge-in](#interruptions-look-abrupt-or-the-avatar-keeps-speaking-after-a-barge-in) ## Mouth barely moves or only idle animation 🟢 **SDK-handled** **Cause:** the model is **starving**. You're sending too little audio, or forwarding tiny TTS fragments one at a time, so the model can't sustain speech at 25 fps and emits idle frames in between. **Fix:** feed audio with a lead: **prime \~1 s, then send the largest chunks you can (\~400 ms)**. See [Optimizing Performance → Feed audio](/guides/optimizing-performance#feed-audio-for-stable-low-latency-lip-sync). `OjinSTVClient` already does this for you, so the simplest fix is to drive the model through the [Python SDK](/models/build-with-python-sdk) or [Pipecat](/models/introduction/integrations). ## Lip-sync drifts out of sync with the audio 🟢 **SDK-handled** **Cause:** on the SDK path, you're adding your own A/V re-sync on top of an already-synced stream; or on the raw API, video is played off the audio clock or frames were reordered during async decode. **Fix:** with the SDK or Pipecat, `output_stream()` is already aligned at 25 fps. **don't re-sync it yourself**. Present each frame as it arrives, or pass each frame's `pts` to your media transport (e.g. WebRTC) and let it handle sync. If you build your own loop on the raw API, drive playback from the audio clock and keep decode order stable. See [Audio and Video Synchronization](/models/introduction/api#audio-and-video-synchronization) in the API Reference. ## Choppy or crackling playback 🟢 **SDK-handled** **Cause:** audio isn't playing at a steady rate, or the event loop is blocked. **Fix:** play audio gaplessly at a steady rate; never stop it just because a video frame is late. Keep your frame handlers light; heavy synchronous work stalls the playback tick. See [Optimizing Performance → Play back at realtime](/guides/optimizing-performance#play-back-at-realtime). ## No video frames at all 🟢 **SDK-handled** **Cause:** you haven't received `SESSION_READY` / `SessionReady` yet, or (SDK) you're reading `frame.rgb` while using a `PassthroughDecoder`. **Fix:** wait for the session-ready signal before sending audio. With the SDK, the default decoder populates `frame.rgb`; a `PassthroughDecoder` leaves `rgb` as `None` and delivers the raw JPEG in `source_bytes` instead. Read that, or keep the default decoder. See [Optimizing Performance → Relay frames](/guides/optimizing-performance#relay-frames-without-re-encoding). ## Growing latency over time 🟢 **SDK-handled** · 🟠 **Your setup** (deployment) **Cause:** your buffer is too large, or frames backed up after a network stall. The model delivers at realtime 25 fps, so the buffer shouldn't grow on its own. Keep only a small jitter buffer. **Fix:** keep the buffer small (a few frames). If frames do back up on a poor connection, trim idle frames (`frame_type == 0`) to recover; never drop speech, start-of-speech, or fade-out frames. The SDK keeps the buffer small for you. Run the client on a backend in **US East** over a stable connection. See [Optimizing Performance → Where to run](/guides/optimizing-performance#where-to-run). ## Frame lag during speech on the raw API 🟠 **Your setup** (raw-API tuning) **Cause:** the model's smoothing filter is adding responsiveness lag on the raw WebSocket protocol. **Fix:** reduce the `speech_filter_amount` parameter; lower is more responsive but less smooth. (The SDK paces output for you, so this only applies when driving the raw API.) ## No backend servers available 🟠 **Your setup** **Cause:** inference capacity is momentarily exhausted. **Fix:** retry shortly with backoff. On the SDK this surfaces as an `ERROR` event with code `NO_BACKEND_SERVER_AVAILABLE`; on the raw API as an `ErrorResponse` (or plain-text message) with `BACKEND_UNAVAILABLE`. ## Connection or authentication fails 🟠 **Your setup** **Fix:** * ✓ Verify your API key and `config_id`, and that the config exists in your [dashboard](https://ojin.ai/dashboard) * ✓ The `Authorization` header must use the **raw API key** (no `Bearer` prefix) * ✓ Ensure your network allows WebSocket connections on port 443 ## Interruptions look abrupt or the avatar keeps speaking after a barge-in 🟢 **SDK-handled** **Cause:** frames already sent or buffered keep playing after you cancel. **Fix:** the SDK's `interrupt()` fades audio and cancels server-side for you. On the raw API, pick an interruption strategy for your latency/smoothness trade-off: clear the buffer for an instant cut, or keep playing video while stopping audio for a smoother look. See [Interruption Handling](/models/introduction/api#interruption-handling) in the API Reference. *** Still stuck? [Contact support](/getting-started/support).