Troubleshooting

Common symptoms when building real-time avatars, with their cause and fix.

🟢 SDK-handled: the Python SDK and Pipecat take care of this for you; it surfaces only on the raw WebSocket API (or if you turn a feature off). The simplest fix is usually "use the SDK."

🟠 Your setup: you own this regardless of integration: deployment, network, credentials, capacity, or raw-API tuning.

Jump to a symptom

Mouth barely moves, idle animation only, or the avatar jumps between utterances

🟢 SDK-handled

Cause: too little audio has arrived, or delivery has stalled. The minimum initial audio is 320 ms for Presence or 200 ms for Portrait. A pause in your TTS or network can still interrupt playback.

Fix: start each turn with the recommended 500 ms for Presence or 400 ms for Portrait, then stream in real time with the largest chunks you can (~400 ms). See Optimizing Performance → Feed audio. With OjinSTVClient, feed audio as your TTS produces it and let the SDK handle batching.

Lip-sync drifts out of sync with the audio

🟢 SDK-handled

Cause: on the SDK path, you're adding your own A/V re-sync on top of an already-synced stream. On the raw API, you're playing your own copy of the TTS audio on its own clock instead of the audio that arrives with each frame, or frames were reordered during async decode.

Fix: with the SDK or Pipecat, output_stream() is already aligned at 25 fps. don't re-sync it yourself. Present each frame as it arrives, or pass each frame's pts to your media transport (e.g. WebRTC) and let it handle sync. If you build your own loop on the raw API, play the audio payload that arrived with each frame and render that frame's image at the same time. The server aligns them for you, so no audio master clock is needed. Keep decode order stable. If playback includes gaps, check that your TTS and network are delivering audio continuously. See Audio and Video Synchronization in the API Reference.

Choppy or crackling playback

🟢 SDK-handled

Cause: audio isn't playing at a steady rate, or the event loop is blocked.

Fix: the SDK emits exactly one audio frame every 40 ms tick (real audio or silence), so its output never starves on its own. Keep your frame handlers light: heavy synchronous work stalls the tick that paces them. On the raw API, play audio gaplessly at a steady rate and never stop it just because a video frame is late. See Optimizing Performance → Play back at realtime.

No video frames at all

🟢 SDK-handled

Cause: you haven't received SESSION_READY / SessionReady yet, or (SDK) you're using a PassthroughDecoder, which suppresses video frames entirely.

Fix: wait for the session-ready signal before sending audio. With the SDK, keep the default decoder: it populates frame.rgb and also carries the raw JPEG in frame.source_bytes, so you can relay without re-encoding. A PassthroughDecoder makes the client emit no video frames at all, so do not use it. See Optimizing Performance → Relay frames.

Growing latency over time

🟢 SDK-handled · 🟠 Your setup (deployment)

Cause: your buffer is too large, or frames backed up after a network stall. The model delivers at realtime 25 fps, so the buffer shouldn't grow on its own. Keep only a small jitter buffer.

Fix: the SDK keeps the buffer small and trims backed-up idle frames for you (idle_buffer_target_frames, default 6), so don't filter output_stream() by frame_type yourself. It fights the client's own catch-up. On the raw API, keep the buffer to a few frames and, if frames back up on a poor connection, trim idle frames (frame_type == 0); never drop speech, start-of-speech, or fade-out frames. Either way, run the client on a backend in US East over a stable connection. See Optimizing Performance → Where to run.

Frame lag during speech on the raw API

🟠 Your setup (raw-API tuning)

Cause: your playback buffer is larger than it needs to be, so every frame waits behind the ones ahead of it.

Fix: shrink the jitter buffer to the smallest size your network tolerates. Output smoothing itself is not caller-tunable: there is no parameter on the wire protocol that changes it. If the lag persists at a small buffer, contact support with a session id rather than trying to tune it from the client.

No backend servers available

🟠 Your setup

Cause: inference capacity is momentarily exhausted.

Fix: retry shortly with backoff. On the SDK this surfaces as an ERROR event with code NO_BACKEND_SERVER_AVAILABLE. On the raw API it arrives either as an ErrorResponse with BACKEND_UNAVAILABLE, or as a bare plain-text frame with no code at all ("No backend servers available. Please try again later."). Handle non-JSON text frames too.

Connection or authentication fails

🟠 Your setup

Fix:

✓ Verify your API key and config_id, and that the config exists in your dashboard
✓ The Authorization header must use the raw API key (no Bearer prefix)
✓ Ensure your network allows WebSocket connections on port 443

Interruptions look abrupt or the avatar keeps speaking after a barge-in

🟢 SDK-handled

Cause: old TTS audio is still being sent, or frames already buffered keep playing after you cancel.

Fix: stop the old TTS producer and call the SDK's interrupt(). Call start_turn() before feeding the next reply. On the raw API, pick an interruption strategy for your latency/smoothness trade-off: clear the buffer for an instant cut, or keep playing video while stopping audio for a smoother look. See Interruption Handling in the API Reference.

Direct WebRTC: the avatar's video is blank right after it joins

🟠 Your setup (viewer UI)

Cause: the avatar publishes its video track when it joins the room, before the model has rendered anything. Nothing is sent on that track until the first frame exists, so a viewer who subscribes in that window gets a track with no video. The wait is the model's startup: under a second on a warm persona, a few seconds on a cold one.

Fix:

✓ Reveal the avatar on the first decoded frame, or on FIRST_FRAME, not on track subscribe
✓ Show your own placeholder until then, such as a still image of the persona
✓ If the wait is longer than you expect, it is cold start. See the session times out before it's ready

A blank element here is the avatar not having started, not a dropped connection. See Direct WebRTC → Events.

Direct WebRTC: the avatar can't join the room

🟠 Your setup (credentials)

Cause: the room rejected the avatar. You get a fatal ERROR whose code names the cause: WEBRTC_AUTH_FAILED (the token doesn't match the room), WEBRTC_NETWORK_FAILED (Ojin couldn't reach the room), or WEBRTC_INVALID_SETTINGS (the room URL or provider was unusable).

Fix:

✓ LiveKit: room_url is your server URL (wss://…), and the token allows room join and publish for the right room, with the identity ojin-avatar
✓ Daily: room_url is the room URL and token is a meeting token for that room
✓ The token hasn't expired
✓ WEBRTC_NETWORK_FAILED: check that your LiveKit or Daily service is reachable and retry

See Direct WebRTC → Prerequisites.

Direct WebRTC: the session times out before it's ready

🟠 Your setup

Cause: the session wasn't ready within webrtc_join_timeout_s (default 10 s). The timeout covers both the room join and the model's cold start, so a cold start alone can exceed it. You get a fatal WEBRTC_JOIN_TIMEOUT.

Fix: raise webrtc_join_timeout_s to about 90 s in production. A cold start alone can take more than 30 s. Retrying usually works too, because the server keeps starting after the client gives up.

Direct WebRTC: the avatar drops out mid-session

🟠 Your setup

Cause: the avatar lost the room after joining it, and you get a fatal WEBRTC_ROOM_LOST. The client never rejoins on its own.

Fix: start a new session. If it recurs, check the room's lifetime and token expiry against your session length.

Direct WebRTC: WEBRTC_NOT_SUPPORTED

🟠 Your setup

Cause: the server returned no room result, so this session can't be published into a room. The client doesn't fall back to the WebSocket; it emits a fatal ERROR and closes the session.

Fix: leave out webrtc= to use the WebSocket mode, or contact support with your config_id.

Direct WebRTC: my bot hears the avatar

🟠 Your setup

Cause: your own voice agent is in the same room and subscribes to the avatar's microphone, so it transcribes the avatar's voice as user speech and replies to itself.

Fix: unsubscribe your bot from the avatar's audio. Identify the avatar with is_avatar_participant() (Daily) or is_avatar_identity() (LiveKit) from ojin. See Direct WebRTC (LiveKit & Daily).

Still stuck? Contact support.

PreviousOptimizing Performance

Last updated 4 days ago

Was this helpful?