fix(openai realtime): treat create_response=false as client-side turn taking - #2205
fix(openai realtime): treat create_response=false as client-side turn taking#2205rosetta-livekit-bot[bot] wants to merge 3 commits into
Conversation
🦋 Changeset detectedLatest commit: 67cc695 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
…arted The rtTurnDetectionEnabled gate sat on a catch-all, so with client-side turn taking every exception from interrupt() was discarded with no log or trace -- in exactly the mode the gate exists to enable. Only the uninterruptible-speech error is expected there; anything else is now logged unconditionally. Python narrows via `except RuntimeError`. JS cannot rethrow the equivalent: realtime plugins emit input_speech_started from call sites that do not all guard against a throwing listener, so an escaping error would abort unrelated session handling. Co-authored-by: Cursor <cursoragent@cursor.com>
…n detection Drop the unreachable `turnDetection !== undefined` clause from the empty-commit gate so it is the single condition Python uses. Both the model constructor and the session constructor always assign a concrete value, so the clause could never be false. Add the case neither language covered: a model built with server_vad whose session sets turnDetectionDisabled must still report input_audio_buffer_commit_empty. That is the path that makes the suppression gate correct rather than over-broad. Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
1 participant
Warning
No CI has run on this PR.
gh api repos/livekit/agents-js/commits/67cc6957/check-runsreturnstotal_count: 0. Nothing has built, type-checked, linted, tested, or license-checked this code. Devin Review andlicense/claare external apps, not GitHub Actions, so their green marks do not indicate any Actions workflow ran.The cause is that this PR targets the feature branch of #2158 rather than
main, and the Test/Build workflows only trigger onpull_requestintonext,main, or0.x. Retargeting is deliberately out of scope while #2158 is open; CI is restored once #2158 lands and this PR is retargeted. See the CI blocker comment on this PR for detail — note that comment cites evidence at heade16f1dad, two commits stale, though its substance still holds at the current head.Known follow-on:
plugins/openai/src/tts.test.tscurrently fails to collect whenOPENAI_API_KEYis set (pre-existing on the base, see Validation), so the first real CI run on this stack is expected to be red for that unrelated reason.Ports livekit/agents#6642 to the JS SDK. OpenAI server VAD can still commit and transcribe audio while
create_response: falseleaves response generation and interruption policy to the client.This also warns when
interrupt_responseremains enabled, suppresses the expected empty-buffer commit race under server VAD, and keeps uninterruptible speech intact for client-side turn taking.Source diff coverage
livekit-agents/livekit/agents/voice/agent_activity.py: ported with one documented divergence toagents/src/voice/agent_activity.ts; gate the unexpected uninterruptible-speech error on resolved server-side turn detection. Divergence: Python narrows toexcept RuntimeError:, so any other exception propagates. JS catches everything and discriminates on the error message string (agents/src/voice/agent_activity.ts:1345-1366), logging a non-matching error at:1357where Python would let it escape. Two target constraints force this:SpeechHandle.interrupt()signals an uninterruptible handle with a plainError, so the message is the only discriminator available without a new error subclass; and rethrowing is unsafe because the Google and Phonic realtime plugins emitinput_speech_startedfrom call sites that do not all guard against a throwing listener, so an escaping error would abort unrelated session handling. A dedicated error subclass is the tracked follow-up.livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py: ported toplugins/openai/src/realtime/realtime_model.ts; resolvecreate_response: falseas client-side turn taking, warn wheninterrupt_responseremains enabled, and suppress the server-VAD empty-buffer commit race.livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py: adapted inplugins/openai/src/realtime/realtime_model.ts; JS already accepts and retains the nativeTurnDetectionTypeobject, includinginterrupt_response, so no conversion helper exists or is needed.tests/test_realtime/test_openai_realtime_model.py: ported toplugins/openai/src/realtime/realtime_model.test.ts; ported the capability, warning, and empty-commit error tests. The legacy Python SDK conversion test is not applicable because JS has no deprecated Pythonsession.TurnDetectionconversion layer and already preserves the field directly.tests/test_realtime_input_speech_interrupt.py: adapted toagents/src/voice/realtime_input_speech_interrupt.test.ts; ported all four behavioral cases using the JS activity/session harness, accounting for JS existing warning-based configuration validation rather than Python constructor rejection.Validation
pnpm exec vitest run agents(113 files, 1528 passed, 5 skipped)env -u OPENAI_API_KEY pnpm test plugins/openai(77 passed, 7 skipped). Theenv -uprefix is load-bearing and not cosmetic: withOPENAI_API_KEYpresent,plugins/openai/src/tts.test.tsfails to collect withA VAD instance is required for gpt-realtime-whisper. That failure is pre-existing and unrelated to this PR — it reproduces on the basedfe73afe, and this PR does not touch that file — but the suite is not green without unsetting the key, so the plain command will fail for a reviewer who has one set.pnpm --filter @livekit/agents buildpnpm --filter @livekit/agents-plugin-openai buildpnpm --filter @livekit/agents lintpnpm --filter @livekit/agents-plugin-openai lintpnpm buildcue-clitext-mode runtime validation: client-triggered greeting and user-turn reply emitted as framework conversation events with no error eventPorted from livekit/agents#6642
Original PR description
Server VAD with
create_response=Falsestill commits and transcribes the user audio, but the reply is the client's call, so the session no longer reports server-side turn detection andallow_interruptions=Falseis honored.Fixes livekit/agents#6635
Cue voice E2E
Verified exact head
67cc695746bd87ea651afbb20fe32b00b4e621deagainstdfe73afed63ce4fa71850bdd12642d50ee1b38e3in voice mode with the real OpenAI Realtime API and Silero VAD. No framework source was modified for the run.Note on the baseline. Because this PR targets a feature branch rather than
main(see the CI blocker comment on this PR), the comparand is the merge base on that branch,dfe73afe— notorigin/main.The scenario is client-side turn taking. Under
turnDetection: { create_response: false }the user speaks and the question is simply whether the assistant ever answers. One script and one predicate across all four runs:The 75s bound is roughly 21x the head arm's observed 3.1s user-to-reply latency, so a baseline timeout reflects a reply that never comes rather than impatience.
67cc6957(cue-2205-head): sessionssid_5d9272a7b1edandsid_68fc7159dab1, both resolved in 7.9s and 8.2s. Decisive event insid_5d9272a7b1edat offset 31 (monotonic_ms12316):conversation_item_addedroleASSISTANT,"The capital city of France is Paris."The second head session committed"Hello! The capital city of France is Paris."— the greeting differs run to run; what matters is that an ASSISTANT item exists in both.dfe73afe(cue-2205-base): sessionssid_0a281ef6669aandsid_8ad16c0eb94b, both timed out at 70.5s and 70.3s. NoASSISTANTconversation_item_addedexists at any offset in either session; the user is left waiting untiluser_state_changed -> US_AWAY, at offset 23 insid_0a281ef6669a(monotonic_ms24240; the second baseline session reaches the same state at 24111).Shared precondition. The baseline arm proves its point by an absence, so it matters that it got as far as head did. Both arms committed a USER
conversation_item_addedcarrying the byte-identical transcript"Hello there, please tell me what the capital city of France is."— at 9211/9316 ms on head and 9242/9114 ms on baseline. Baseline heard the user, transcribed the turn, committed it, and then stood down without replying, which is precisely the reported bug.Arm attribution. Each arm ran under a distinct
agentName, and each session is self-identifying: abuild_identityevent reads the witness off the loaded code under acreate_response: falseconfig. Head reportscapabilitiesTurnDetection: false, rtTurnDetectionEnabled: false; baseline reportstrue, true. That value is exactly what this PR changes, so a session cannot be attributed to the wrong arm by mislabeling. Each event also carries itsgitSha, worktree, and resolvedagents/dist/index.cjsand openai plugin paths.Limitations.
input_audio_buffer_commit_emptysuppression and thecreate_response/interrupt_responseconfig warning never fired in these runs, and the narrowed error path was not triggered.agents/src/voice/agent_activity.tsis covered by the new unit tests inrealtime_input_speech_interrupt.test.ts, not by this run. This evidence should not be read as whole-PR E2E coverage.