feat: add DuplexModel for full-duplex speech models - #6677
Draft
longcw wants to merge 2 commits into
Draft
Conversation
A full-duplex speech model listens and speaks at the same time. Its audio streams continuously whether or not it is speaking, it decides for itself when to talk, and it can be neither interrupted nor truncated. None of that fits RealtimeModel, whose surface is built around the client driving each response. DuplexModel / DuplexSession give plugin authors a surface carrying no method the model cannot perform. DuplexRealtimeAdapter runs one inside an AgentSession by segmenting its continuous output into turns and presenting them as an ordinary RealtimeSession, so the voice pipeline needs no duplex-specific path. An energy gate decides what reaches the room, since a model that emits unconditionally would otherwise publish silence forever. Its floor is the quietest frame of the recent past, which speech cannot drag upward, and every duration is measured in audio rather than wall clock, so one set of defaults ports across providers and frame sizes. A turn closes when the model reports it ended, or when its transcript catches up with the audio that carried sound; a liveness bound covers a model that can do neither. Output the model never transcribes still plays, it simply produces no chat item, which is what makes backchannels audible. Two capabilities are added to RealtimeCapabilities, both defaulted so existing realtime models are unaffected: - server_barge_in, for a model that stops its own output when the user speaks. The framework then leaves playback alone on its speech-started event and only interrupts from an explicitly configured VAD or interruption detector. - manual_response_creation, for whether generate_reply may ask the model to speak. When false the reply is skipped rather than dispatching a call that can only fail. Also releases the transcription synchronizer's trailing word once a timed span closes, rather than holding it for a delimiter that a server-driven model only sends with its next turn.
A duplex model previously had to be wrapped in its adapter by hand before it could be passed as an llm. Accept it directly and wrap it where it is stored, so nothing downstream ever sees one. Storage is the right place rather than AgentActivity: _update_models rejects swapping to or from a RealtimeModel while running, because such a model owns a live session. A DuplexModel wrapped any later would not match that check and would slip through it. Annotating _llm makes the invariant checkable rather than conventional.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A full-duplex speech model listens and speaks at the same time. Its audio streams continuously whether or not it is speaking, it decides for itself when to talk, and it can be neither interrupted nor truncated.
That does not fit
RealtimeModel, whose surface assumes a response the client can act on. Server-side turn detection decides when a response starts, but the response is still an object the client can trigger, cancel, truncate and commit against. A duplex model has none of that, and its audio neither begins nor ends with one.DuplexModel/DuplexSessiongive plugin authors a surface carrying only what such a model can actually do.DuplexRealtimeAdapterruns one inside anAgentSession.Why an adapter:
GenerationCreatedEvents, soSpeechHandle, speech scheduling,drain()andagent_stateneed no duplex-specific path. The only changes outside the adapter are two capability flags:server_barge_in, for a model that stops its own output when the user speaks, andmanual_response_creation, for whethergenerate_replymay ask a model to speak. Both default to current behaviour.AgentSession(llm=...)andAgent(llm=...)accept aDuplexModeldirectly and wrap it internally.TODO: client delegation — a duplex model handing its reasoning and tool work back to the agent, answered through the normal text pipeline.