Skip to content

Add realtime voice agent WebSocket support - #50348

Closed
Mike Guo (guotuofeng) wants to merge 29 commits into
mainfrom
myguo/websocket
Closed

Add realtime voice agent WebSocket support#50348
Mike Guo (guotuofeng) wants to merge 29 commits into
mainfrom
myguo/websocket

Conversation

@guotuofeng

@guotuofeng Mike Guo (guotuofeng) commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • Regenerate the Azure AI Agents and Projects clients from the updated Foundry specification.
  • Add realtime voice-agent WebSocket support, typed realtime protocol models, and persisted voice-conversation APIs.
  • Add Python-parity samples across agents, tools, workflows, optimization, telemetry, memory, and voice scenarios.
  • Add deterministic protocol, tool, polling, streaming, voice, and WebSocket tests.
  • Improve long-running-operation status handling and sample resource cleanup.

Handwritten files

The handwritten WebSocket implementation is contained in these production Java files:

  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketAsyncClient.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/VoiceAgentWebSocketSessionClient.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/VoiceAgentWebSocketSessionAsyncClient.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentWebSocketConnectionOptions.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketClientConfiguration.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketHandshakeHandler.java
  • sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketHttpResponse.java
  • sdk/ai/azure-ai-agents/src/main/java/module-info.java

These files implement authenticated WS/WSS connections, preview headers, subprotocol negotiation, proxy configuration, typed event send/receive, text and PCM16 audio operations, response cancellation, function-tool output, bounded buffering, close semantics, and handshake error mapping.

Other handwritten changes include regeneration-safe AST customizations, polling helpers, Maven/configuration updates, README/changelog updates, and test-recording metadata.

Auto-generated files

  • Regenerated Agents and Projects clients, builders, and REST implementations.
  • Added generated realtime client/server event and conversation-item models.
  • Added generated voice-agent, persisted-conversation, audio, telephony, toolbox, optimization, routine, and agent-insight models.
  • Updated generated implementation request types and metadata JSON files.

Examples

  • Agent lifecycle, retrieval, structured output, conversations, and Responses API usage.
  • Workflow-agent, optimization, streaming, and telemetry scenarios.
  • Tool samples covering File Search, Code Interpreter, Web Search, MCP, A2A, Work IQ, Fabric, SharePoint, browser automation, Azure Functions, OpenAPI, image generation, and toolboxes.
  • Memory-store CRUD, item management, conversational updates, and Memory Search samples, organized under the memory package.
  • Voice-agent lifecycle, versions, guided generation, tools, persisted conversation/audio retrieval, realtime text, realtime audio, and client-side function-tool sessions.
  • Azure AI Projects routine samples updated for the regenerated surface.
  • Removed or consolidated redundant agent, conversation, and OpenAPI samples.

Tests

  • Added deterministic HTTP fixtures and protocol-model serialization coverage.
  • Added tool behavior, streaming event, preview-header, and optimization polling tests.
  • Added memory-store sync/async coverage with updated playback recordings.
  • Added voice-agent definition and persisted-conversation tests.
  • Added deterministic WS/WSS loopback tests for authentication, headers, query options, typed events, text/audio/function operations, ordering, buffering, malformed frames, close behavior, and rejected handshakes.
  • Added local TLS certificate fixtures for WSS tests.

Related specification

Validation

  • 307 tests passed with 0 failures and 20 expected skips.
  • Java 8 and current-JDK compilation passed.
  • Maven verify passed, including Javadocs, Checkstyle, codesnippet, coverage, Revapi, and Spotless.
  • CSpell, whitespace, and link validation passed.
  • Repeated TypeSpec generation was byte-stable.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 3, 2026 12:40
@azure-pipelines

azure-pipelines Bot commented Sep 3, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
35 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI reviewed Sep 3, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

JPMS requires transitive was dropped for OpenAI dependencies and the sync WebSocket client can throw an unclear NPE when options is null.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR expands the Azure AI Agents/Projects Java SDK surface to support Foundry “realtime voice agents” over WebSockets, alongside regenerated clients/models and updated samples/tests to maintain parity with newer service capabilities.

Changes:

  • Added preview voice-agent WebSocket client/session APIs and supporting realtime/voice models (typed events, transport options, etc.).
  • Regenerated Agents and Projects clients/models and updated many samples/tests to align with updated request/response shapes (for example CreateAgentVersionInput, routine dispatch authorization).
  • Updated spell-check dictionary and changelog entries to reflect the new features and terminology.
File summaries
File Description
sdk/ai/cspell.yml Adds new allowed terms related to voice/realtime features.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesTimerTriggerSample.java Updates routine creation sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesTimerTriggerAsyncSample.java Updates async routine creation sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesScheduleTriggerSample.java Updates scheduled routine sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesScheduleTriggerAsyncSample.java Updates async scheduled routine sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesSample.java Updates routines CRUD sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesManualDispatchSample.java Updates manual dispatch sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesManualDispatchAsyncSample.java Updates async manual dispatch sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/RoutinesAsyncSample.java Updates async routines sample to include dispatch authorization.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/AgentsSample.java Updates agent creation sample to use CreateAgentVersionInput and revised agent name.
sdk/ai/azure-ai-projects/src/samples/java/com/azure/ai/projects/AgentsAsyncSample.java Updates async agent creation sample to use CreateAgentVersionInput.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/SimulationSeedDataGenerationJobOptions.java Updates model JavaDoc for regenerated data generation options.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/RoutineDispatchIdentity.java Adds new expandable enum for routine dispatch authorization identity.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/AgentInsightStatus.java Adds new expandable enum for agent insight status.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/AgentInsightSeverity.java Adds new expandable enum for agent insight severity.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/AgentInsightRunTrigger.java Adds new expandable enum for agent insight run trigger.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/AgentInsightPromptSurface.java Adds new expandable enum for agent insight prompt surface.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/models/AgentInsightOverviewSource.java Adds new expandable enum for agent insight overview source.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/implementation/models/FoundryFeaturesOptInKeys.java Adds new Foundry-Features opt-in keys for preview capabilities.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/implementation/BetaRoutinesImpl.java Updates generated REST docs to include routine authorization shape.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/implementation/AIProjectClientImpl.java Wires new beta “agent insight monitors” operations into the inner client.
sdk/ai/azure-ai-projects/src/main/java/com/azure/ai/projects/AIProjectClientBuilder.java Exposes beta Agent Insight Monitors clients from the builder.
sdk/ai/azure-ai-agents/tsp-location.yaml Updates TypeSpec source commit and expands additionalDirectories, including voice-agents.
sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/tools/WorkIQSamplesTestBase.java Updates test base to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/tools/FabricIQSamplesTestBase.java Updates test base to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/test/java/com/azure/ai/agents/OpenAIResponseFixtures.java Adds test JSON fixture helpers for deterministic OpenAI-like responses.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/WorkflowSampleUtils.java Adds helper to construct workflow YAML used by samples.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/voice/VoiceAgentSampleUtils.java Adds helper for creating voice-agent definitions used in samples.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/WorkIQSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/WorkIQAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/WebSearchAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/SharePointGroundingSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/SharePointGroundingAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/OpenApiWithConnectionSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/OpenApiWithConnectionAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/OpenApiSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/OpenApiSample.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/OpenApiAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/MemorySearchSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/MemorySearchAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/McpWithConnectionSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/McpWithConnectionAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/McpSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/McpAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/ImageGenerationSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/ImageGenerationAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FunctionCallSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FunctionCallAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FileSearchSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FileSearchAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FabricSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FabricIQSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FabricIQAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/FabricAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/CustomCodeInterpreterSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/CustomCodeInterpreterAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/ComputerUseSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/ComputerUseAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/CodeInterpreterSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/CodeInterpreterAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/BrowserAutomationSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/BrowserAutomationAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/BingGroundingSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/BingGroundingAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/BingCustomSearchSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/BingCustomSearchAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/AzureFunctionSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/AzureFunctionAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/AzureAISearchSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/AzureAISearchAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/AgentToAgentSync.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/AgentToAgentAsync.java Updates async sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/telemetry/AgentConsoleTracingSample.java Adds new tracing sample demonstrating console export.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/telemetry/AgentConsoleTracingCustomAttributesSample.java Adds tracing sample demonstrating custom span attributes.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/telemetry/AgentAzureMonitorTracingSample.java Adds tracing sample demonstrating Azure Monitor export.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/streaming/SimpleStreamingSync.java Updates streaming sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/streaming/SimpleStreamingAsync.java Updates async streaming sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/streaming/FunctionCallStreamingSync.java Updates streaming function-call sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/streaming/FunctionCallStreamingAsync.java Updates async streaming function-call sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/streaming/CodeInterpreterStreamingSync.java Updates streaming code interpreter sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/streaming/CodeInterpreterStreamingAsync.java Updates async streaming code interpreter sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/ReadmeSamples.java Updates README code snippets to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/optimization/AgentOptimizationSampleUtils.java Adds helper utilities for agent optimization samples.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/MultipleAgentsConversation.java Updates helper to create agents via CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/MemorySearchAgent.java Updates sample agent creation to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/hostedagents/SessionLogStreamAsyncSample.java Ensures OpenAI request creation is deferred to subscription time.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/hostedagents/AgentEndpointAsyncSample.java Ensures OpenAI request creation is deferred to subscription time.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/CreateResponseWithStructuredInput.java Updates structured-input sample to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/conversations/GetConversation.java Improves conversation sample to create/retrieve/delete a conversation with clearer setup docs.
sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/agents/CreateAgent.java Updates agent creation sample to use CreateAgentVersionInput.
sdk/ai/azure-ai-agents/src/main/java/module-info.java Updates JPMS module dependencies for new realtime/WebSocket implementation.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceResponseBaseStatus.java Adds generated enum for voice response lifecycle status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceResponseBaseOutputModality.java Adds generated enum for voice response output modality.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceResponseBaseObject1.java Adds generated enum for response object discriminator.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceResponseBaseObject.java Adds generated enum variant for response object discriminator.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceModelType.java Adds generated expandable enum for voice model hosting type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAudioRole.java Adds generated expandable enum for voice audio participant roles.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAudioContainerFormat.java Adds generated expandable enum for audio container format.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAudioCodec.java Adds generated expandable enum for audio codec.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentTransport.java Adds generated expandable enum for voice transport (WebSocket/WebRTC).
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentSystemToolName.java Adds generated expandable enum for system tool names.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentSessionIncludeOption.java Adds generated enum for include options.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentResponseCreateParamsConversation.java Adds generated expandable enum for response create conversation option.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentInterimResponseTrigger.java Adds generated expandable enum for interim response triggers.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentFunctionToolType.java Adds generated enum for function tool type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentEchoCancellationReferenceSource.java Adds generated enum for echo cancellation reference source.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentAvatarType.java Adds generated expandable enum for avatar type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentAudioTimestampType.java Adds generated enum for audio timestamp type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/VoiceAgentAnimationOutputType.java Adds generated enum for animation output types.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ToolType.java Adds new tool type discriminator value(s).
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ToolChoiceOptions.java Adds enum to represent tool-choice mode.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ToolboxToolType.java Extends toolbox tool types with new enum values.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ToolboxTool.java Extends toolbox tool discriminator parsing for new tool types.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/Tool.java Extends tool discriminator parsing for new tool types.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/TelephonyProvider.java Adds generated expandable enum for telephony providers.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/TelephonyCallTraceMode.java Adds generated expandable enum for telephony trace modes.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/TelephonyCallDurationBasis.java Adds generated expandable enum for call duration basis.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/TelephonyBindingStatus.java Adds generated expandable enum for telephony binding status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeTurnDetectionType.java Adds generated expandable enum for realtime turn detection type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeServerEventResponseContentPartDonePartType.java Adds generated enum for realtime response content part “done” type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeServerEventResponseContentPartAddedPartType.java Adds generated enum for realtime response content part “added” type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeServerEventRateLimitsUpdatedRateLimitsName.java Adds generated enum for realtime rate limit names.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeServerEventConversationCreatedConversationObject.java Adds generated enum for realtime conversation object discriminator.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeResponseStatusDetailsType.java Adds generated enum for status details type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeResponseStatusDetailsReason.java Adds generated enum for status details reason.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemObject.java Adds generated expandable enum for conversation item object discriminator.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageUserStatus.java Adds generated enum for user message status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageUserContentType.java Adds generated enum for user message content type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageUserContentDetail.java Adds generated enum for message content detail.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageSystemStatus.java Adds generated enum for system message status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageSystemContentType.java Adds generated enum for system message content type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageAssistantStatus.java Adds generated enum for assistant message status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemMessageAssistantContentType.java Adds generated enum for assistant message content type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemFunctionCallStatus.java Adds generated enum for function call item status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeConversationItemFunctionCallOutputStatus.java Adds generated enum for function call output item status.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeClientEventSessionUpdateSessionTruncation.java Adds generated enum for session truncation option.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeClientEventSessionUpdateSessionOutputModality.java Adds generated enum for session output modality.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RealtimeAudioFormatsAudioPcmRate.java Adds generated enum for PCM sample rate representation.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RaiInvocationMode.java Adds generated expandable enum for RAI invocation mode.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/RaiInvocationContentType.java Adds generated expandable enum for RAI invocation content type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/NoiseReductionType.java Adds generated enum for noise reduction mode.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ModelRouterMode.java Adds generated expandable enum for model router modes.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ImageDetail.java Adds generated enum for image detail selection.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/FileInputDetail.java Adds generated enum for file input detail selection.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/DigitalWorkerType.java Adds generated expandable enum for digital worker type.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/AzureCreateResponseDetails.java Adds response details field for model selection metadata.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/AgentKind.java Adds new agent kind discriminator value for voice agents.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/AgentEndpointProtocol.java Adds endpoint protocol discriminator value for voice.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/AgentEndpointConfig.java Adds new server-populated publish approval status field.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/AgentDefinition.java Extends agent definition discriminator parsing to include voice agents.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/models/ActivityProtocolConfiguration.java Adds access boundary parsing and getter.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketHandshakeHandler.java Adds Netty handler to surface WebSocket upgrade rejection details.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/models/UpdateAgentRequest.java Updates docs to include voice agent definitions.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/models/FoundryFeaturesOptInKeys.java Adds new Foundry-Features opt-in keys for agents package.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/models/AgentDefinitionOptInKeys.java Adds preview opt-in key for digital worker.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/JsonMergePatchHelper.java Adds JsonMergePatch accessor support for telephony binding update requests.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/AgentsClientImpl.java Wires new beta “agent endpoint conversations” operations into the inner client.
sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java Adds sync WebSocket client wrapper over the async implementation.
sdk/ai/azure-ai-agents/CHANGELOG.md Documents the new preview WebSocket features and new voice-related samples.
Review details
  • Files reviewed: 129/579 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java
Comment thread sdk/ai/azure-ai-agents/src/main/java/module-info.java Outdated
Comment thread ...jects/src/main/java/com/azure/ai/projects/models/SimulationSeedDataGenerationJobOptions.java
@guotuofeng
Mike Guo (guotuofeng) force-pushed the myguo/websocket branch 2 times, most recently from e8df16b to fc65f56 Compare September 9, 2026 09:20
alzimmermsft
Alan Zimmer (alzimmermsft) reviewed Sep 9, 2026
View reviewed changes
Comment thread ...azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/AgentsServicePollUtils.java Outdated
Comment thread sdk/ai/azure-ai-agents/customizations/src/main/java/AgentsCustomizations.java Outdated
Comment thread sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java
Comment thread sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketClient.java Outdated
Comment thread sdk/ai/azure-ai-agents/src/main/java/module-info.java
Comment thread ...azure-ai-agents/src/main/java/com/azure/ai/agents/VoiceAgentWebSocketSessionAsyncClient.java Outdated
Comment thread ...azure-ai-agents/src/main/java/com/azure/ai/agents/VoiceAgentWebSocketSessionAsyncClient.java
Comment thread ...azure-ai-agents/src/main/java/com/azure/ai/agents/VoiceAgentWebSocketSessionAsyncClient.java
Comment thread ...azure-ai-agents/src/main/java/com/azure/ai/agents/VoiceAgentWebSocketSessionAsyncClient.java
Mike Guo (guotuofeng) and others added 14 commits September 10, 2026 15:27
@guotuofeng @claude
Add realtime voice agent WebSocket support
6d1ed80 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng @claude
Fix voice agent preview headers and sample cleanup (#50359)
7e5f373 
* Update voice agent generation pin

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix voice agent sample cleanup and preview headers

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
@guotuofeng @claude
Fix WebSocket review and CI issues
9575a11 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng
Fix com.azure.ai.agents.agents samples errors
fafad8a
@guotuofeng @claude
Fix Agents CI and polling status handling
596b42c 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng @claude
Fix workflow and conversation samples
ca201ac 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng @claude
Clean up agents samples
dc83626 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng @claude
Document realtime voice WebSocket examples
994f38f 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng @claude
Organize and improve memory samples
b80341d 
Co-Authored-By: Claude <noreply@anthropic.com>
@guotuofeng
fix part of tools sample errors
4d03f85
@guotuofeng
Regenerate AI Agents and Projects SDKs and preserve customizations
8aad008 
Regenerate both Java SDKs from azure-rest-api-specs commit b25b27f1034082e994bdb68e33eb52bdfae2f3e9.

Preserve final voice-tool discriminator fields and Skills response helper naming through generation customizations. Update voice-tool samples and tests, AI spelling dictionary, and release notes for telephony, session affinity, and intentional API breaks.

Validated regeneration, builds, Checkstyle, playback tests, README, spelling, and changelogs. Revapi continues to report the six intentional compatibility breaks against 2.4.0.
@guotuofeng
Fix Revapi checks for preview API changes
d0f2733
@guotuofeng
Remove obsolete APIView properties files
89a87d2 
Remove legacy APIView metadata resources superseded by generated metadata JSON files.
@guotuofeng
Fix memory store tests and voice deserialization
8131b06 
Use normalized memory store names in tests and preserve voice AgentDefinition polymorphic deserialization through code generation customization.
Mike Guo (guotuofeng) and others added 5 commits September 10, 2026 15:27
@guotuofeng
Regenerate AI Agents and Projects SDKs
199a108
@guotuofeng @claude
Fix AI Agents README sample links
bcd3a7d 
Co-Authored-By: Claude Code <noreply@anthropic.com>
@guotuofeng
Address voice WebSocket review feedback
415db01 
Implement an independent OkHttp-based synchronous WebSocket transport, refine async timeout and error handling, simplify polling status mapping, and apply the expected generated model update.
@guotuofeng
Fix Projects Revapi preview exclusions
26de4f2 
Document intentional Agent Insights preview changes for highlighted trace duration and token count APIs.
@guotuofeng
Move PR 50348 changes to unreleased
c9ffd87
@guotuofeng

Mike Guo (guotuofeng) commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Superseded by GitHub Stack #50411, reconstructed into reviewable, buildable layers:

  1. Regenerate Azure AI Projects client #50407 — Azure AI Projects generated foundation
  2. Add voice agent WebSocket transport, tests, samples, and AI Projects parity #50408 — Azure AI Agents generated surface and handwritten WebSocket production implementation
  3. Add realtime voice agent WebSocket tests #50409 — WebSocket and protocol tests
  4. Add realtime voice agent samples #50410 — Voice samples and documentation

The stack tip reproduces all 469 changed paths from this PR, and the affected module trees are content-identical. The original head is preserved as branch myguo/websocket-backup-20260911.

@guotuofeng

Mike Guo (guotuofeng) commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Stack refinement: replacement stack is now #50413. The previous combined Agents PR was split further into #50412 (generated Agents foundation) and #50408 (14-file handwritten WebSocket transport and required builder wiring). The remaining layers are unchanged: #50407, #50409, and #50410.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alzimmermsft Alan Zimmer (alzimmermsft) alzimmermsft left review comments
Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@dargilco Darren Cohen (dargilco) Awaiting requested review from dargilco dargilco is a code owner
@glecaros Gerardo Lecaros (glecaros) Awaiting requested review from glecaros glecaros is a code owner
@jpalvarezl Jose Alvarez (jpalvarezl) Awaiting requested review from jpalvarezl jpalvarezl is a code owner
@kaylieee Kaylie (kaylieee) Awaiting requested review from kaylieee kaylieee is a code owner

Assignees

No one assigned

Labels

AI Agents AI Projects AI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@guotuofeng @alzimmermsft