Fix voice agent WebSocket configuration

sdk/ai/azure-ai-agents/CHANGELOG.md

@@ -10,7 +10,7 @@
 - Added raw JSON WebSocket sends, complete unknown-event payloads, UTF-8 binary JSON reception, configurable receive
   limits and overflow policies, and opt-in recovery from malformed events.
 - Added custom WebSocket close codes and reasons, and per-event synchronous receive timeouts.
-- Added realtime handshake options for session IDs, structured inputs, API versions, credential scopes, preview features, extra headers and query parameters, and same-host secure connection URL overrides.
+- Added realtime session options for session IDs, structured inputs, persistence, and agent version selection.
 
 - Added preview `BetaVoiceAgentsTelephonyClient` and `BetaVoiceAgentsTelephonyAsyncClient` for outbound call jobs and campaign management, including recipient import, validation, publishing, pausing, resuming, and cancellation.
 - Added preview `BetaVoiceAgentsConversationsClient` and `BetaVoiceAgentsConversationsAsyncClient` for managing
@@ -32,6 +32,8 @@
 ### Bugs Fixed
 
 - Reject insecure voice-agent WebSocket URLs before token acquisition to prevent sending credentials over plaintext.
+- Reject HTTP client, pipeline, policy, and retry builder settings that native WebSocket transports cannot honor.
+- Enforce the single-iterator contract of synchronous voice-agent event streams.
 - Made synchronous voice-agent receive-buffer overflow signaling atomic across concurrent callbacks.
 - Fixed polling for telephony operations that return the `cancelled` status spelling.
 

sdk/ai/azure-ai-agents/README.md

@@ -120,22 +120,21 @@ ConversationService conversationService = openAIClient.conversations();
 ### Realtime connection options
 
 Use `VoiceAgentWebSocketConnectionOptions` with the synchronous or asynchronous beta voice-agent client's
-`openWebSocketSession` method to set session IDs, agent version overrides, structured inputs, API versions, credential
-scopes, preview features, and extra handshake headers or query parameters.
+`openWebSocketSession` method to configure session behavior such as session IDs, agent version selection, structured
+inputs, persistence, buffering, and timeouts.
 
 ```java
 VoiceAgentWebSocketConnectionOptions options = new VoiceAgentWebSocketConnectionOptions()
     .setAgentSessionId("session-id")
     .setAgentVersionOverride("2")
     .setStructuredInputs("{\"language\":\"en\"}")
-    .setExtraHeaders(Collections.singletonMap("User-Agent", "my-application/1.0"));
+    .setStoreEnabled(true);
 ```
 
-Extra query parameters and non-protected headers override defaults. Authentication and WebSocket protocol
-headers remain transport-controlled. An explicitly empty `Foundry-Features` value is preserved.
-`setConnectionUrl` accepts a full `wss://` URI on the project endpoint's host and port, with no user information
-or fragment. Existing query parameters are preserved unless overridden. URL validation happens before token
-acquisition; cross-host overrides are rejected to prevent credentials from being sent to another host.
+The SDK owns the WebSocket route, API version, authentication scope, transport, and preview feature headers. Endpoint,
+credential, service version, configuration-based proxy settings, and `ClientOptions` are reused from
+`AgentsClientBuilder`. Custom HTTP clients, pipelines, policies, and retry settings are rejected when building a
+WebSocket client because the native WebSocket transports cannot apply them.
 
 ### Agent version drafts
 
@@ -995,8 +994,8 @@ BetaVoiceAgentWebSocketAsyncClient realtimeAsyncClient
 
 #### Send a synchronous text turn
 
-Connections require an `https://` or `wss://` project endpoint. Insecure endpoints and untrusted connection URL overrides
-are rejected before acquiring a token. This also applies to localhost; use certificate-verified TLS for local servers.
+Connections require an `https://` or `wss://` project endpoint. Insecure endpoints are rejected before acquiring a
+token. This also applies to localhost; use certificate-verified TLS for local servers.
 
 Unknown server event types are returned as `RawRealtimeServerEvent`; `getRawEvent()` preserves the complete JSON object.
 Use `sendEvent(BinaryData)` to send raw JSON objects, including event types or fields not modeled by this SDK. Both

sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/AgentsClientBuilder.java

@@ -678,8 +678,14 @@ public BetaMemoryStoresClient buildBetaMemoryStoresClient() {
 
         /**
          * Builds an asynchronous client for realtime voice-agent WebSocket sessions.
+         * <p>
+         * Endpoint, credential, service version, configuration-based proxy settings, and client options are applied to
+         * WebSocket handshakes. Custom HTTP clients, pipelines, policies, and retry configuration are not compatible
+         * with the native WebSocket transport and cause this method to fail rather than being silently ignored.
+         * HTTP log options contribute to the user agent but do not configure WebSocket frame logging.
          *
          * @return an asynchronous voice-agent WebSocket client.
+         * @throws IllegalStateException if unsupported HTTP pipeline configuration is present.
          */
         @Beta
         public BetaVoiceAgentWebSocketAsyncClient buildBetaVoiceAgentWebSocketAsyncClient() {
@@ -688,8 +694,14 @@ public BetaVoiceAgentWebSocketAsyncClient buildBetaVoiceAgentWebSocketAsyncClien
 
         /**
          * Builds a synchronous client for realtime voice-agent WebSocket sessions.
+         * <p>
+         * Endpoint, credential, service version, configuration-based proxy settings, and client options are applied to
+         * WebSocket handshakes. Custom HTTP clients, pipelines, policies, and retry configuration are not compatible
+         * with the native WebSocket transport and cause this method to fail rather than being silently ignored.
+         * HTTP log options contribute to the user agent but do not configure WebSocket frame logging.
          *
          * @return a synchronous voice-agent WebSocket client.
+         * @throws IllegalStateException if unsupported HTTP pipeline configuration is present.
          */
         @Beta
         public BetaVoiceAgentWebSocketClient buildBetaVoiceAgentWebSocketClient() {
@@ -821,10 +833,41 @@ private BetaVoiceAgentsTelephonyClient buildBetaVoiceAgentsTelephonyClient() {
         return new BetaVoiceAgentsTelephonyClient(buildInnerClient().getBetaVoiceAgentsTelephonies());
     }
 
+    /**
+     * Creates the parallel configuration path required by the native WebSocket transports. Azure Core's
+     * {@link HttpClient} and {@link HttpPipeline} abstractions don't expose WebSocket session operations, so these
+     * clients can't reuse the generated HTTP pipeline directly. Compatible builder settings are adapted for the
+     * WebSocket handshake and must remain aligned with {@code createHttpPipeline()} when the TypeSpec emitter changes.
+     * HTTP transport, pipeline, policy, and retry settings are rejected rather than silently ignored.
+     *
+     * @return the voice-agent WebSocket client configuration.
+     * @throws IllegalStateException if unsupported HTTP pipeline configuration is present.
+     */
     private VoiceAgentWebSocketClientConfiguration createVoiceAgentWebSocketConfiguration() {
         validateClient();
         Objects.requireNonNull(tokenCredential,
             "'credential' must be configured to build a voice-agent WebSocket client.");
+        List<String> unsupportedSettings = new ArrayList<>();
+        if (httpClient != null) {
+            unsupportedSettings.add("httpClient");
+        }
+        if (pipeline != null) {
+            unsupportedSettings.add("pipeline");
+        }
+        if (!pipelinePolicies.isEmpty()) {
+            unsupportedSettings.add("addPolicy");
+        }
+        if (retryOptions != null) {
+            unsupportedSettings.add("retryOptions");
+        }
+        if (retryPolicy != null) {
+            unsupportedSettings.add("retryPolicy");
+        }
+        if (!unsupportedSettings.isEmpty()) {
+            throw LOGGER.logExceptionAsError(
+                new IllegalStateException("Voice-agent WebSocket clients do not support these HTTP builder settings: "
+                    + String.join(", ", unsupportedSettings) + "."));
+        }
         Configuration buildConfiguration
             = configuration == null ? Configuration.getGlobalConfiguration() : configuration;
         ClientOptions localClientOptions = clientOptions == null ? new ClientOptions() : clientOptions;

sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionAsyncClient.java

@@ -118,7 +118,7 @@ Mono<Void> connect() {
                 return Mono.error(new IllegalStateException("The voice-agent session has already been started."));
             }
 
-            TokenRequestContext tokenContext = VoiceAgentWebSocketUtils.createTokenRequestContext(options);
+            TokenRequestContext tokenContext = VoiceAgentWebSocketUtils.createTokenRequestContext();
             return configuration.getCredential().getToken(tokenContext).map(AccessToken::getToken).flatMap(token -> {
                 Disposable connection = openWebSocket(token).subscribe(unused -> {
                 }, this::terminateWithError, this::terminateNormally);

sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/BetaVoiceAgentWebSocketSessionClient.java

@@ -32,13 +32,16 @@
 import com.azure.core.util.logging.ClientLogger;
 import java.io.IOException;
 import java.net.Proxy;
+import java.net.ProxySelector;
+import java.net.SocketAddress;
 import java.net.URI;
 import java.nio.charset.CharacterCodingException;
 import java.nio.charset.StandardCharsets;
 import java.time.Duration;
 import java.util.Base64;
 import java.util.Collections;
 import java.util.Iterator;
+import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.Objects;
 import java.util.concurrent.ArrayBlockingQueue;
@@ -48,6 +51,7 @@
 import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
 import okhttp3.Credentials;
 import okhttp3.OkHttpClient;
 import okhttp3.Request;
@@ -87,7 +91,7 @@ private BetaVoiceAgentWebSocketSessionClient(VoiceAgentWebSocketClientConfigurat
         this.events = new ArrayBlockingQueue<>(receiveBufferCapacity + 1);
         this.websocketUri = VoiceAgentWebSocketUtils.buildWebSocketUri(configuration, agentName, options);
         String token = configuration.getCredential()
-            .getTokenSync(VoiceAgentWebSocketUtils.createTokenRequestContext(options))
+            .getTokenSync(VoiceAgentWebSocketUtils.createTokenRequestContext())
             .getToken();
         this.httpClient = createHttpClient(configuration, options);
         Request.Builder request = new Request.Builder().url(websocketUri.toString())
@@ -173,7 +177,14 @@ public IterableStream<RealtimeServerEvent> receiveEvents(Duration timeout) {
             throw LOGGER.logExceptionAsError(
                 new IllegalStateException("Only one receiveEvents iterator is supported per session."));
         }
-        return IterableStream.of(() -> new EventIterator(events, timeout));
+        AtomicBoolean iteratorCreated = new AtomicBoolean();
+        return IterableStream.of(() -> {
+            if (!iteratorCreated.compareAndSet(false, true)) {
+                throw LOGGER.logExceptionAsError(
+                    new IllegalStateException("The receiveEvents stream may only be iterated once."));
+            }
+            return new EventIterator(events, timeout);
+        });
     }
 
     /**
@@ -445,7 +456,7 @@ private void shutdownHttpClient() {
         }
     }
 
-    private static OkHttpClient createHttpClient(VoiceAgentWebSocketClientConfiguration configuration,
+    static OkHttpClient createHttpClient(VoiceAgentWebSocketClientConfiguration configuration,
         VoiceAgentWebSocketConnectionOptions options) {
         OkHttpClient.Builder builder = new OkHttpClient.Builder();
         builder.connectTimeout(options.getHandshakeTimeout().toMillis(), TimeUnit.MILLISECONDS)
@@ -455,7 +466,29 @@ private static OkHttpClient createHttpClient(VoiceAgentWebSocketClientConfigurat
         if (proxyOptions != null) {
             Proxy.Type proxyType = proxyOptions.getType() == ProxyOptions.Type.SOCKS4
                 || proxyOptions.getType() == ProxyOptions.Type.SOCKS5 ? Proxy.Type.SOCKS : Proxy.Type.HTTP;
-            builder.proxy(new Proxy(proxyType, proxyOptions.getAddress()));
+            Proxy proxy = new Proxy(proxyType, proxyOptions.getAddress());
+            if (proxyOptions.getNonProxyHosts() == null) {
+                builder.proxy(proxy);
+            } else {
+                Pattern nonProxyHosts = Pattern.compile(proxyOptions.getNonProxyHosts(), Pattern.CASE_INSENSITIVE);
+                builder.proxySelector(new ProxySelector() {
+                    @Override
+                    public List<Proxy> select(URI uri) {
+                        return Collections
+                            .singletonList(uri.getHost() != null && nonProxyHosts.matcher(uri.getHost()).matches()
+                                ? Proxy.NO_PROXY
+                                : proxy);
+                    }
+
+                    @Override
+                    public void connectFailed(URI uri, SocketAddress address, IOException error) {
+                        LOGGER.atVerbose()
+                            .addKeyValue("uri", uri)
+                            .addKeyValue("proxyAddress", address)
+                            .log("Failed to connect through the configured proxy.");
+                    }
+                });
+            }
             if (proxyOptions.getUsername() != null) {
                 builder.proxyAuthenticator((route, response) -> response.request()
                     .newBuilder()

sdk/ai/azure-ai-agents/src/main/java/com/azure/ai/agents/implementation/realtime/VoiceAgentWebSocketUtils.java

@@ -3,9 +3,9 @@
 
 package com.azure.ai.agents.implementation.realtime;
 
+import com.azure.ai.agents.implementation.models.AgentDefinitionOptInKeys;
 import com.azure.ai.agents.models.RawRealtimeServerEvent;
 import com.azure.ai.agents.models.RealtimeServerEvent;
-import com.azure.ai.agents.models.VoiceAgentTransport;
 import com.azure.ai.agents.models.VoiceAgentWebSocketConnectionOptions;
 import com.azure.core.credential.TokenRequestContext;
 import com.azure.core.http.HttpHeader;
@@ -87,6 +87,7 @@ public static void validateClose(int code, String reason) {
     private static boolean isProtectedHeader(String name) {
         String lower = name.toLowerCase(Locale.ROOT);
         return "authorization".equals(lower)
+            || "user-agent".equals(lower)
             || "host".equals(lower)
             || "upgrade".equals(lower)
             || "connection".equals(lower)
@@ -112,28 +113,10 @@ public static URI buildWebSocketUri(VoiceAgentWebSocketClientConfiguration confi
         String basePath = endpoint.getRawPath() == null ? "" : endpoint.getRawPath().replaceAll("/$", "");
         String path = basePath + "/agents/" + encode(agentName) + "/endpoint/protocols/voice";
         URI baseUri = URI.create(scheme + "://" + endpoint.getRawAuthority() + path);
-        if (options.getConnectionUrl() != null) {
-            baseUri = options.getConnectionUrl();
-            int endpointPort = endpoint.getPort() == -1 ? 443 : endpoint.getPort();
-            int overridePort = baseUri.getPort() == -1 ? 443 : baseUri.getPort();
-            if (!"wss".equalsIgnoreCase(baseUri.getScheme())
-                || baseUri.getHost() == null
-                || !baseUri.getHost().equalsIgnoreCase(endpoint.getHost())
-                || endpointPort != overridePort
-                || baseUri.getRawUserInfo() != null
-                || baseUri.getRawFragment() != null) {
-                throw new IllegalArgumentException(
-                    "Connection URL must be a wss URL on the project endpoint's host and port, without user information or a fragment.");
-            }
-        }
         UrlBuilder url = UrlBuilder.parse(baseUri.toString());
-        url.setQueryParameter("api-version",
-            encode(options.getApiVersion() == null ? configuration.getApiVersion() : options.getApiVersion()));
+        url.setQueryParameter("api-version", encode(configuration.getApiVersion()));
         url.setQueryParameter("x-ms-client-sdk", encode(configuration.getUserAgent()));
-        VoiceAgentTransport transport = options.getTransport();
-        if (transport != null) {
-            url.setQueryParameter("transport", encode(transport.toString()));
-        }
+        url.setQueryParameter("transport", "websocket");
         if (options.isStoreEnabled() != null) {
             url.setQueryParameter("store", options.isStoreEnabled().toString());
         }
@@ -143,14 +126,11 @@ public static URI buildWebSocketUri(VoiceAgentWebSocketClientConfiguration confi
         if (options.getAgentSessionId() != null) {
             url.setQueryParameter("agent_session_id", encode(options.getAgentSessionId()));
         }
-        options.getExtraQuery().forEach((name, value) -> url.setQueryParameter(encode(name), encode(value)));
         return URI.create(url.toString());
     }
 
-    public static TokenRequestContext createTokenRequestContext(VoiceAgentWebSocketConnectionOptions options) {
-        return options.getCredentialScopes() == null || options.getCredentialScopes().isEmpty()
-            ? new TokenRequestContext().addScopes(TOKEN_SCOPE)
-            : new TokenRequestContext().setScopes(options.getCredentialScopes());
+    public static TokenRequestContext createTokenRequestContext() {
+        return new TokenRequestContext().addScopes(TOKEN_SCOPE);
     }
 
     public static HttpHeaders buildHeaders(VoiceAgentWebSocketClientConfiguration configuration,
@@ -163,15 +143,11 @@ public static HttpHeaders buildHeaders(VoiceAgentWebSocketClientConfiguration co
                 }
             }
         }
-        headers.set(HttpHeaderName.fromString("Foundry-Features"), options.getFoundryFeatures());
+        headers.set(HttpHeaderName.fromString("Foundry-Features"),
+            AgentDefinitionOptInKeys.VOICE_AGENTS_V1_PREVIEW.toString());
         if (options.getStructuredInputs() != null) {
             headers.set(HttpHeaderName.fromString("x-ms-voice-structured-inputs"), options.getStructuredInputs());
         }
-        options.getExtraHeaders().forEach((name, value) -> {
-            if (!isProtectedHeader(name) || "Foundry-Features".equalsIgnoreCase(name)) {
-                headers.set(HttpHeaderName.fromString(name), value);
-            }
-        });
         return headers.set(HttpHeaderName.AUTHORIZATION, "Bearer " + token);
     }