[giannik]

Fixes #263

The OpenAI provider speaks, so a cascaded realtime deployment can use an OpenAI model such as gpt-4o-mini-tts as its speaking leg, and the voice picker lists that leg's voices.

Changes

• OpenAIClientProvider.GetTextToSpeechClientAsync returns the SDK's client (AudioClient.AsITextToSpeechClient(), Microsoft.Extensions.AI.OpenAI):
  • it streams where the model can (gpt-4o-mini-tts), and answers in one piece for tts-1 / tts-1-hd;
  • like speech-to-text, a missing deployment name falls back to the connection's TextToSpeechDeploymentName.
• OpenAITextToSpeechFormatClient (internal) handles the one name the cascade and OpenAI disagree on:
  • the cascade asks its speaking leg for Pcm{rate}, Azure Speech's naming (CascadedRealtimeClient.FormatAudioFormat). OpenAI calls the same audio pcm, refuses a name it does not know, and produces raw PCM only at 24 kHz;
  • so Pcm24000 is sent as pcm, and raw PCM at any other rate is refused with a clear message, since it would play at the wrong speed;
  • every other format name is passed on unchanged.
• OpenAIClientProvider.GetSpeechVoicesAsync lists the model's voices (OpenAITextToSpeechVoices), per OpenAI's text-to-speech guide:
  • thirteen for gpt-4o-mini-tts;
  • nine for tts-1 and tts-1-hd.
• Docs:
  • providers/openai.md: the voices, the raw PCM rate, and gpt-4o-mini-tts in the models table;
  • a changelog entry under cascaded realtime deployments.

Tests

OpenAITextToSpeechTests, 8:

• A loopback stand-in for OpenAI keeps the request the cascade's Pcm24000 produces: response_format: pcm, the voice and the model. The streamed audio comes back as audio/l16, the raw PCM the cascade plays.
• Pcm8000, Pcm16000 and pcm48000 are refused before any request.
• The two voice lists.
• A missing deployment name throws.

Without the format translation, the request test and the three refused rates fail.

The full test project passes on our fork (4,665 passed). The only 2 failures there are the known ones under a Greek locale (EmbeddedResourceAIProfileTemplateProviderTests, OpenXmlIngestionDocumentReaderTests), and they fail without this change too. On main the files this change touches are the same as on our fork, except for an unrelated change of ours to GetRealtimeClientAsync.

Verified on live calls

The calls ran through CrestApps.OrchardCore's AI voice agent on Telnyx, with a cascade of gpt-4o-transcribe, gpt-4.1-mini and gpt-4o-mini-tts.

• Before: the session failed to open ("The provider 'OpenAI' does not support text-to-speech"), and the voice picker was empty for the speaking leg.
• After:
  • the session opened 1.6–1.7 s after the answer;
  • the opening, an answer and the goodbye were spoken;
  • the call concluded normally;
  • the profile's voice picker listed the thirteen voices.

Not in this change

• Speaking pace. At its default speed, gpt-4o-mini-tts spoke at about 145–160 words a minute on our calls, against 175–245 for gpt-realtime. A trial with speed 1.2 and a short style instructions brought it to 170–220, which a caller judged a normal pace. The cascade passes neither today. That would be a separate change; I can open an issue to discuss where such a setting belongs.
• Azure OpenAI's speech deployments. They could reuse the same client once Move the Azure OpenAI provider onto the OpenAI SDK #255 moves that provider onto the OpenAI SDK.

🤖 Generated with Claude Code