Skip to content

Resample OpenAI text-to-speech PCM when a cascaded realtime session is not 24 kHz - #281

Merged
MikeAlhayek merged 1 commit into
mainfrom
fix/279-resample-openai-tts
Oct 19, 2026
Merged

MikeAlhayek merged 1 commit into
mainfrom
fix/279-resample-openai-tts

Conversation

@MikeAlhayek

Copy link
Copy Markdown
Member

Fixes #279

OpenAI produces raw PCM at 24 kHz only, so a cascaded realtime session at another output rate (8 kHz or 16 kHz for a phone line) failed on its first spoken reply with a NotSupportedException. The OpenAI speaking leg now serves a cascade at any output rate.

Changes

  • PcmResampler (new, internal): converts a stream of 16-bit mono PCM between rates, a chunk at a time.
    • A windowed-sinc (Blackman) filter evaluated at the exact rational ratio of the two rates, one filter per phase, each normalised to unity gain at DC.
    • Going down in rate, the cutoff follows the lower rate, so what a phone line cannot carry is removed instead of folding back as noise.
    • The filter history, the position between input samples, and a byte left over from an odd-length chunk all carry from one chunk to the next, so where the stream is split makes no difference to the audio. Flush returns the last samples once the stream ends.
  • OpenAITextToSpeechFormatClient: Pcm24000 is passed through as OpenAI's pcm, as before. Any other Pcm{rate} is asked for as pcm (24 kHz) and converted, both for GetAudioAsync and as GetStreamingAudioAsync streams. The tail comes in one last update of its own. The refusal is gone.
  • CrestApps.Core.AI.OpenAI exposes its internals to the test project, as CrestApps.Core.AI already does.
  • Docs: providers/openai.md and a changelog entry.

Tests

  • PcmResamplerTests:
    • the output has one sample for every point in time the input covered (8, 16, 22.05, 44.1 and 48 kHz);
    • a 1 kHz tone keeps its pitch (within 1%) and level (within 3%) at 8, 16, 44.1 and 48 kHz;
    • a 10 kHz tone resampled to 8 kHz is attenuated by more than 40 dB, so nothing aliases into the phone band;
    • a steady level stays within ±1;
    • random chunk sizes, including chunks that end half-way through a sample, give exactly the same output as one piece;
    • full-scale audio is clipped rather than wrapped around.
  • OpenAITextToSpeechTests: the refusal test is replaced by one that asks for Pcm8000, Pcm16000 and pcm48000 through a loopback stand-in for OpenAI, checks the request still says "response_format":"pcm", and checks that 100 ms at 24 kHz comes back as 100 ms at the rate asked for.
  • The full test project passes locally (4771).

Not verified against the live OpenAI API: there is no OpenAI key in the local configuration. The request shape is the one #272 already sends for 24 kHz.

🤖 Generated with Claude Code

…s not 24 kHz

OpenAI produces raw PCM at 24 kHz only, so a cascade at another output rate
(8 kHz or 16 kHz for a phone line) failed on its first spoken reply.
OpenAITextToSpeechFormatClient now asks for 24 kHz and converts the audio to
the rate asked for with a streaming windowed-sinc resampler that carries its
filter history, fractional position and any odd byte across chunks.

Fixes #279
@MikeAlhayek
MikeAlhayek merged commit c5b95f6 into main Oct 19, 2026
10 checks passed
@MikeAlhayek
MikeAlhayek deleted the fix/279-resample-openai-tts branch October 19, 2026 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resample OpenAI text-to-speech PCM when a cascaded realtime session is not 24 kHz

1 participant