Repository navigation
Resample OpenAI text-to-speech PCM when a cascaded realtime session is not 24 kHz - #281
Merged
Merged
Conversation
…s not 24 kHz OpenAI produces raw PCM at 24 kHz only, so a cascade at another output rate (8 kHz or 16 kHz for a phone line) failed on its first spoken reply. OpenAITextToSpeechFormatClient now asks for 24 kHz and converts the audio to the rate asked for with a streaming windowed-sinc resampler that carries its filter history, fractional position and any odd byte across chunks. Fixes #279
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #279
OpenAI produces raw PCM at 24 kHz only, so a cascaded realtime session at another output rate (8 kHz or 16 kHz for a phone line) failed on its first spoken reply with a
NotSupportedException. The OpenAI speaking leg now serves a cascade at any output rate.Changes
PcmResampler(new, internal): converts a stream of 16-bit mono PCM between rates, a chunk at a time.Flushreturns the last samples once the stream ends.OpenAITextToSpeechFormatClient:Pcm24000is passed through as OpenAI'spcm, as before. Any otherPcm{rate}is asked for aspcm(24 kHz) and converted, both forGetAudioAsyncand asGetStreamingAudioAsyncstreams. The tail comes in one last update of its own. The refusal is gone.CrestApps.Core.AI.OpenAIexposes its internals to the test project, asCrestApps.Core.AIalready does.providers/openai.mdand a changelog entry.Tests
PcmResamplerTests:OpenAITextToSpeechTests: the refusal test is replaced by one that asks forPcm8000,Pcm16000andpcm48000through a loopback stand-in for OpenAI, checks the request still says"response_format":"pcm", and checks that 100 ms at 24 kHz comes back as 100 ms at the rate asked for.Not verified against the live OpenAI API: there is no OpenAI key in the local configuration. The request shape is the one #272 already sends for 24 kHz.
🤖 Generated with Claude Code