Repository navigation
kokoro: make the GPU path actually fast (+ streaming) - #189
Merged
Merged
Conversation
chunk_forward_pass built a new KPipeline every request, which reloads the g2p and spacy models, ~0.9s each time. cache them per lang_code, clear on unload.
on GPU the whole kokoro IR recompiles kernels for every new output length, 7-11s per request. fix: - text front end (bert, durations, f0/n) runs as an openvino CPU graph, token count changes every request so it can't live on the GPU - vocoder compiled at fixed frame buckets (64..512), static shapes, warmed at load. inputs padded with reflect (zeros skew adain's time stats) and trimmed - longer inputs are cut in pauses and decoded in pieces - ConvTranspose1d rewritten polyphase and snake alpha folded into the neighbouring layers, both exact. static reshape fails without the first, and they move the vocoder off the slow reference kernels - SineGen uses exact sizes, the float scale factor broke above ~1536 frames - converted IRs cached in cache_dir so reloads skip conversion B50: median 0.17s per sentence vs 10.6s before, ~2.5GB VRAM.
openarc_tts.kokoro.stream=true returns audio/L16 (24kHz mono) chunk by chunk as each text chunk finishes instead of waiting for the whole file. first chunk is kept short so audio starts in ~0.1-0.25s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
follow up to #186. builds on #186 and #188, so those commits show up here until they merge.
#186 gets kokoro onto the GPU, but on its own it's slow: openvino recompiles kernels for every new output length, so every new sentence costs 11-13s. this fixes that:
also adds streaming:
openarc_tts.kokoro.stream=truereturns audio/L16 (24kHz mono) chunk by chunk instead of the whole file.B50, sentences it hadn't seen:
30s of audio renders in 0.77s, streaming starts playing at 0.18s. sounds the same as CPU to me.
costs ~2.5GB VRAM and ~80s for the first load. set
cache_dirandNEO_CACHE_PERSISTENT=1+NEO_CACHE_DIRso restarts don't recompile, andOMP_NUM_THREADS=4or torch spins ~13 cores per request. only tested on a B50.