KittenTTS module. by jsett · Pull Request #1107 · brailcom/speechd

jsett · 2026-08-12T21:18:20Z

KittenTTS Module.

This is a speech dispatch model for running Kitten TTS. Kitten TTS is a deep learning model which provides high quality natural sounding TTS generation using models ranging from 15M to 80M parameters. Due to its small size it is able to run in real-time on CPU. The goal of this project is to integrate Kitten TTS with speech dispatch while maintaining its near real-time speech generation, with special attention being place on reading of long text's such as ebooks. To achieve this the original python code was rewrote into c and tightly integrated into a speech dispatch model.

There is a lot here so let me give a quick summary of what all is going on here.

kitten_server.c:

This file is for handling the protocol and follows some what closely to the example modules for async servers with speech dispatch handling the audio. By design I made sure very little work is done in any function in this thread. Any long running code should be handed to an async queue and ran on one of the threads in the kitten_worker.c file.

kitten_worker.c:

This is where the handling of long running tasks is done. We create two different threads here one to one to handle passing audio back to the server(since the module_tts_output_server function can block and we want the generation to continue while we are outputting audio to the server). The other thread is dedicated to the generation of audio by the model. We synchronize all this using two GasyncQueue, one for handling incoming speak requests, and the other to handle the outputted audio from our model. we also keep track of how much audio we have generated and played so that long speak commands don't run the cpu unnecessarily hard (For example I have seen that Okular will in some cases send an entire book's text in a single speak command). There is also code here for parsing ssml text using libxml.

kitten_model.c:

The handles everything we need to do to generated output from onnx using our model. The most important functions here are init_voice_style to handle loading voice styles. reload_models_and_voices: to handle changing the voice style. And kitten_speak for generating audio.

kitten_downloader.c:

This handles downloading the model+voice styles if its not already on our computer. If it download it does verify the file against a sha256, but that verification is not strongly enforce and is more of a warning.

…issue when processing text over 400 char. Fix an issue when we get unmapped chars.

sthibaul

Thanks for the nice work!

In the future, we will probably want to integrate various onnx-based voices, so code will be useful to share between modules, but your writing seems already quite well structured so that it will be convenient to do.

There are just a few changes that need to happen before we can integrate this.

sthibaul · 2026-08-13T23:00:51Z

+PKG_CHECK_MODULES([LIBCURL], [libcurl])
+AC_SUBST([LIBCURL_CFLAGS])
+AC_SUBST([LIBCURL_LIBS])
+


We want to make these optional, and just build speech-dispatcher without KittenTTS support. See how it is done e.g. for the espeak_ng module, we only build it if espeak-ng is available.

Also, we want to make the CI exercise building the KittenTTS module, i.e. add to .github/workflows/ci.yml the apt-get dependencies for these: libonnxruntime-dev, libxml2-dev, libcurl4-gnutls-dev.

I should be able to fix this.

I have updated the code to fix this.

sthibaul · 2026-08-13T23:04:14Z

+
+    *msg = strdup("ok!");
+
+	return 0;


We really want the init function to report any error that renders the module non-working. Otherwise if the init function returns success, the module which show up as a choice for users, they will try to select it, and then the computer will suddenly be completely silent and the user be at a loss.

This should not be to hard to fix.

I have updated the code to fix this.

sthibaul · 2026-08-13T23:07:04Z

+    g_ort = OrtGetApiBase()->GetApi(ORT_API_VERSION);
+    if (!g_ort) {
+        fprintf(stderr, "Failed to initialize ONNX Runtime API.\n");
+        return -1;


we do want to report this in the module initialization function

same thing.

I have updated the code to fix this.

sthibaul · 2026-08-13T23:12:28Z

+    return 0;
+}
+
+// This is the same as spd_pthread_create, I move it in here because including $(common_SOURCES) was creating circular dependences for me.


What kind of circular dependences were showing up? We should be able to fix them rather than duplicating code.

So, I can see that cicero and dummy both use spd_pthread_create successfully and they build fine but when I try and model my compile flags and includes to match I get errors. Either stuff has "undefined reference" or "multiple definition" depending on what I try. It could be some issue with how I structured my .h file or something is sneaking references into the build files. To be honest I really don't know, It's very confusing for me. This seems to be the only function that if I try and use it cause issues. I will try looking at it again, but I really don’t know what causing this issue.

So I had some time to look at this and I don't think this is an issue with my code. It seems that its probably something with module_utils.h. The other modules that include spd_pthread_create such as dummy.c, get away with it because they don't have the module_loop and module_* functions that seem to get multiple definitions.
For example you can break dummy.c and recreated this issue by adding in

int module_debug(int enable, const char *file) { return 0; }

Fixing this would likely require changing module_utils which is outside the scope of what am working on. So I don't have a solution other then coping in spd_pthread_create.

sthibaul · 2026-08-13T23:13:08Z

@@ -0,0 +1,133 @@
+#include <glib/gstdio.h>


We need a copyright header at the beginning of every added file, to specify the licencing of the file and the copyright holders

For the code I wrote which are all the kitten_* files, I will probably place them under a MIT License. My primary desire is to keep the stuff that may be reusable on another project in a highly permissive license for example kitten_model.c very well may be reusable else where. I don't think this would cause any with speechd, even thought its under GPL, but let me know if that is an issue. Also note I am not the copyright owner for the model. According to its hugging face page Its under the apache-2.0.

Indeed, MIT will be completely fine.

Also note I am not the copyright owner for the model

Sure, that's not a problem, people will be able to download them fine.

BTW, I'm however thinking: distributions will probably like to be able to install the models from packages, so they are already there and users don't have to download them before use. It would thus be useful to also try to look for the models somewhere in /usr/share/speech-dispatcher.

I have updated the headers with the license.

I also added code to check if the models are in /usr/share/speech-dispatcher/models/kitten and use that path if they are.

sthibaul · 2026-08-13T23:14:23Z

+
+#define PAD "$"
+#define PUNCTUATION ";:,.!?¡¿—…\"«»\"\" "
+#define LETTERS "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"


does this TTS only support english? Can it support more in the future?

I think this model only supports English. But I think that the architecture it uses supports multiple languages. According to the StyleTTS2 readme "Non-English dataset: You can train on any language you want". Also Kokoro which is very similar and also based off StyleTTS2 doe support multiple languages according to their docs (Of course Kokoro is a much larger almost 4x as large ). But it would not surprise me if with minimal changes Kokoro could be supported. I choose this model primarily because of its size and my primary goal was to have real-time generation on CPU.

sthibaul · 2026-08-13T23:16:43Z

+// also verifys using sha256 and checks file size.
+int download_models(void) {
+    // Build absolute destination directory path: ~/.config/speech-dispatcher/extra/
+    char *target_dir = g_build_filename(home_dir, TARGET_SUBDIR, NULL);


it should rather go to .cache/speech-dispatcher, and I'd say KittenTTS rather than an non-informative "extra"

This should be an easy change.

I updated this.

sthibaul · 2026-08-13T23:21:55Z

+	fprintf(stderr, "pausing\n");
+
+    // does not seem there is a resume function so
+    // pause will be handled the same as stop.


as mentioned in the module output documentation, pause is indeed similar to stop, but the difference is that it is preferrable to stop at a place where the indexing position is available, so that if the screen reader brings the cursor to that position, the feedback to the user is coherent with that positionning: not before, not after, exactly where the audio feedback finished.

(of course, that's a refinement that we can integrate later, doing the same as stop is fine for now)

…dencies to the CI

jsett added 9 commits August 5, 2026 20:01

kitten tts module for speechd

de1ef6c

Remove the buffering code. Everything runs fine without it. Fixed an …

8d7a4db

…issue when processing text over 400 char. Fix an issue when we get unmapped chars.

Added instructions into the readme on building/install

01cb7d1

small bug

2b2a6de

change stuff in model_change_voice to use Gstring's instead of char*'s.

a5ab64f

mark need to be a char*

3a611e6

updated readme.

9f5bf7e

updated readme

a284a5c

removed some old dependencies.

d188bf5

sthibaul requested changes Aug 13, 2026

View reviewed changes

jsett added 5 commits August 14, 2026 16:08

Updated the make and configure file for optional support. Added depen…

5a8baa9

…dencies to the CI

make sure that if any of our init's fail it get reported as a failure.

4a082cc

placed in license text

0249b33

added checks for a distro path.

b41356a

forgot to change this comment

acebf8e

Uh oh!

Conversation

jsett commented Aug 12, 2026

KittenTTS Module.

kitten_server.c:

kitten_worker.c:

kitten_model.c:

kitten_downloader.c:

Uh oh!

sthibaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants