[Tracking] Complete audio task coverage across CPU, GPU, and NPU backends

[mergennachin]

Make ExecuTorch the go-to place for common audio tasks across CPU, GPU, and NPU backends. Prioritize task coverage, current model quality, MLX-Audio support, and Hugging Face adoption.

Implementation priority below. Evaluation candidates are conditional on a measured benefit over the selected model or existing examples.

1. Qwen3-ASR 0.6B — multilingual transcription; evaluate 1.7B.
2. Qwen3-ForcedAligner 0.6B — word-level alignment and timestamps.
3. Qwen3-TTS — CustomVoice 0.6B, Base voice cloning, then 1.7B VoiceDesign.
4. Nemotron 3.5 ASR streaming 0.6B — live transcription.
5. Smart Turn v3 — conversational end-of-turn detection.
6. Nemotron 3 Diarization — speaker diarization; extend backend coverage (Enable Nemotron-3-Diarization on MLX, CUDA, XNNPACK, and Vulkan #23131).
7. Parakeet / Whisper — transcription and translation; extend existing examples.
8. Silero VAD — speech activity detection; extend backend coverage.
9. Supertonic 3 — lightweight TTS; extend backend coverage.
10. Voxtral / Voxtral Realtime / Voxtral TTS — audio understanding, live ASR, and TTS; extend existing examples.
11. LFM2.5-Audio 1.5B — speech-to-speech interaction.
12. DeepFilterNet3 — speech enhancement and denoising.
13. SAM-Audio — prompted sound extraction.
14. DialogueSidon — overlapping speaker separation.
15. Mel-Band-RoFormer — vocal/instrumental separation.
16. MOSS-Music — music understanding and lyrics transcription.
17. MiniMax Music 3 — music and song generation.
18. VoxCPM2 — evaluate additional voice-cloning/design quality and control.
19. Granite Speech 5.0 470M TurboCTC — evaluate compact English ASR.
20. Canary v2 — evaluate uncovered speech-translation needs.
21. VibeVoice-ASR Streaming — evaluate joint transcription/diarization beyond composed pipelines.
22. MossFormer2 SE — evaluate enhancement quality beyond DeepFilterNet3.

Use available MLX-Audio references for initial comparisons, then validate across CPU, GPU, and NPU backends and track quality, performance, and backend gaps.

cc @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng @metascroy

[msluszniak]

@mergennachin are there similar aggregates for other classes of models?

[mergennachin]

@msluszniak what kind of tasks and modalities are you interested in?

I just used mlx-audio and HF downloads as a proxy of what people want.

We can certainly expand to other tasks.

The reason I think this exercise is important was:

• I want ExecuTorch to be the go-to place for cross-platform/backend audio tasks
• I want to harden our backends along the way and fill the gaps.

[msluszniak]

From what we have right now, I feel we have three groups of models.

1. Audio (TTS, STT, audio input for LLMs, VAD, S2S, etc.)
2. Computer Vision (object detection, instance / semantic / sam segmentation, diffusion models, OCR, keypoint detections etc.)
3. LLM-like (LLMs, privacy filters, text / image embeddings, etc.)

From my experience with delegations we are mostly interested in the 1st and 2nd group. The third one is the hottest topic overall, but there are huge communities that are doing it really well, have backend coverage and 0-day support on almost every OSS model, so it's hard to compete with them, like llama.cpp etc. Making delegates faster than them is a process and as a whole will take a lot of time. The first and second groups look for me like more immediate wins since there are no such good alternatives for them in cross-platform ecosystem. Also NPU / ANE utilization on CV gives much bigger gains than on i.e. LLMs.

[msluszniak]

I want to harden our backends along the way and fill the gaps.

Yeah, currently I'm doing sweep over our models on HF and also trying to export them on multiple backends and upstream every bug fix / speedup to ET to make the process of bumping ET version in our tooling easier, but at the same time has these fixes immediately.

[mergennachin]

@msluszniak - yeap agree with you. for 3) we are bringing our backends on-par on LLM with other frameworks but there's a significant reservation from people to switch from unless we are (a) on-par or better on performance (b) bring additional capabilities down the line (which brings me 1st and 2nd groups of models you mentioned)

for 2, yeah, historically we have tested on "classical" cv models but we haven't kept up with the new hotness in the CV world much and relied on partners like yourself to give us feedback.

if there are concrete models and tasks in CV that we are not covering let us know, we can do a similar exercise of model diversity in this group and expand our coverage.

[mergennachin]

Yeah, currently I'm doing sweep over our models on HF and also trying to export them on multiple backends and upstream every bug fix / speedup to ET to make the process of bumping ET version in our tooling easier, but at the same time has these fixes immediately.

Yes, I figured. And we're deeply appreciative of your work.

[msluszniak]

if there are concrete models and tasks in CV that we are not covering let us know, we can do a similar exercise of model diversity in this group and expand our coverage.

I will think about the list of models we want to support in the near future and let you know here or in more accurate thread.