How to use the VAD API?

[lekker-release-radar]

I cannot find any informations about using the VAD (voice activity detection) API #4204

The current situation is: when I pass a silent audio file to the whisper API, it returns "Thank you.".
I like to use the VAD API to prefent this.
Any ideas?

[Maxaubert]

There's a documented POST /v1/vad (alias /vad) endpoint for exactly this. It's not wired into the Whisper/transcription flow automatically, so you call it yourself first and skip Whisper when it comes back empty.

Request/response

// POST /v1/vad
{
  "model": "silero-vad",
  "audio": [0.001, -0.002, ...]   // float32[], 16kHz mono PCM samples
}

{
  "segments": [
    { "start": 0.5, "end": 2.3 },
    { "start": 3.1, "end": 5.8 }
  ]
}

If segments is empty, there's no speech in the clip, so you skip the Whisper call instead of feeding it silence (which is why you're getting hallucinated text like "Thank you.").

Model setup

Add a model config:

name: silero-vad
backend: silero-vad

LocalAI will pull the gallery entry (backed by https://huggingface.co/onnx-community/silero-vad, see gallery/index.yaml) the first time it's loaded.

If you're already running the audio.cpp backend for TTS/ASR, you don't need a separate model at all: it bundles the silero_vad (and marblenet_vad) assets, so model: bundled:silero_vad with the family:silero_vad option works out of the box (see docs/content/features/audio-cpp.md).

Full example (Python)

import soundfile as sf
import numpy as np
import requests

audio, sr = sf.read("speech.wav")  # must be 16kHz mono
if audio.ndim > 1:
    audio = audio.mean(axis=1)
samples = audio.astype(np.float32).tolist()

vad = requests.post(
    "http://localhost:8080/v1/vad",
    json={"model": "silero-vad", "audio": samples},
).json()

if not vad["segments"]:
    print("no speech detected, skipping whisper")
else:
    # proceed to /v1/audio/transcriptions as usual
    ...

Source and doc references:

• Docs: docs/content/features/voice-activity-detection.md
• Handler: core/http/endpoints/localai/vad.go
• Request/response schema: core/schema/localai.go#L123-L135
• The PR that introduced it: #4204