Voice Activity Detection: The Cheap Gate That Makes Your Speech Pipeline Fast

Don't pay a transcription model to listen to silence.

We've been experimenting with audio models lately — feeding real recordings into speech-to-text and seeing what comes back. The microphone is dumb and always on; the speech-to-text is not dumb and not free. And when I looked at a real recording, the uncomfortable truth was that more than half of it was silence — pauses between sentences, dead air, the gap while someone thinks. If I ship all of that to the transcriber, I pay full price to turn nothing into nothing.

That gap between "always recording" and "only speech matters" is exactly what voice activity detection fills. VAD is the cheap gate you put in front of the expensive stuff. It doesn't transcribe, it doesn't identify who's talking — it answers one small question, over and over: is the current slice of audio speech, or not? Get that answer 32 milliseconds at a time and suddenly you only spend your real compute on the parts that carry words.

Where the gate earns its keep

Once you have a reliable speech/not-speech signal, a surprising number of problems get easier:

• Cost and latency. Whisper, or whatever ASR you run, only sees the speech segments. On the clip I use in the example below, 55% of the audio never needs to reach the transcriber. That's not a rounding error — that's more than half your bill and half your latency.
• Segmentation and turn-taking. VAD gives you the boundaries of each utterance for free, which is what you need for endpointing ("has the user finished talking?"), for chunking long audio into sentence-sized pieces, and for knowing when to stop recording.
• Cleaner everything. Diarization, wake-word pipelines, call recording, live captioning — they all behave better when they aren't chewing on dead air.

The naive version of this is an energy threshold: if the audio is loud enough, call it speech. It works in a quiet room and falls apart the moment there's an air conditioner, a keyboard, or a truck outside — steady noise reads as "loud," and quiet speech reads as "silence." The classic step up was WebRTC VAD, a small statistical model that shipped inside every browser's audio stack; it's fast and it's fine, but it's old and it trips on noise too.

What changed the game is that a genuinely good neural VAD now fits in about two megabytes and runs on a CPU in real time. That's Silero VAD, and because it ships as an ONNX model, you can run it straight from C# with no Python anywhere in the loop.

Doing it in C#

I put a complete, minimal example here: github.com/egarim/vad-dotnet. It's a WAV reader, a Silero wrapper, a segmenter, and a CLI — small enough to read in one sitting. The only dependency is Microsoft.ML.OnnxRuntime.

The model is stateful and works on fixed windows. At 16 kHz you feed it 512 samples at a time — 32 ms — and it hands back a probability that the window is speech, carrying an internal LSTM state from one call to the next. The core of the wrapper is genuinely this small:

using var vad = new SileroVad("models/silero_vad.onnx");   // 16 kHz
foreach (var window in audio.Chunk(vad.WindowSamples))      // 512-sample slices
{
    float p = vad.Process(window);                          // P(speech) for this 32 ms
    // ... hand p to the segmenter ...
}

The probabilities are noisy frame to frame, so you don't threshold them directly — you run them through a little hysteresis gate: open at 0.5, close at 0.35, require a minimum silence before you actually end a segment, and pad each side so you don't clip the first or last phoneme. That's the same logic Silero's own get_speech_timestamps uses, and it's about forty lines in the example. Point the CLI at the bundled clip and it prints:

segments   : 2
     1.12s ->    2.40s  (1.28s)
     4.13s ->    6.17s  (2.04s)
speech     : 3.32s (45.4%)
silence    : 3.99s (54.6%)
=> you can skip 54.6% of this audio before it ever reaches your ASR.

Those are the two spoken phrases in a clip that's otherwise silence, found to the frame.

The detail that will cost you an afternoon

Here's the part I wish someone had put in bold, because I burned real time on it and so does everyone else. Silero v5's ONNX model takes three inputs — input, state, sr — and the obvious thing to do is feed it your 512-sample window as input. Do that and every probability comes back essentially zero. The model looks broken. Your audio is fine, the peaks are there, and it insists there's no speech anywhere.

The model does not want a bare 512-sample window. It wants the 64 samples from the tail of the previous window glued in front of the current one — 576 samples per call — and it expects you to keep feeding its state back to it between calls. That context stub is how it hears across window boundaries, and without it the network is starved. The official Python wrapper does this quietly, deep in a helper, which is why the trap survives contact with the docs. Two smaller companions to the same trap: the sr input is a genuine scalar (rank-0) tensor, not shape [1], and the window size is fixed — 512 at 16 kHz, 256 at 8 kHz, and those are the only two sample rates it knows.

In the example, SileroVad.Process carries the 64-sample context and the state so you never see any of it — but that one detail is most of the reason the repo exists.

It runs where the audio is

The nicest property for real systems: this is streaming, not batch. You don't need the whole file. Feed Process your live 32 ms chunks as they arrive from the microphone and you can gate the recorder in real time — start writing when speech opens, stop when it closes, and never hand your transcriber a second of silence. Call Reset() between independent streams and the state clears. On a CPU, a 2 MB model keeps up with real time without noticing.

For us this is the front door to any audio pipeline we're experimenting with: the source streams audio, the VAD decides what's worth keeping, and only the speech reaches the model that costs money. Any time you're about to run something expensive on audio, put the cheap gate in front of it first. The code is on GitHub; clone it, run get-model.sh, point it at a recording of your own, and watch how much of it was never worth transcribing.