Voice agents are judged less by their vocabulary than by their timing. A capable agent that waits two seconds before responding feels broken; one that listens, responds quickly, and stops speaking when interrupted feels usable. For builders in India, low latency real time audio streaming for AI agents requires more than choosing a fast model. It requires coordinated decisions across transport, audio capture, turn detection, inference, playback, and production operations.
The target should not be a single headline number. Track the complete interaction: microphone capture to server receipt, speech endpoint detection, transcript availability, model time to first token, TTS time to first audio, and audio playback start. A strong production target is often under 500 ms from end of user speech to first agent audio, while keeping interruption response close to 100–200 ms. Measure these separately; otherwise a fast model can hide a slow network or an overlong silence timeout.
Design the pipeline for overlap
A production voice agent typically contains these stages:
- Capture: The client records microphone audio, usually as Opus through WebRTC.
- Transport: Audio and control events travel over a persistent, bidirectional connection.
- Turn detection: Voice activity detection (VAD) estimates speech start and end.
- Streaming ASR: Partial transcripts arrive while the user is still speaking.
- Reasoning: The agent begins planning as soon as the transcript is stable enough.
- Streaming TTS: Audio is synthesised sentence fragments at a time.
- Playback: The client queues a small amount of audio and plays it immediately.
Avoid a file-based workflow that waits for a complete recording, transcript, model response, and audio file. Instead, pipeline the stages. Send interim ASR results to the dialogue manager, stream model tokens, and start TTS at a safe phrase boundary. The objective is not merely faster components; it is less idle time between components.
A useful latency budget might allocate 50–100 ms to capture and transport, 50–150 ms to endpointing and ASR stabilisation, 100–250 ms to model first-token generation, and 80–200 ms to TTS and playback. These figures vary by provider and language, but the budgeting exercise exposes unrealistic assumptions early.
Choose transport based on interaction requirements
WebRTC is usually the best default for live browser or mobile voice. It provides low-latency media transport, adaptive behaviour, jitter handling, echo cancellation, and packet-loss concealment. Opus is well suited to speech because it can maintain intelligibility at modest bitrates and adapt to changing network conditions.
WebSockets remain useful for signalling, transcripts, tool events, and systems where audio is already encoded and controlled by the application. They are also simpler for server-to-server integrations. However, TCP retransmission and head-of-line blocking can create stalls when packets are lost. Do not assume that moving audio into a WebSocket automatically delivers real-time performance.
Keep the media path and control path explicit. Audio packets should not wait behind large transcripts, tool results, or logs. Use sequence numbers, timestamps, session identifiers, and event types such as speech_started, speech_stopped, response_started, and response_cancelled. This makes retries and debugging safer than relying on implicit connection state.
Make turn-taking and barge-in first-class features
Silence detection is a product decision as much as an ML decision. A long endpointing timeout makes the agent feel slow; an aggressive timeout cuts users off. Tune thresholds by language, microphone quality, background noise, and use case. Indian deployments should test English, Hindi, Hinglish, and regional-language speech rather than assuming one VAD profile works everywhere.
Barge-in requires a complete cancellation path. When the user begins speaking, the system should:
- stop or fade the currently playing audio;
- cancel pending TTS generation;
- discard unsent audio from the playback buffer;
- cancel or mark obsolete model and tool requests;
- preserve the conversation state without duplicating the interrupted sentence.
Client-side VAD can reduce unnecessary uplink traffic and improve perceived responsiveness, while server-side VAD remains valuable for authoritative session state. Use both carefully: a client decision should not permanently prevent the server from detecting speech in noisy conditions.
Stream ASR, LLM, and TTS safely
Streaming does not mean forwarding every partial result directly to the model. Interim ASR hypotheses can change substantially, especially with code-switching, names, addresses, and noisy audio. Use partial transcripts for early intent hints, but commit turns only when confidence, punctuation, or endpointing rules justify it.
For TTS, begin at semantic boundaries rather than after an arbitrary token count. A short clause is usually a better unit than five words because it avoids awkward pauses and reduces the chance of speaking text that the model later revises. Maintain a small playback buffer for smoothness, but keep it short enough to support immediate interruption.
Tool use introduces another source of delay. Let the agent acknowledge a request quickly when appropriate, but do not speak unsupported claims while a backend action is pending. For transactional actions, expose clear states such as “checking,” “confirmed,” or “unable to complete.” This is especially important for fintech customer onboarding with voice agents, where latency cannot come at the cost of consent, verification, or auditability.
Engineer for Indian networks and languages
Test from actual Indian connectivity conditions: congested 4G, variable Wi-Fi, low-end Android devices, and handoffs between networks. Deploy media relays and inference close to users where possible, including Indian cloud regions, but measure the complete route rather than assuming geographic proximity guarantees low latency.
Practical measures include:
- use Opus with adaptive bitrate and sensible packetisation;
- keep audio frames small enough for responsiveness without excessive overhead;
- use regional media servers and avoid unnecessary US or European round trips;
- preload models and keep inference workers warm;
- prefer compact, language-appropriate ASR and TTS models when quality permits;
- cache prompts, pronunciation dictionaries, and frequently used responses;
- degrade gracefully to audio-only or a text fallback when conditions deteriorate.
Language quality is part of latency. A recogniser that repeatedly mishears Hindi-English names creates clarification turns, increasing the effective interaction time even if network metrics look excellent. Domain vocabulary, transliteration handling, and pronunciation tests should be included in performance benchmarks. For customer-facing deployments, patterns from multilingual voice agents for restaurants in India are relevant: menus, addresses, names, and noisy environments expose weaknesses quickly.
Measure what users experience
Instrument every session with timestamps for capture, packet arrival, speech start, speech end, first partial transcript, final transcript, first model token, first TTS byte, playback start, interruption, and completion. Report p50, p95, and p99—not just averages. Segment results by device, carrier, region, language, model, and network type.
Track quality alongside speed:
- word error rate and correction rate for ASR;
- interruption success rate;
- percentage of responses abandoned before playback;
- audio underruns and packet loss;
- first-audio latency and total response duration;
- task completion and repeat-question rates.
Replay production-like audio in load tests. Synthetic latency tests often miss echo, code-switching, background television, and user hesitation. Add tracing across the media gateway, ASR, model, TTS, and tool services, while redacting or protecting sensitive audio and transcripts.
Security, privacy, and reliability
Voice data can contain identity, health, financial, and location information. Define retention periods, encrypt media in transit and at rest, restrict transcript access, and make recording consent explicit. Healthcare use cases need stronger controls; builders can compare these requirements with guidance on patient follow-up with voice agents in India and HIPAA-compliant voice agents for hospitals, while also accounting for applicable Indian privacy obligations.
Use connection timeouts, backpressure, circuit breakers, provider fallbacks, and idempotent tool calls. A fallback should preserve the task—not merely return an error. For example, if TTS is unavailable, offer text or a callback workflow; if a model is overloaded, route to a smaller model with a constrained prompt.
A practical build sequence
Start with one narrow workflow and a measurable latency budget. Then:
1. Build a WebRTC or equivalent duplex audio path with reliable session events.
2. Add VAD, streaming ASR, and transcript observability before adding complex tools.
3. Implement streaming response generation and cancellable TTS.
4. Test barge-in, packet loss, reconnects, and device sleep conditions.
5. Benchmark Indian languages, carriers, and low-end devices.
6. Add privacy controls, fallbacks, rate limits, and production tracing.
7. Expand to workflows such as real-estate lead qualification with a voice agent only after the core interaction is stable.
The winning architecture is rarely the one with the fastest isolated model. It is the one that overlaps work, limits buffering, handles interruption cleanly, and remains intelligible when the network or a downstream provider fails.