A voice agent succeeds or fails in the gaps between a caller’s words and the system’s response. If transcription arrives late, the model waits. If speech-to-text (STT) mishears a name, amount, address, or intent, every downstream step becomes unreliable. For Indian deployments, the engineering challenge is sharper: calls may involve 8 kHz telephony audio, background noise, regional accents, and frequent switching between English and an Indian language.
The right goal is not simply “the fastest model.” It is a pipeline that starts recognising speech early, detects when a user has finished, streams an answer safely, and recovers when confidence is low. This guide explains how to design that system, measure it, and operate it in production as of 2026.
Define latency and accuracy targets first
Do not treat latency as one number. Measure the full conversational loop:
- Audio-to-partial transcript: time from captured speech to the first useful STT fragment.
- End-of-turn detection: time taken to decide that the caller has finished speaking.
- Time to first token: delay between the request reaching the LLM and its first response token.
- Time to first audio: time until the caller hears the first synthesised words.
- Barge-in recovery: time required to stop the agent when the caller interrupts.
For a natural interaction, target first audio within roughly 500–800 milliseconds after turn completion, then optimise toward faster partial responses where the use case permits. Track p50, p95, and p99 values rather than averages; a system that feels fast in testing but stalls on busy Indian call windows will quickly lose trust.
Accuracy also needs a practical definition. Report word error rate (WER), but separately measure entity accuracy for names, account numbers, PINs, locations, product codes, and rupee amounts. A transcript can have a respectable WER while still corrupting the one field that matters to a banking or delivery workflow.
Use a streaming architecture, not a batch pipeline
A production voice agent typically connects a media layer, streaming STT, turn detection, an LLM or workflow engine, and streaming text-to-speech. Audio should move continuously rather than wait for a complete utterance.
A useful flow is:
- Capture microphone or telephony audio through WebRTC, SIP, or a managed media gateway.
- Resample and normalise audio without introducing unnecessary buffering.
- Send short audio frames to a streaming STT endpoint.
- Maintain partial and final transcripts separately.
- Start intent classification or retrieval as soon as the transcript is stable enough.
- Stream the LLM response into TTS at sentence or phrase boundaries.
- Play audio immediately, while supporting interruption and cancellation.
Keep the audio path and business-logic path loosely coupled. If a CRM lookup takes 700 milliseconds, the media loop should still detect a barge-in and stop playback. For teams evaluating the broader stack, first understand what a voice agent is and distinguish it from a scripted voicebot before selecting vendors.
Select STT for your actual Indian traffic
Benchmark models on representative recordings, not vendor demos. Build a test set covering:
- Hindi-English and other code-switched conversations.
- Regional pronunciation differences across cities and states.
- 8 kHz contact-centre audio and mobile-network compression.
- Call-centre noise, vehicle noise, fans, and multiple speakers.
- Names, addresses, dates, policy numbers, order IDs, and rupee values.
- Fast speech, hesitations, repetitions, and incomplete sentences.
Streaming cloud STT can offer strong latency and managed scaling. Self-hosted options based on Whisper-class models can improve data control and customisation, but require GPU capacity, model serving, monitoring, and failover planning. Do not assume a larger model is automatically better: a smaller model with Indian-language coverage and domain adaptation may outperform a general model in your calls.
Use phrase hints, custom vocabulary, or grammar constraints where supported. Apply them carefully; over-biasing can force the recogniser toward a likely but incorrect term. For sensitive actions, confirm the recognised value with the caller instead of silently trusting transcription.
Reduce latency at each stage
Tune turn detection
The end of a user’s turn is often the largest avoidable delay. A fixed 500-millisecond silence timeout is simple but performs poorly across callers. Combine voice activity detection with speech probability, punctuation stability, semantic completion, and the conversation state.
Use shorter waits after commands such as “cancel” or “yes,” and longer waits when the caller is dictating an address. Always allow interruption. A turn detector should be evaluated for both false cut-offs and late responses; reducing silence too aggressively makes the agent talk over people.
Stream the model and speech output
Set a strict response policy so the LLM does not generate long explanations in a voice channel. Begin TTS when a safe phrase or clause is available, but avoid speaking half a number, medical instruction, or transaction amount. Sentence-aware buffering usually provides a better balance than token-by-token audio.
Cache greetings, confirmations, and common prompts. Keep tool calls parallel where possible, and use deterministic workflows for high-risk operations instead of asking the LLM to improvise every step.
Keep infrastructure close to callers
Deploy media gateways, STT connections, application servers, and databases in the same Indian region where possible. Mumbai, Hyderabad, and Delhi-NCR placements may be useful depending on provider availability and customer geography. Measure network RTT from real mobile and carrier paths; cloud-region labels alone do not guarantee low latency.
Use WebRTC for browser and app conversations when appropriate, and SIP or telephony providers for phone calls. Jitter buffers, packet-loss recovery, and echo cancellation can matter more than shaving a few milliseconds from model inference.
Improve accuracy with audio and conversation design
Before changing models, fix the input. Use echo cancellation, noise suppression, automatic gain control, and consistent sample rates. Preserve the original audio for debugging subject to consent, retention, and privacy requirements. Detect overlapping speech and multiple speakers rather than forcing every recording through a single-speaker recogniser.
Design prompts and workflows around uncertainty. If STT confidence is low or two candidate values differ, ask a focused confirmation: “Did you say 15,000 rupees or 50,000 rupees?” For names and addresses, use spell-back or repeat-back flows. Never expose raw transcripts to customers when a structured confirmation would be safer.
For healthcare, finance, and insurance, apply explicit consent, access controls, retention limits, and audit logs. A healthcare deployment should also be reviewed alongside guidance on voice agents in Indian healthcare, especially when calls involve patient identifiers or clinical information.
Test with a production scorecard
Run offline replay tests and live canaries. Track:
- WER and character error rate by language, accent, and channel.
- Accuracy for critical entities and completed tasks.
- Time to first partial transcript and first audio.
- Interruption success rate and accidental cut-offs.
- Handoff rate, abandonment, repeat-question rate, and containment.
- Cost per connected minute and cost per successfully completed task.
Label failures by root cause: audio quality, STT, turn detection, retrieval, tool latency, LLM behaviour, or TTS. This prevents teams from buying a new model when the real problem is a 400-millisecond silence timeout or an overloaded database.
Build for scale, privacy, and failure
Use regional failover, connection limits, backpressure, and graceful degradation. If premium STT is unavailable, route to a fallback model or transfer to a human rather than continuing with unreliable recognition. Keep secrets out of prompts, encrypt recordings and transcripts, and separate personally identifiable information from analytics wherever practical.
Plan unit economics early. Streaming STT, LLM tokens, TTS characters, telephony minutes, GPU inference, storage, and human handoffs all contribute to cost. A voice agent pricing and ROI analysis should use completed outcomes—not only per-minute vendor rates.
A practical implementation sequence
Start with one narrow workflow and a labelled evaluation set. Then:
1. Establish latency, entity-accuracy, and task-completion baselines.
2. Add streaming STT and partial-transcript handling.
3. Tune VAD, turn detection, and barge-in behaviour.
4. Introduce streaming LLM and TTS with safe phrase buffering.
5. Test Indian languages, code-switching, telephony audio, and real noise.
6. Add observability, consent controls, regional deployment, and failover.
7. Expand workflows only after the first one is reliable.
If the project needs custom integrations, language evaluation, or on-premise deployment, use a structured process to hire voice agent developers. The strongest systems are not defined by a single STT vendor; they are defined by disciplined measurement, careful conversation design, and reliable recovery when speech is ambiguous.