Voice AI Latency: Find the Delays That Interrupt the Conversation

The sub-500ms threshold: the psychophysics of human conversational timing

In text-based conversational interfaces, latency is an ergonomic convenience. When an engineer queries an LLM in a terminal or web browser, a time-to-first-token of two seconds is perfectly acceptable because the user's visual attention is occupied by a typing cursor or streaming markdown tokens. In spoken voice communication, however, latency is not a convenience; it is the fundamental parameter that defines whether a conversation can occur at all.

Psycholinguistic research into human conversational dynamics reveals that natural turn-taking between native speakers occurs within an astonishingly tight temporal window: typically between 200 and 450 milliseconds. When one speaker finishes a sentence, the listener begins formulating their response hundreds of milliseconds before the final phoneme is uttered, allowing the reply to begin almost seamlessly after the pause.

When an automated voice system introduces delay, human conversational behavior breaks down in predictable, structural stages. At round-trip latencies between 500ms and 800ms, the interaction feels noticeably robotic and hesitant, though human speakers can still adapt by deliberately pausing. When latency crosses 1,200ms, the system enters the collision zone: assuming the agent did not hear them, the caller begins to speak again (‘Hello? Are you there?’), precisely as the agent finally begins outputting its delayed response. Both parties speak simultaneously, creating acoustic confusion and prompt corruption.

In addition to verbal collisions, latency above one second triggers acute cognitive fatigue. In empirical usability testing of voice interfaces, human callers report intense feelings of anxiety, conversational friction, and loss of control when interacting with a lagging speech system. Callers frequently begin shouting, repeating words, or over-enunciating syllables in a frustrating attempt to force the system to respond, which degrades transcription accuracy even further.

At the 2,000ms to 4,000ms latencies common in naive API wrappers—where a web framework records a full audio file, uploads it to a cloud REST endpoint, waits for transcription, calls a standard LLM API, and then sends the complete text to a text-to-speech service—conversational voice is functionally unusable. Callers abandon the interaction within thirty seconds, resulting in disastrous customer experience metrics. Engineering a production Voice AI system requires treating every single millisecond as an unrecoverable operational budget.

References: LiveKit — Real-time Voice and Agent ArchitectureIETF RFC 8829 — JavaScript Session Establishment Protocol (JSEP / WebRTC)

The latency arithmetic of cascaded voice pipelines (STT -> LLM -> TTS)

Most production Voice AI systems deployed today use a cascaded architecture: an incoming audio stream is processed by a Speech-to-Text (STT) engine, the transcript is processed by a Large Language Model (LLM), and the resulting text tokens are converted into audio by a Text-to-Speech (TTS) synthesizer. To achieve an end-to-end roundtrip under 500ms, every component in this cascade must operate in a streaming, chunked pipeline rather than a batch sequential model.

Component 1: Voice Activity Detection (VAD) and Turn Endpoints (80ms to 150ms). The pipeline cannot begin processing until it determines that the speaker has finished speaking. If the VAD threshold is too aggressive (e.g. 50ms), the system interrupts callers whenever they pause to breathe or think mid-sentence. If the threshold is too conservative (e.g. 500ms), you have spent your entire latency budget before transcription even finishes. Modern architectures use lightweight machine learning VAD models (such as Silero VAD) running at 20ms frame intervals, combined with semantic end-of-turn classification to trigger completion in 100ms to 120ms.

Component 2: Streaming Speech-to-Text (140ms to 220ms). Waiting for an entire audio file to upload is fatal to latency. Production systems stream raw audio frames (16kHz linear PCM or Opus) over persistent WebSockets or WebRTC data tracks to low-latency transcription engines like Deepgram Nova-2 or version-pinned Whisper Turbo models. By evaluating partial interim transcripts in real time, the STT engine emits the final word token within 150ms of the VAD endpoint trigger.

Component 3: Time-to-First-Token (TTFT) LLM Inference (100ms to 180ms). Traditional multi-tenant cloud APIs with 800ms cold starts cannot be used in voice pipelines. Production voice architectures route requests to dedicated high-throughput inference engines (such as vLLM on private cloud GPUs, Groq LPU instances, or Cerebras wafer-scale clusters) running optimized 8B-to-70B parameter models with FP8 quantization and KV-cache pre-allocation. These engines emit the first response token in under 120ms.

Component 4: Chunked Streaming Text-to-Speech (90ms to 160ms). Waiting for the LLM to finish generating a 30-word response before synthesizing audio adds another 1,500ms of latency. Streaming TTS engines (such as Cartesia Sonic, ElevenLabs Turbo v2, or FastPitch implementations) begin synthesizing audio as soon as the first complete clause (typically 4 to 8 words) is emitted by the language model. The first playable audio chunk is emitted within 100ms of receiving the initial text tokens.

Component 5: Network Transport and Telephony Buffering (30ms to 60ms). Transmitting audio over standard WebRTC UDP tracks introduces minimal network latency (typically 20ms to 40ms under normal broadband or 5G conditions), plus a 20ms jitter buffer. Summing the cascade: 100ms (VAD) + 150ms (STT) + 120ms (LLM TTFT) + 100ms (TTS TTFT) + 30ms (Transport) yields a total speech-to-speech roundtrip latency of approximately 500ms. Achieving this benchmark requires flawless synchronization across every transport boundary.

References: LiveKit — Real-time Voice and Agent ArchitectureIETF RFC 8829 — JavaScript Session Establishment Protocol (JSEP / WebRTC)

Cascaded pipelines versus native Speech-to-Speech (S2S) models

A critical architectural decision facing engineering leaders in 2026 is choosing between a cascaded pipeline (STT + LLM + TTS) and an end-to-end native Speech-to-Speech (S2S) model, such as GPT-4o Realtime or Gemini Live. Both paradigms present profound architectural trade-offs.

References: LiveKit — Real-time Voice and Agent ArchitectureGoogle Search Central — AI features and your website

Barge-in mechanics: handling interruptions without state corruption

In human conversation, turn-taking is not strictly sequential; listeners frequently interrupt, interject, or acknowledge understanding while the other party is still speaking. A voice agent that cannot be interrupted—one that relentlessly speaks over a caller for twenty seconds while the caller shouts ‘Stop, that’s not what I meant’—is an operational failure.

Implementing real-time interruption handling (commonly known as barge-in) requires coordinating three distinct sub-systems: Acoustic Echo Cancellation (AEC), frame-level voice detection, and instant buffer purge.

First, consider the echo problem. When the voice agent speaks through the caller's phone speaker, that audio is picked up by the phone microphone and transmitted back to the server. If the server cannot distinguish between the agent's own reverberated voice and the caller's new speech, the agent will interrupt itself after uttering its first word. Implementing robust hardware-level or software-level Acoustic Echo Cancellation (via WebRTC APM or carrier echo cancellers) is an absolute prerequisite for reliable barge-in.

Second, consider interruption detection. While the agent is actively streaming audio to the client, the server continuously processes incoming audio frames through a rapid VAD classifier. The moment human vocal energy is detected for more than 60ms to 80ms (filtering out transient background clicks or coughs), the server declares a barge-in event.

Third, consider state rollback and buffer clearance. The instant barge-in triggers, the server must execute three atomic operations in under 40 milliseconds: send a cancellation message over the WebRTC data channel to immediately mute and discard all buffered client-side audio; terminate the downstream TTS synthesis task; and send an abort signal to the language model generation stream. Furthermore, if the cancelled turn had initiated a speculative database mutation or tool call, the state machine must safely roll back the transaction to maintain data integrity.

References: IETF RFC 8829 — JavaScript Session Establishment Protocol (JSEP / WebRTC)LiveKit — Real-time Voice and Agent Architecture

Deterministic tool calling in voice: avoiding conversational deadlocks

In text-based AI applications, executing a tool call that takes 1.5 seconds to query a backend database or CRM API is imperceptible to the user. In a voice conversation, 1.5 seconds of dead air on a telephone line feels like an eternity. Callers immediately assume the call has disconnected or the system has crashed.

This architectural constraint creates a dilemma: enterprise voice agents must execute real business logic (such as booking appointments, verifying account balances, or dispatching technicians) without introducing silent deadlocks into the conversation. Production engineering teams resolve this using three core architectural patterns.

Pattern 1: Speculative Pre-Fetching. When a caller begins describing an inquiry—for example, ‘I need to schedule a service visit for my property in North Austin’—the agent’s semantic routing layer initiates an asynchronous background lookup of Austin service technician schedules before the caller has even finished speaking. By the time the user completes their turn and asks ‘Do you have anything open on Thursday?’, the relevant calendar records are already cached in local memory, allowing the model to respond in 300ms without an in-turn API delay.

Pattern 2: Conversational Fillers and Asynchronous Acknowledgments. When a backend query cannot be pre-fetched and requires more than 600ms of execution time, the agent must acknowledge the task verbally before executing the blocking call. The agent emits a natural conversational filler—such as ‘Let me pull up that account record right now’ or ‘Checking availability for Thursday morning...’—which immediately begins audio playback to satisfy the user’s temporal expectation. The actual database query executes concurrently with audio playback, so that the subsequent confirmation turn is ready the instant the filler phrase concludes.

Pattern 3: Decoupled Post-Call Mutations. Whenever possible, heavy state mutations—such as generating PDF invoices, sending calendar invite emails, or syncing CRM opportunities across multiple third-party webhooks—should be completely decoupled from the live call audio loop. The voice agent verifies the required parameters with the caller, writes a lightweight intake record to an in-memory queue, confirms the action verbally, and offloads the heavy integration work to asynchronous background workers after call teardown.

References: LiveKit — Real-time Voice and Agent ArchitectureGoogle Search Central — Creating helpful, reliable, people-first content

Telephony infrastructure: bridging SIP trunks to WebRTC agent runtimes

While modern web browsers and mobile applications communicate effortlessly over WebRTC, the overwhelming majority of commercial enterprise voice traffic still originates from traditional telecommunications networks via SIP (Session Initiation Protocol) trunks and Public Switched Telephone Networks (PSTN). Bridging legacy telephony to low-latency AI agent runtimes requires a specialized media gateway layer.

In traditional telecommunications, phone calls operate over carrier SIP trunks using narrowband G.711 codecs (8kHz sampling rate) with rigid packetization intervals (typically 20ms RTP packets). Modern speech models and neural audio synthesizers, by contrast, operate at wideband or fullband sampling rates (16kHz, 24kHz, or 48kHz Opus) over UDP-based WebRTC media streams. A telephony gateway (such as LiveKit SIP Gateway, Kamailio, Asterisk, or FreeSWITCH) sits at the perimeter of the infrastructure to negotiate this translation.

The gateway performs three essential real-time transformations: it negotiates session establishment via RFC 3261 SIP signaling, bridges RTP carrier audio into encrypted WebRTC media tracks (RFC 8829), and executes real-time codec resampling between G.711 and 16kHz Opus. Crucially, the gateway must manage jitter buffers dynamically: network latency on cellular networks can vary wildly from packet to packet, and a poorly tuned jitter buffer will either introduce artificial delay or drop audio frames, resulting in robotic speech and degraded transcription accuracy.

Furthermore, telephony gateways must handle telecommunication-specific signaling events that do not exist in web browsers. This includes Dual-Tone Multi-Frequency (DTMF) touch-tone keypress detection (essential for callers entering credit card numbers, account IDs, or menu selections in noisy environments), carrier hangup detection, and answering machine detection (AMD) for outbound notification workflows.

References: IETF RFC 3261 — SIP: Session Initiation ProtocolIETF RFC 8829 — JavaScript Session Establishment Protocol (JSEP / WebRTC)LiveKit — Real-time Voice and Agent Architecture

Audio transport physics: why WebRTC UDP defeats TCP WebSockets under packet loss

A subtle engineering trap that undermines many voice agent prototypes is relying on standard TCP WebSockets for real-time audio transport. In controlled office environments with fiber internet connections, WebSockets appear to work adequately. In production mobile environments, where callers are driving through cellular dead zones or walking across Wi-Fi handoffs, TCP transport causes catastrophic failure.

The reason is rooted in transport protocol physics. TCP enforces guaranteed in-order packet delivery through head-of-line blocking. If a single audio packet is dropped due to cellular jitter, the TCP stack halts all subsequent packet processing while waiting for the dropped packet to be retransmitted. In a file download, this is desirable; in live human conversation, it is fatal. A 300ms packet retransmission delay freezes the audio stream, followed by an unnatural burst of sped-up audio as buffered packets are delivered all at once.

WebRTC, standardized in RFC 8829, solves this by running audio exclusively over UDP via the Secure Real-time Transport Protocol (SRTP). UDP prioritizes timeliness over completeness: if an audio packet is dropped, the protocol simply discards it and continues playing the next arriving frame. WebRTC implementations use sophisticated Packet Loss Concealment (PLC) algorithms and Forward Error Correction (FEC) within the Opus codec to mathematically interpolate missing audio waveforms, resulting in seamless, glitch-free audio even under 15% to 20% packet loss.

Furthermore, WebRTC incorporates dynamic bandwidth estimation and congestion control (such as Google Congestion Control or BBR for media). When cellular signal quality degrades, the WebRTC media engine automatically downsamples the Opus bitrate from 32kbps to 12kbps without dropping the call or increasing conversational round-trip latency.

References: IETF RFC 8829 — JavaScript Session Establishment Protocol (JSEP / WebRTC)LiveKit — Real-time Voice and Agent Architecture

Audio sentencizing and prosody: balancing chunk sizes against acoustic realism

In streaming text-to-speech synthesis, there is a fundamental acoustic tension between low latency and natural prosodic inflection. If your streaming pipeline sends text tokens to the TTS engine word by word, the model has zero context about the overall sentence structure. It cannot predict whether a phrase is a question, a command, or an empathetic observation. The resulting voice output sounds like a disjointed, robotic GPS navigation voice from 2005.

Conversely, if the pipeline waits for the language model to complete a full 35-word grammatical sentence before sending it to the TTS synthesizer, conversational latency explodes. Even on the fastest hosted inference engines, generating 35 tokens requires 400ms to 600ms of compute time, completely consuming your latency budget before the first audio byte is synthesized.

Production voice architectures resolve this tension through a dynamic sentencizer algorithm. The sentencizer operates on an adaptive chunking threshold: for the very first audio chunk of an agent's turn, it aggressively triggers synthesis on the first sub-clause boundary (typically 4 to 7 words, or the first comma, dash, or conjunction). This ensures that initial audio playback begins in 100ms to 140ms, satisfying the caller's temporal expectation.

While the first audio chunk is playing over the phone speaker, the sentencizer switches to larger, full-sentence chunking thresholds (12 to 20 words) for all subsequent turns in that passage. Because the user is already listening to audio playback, the system has a generous buffer of 1.5 to 2.5 seconds of playback time to synthesize the remainder of the response. This hybrid chunking strategy delivers the holy grail of Voice AI engineering: instant conversational turn-taking combined with rich, human-grade prosodic realism.

References: LiveKit — Real-time Voice and Agent ArchitectureGoogle Search Central — AI features and your website

The human escalation protocol: seamless warm transfers

A foundational principle of enterprise Voice AI engineering is the recognition that automated agents must never become conversational dead ends. When a caller presents a complex edge case, displays high emotional distress, or explicitly requests to speak with a human, the system must execute an immediate, graceful escalation.

In production telephony architectures, this is implemented using the SIP REFER protocol standard (RFC 3515) to execute an attended warm transfer. When the escalation threshold is triggered, the AI agent does not simply drop the caller into an unmonitored queue. The system places the caller on a brief polite hold, bridges a new outbound leg to your corporate PBX or contact center queue (such as Genesys, Five9, or Twilio Flex), and transmits a structured context payload.

The context payload—typically delivered via an internal webhook or SIP user-to-user headers—carries the full structured transcript of the interaction, the caller's authenticated identity, the specific unresolved intent, and a concise 3-bullet executive summary generated by the agent. When the human agent answers the phone, their computer screen immediately displays the complete context, allowing them to greet the caller by name and resolve the problem without forcing the customer to repeat their entire story from the beginning.

Beyond passing structured intake context, a robust warm transfer architecture requires carrier-grade exception handling. If the destination corporate PBX queue returns a SIP 486 Busy Here, 503 Service Unavailable, or fails to answer within a configurable 20-second timeout, the voice agent must catch the SIP signaling error, resume the audio bridge with the caller, apologize gracefully, and offer to schedule an automated callback or dispatch an immediate SMS notification. Under no circumstances should an escalation failure result in an abrupt call disconnection or silent line drop.

References: IETF RFC 3261 — SIP: Session Initiation ProtocolGoogle Search Central — Creating helpful, reliable, people-first content

Decision matrix: choosing your voice architecture for enterprise operations

Use the following engineering decision matrix to select the appropriate voice architecture, model tier, and telephony infrastructure for your specific operational requirements.

References: LiveKit — Real-time Voice and Agent ArchitectureIETF RFC 8829 — JavaScript Session Establishment Protocol (JSEP / WebRTC)IETF RFC 3261 — SIP: Session Initiation Protocol