AI Voice AI Systems

Watch a spoken question travel through speech recognition, language understanding and speech synthesis, then follow a natural back-and-forth conversation with an assistant.

A real-time voice AI agent works in a loop: it captures audio, detects the end of the user's turn with voice activity detection, transcribes the speech (ASR), reasons and calls tools with a language model, synthesises the reply (TTS), and stops speaking immediately if the user interrupts (barge-in). A real-time voice assistant is a streaming pipeline wrapped in a turn-taking state machine. Microphone audio is cut into short frames and streamed to a speech recogniser while the user is still talking; a voice-activity detector and an endpointing rule decide when the user has finished; a language model (usually with tools) decides what to say or do; a speech synthesiser turns the reply into audio in small chunks so playback can begin before the reply is complete; and the whole loop must stop talking within a fraction of a second if the user cuts in. Two things separate voice from text chat. Time is part of the interface: a delay that is invisible in a chat window is audible as an awkward silence. A voice agent is also not a transcription service: transcription converts audio to text and stops, while a voice agent must decide when to answer, act, and yield the floor (the recognition model itself is taught in Audio to Text). And the channel is lossy and sequential: the recogniser can mishear a name or a number, the listener cannot skim back, and the assistant cannot show a confirmation button.

This lesson follows one realistic conversation through the system, then examines each stage: the audio input path, the two broad architectures, the conversation and tool layer, speech output, the turn-taking state machine, a worked latency budget, the transport, accuracy, privacy, evaluation, and a reference implementation. The latency and accuracy numbers throughout are illustrative values chosen to make the arithmetic traceable, not measurements of any product or vendor. The companion visual lesson shows the basic listen-understand-speak loop; here we are concerned with what makes that loop work live.

A traveller says, to an airline's phone assistant: "Check my flight... actually tomorrow, not today." The sentence contains a pause and a self-correction, and a text chatbot never has to deal with either. The user types a complete message and presses send; the end of the turn is explicit. In speech the end of the turn is inferred. Suppose the speaker says "Check my flight" over 1,100 ms, pauses for 380 ms while deciding how to phrase the correction, and then says "actually tomorrow, not today" over the next 1,420 ms. If the system declares the turn finished after 200 ms of silence, it fires at t = 1,300 ms with the transcript "check my flight" and starts working on the wrong request. The user resumes at t = 1,480 ms. The assistant must now notice that a turn it thought was over is not over, throw away the work in flight, and wait for the real request.

Everything in a voice system is built around handling this kind of ambiguity in real time. Streaming recognition produces partial transcripts (hypotheses that may still change: "check my flight" could become "check my flights") and later a final transcript for a finished utterance. Turn-taking logic decides when the user is done. Barge-in is the user speaking while the assistant is speaking, and the assistant is expected to stop almost immediately, because a person who is interrupted by an assistant that keeps talking has no way to regain control other than shouting. And because every stage can be wrong, the system confirms anything it is about to act on. Later we will trace this same conversation through the state machine and a timeline with exact timestamps.

There are two broad ways to build the middle of the system. In a cascaded architecture, three components with narrow jobs are chained: automatic speech recognition (ASR) turns audio into text, a language model or dialogue manager decides on a reply (calling tools where needed), and text-to-speech (TTS) turns the reply into audio. In a speech-native (end-to-end) architecture, a single multimodal model accepts audio and produces audio, and may also produce text and tool calls as side outputs. The conceptual background on models that take audio as input is in Multimodal AI and Audio to Text.

The trade-offs are real but not one-sided, and the right choice depends on what the product values.

Observability and control. The cascade gives you inspectable seams. You can store the transcript, run a regex or classifier over it for PII, unit-test the dialogue layer with text fixtures, swap the ASR vendor for a domain-adapted one, and apply a deterministic policy between "model proposed a refund" and "refund executed". A speech-native model offers fewer natural seams, so more of the behaviour is governed by the model itself; you can still capture transcripts if the model emits them, but you should check whether the transcript reflects what the model actually "heard" or is produced by a separate recogniser. Either way, tool authorisation belongs in the application, not in the model.

Latency. The cascade pays for serial hand-offs and must be engineered with streaming overlap (covered in the latency section). A speech-native model can in principle skip the explicit text boundaries and react to audio directly, which may shorten the path, but its first-audio latency still depends on the model, the serving stack and the network, so no architecture is faster by definition.

Voice quality and paralinguistics. A cascade discards everything in the audio that is not in the words: tone, hesitation, emphasis, laughter. A speech-native model can in principle perceive and produce these. Conversely, a dedicated TTS system can offer a consistent branded voice and fine-grained voice settings, which a general speech model may or may not expose.

Deployment complexity. The cascade is three services with three failure modes, three scaling profiles and an orchestrator in the middle, but each is a mature component you can version independently. The native model is one dependency, but a larger blast radius when it misbehaves, and fewer ways to patch a specific failure (such as a mispronounced product name) without waiting for the model provider.

There is no universal winner, and many production systems are hybrids: a speech-native model for the conversational loop while a separate recogniser produces an auditable transcript, or a cascade with a streaming ASR that feeds partial transcripts into an LLM before the user finishes. The rest of this lesson uses the cascade as the running example because its seams are explicit and therefore easiest to teach; the turn-taking machine, latency reasoning, transport, privacy and evaluation concerns apply to both.

Frames, sample rate and encoding

A microphone produces a continuous waveform which is sampled at a fixed rate. Speech recognisers commonly work with 16,000 samples per second (16 kHz), because most of the information in speech lies below 8 kHz; telephone networks use 8 kHz. The unit of streaming is a frame, typically 10 to 30 ms. For illustrative arithmetic: 16 kHz, 16-bit mono PCM is 16,000 samples/s × 2 bytes = 32,000 bytes/s, or 256 kbit/s, and a 20 ms frame holds 320 samples = 640 bytes. Narrowband telephony at 8 kHz with 8-bit companded encoding is 8,000 × 8 = 64 kbit/s with 160-byte 20 ms frames. Lossy speech codecs reduce these rates considerably at some cost in fidelity. The sample rate and codec are not cosmetic: a recogniser trained on 16 kHz audio that receives up-sampled 8 kHz telephone audio has lost the high-frequency detail that separates, for example, an "f" from an "s", and its accuracy on that channel will be worse than on a headset.

Cleaning the signal

Before recognition the audio is typically conditioned: echo cancellation removes the assistant's own voice from the microphone signal (essential on speakerphone, otherwise the assistant "hears" itself and triggers its own barge-in), noise suppression attenuates steady background sound, and automatic gain control evens out volume. These steps help, but they are not free: aggressive noise suppression can clip quiet consonants and make recognition worse, so the pre-processing should be evaluated against recognition accuracy on realistic audio rather than against how pleasant the cleaned audio sounds.

VAD, endpointing and partial transcripts

Voice activity detection (VAD) classifies each frame as speech or non-speech. It answers "is someone talking right now?" and nothing more. Endpointing is the decision built on top of it: "has the user finished their turn?" A simple endpointer declares the turn over after a fixed silence (for example 200 ms, 500 ms or 700 ms); better ones combine silence with the content of the partial transcript, so that "I'd like to book a flight to" is treated as unfinished while "yes" is treated as complete.

Streaming ASR consumes frames as they arrive and emits events: interim (partial) results that are revised as more audio arrives, and a final result when the recogniser commits a segment. For our traveller, the stream looks like this (illustrative event log, times in ms from the start of the utterance):

t=   420  interim  "check"
t=   700  interim  "check my fly"
t=   980  interim  "check my flight"
t=  1100  vad      speech_end
t=  1300  endpoint silence=200ms             <- premature: user was only pausing
t=  1480  vad      speech_start               (raw VAD flips back to speech)
t=  1540  vad      speech_confirmed           (min speech 60 ms, filters clicks)
t=  2000  interim  "check my flight actually tomorrow"
t=  2900  vad      speech_end
t=  3100  endpoint silence=200ms
t=  3250  final    "check my flight actually tomorrow not today"

Note two details. The interim text "check my fly" is not wrong, it is early, and downstream components must treat interim text as provisional. And the transcript has no punctuation or self-repair markup: "actually tomorrow, not today" appears as a continuation of the sentence. Understanding that "actually" replaces the earlier constraint is the dialogue layer's job, not the recogniser's.

The endpointing trade-off

The silence threshold trades responsiveness against interruption. A short threshold (such as 200 ms) lets the assistant answer quickly, but speakers routinely pause for longer than that mid-sentence to think, to breathe, or between "check my flight" and a correction. Each such pause becomes a false endpoint: the assistant starts to respond to half a request, and if it speaks, it talks over the user. A long threshold (such as 700 ms) is safe against that but adds 500 ms to every single turn, which users experience as lag. No fixed value is right for every utterance, because the cost of waiting is the same but the likelihood of an unfinished thought varies: after a complete-sounding "yes" the user is almost certainly done, while after "to" they almost certainly are not.

Slide the threshold across the traveller's 380 ms pause to see where the false endpoint appears and what each choice costs every turn.

Practical systems therefore use adaptive endpointing (shorter thresholds when the partial transcript looks grammatically and semantically complete, longer when it ends mid-phrase), and they make the cost of a false endpoint small. The key design move is that work started at an endpoint must be cancellable and, until the assistant actually speaks, free of side effects. A false endpoint that only started a read-only lookup and an LLM call is cheap to cancel; one that spoke or modified a booking is not. This asymmetry is the reason the tool layer later distinguishes read tools from write tools.

The pipeline stages are easy to draw as a line; the hard part is that the line can be interrupted at any point. The orchestrator that sits above ASR, model and TTS is best written as an explicit state machine, because "what should happen when the user starts speaking right now?" has a different answer in every state. Seven states are enough for most assistants.

The transitions that matter are defined by what is cancelled when they fire:

LISTENING to USER_SPEAKING happens on confirmed speech (VAD with a minimum duration so a door slam does not count). ASR streaming begins and any idle timers are cleared.

USER_SPEAKING to PROCESSING happens at the endpoint. The final transcript is assembled, the conversation context is built and the model request is issued. Nothing has been said aloud yet, so everything started here is cancellable at no user-visible cost.

PROCESSING to USER_SPEAKING is the false-endpoint repair. When speech resumes while the model is still working (as at t = 1,540 ms in our trace), the in-flight model request is aborted through its cancellation token, partial model output is discarded, and the next final transcript replaces the earlier one. A robust system also merges the two segments into one user turn, otherwise the conversation history records two fragments ("check my flight", "actually tomorrow not today") instead of the single request the user made.

PROCESSING to WAITING_FOR_TOOL and back models the tool round trip. While waiting, the user may still speak, so the machine must accept speech here too; a good assistant may say a brief filler ("one moment") if the tool is slow, but that filler must itself be interruptible.

ASSISTANT_SPEAKING to INTERRUPTED is barge-in. Five things must happen, in this order of importance: stop playback and flush the client's audio buffer (audio already sent but not yet played would otherwise keep coming out); abort TTS generation and the model stream; record which part of the reply the user actually heard; mark any not-yet-executed plan as cancelled; and hand the floor to the user. The recorded "heard so far" text matters. If the assistant planned "Your flight tomorrow departs at 9:40 and is on time. I can also move you to the 14:15 departure." and was cut off after "departs", the conversation history should say that only "Your flight tomorrow departs" was spoken. Otherwise the model will later assume the user already knows the departure is on time.

ERROR collects ASR disconnects, model timeouts, TTS failures, and tool errors. Recovery means saying something short and honest ("Sorry, I lost you for a moment. Could you say that again?") rather than silence, and returning to LISTENING with a clean cancellation state.

Two refinements are worth having from the start. First, not every sound is a barge-in: "mm-hm" backchannels, coughs and a second speaker in the room should not stop the assistant. A common approach is to require a minimum speech duration (or some recognised words) before cancelling, which is a deliberate trade between stopping promptly and stopping for the wrong reasons. Second, every transition should carry a turn id, so that a late result from a cancelled turn (a TTS chunk that was already in flight) is recognised as stale and dropped instead of being played.

Once a final transcript exists, the voice system looks much like a chat assistant with a tool layer, and the mechanics of conversation state, system prompts and tool calling apply unchanged. What changes is the input quality and the cost of mistakes. The transcript is noisy and unpunctuated, references are spoken rather than typed, and there is no screen on which to show options.

Reference resolution and self-correction

"Check my flight... actually tomorrow, not today" contains three things the dialogue layer must resolve. "My flight" is not an identifier: it means one of the bookings attached to the authenticated caller. "Tomorrow" and "today" are relative dates that need the caller's time zone and the current date. And "actually... not today" retracts a constraint rather than adding one. A model given the whole transcript plus the context (customer id, current date in the customer's zone, list of upcoming bookings) handles this well, and the output should be a structured tool call rather than free text, so that the application can validate it:

{
  "tool": "get_flight_status",
  "arguments": {
    "booking_id": "bk_8841",
    "travel_date": "2026-10-29"
  }
}

When the references are genuinely ambiguous, such as when the caller has two flights tomorrow, the correct move is a short clarifying question, offered with a bounded choice: "You have two flights tomorrow, one to Pune at 9:40 and one to Delhi at 18:05. Which one?" Voice favours closed questions with at most two or three options, since the listener has to hold them in memory.

Read tools and write tools are different risk classes

Looking up a flight status is a read: it has no side effect, it is idempotent, and it can be issued speculatively and cancelled. Changing the booking is a write: it has consequences, it may cost money, and it must not be triggered by a transcript the system is not sure about. The distinction is a property of the tool registry, enforced by the application. A prompt that says "never change a booking without confirmation" is guidance to the model; the guarantee has to be in code that refuses to execute a write tool unless a confirmation state is present for exactly that action.

Voice sharpens the need. Recognition errors on names, dates and numbers are common enough that "the user said it" is weaker evidence than "the user typed it". For the write path the usual design is: the model proposes an action; the application validates it against the caller's authorisation and business rules; the assistant reads the exact action back; the caller confirms; and only then does the application execute with an idempotency key and speak the result it actually received, not the result it expected. Human-in-the-loop and guardrail patterns describe the general approach; voice adds the constraint that the read-back must be short and the confirmation must be unambiguous.

Permission gates: lookup versus an unauthorised change

Suppose a second caller phones in and says: "Cancel the booking for Anita Rao, reference K7Q2LM." The recogniser may transcribe the name and reference perfectly, and the model may produce a perfectly formed cancel_booking call. It must still not execute, because nothing in the transcript proves that this caller is entitled to act on that booking. Authorisation is decided from the session's authenticated identity (for example a customer logged in through the app that opened the call, or a one-time code sent to the phone number on file), checked against ownership of the booking id on the server. A sound voice, a known name, or the caller reciting a booking reference is not authentication; references are routinely shared on itineraries. The orchestrator's response to the unauthorised request is deterministic: refuse the tool call, say what the caller can do (verify identity through the app, or be transferred to an agent), and log the attempt. The model's text about what it "can't do" is cosmetic; the denial is in the tool-execution layer. See AI Security for the broader view, and AI Agents in Business for how bounded actions are designed.

Text-to-speech is not a single call that returns a finished file. In a live system it is a stream, and the shape of that stream determines both how soon the user hears something and how cleanly the assistant can stop.

Chunking: word by word, sentence by sentence, or whole reply

An LLM generates text incrementally, so the TTS can start before the reply is finished. The question is how much text to hand over at once. Synthesising the whole reply gives the best prosody but makes time-to-first-audio proportional to reply length: a 14-word reply at about 150 words per minute is 14 / 150 × 60 = 5.6 s of speech, and generating it all first delays the first sound by the full synthesis time. Synthesising word by word minimises delay but ruins the intonation: a speaker cannot give "tomorrow" the right stress without knowing whether it ends a statement or a question, and the joins between separately synthesised words are audible. Clause or sentence chunks are the usual compromise: wait for the first natural boundary (comma, full stop, or a short word-count limit), synthesise that, and keep a small look-ahead so the next chunk is ready before the first finishes playing.

The first chunk should therefore be short and written for speaking: the response policy can instruct the model to begin with a brief direct answer ("Tomorrow's flight is on time.") followed by detail. This is a system-design choice as much as a prompt choice, because it shapes how early synthesis can start.

Buffering and backpressure

Synthesis is usually faster than real time. If the TTS generates audio at, say, 5 times real time (illustrative), the 5.6 s reply is ready in about 1.1 s while playback takes 5.6 s. Pushing all of it to the client immediately causes two problems. The client holds seconds of audio that will be useless if the user interrupts, and the server has done work (and may have paid for it) that is thrown away. The remedy is bounded look-ahead: keep, for example, about 1 s of audio ahead of the playhead and apply backpressure to the synthesiser beyond that. A smaller buffer makes interruption cleaner and cheaper; a larger one protects against network jitter. Chosen well, it is a few hundred milliseconds above the jitter you actually observe.

Prosody, language and voice settings

Voice, speaking rate, language and (where the TTS supports it) style are per-utterance settings. Numbers, dates and identifiers need explicit handling because TTS systems normalise them differently: "9:40" might be read as "nine forty", "nine hours forty" or "nine colon forty" unless the text is normalised first, and a booking reference like "K7Q2LM" should be spelled out with pauses. It is common to convert such fields to speech-friendly text in the application (a deterministic formatter) rather than hoping the model writes them that way. For a multilingual assistant, the TTS voice must match the language of the reply, which means the orchestrator tracks the conversation language and switches voice (or uses a multilingual voice) explicitly.

Cancelling speech cleanly

The server-side cancellation described in the state-machine section has a client half. The client must stop playback and empty its queue on a cancel event, and it must report how much of the current audio was actually played so that the transcript of the assistant turn can be truncated to what the user heard. If synthesis is chunked and the text-to-audio alignment is known, "audio played = 2,330 ms" maps back to a character position in the reply; if alignment is not available the safest approximation is to truncate at the last fully played chunk. An orchestrator that cannot reconstruct what was heard should say so in the history rather than assume the whole reply was delivered.

The figure shows how the look-ahead and the client flush together decide how long the assistant keeps talking after a barge-in.

Conversational turn gaps between people are short, on the order of a few hundred milliseconds, so users judge a voice assistant by how soon it begins to speak after they stop. The metric to manage is time to first audio: the interval from the end of the user's speech to the first sound they hear. All numbers below are illustrative values chosen to make the arithmetic clear. They are not benchmarks, and your own stack must be measured.

The serial pipeline

Suppose each stage waits for the previous one to finish: endpointing 200 ms (the silence the system waits before deciding the user is done), ASR finalisation 150 ms (the recogniser commits the last words), model first response 450 ms (time until the first sentence-sized chunk of reply), and TTS first audio 180 ms. The total is 200 + 150 + 450 + 180 = 980 ms. This excludes the network and the playout buffer, which we estimate separately below.

Where the time can be recovered

Overlap model start with ASR finalisation. The recogniser's last partial transcript is usually identical to its final one. The orchestrator can start the model call at the endpoint using the stable partial transcript, while finalisation completes in parallel. If the final transcript matches the one used, nothing is lost; the timeline becomes 200 (endpoint) + 450 (model) + 180 (TTS) = 830 ms, a saving of the 150 ms finalisation. If the final transcript differs, the speculative call is cancelled and restarted from the final text: 200 + 150 + 450 + 180 = 980 ms, no worse than serial (ignoring the wasted compute). If the speculation matches 80 % of the time (an assumption), the expected time to first audio is 0.8 × 830 + 0.2 × 980 = 664 + 196 = 860 ms.

Stream the model into TTS. The "model first response" of 450 ms already assumes that TTS starts on the first chunk rather than after the whole reply. If the model needed to finish a 60-token reply before TTS started, the model segment would be much longer. An illustrative decomposition of the 450 ms is 250 ms to the first token (prompt processing and queueing) plus 200 ms to generate the 8 tokens of a first clause at 40 tokens per second (8 / 40 = 0.2 s). Prompt size, prefix caching and model size all feed the first component; see Inference Infrastructure and Caching in AI Systems.

Shorten the endpoint carefully. Dropping the silence threshold from 200 ms to, say, 120 ms saves 80 ms on every turn but increases false endpoints, each of which costs a cancelled model call and possibly a talk-over. That is a trade to evaluate with the false-endpointing metric (below), not a free win.

Add the network and playout. The user hears the sound later than the server generates it. With an illustrative 40 ms uplink, 40 ms downlink and a 60 ms jitter buffer (three 20 ms packets), the user-perceived time is the server figure plus 140 ms: 980 + 140 = 1,120 ms serial, or 830 + 140 = 970 ms overlapped (when the speculation hits). The uplink delay really sits at the front of the chain (the audio must reach the server before it can be recognised), but for an order-of-magnitude budget the sum is what matters.

Mask unavoidable delay honestly. If a tool call takes 800 ms, the assistant can say a short acknowledgement ("Let me check that.") while it waits. That does not reduce the true latency, and it should not be used to hide delays longer than the filler lasts, but it keeps the line from feeling dead. The filler is itself a TTS stream and must be cancellable on barge-in.

With the state machine and latency model in hand, we can replay the conversation with the timestamps from the opening. Times are milliseconds on the session clock and are illustrative; stage durations use the serial budget so the arithmetic is easy to follow.

t = 1,300. After 200 ms of silence the endpointer fires. The orchestrator moves to PROCESSING with the transcript "check my flight" and issues a model request. Because this is a read-only situation the orchestrator may also prefetch the caller's upcoming bookings (a caller-scoped read, with no date filter, so the result remains useful whatever the user says next). The prefetch returns by about t = 1,420.

t = 1,480 to 1,540. Raw VAD flips back to speech at 1,480, and speech is confirmed at 1,540 once it has lasted 60 ms. The orchestrator moves PROCESSING to USER_SPEAKING, aborts the in-flight model request via its cancellation token and marks turn 1's output stale. The prefetched bookings stay cached (a read, harmless, and still valid). Nothing was said aloud, so the user sees no effect, apart from the system's compute being wasted.

t = 2,900 to 3,250. The speaker finishes; 200 ms later the endpoint fires (3,100); ASR finalisation completes at 3,250 with "check my flight actually tomorrow not today".

t = 3,250 to 3,880. The model, given the transcript plus date context and the cached bookings, resolves "tomorrow" to 2026-10-29 and calls get_flight_status for the matching booking (the result is part of the 450 ms model segment in this illustration). The first reply chunk ("Tomorrow's flight to Pune, EX214, is on time.") goes to TTS, and first audio plays at 3,880 ms: 3,880 − 2,900 = 980 ms after the user stopped speaking, matching the serial budget.

t = 6,100 to 6,210. While the assistant is saying the second chunk the user speaks again ("wait, the later one"), beginning at 6,100. VAD confirms speech at 6,160. The orchestrator moves to INTERRUPTED: the TTS stream and model stream are aborted, a cancel message tells the client to stop and flush playback, and playback stops at about 6,210 ms, roughly 50 ms after confirmation and 110 ms after the user's first sound (assumed: 30 ms for the cancel to propagate and 20 ms to flush). The client reports the amount played; the history is truncated to the heard text, any undelivered offer in the cancelled plan is discarded, and the machine moves to USER_SPEAKING with the new request, which will enter the write path (read-back and confirmation) as described earlier.

A voice session carries two different kinds of traffic that should not be confused. The media plane is continuous audio in both directions, which is latency-sensitive and tolerant of small losses: a dropped 20 ms frame is a glitch, a late frame is useless. The control plane carries application events (ASR transcripts, state changes, tool results, cancel messages, playback reports), which are small, must arrive reliably and in order, and are not substituted by retransmitting audio.

WebRTC versus WebSocket

WebRTC was designed for real-time media between browsers and servers: it uses UDP-based transport with congestion control, built-in echo cancellation and codec negotiation in browsers, and a jitter buffer on the receiving side. A late packet is discarded rather than stalling the stream, which is the right behaviour for audio. It adds signalling and NAT-traversal machinery to operate. A WebSocket runs over TCP: it is simple to deploy through ordinary infrastructure and gives an ordered, reliable stream, but TCP retransmits lost packets and delivers them in order, so one lost packet can hold back everything behind it (head-of-line blocking). On a good network the difference is small; on a lossy mobile link it appears as bursts of delay. Many server-side voice agents use WebSockets for simplicity, particularly for server-to-server telephony bridges, while browser-facing products often prefer WebRTC for the media path. This comparison is a generic description of the two transports, not a statement about any specific vendor's API.

Loss, jitter and reconnection

Packets arrive with variable delay (jitter). The receiver holds a small jitter buffer, which reorders packets and absorbs delay variation: with 20 ms frames, a 60 ms buffer holds three frames and tolerates up to about 60 ms of delay variation at the cost of 60 ms of added latency. Frames that arrive too late are dropped or concealed. Adaptive buffers grow when the link is bad and shrink when it recovers. For inbound audio the equivalent problem is that the ASR receives gaps; the recogniser should be given silence or concealment frames to keep timing aligned rather than a stream that silently compresses time.

Connections drop. On reconnect, the client needs a session id so the server can resume with the existing conversation state, and the machine must have a defined behaviour for a call that dropped mid-turn: discard the partial turn, speak a short recovery message, and never replay a write action that was awaiting confirmation. A pending confirmation should expire after a short time, because a "yes" arriving after a reconnect may refer to something else entirely.

Finally, synchronisation across the two planes. Control events should carry the turn id and, where useful, a position on the audio timeline, so a "stop playing" cancel (which travels on the control plane) applies to the audio of the turn it was intended for and does not clip the start of the next. Unbounded queues on the downlink are the typical cause of "the assistant kept talking for two more seconds after I interrupted": the server sent audio faster than real time, the client buffered it, and the cancel only stopped new audio from arriving. Bounded look-ahead, described under speech output, plus an explicit client flush fixes it.

Recognition errors are inevitable and uneven. The system should assume they will occur and be designed so that an error in a low-stakes word is harmless while an error in a critical one is caught.

Accents, dialects and noise. Accuracy varies with how well the training data represents the speaker and channel. A recogniser can perform very differently for different accents, ages and recording conditions, so evaluating on an average across all callers hides the groups for which the product fails. Background noise, music and overlapping speech degrade accuracy, and echo from speakerphones is a classic self-inflicted cause. Evaluate on audio that resembles your real channel (including 8 kHz telephony if that is what you serve).

Proper names, numbers and disfluency. Names, product terms and flight numbers are rare words, and general recognisers often turn them into common ones. Many recognisers accept custom vocabulary or phrase hints so that known terms (airport codes, product names) can be boosted; check what your provider supports. Numbers are particularly dangerous: "forty" and "fourteen" differ by a small acoustic cue, and "nine forty" can also become "nine fourteen". Speech also contains fillers, restarts and self-corrections ("uh, the, the ninth, I mean the tenth"); the dialogue layer should be robust to them rather than expecting a clean sentence.

Code-switching and multilingual speech. Bilingual speakers switch languages mid-sentence ("book karo ek ticket for tomorrow"). A recogniser configured for one language may garble the other part, and language identification at utterance level may not catch switches inside it. Decide per product whether to support mixed language explicitly, and test it with realistic audio; do not assume that a model strong in two languages is strong in their mixture. The reply language and TTS voice must follow the same policy.

Speaker diarisation. When more than one person speaks on a channel (a conference call, a home speaker in a kitchen), diarisation assigns speech segments to speakers. It is useful for meeting transcription and call analytics, but an assistant acting on commands should not rely on it to decide who is authorised.

Confirming critical entities. The practical defence is to confirm what matters and to make confirmation cheap for what does not. Word error rate, defined below, treats all words alike, but a wrong "the" costs nothing and a wrong digit in an amount is a failure. A useful rule is to classify entities by consequence: confirm amounts, dates of write actions, destinations, account identifiers and anything irreversible; accept low-stakes words without ceremony. When the recogniser reports low confidence on an entity, the dialogue should ask for it again in a different form (spell it, give the digits one at a time) rather than repeat the same prompt that failed.

Audio carries more than the words: it is a biometric-adjacent signal, it reveals accent, health and emotion, and it often captures third parties in the background. Privacy design therefore starts with the question of what the system must keep at all.

Consent and retention. Tell callers that the call is handled by an AI and, where applicable, recorded; the exact legal requirements vary by jurisdiction (consent rules for recording calls differ between countries and, in some countries, between regions, and voice data that identifies a person can fall under privacy law such as the GDPR), so involve legal counsel rather than copying a generic notice; this is general information, not legal advice. Decide what to store: the raw audio, the transcript, only derived metrics, or nothing. A sensible default is to avoid retaining raw audio unless there is a defined purpose, and to give audio a shorter retention period than the structured outcome of the call. If audio or transcripts are used to improve models, that is a separate purpose that needs separate consent and a separate pipeline. Provider data-handling terms differ and must be read for the specific vendor.

Sensitive speech and redaction. Callers read out card numbers, dates of birth, addresses and medical details. Transcript redaction (replacing such spans with placeholders before storage and before sending to downstream services) is more reliable when the system knows which dialogue turn it is in: while the assistant is collecting a card number, the next utterance is flagged sensitive and is kept out of logs entirely. For payments, a common approach is to move the caller to a payment-specific flow so that the model never sees the card number. Redaction by pattern or classifier is a mitigation and will miss things, which is why minimising collection is stronger than cleaning up afterwards.

Voice cloning and impersonation. Modern synthesis can imitate a person's voice from a short sample, so the voice of someone who sounds like the account holder is no proof of identity, and a convincing voice on the phone is not evidence of a legitimate request. Equally, a product that clones or imitates voices needs explicit permission from the person being cloned and controls to prevent misuse. Do not use one's own assistant voice to impersonate a human without disclosure.

Speaker verification has limits. Voice biometrics can score whether a voice matches an enrolled one, but the score is probabilistic, it degrades with noise and illness, and it is vulnerable to replayed or synthesised audio. It can be one factor in a risk-based decision, such as deciding when to trigger step-up verification, but it should not be the sole gate for high-risk actions.

Tool authorisation is not a voice property. As the tool layer showed, permissions come from authenticated session identity and server-side checks on each tool call, and a transcript is untrusted input. That holds also for instructions embedded in what the assistant retrieves or hears (a recording playing in the background saying "transfer all funds" should have no authority); see AI Security and HITL & Guardrails.

Voice systems fail in ways a text benchmark never sees, so evaluation needs metrics at several levels, tied to the stages and states discussed above.

Word error rate (WER) is the standard ASR metric: WER = (S + D + I) / N, where S, D and I are the substituted, deleted and inserted words in the best alignment between the reference and the hypothesis, and N is the number of reference words. For the reference "move me to the nine forty flight" (N = 7) and the hypothesis "move me to the nine fourteen flight", there is one substitution, so WER = 1/7 = 14.3 %. The hypothesis "move me to a nine forty flight" also scores 1/7 = 14.3 %, yet the first error moves the booking to the wrong departure and the second changes nothing. WER cannot tell them apart, so it must be complemented by an entity-level error rate (did the date, number or name come out right?) for the entities your tasks depend on. WER is useful for choosing between recognisers and for tracking regressions on a fixed audio test set; it is not a measure of product quality.

def wer(reference: str, hypothesis: str) -> float:
    ref, hyp = reference.split(), hypothesis.split()
    d = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
    for i in range(len(ref) + 1):
        d[i][0] = i
    for j in range(len(hyp) + 1):
        d[0][j] = j
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            substitution = d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1])
            d[i][j] = min(substitution, d[i - 1][j] + 1, d[i][j - 1] + 1)
    return d[-1][-1] / len(ref)

print(wer("move me to the nine forty flight",
          "move me to the nine fourteen flight"))  # 0.1428... = 1/7

End-to-end task success is the metric the business cares about: did the caller get the flight status, or complete the rebooking correctly? It is measured by replaying scripted scenarios (including noisy and accented audio) and by reviewing sampled real calls, and it should be broken down by task type, because a fine average can hide a failing rebooking path. With 200 evaluated calls and 164 resolved correctly (illustrative), success is 164 / 200 = 82 %; the number only means something alongside the breakdown by failure cause: ASR, understanding, tool, TTS, or user abandonment.

Latency is reported as time to first audio at p50 and p95, per stage and end to end, since the user's experience is the tail. False endpointing is the fraction of turns where the system started responding before the user had finished: if 14 of 200 turns were cut off, the rate is 7 %. It is the direct counterweight to shortening the endpoint threshold; a change that improves p50 latency by 80 ms but doubles false endpoints is probably not an improvement. The opposite failure, late endpointing (unusually long silences before the system responds), is measured by the distribution of the endpoint wait.

Interruption handling measures barge-in: the delay from the user's speech onset to playback stopping (in our trace, 110 ms), the proportion of genuine interruptions that were honoured, and the proportion of false barge-ins (backchannels, echo, noise) that wrongly stopped the assistant. It should also check the history: after an interruption, does the recorded assistant turn contain only what was actually heard? Conversation repair counts how often the user had to repeat or correct themselves ("no, I said Pune") and how often the assistant recovered with a good clarification.

TTS intelligibility and quality are assessed by listening tests and, for intelligibility, by feeding synthesised audio back through a recogniser (round-trip WER), keeping in mind that the recogniser error and TTS error mix. Pronunciation of domain terms is checked against a fixed list. User satisfaction comes from surveys and from behavioural signals such as abandonment, transfers to a human, and repeated requests, with the usual caveat that the signals are noisy proxies.

Instrument the state machine itself. Log each transition with the turn id, the session clock and the reason (for example PROCESSING → USER_SPEAKING: speech resumed, 240 ms after endpoint), and trace each turn as a span tree across ASR, model, tool and TTS. That makes "why did it cut me off at 14:02?" answerable from a log line instead of a call recording. Keep transcripts and audio under the retention rules from the privacy section; the instrumentation should capture metadata first and content only where permitted. For general practices see AI Observability.

The pseudocode below puts the pieces together: VAD and ASR events drive the state machine, each turn has a cancellation token and an id, tool calls are classified as read or write, writes require a pending confirmation bound to the exact action, and TTS chunks are dropped if their turn has been cancelled. It is illustrative TypeScript-style pseudocode, not a working library: the asr, llm,tts, player and bookings objects are hypothetical interfaces, and error handling is abbreviated. Treat the structure as the point.

type State = 'LISTENING' | 'USER_SPEAKING' | 'PROCESSING' | 'WAITING_FOR_TOOL'
           | 'ASSISTANT_SPEAKING' | 'INTERRUPTED' | 'ERROR';

interface PendingAction { tool: string; args: unknown; readBack: string; expiresAt: number }

class VoiceSession {
  state: State = 'LISTENING';
  turnId = 0;
  abort = new AbortController();          // cancellation token for the current turn
  pending?: PendingAction;                // write awaiting confirmation (at most one)
  heardText = '';                         // what the user actually heard of the reply

  constructor(
    private caller: { id: string },       // identity established OUTSIDE the transcript
    private asr: Asr, private llm: Llm, private tts: Tts,
    private player: ClientPlayer, private bookings: BookingApi,
  ) {}

  // ---- events from VAD / ASR / client ----
  onSpeechConfirmed() {
    if (this.state === 'ASSISTANT_SPEAKING') return this.bargeIn();
    if (this.state === 'PROCESSING' || this.state === 'WAITING_FOR_TOOL') {
      this.cancelTurn();                  // premature endpoint: abort model work
    }
    this.state = 'USER_SPEAKING';
  }

  onEndpoint(transcript: string) {        // silence threshold reached
    this.state = 'PROCESSING';
    void this.runTurn(++this.turnId, transcript);
  }

  private bargeIn() {
    this.state = 'INTERRUPTED';
    this.cancelTurn();                    // abort LLM + TTS streams
    this.player.stopAndFlush();           // client stops and empties its buffer
    this.heardText = this.player.playedText();  // truncate the history to what was heard
    this.state = 'USER_SPEAKING';
  }

  private cancelTurn() {
    this.abort.abort();
    this.abort = new AbortController();   // fresh token for the next turn
  }

  // ---- one turn ----
  private async runTurn(id: number, transcript: string) {
    await this.runTurnBody(id, transcript);
    // Back to listening only if this turn is current and nobody took the floor meanwhile.
    const busy = ['PROCESSING', 'WAITING_FOR_TOOL', 'ASSISTANT_SPEAKING'];
    if (id === this.turnId && busy.includes(this.state)) this.state = 'LISTENING';
  }

  private async runTurnBody(id: number, transcript: string) {
    const signal = this.abort.signal;
    try {
      // A confirmation is handled by the application, not left to the model.
      if (this.pending) {
        const verdict = await this.llm.classifyConfirmation(transcript, this.pending.readBack, { signal });
        if (Date.now() > this.pending.expiresAt) { this.pending = undefined; }
        else if (verdict === 'yes') return await this.executePending(id, signal);
        else if (verdict === 'no') { this.pending = undefined; return this.say(id, 'Okay, I will leave it as it is.', signal); }
        // 'unclear' falls through: ask again rather than guess
      }

      const stream = this.llm.respond({ caller: this.caller.id, transcript }, { signal });
      for await (const event of stream) {
        if (id !== this.turnId) return;   // stale turn

        if (event.type === 'tool_call') {
          this.state = 'WAITING_FOR_TOOL';
          const result = await this.handleToolCall(event, signal);
          stream.provide(event.id, result);   // model continues with the tool result
        } else if (event.type === 'text_chunk') {
          await this.say(id, event.text, signal);
        }
      }
    } catch (e) {
      if (signal.aborted) return;         // cancelled, not an error
      this.state = 'ERROR';
      await this.say(id, 'Sorry, something went wrong. Could you say that again?', undefined);
      this.state = 'LISTENING';
    }
  }

  private async handleToolCall(call: ToolCall, signal: AbortSignal) {
    if (call.name === 'get_flight_status') {
      // read tool: caller-scoped, no side effects, safe to cancel or retry
      return this.bookings.getStatus(this.caller.id, call.args, { signal });
    }
    if (call.name === 'change_booking') {
      // write tool: authorise, validate, then ask. Never executed from the transcript alone.
      if (!(await this.bookings.ownedBy(this.caller.id, call.args.bookingId))) {
        return { error: 'not_authorised' };   // deterministic denial
      }
      const readBack = await this.bookings.describeChange(call.args);   // e.g. "EX218 on 29 Oct at 14:15"
      this.pending = { tool: call.name, args: call.args, readBack, expiresAt: Date.now() + 30_000 };
      return { status: 'awaiting_confirmation', readBack };
    }
    return { error: 'unknown_tool' };
  }

  private async executePending(id: number, signal: AbortSignal) {
    const action = this.pending!;
    this.pending = undefined;
    const key = 'voice-' + this.caller.id + '-' + id;  // idempotency key for retries
    const outcome = await this.bookings.change(this.caller.id, action.args, key);
    // Speak the result that came back, not the one that was expected.
    const text = outcome.ok ? 'Done. You are now on ' + outcome.summary + '.'
                            : 'I could not change that booking. ' + outcome.reason;
    return this.say(id, text, signal);
  }

  // ---- speech out: chunked TTS with bounded look-ahead ----
  private async say(id: number, text: string, signal: AbortSignal | undefined) {
    this.state = 'ASSISTANT_SPEAKING';
    for await (const audio of this.tts.stream(text, { signal })) {
      if (id !== this.turnId || signal?.aborted) return;       // drop stale chunks
      await this.player.enqueue(audio, { maxLookaheadMs: 1000 }); // backpressure
    }
  }
}

Several design decisions here deserve a note. The caller identity is a constructor argument: it was established by the call setup (a signed-in app session or a verification step), and nothing the model says can change it. The model can propose change_booking with any arguments, but the handler checks ownership and builds the read-back itself, so the sentence the caller agrees to is generated from structured data, not from the model's free text. The stored pending action expires after 30 s (an illustrative value) so a late "yes" cannot confirm a stale proposal, and the action that executes is the stored one, so the arguments cannot be silently altered between confirmation and execution. The idempotency key lets the booking API de-duplicate if the request is retried after a timeout; the broader pattern is described in Idempotency. And an aborted turn never speaks, because every chunk is checked against the current turn id.

What is not shown but must exist in a real system: a barge-in threshold to ignore backchannels; text normalisation before TTS; reconnection and resume; per-stage timeouts with a fallback message; rate limits; and persistence of the transcript according to the retention policy. The design also keeps a good deal of behaviour deterministic on purpose: it should be possible to say, from reading the handler code alone, which actions the assistant can perform and what stops it from performing them without permission. When the task is narrow (say, only reading the flight status), a menu-driven phone tree or a rules-based intent classifier may serve better than an LLM at lower cost and risk; the generative part earns its place when callers phrase things in ways a fixed grammar cannot anticipate.

1. 1

   Voice is full duplex: model who holds the floor as an explicit state machine, so every transition says what is cancelled (the in-flight model request on a false endpoint, playback and the client buffer on barge-in).

2. 2

   The endpointing threshold trades responsiveness against interruption. A 200 ms silence threshold would fire inside a 380 ms pause, so use adaptive endpointing and repair false endpoints by aborting the stale request.

3. 3

   Choose cascaded or speech-native on observability, latency, paralinguistics and deployment cost. The cascade gives inspectable seams for PII checks and tests, and hybrids are common.

4. 4

   Spoken input is weaker evidence than typed input: read back dates, amounts and names before any write, and keep reads (idempotent, cancellable) separate from writes (confirm first).

5. 5

   Budget latency per stage and stream everything: start the model on the last partial transcript, stream model output into TTS in short first chunks, and track p50 and p95 time-to-first-audio.

6. 6

   Treat audio as personal data: minimise what you keep, set shorter retention for raw audio than for the structured outcome, redact sensitive turns, and keep model improvement a separate consented purpose.

The assistant above can answer a status question and, with a confirmation, change a booking. That is the boundary of this chapter. The voice layer contributes the real-time audio path, turn-taking and the spoken confirmation, and the pieces that decide whether an action is permitted, how it is retried safely and how its outcome is verified are exactly the ones that matter once the assistant is wired into more systems: refunds, rescheduling, case creation, escalation to a person with the conversation attached. Those are design problems about bounded autonomy rather than about speech, and they appear in the same form whether the front end is a microphone or a chat box. The latency, authorisation and cancellation disciplines developed here transfer directly; the next lesson treats the action side on its own terms.