Automatic Speech Recognition (ASR)

Watch speech move from a raw waveform into patterns, candidate words, and a readable transcript. The animation is a conceptual ASR walkthrough, not a claim that every modern speech recognizer implements one identical pipeline internally.

Automatic speech recognition (ASR), also called speech-to-text or audio-to-text, is a pipeline that converts a pressure wave sampled thousands of times a second into a sequence of text tokens. The audio is digitised, cleaned and cut into short overlapping frames; each frame is turned into acoustic features such as a mel-spectrogram; a neural acoustic model maps those features to probabilities over characters, word pieces or words; and a decoder searches for the most probable token sequence, usually combining that acoustic evidence with a language prior, then formats the result with punctuation, casing and timestamps. The output is the most probable transcript given the evidence and the priors, not a recording of what was said, and for high-stakes fields such as amounts, dates and negations it needs to be treated as a hypothesis.

The rest of the chapter follows one real-shaped problem through every stage, then looks at what changes when the audio arrives live, who spoke when, how to measure quality with correct arithmetic, and how to build the service around the model. Speech recognition is one component of the larger multimodal systems described elsewhere; this lesson stops at text. The interactive voice-assistant pipeline built around an ASR model is covered in AI Voice AI Systems.

A support line records this exchange. All content is fictional and illustrative. The customer says, over a kitchen extractor fan: “I was charged twice on October twenty-first, fifteen dollars each time.” Half a second before she finishes, the agent starts: “Sorry, which card was that?” The two voices overlap for about half a second, and the fan raises the noise floor exactly where the consonants of twenty-first and the final n of fifteen live.

If you only measure how many words came out right, the transcript “I was charged twice on October twenty first, fifty dollars each time. Sorry, which card was that?” looks excellent: eleven of twelve customer words are correct. Yet it has changed the amount from $15 to $50, and it has merged two speakers into one line so that “which card was that?” appears to be the customer's own sentence. A transcript used for a dispute has to preserve more than words:

• the exact entities: amounts, dates, card suffixes, ticket numbers, and the negations that flip a claim;
• who said each part, attributed by speaker turn rather than by guesswork;
• where each part sits in time, so a reviewer can jump to 00:04 and listen;
• an honest signal of which spans were uncertain, so the downstream step knows to verify them.

This is the same incident that the screenshot and screen recording lessons use: the customer's spoken date is one piece of evidence, the payment error in the image is another, and the account API is the authority. An ASR transcript is evidence of what was heard, never a verified fact about the account. We will return to this clip at every stage and finally decide which of its fields may be acted on automatically.

Sound reaches the system as a sequence of numbers, and every earlier choice limits what the model can recover. A microphone converts air pressure to voltage; an analog-to-digital converter measures that voltage sample rate times per second and rounds each measurement to one of a fixed number of levels (quantization, commonly 16 bits). Phone and conferencing paths then usually apply a codec, which trades bitrate for fidelity, and may mix several channels into one. The result that arrives at your service is a file or stream in some container (WAV, Ogg/Opus, MP3 and so on) with its own sample rate, channel count and bit depth.

Three numbers people confuse

The sample rate is how many amplitude measurements are taken per second (Hz). The bitrate is how many bits per second the stored or transmitted form uses, which depends on the sample rate, bit depth, channel count and any compression. The model input format is whatever the acoustic front end expects, for instance 16 kHz mono floating-point samples, and is independent of what the customer's device captured. A 48 kHz stereo upload must be downmixed and resampled to the model's format, and resampling cannot restore frequencies that were never recorded.

The Nyquist limit explains why 16 kHz is a common speech setting: a sample rate of 16,000 Hz can represent frequencies up to 8,000 Hz, which covers the energy that distinguishes most speech sounds. Narrowband telephony, historically sampled at 8 kHz, caps at 4 kHz, which is why “fifteen” and “fifty” (whose distinction leans on a quiet word-final nasal and a different stressed syllable) and s versus f are harder on phone audio.

mono PCM, 16-bit, 16 kHz, 60 s

samples      = 16,000 samples/s x 60 s            = 960,000 samples
bytes        = 960,000 samples x 2 bytes/sample   = 1,920,000 bytes  (~1.92 MB decimal)
bitrate      = 16,000 x 16 bits                   = 256,000 bit/s = 256 kbit/s

same minute encoded at an illustrative 24 kbit/s speech codec:
bytes        = 24,000 bit/s x 60 s / 8            = 180,000 bytes (~0.18 MB)

What each cleaning step is for, and what it can damage

Normalisation scales amplitude into a consistent range so a quiet caller and a loud one reach the model at similar levels. It is cheap and almost always safe, except that gain also amplifies the noise floor. Voice activity detection (VAD) classifies short windows as speech or non-speech. Dropping silence saves compute, defines utterance boundaries for streaming, and avoids the hallucinated words that some models produce on long silences. A VAD tuned too aggressively clips a soft word-initial consonant (the t of “twenty”) or a trailing n, and the clipped sound can never be recovered downstream.

Chunking splits long recordings into pieces a model can process, either at VAD pauses or with overlap so a word straddling the cut is seen whole in at least one chunk; stitching the overlap back together is itself a source of duplicated or lost words. Denoising or speech enhancement is optional and deserves suspicion: it is trained to produce audio that sounds cleaner to people, and it can suppress quiet speech or introduce artefacts the ASR model never saw in training. Many modern ASR models are trained on noisy data and do better with the raw signal than with an aggressively enhanced copy, so treat enhancement as a hypothesis to test on your own recordings, comparing error rates with and without it.

A raw waveform at 16 kHz is 16,000 numbers per second whose individual values tell you almost nothing about which phoneme is being spoken: the same vowel produces very different sample sequences depending on pitch and phase. What distinguishes speech sounds is how energy is distributed across frequencies and how that distribution changes over tens of milliseconds. The short-time Fourier transform exposes this. Take a window of 25 ms (400 samples at 16 kHz), apply a taper, compute an FFT, and record the magnitude per frequency bin; then slide forward by a 10 ms hop (160 samples) and repeat. Stacking the columns gives a spectrogram, a time-frequency image.

Human hearing resolves low frequencies more finely than high ones, so the frequency axis is usually warped to the mel scale and the magnitudes are compressed with a logarithm, giving a log-mel spectrogram of, for example, 80 bands. Some modern models learn directly from waveforms or from self-supervised encoders, but log-mel input remains the common baseline and is the right mental model.

import numpy as np

sr, win_ms, hop_ms, n_fft, n_mels = 16_000, 25, 10, 512, 80
win, hop = sr * win_ms // 1000, sr * hop_ms // 1000      # 400, 160 samples

def num_frames(n_samples: int) -> int:
    return 1 + (n_samples - win) // hop                    # no padding

def hz_to_mel(f):  # HTK-style mel scale
    return 2595 * np.log10(1 + f / 700)

for seconds in (10, 60):
    print(seconds, "s ->", num_frames(sr * seconds), "frames x", n_mels, "mel bins")
print("FFT bins:", n_fft // 2 + 1, "bin width Hz:", sr / n_fft)
print("mel(1000 Hz) =", round(float(hz_to_mel(1000))), " mel(8000 Hz) =", round(float(hz_to_mel(8000))))

# 10 s -> 998 frames x 80 mel bins
# 60 s -> 5998 frames x 80 mel bins
# FFT bins: 257 bin width Hz: 31.25
# mel(1000 Hz) = 1000  mel(8000 Hz) = 2840

These shapes matter operationally. A 10 s clip is a [998, 80] matrix; encoders usually reduce the time axis further, for example with strided convolutions that shorten it by 4x, giving roughly 250 encoder frames of 40 ms each (illustrative; the factor is a design choice). Everything downstream, from attention cost to streaming latency, scales with this frame count, so the hop length and subsampling factor are real latency and compute decisions, not implementation details.

One frame is far too short to identify a phoneme: a single 25 ms slice of the vowel in “twice” looks like many other vowels. The information is in the trajectory. A stop consonant like t is a closure (near silence for tens of milliseconds) followed by a burst, and a diphthong like the ai in “twice” is a formant glide across perhaps 100 to 150 ms, so one phoneme spans roughly ten to fifteen 10 ms frames and the acoustic model must look at context on both sides. This is also why the model cannot decide frame by frame in isolation, and why a streaming system that only sees the past has less information than one that can wait.

The acoustic model has a structural problem. The input has about 100 frames per second; the output has a handful of tokens per second, and the model is not told which frames belong to which token. Different ASR families differ mainly in how they solve this alignment between a long frame sequence and a short token sequence, and that choice drives their latency, their vocabulary of failure modes and what they need around them. No family is the right answer everywhere; the choice follows from whether you need live partial results, how much domain customisation you need, and what you can afford to run.

CTC: per-frame labels and a collapse rule

Connectionist temporal classification (CTC) adds a special blank token and has the encoder emit one distribution per frame. Any frame-level path is turned into text by first merging consecutive repeats and then deleting blanks. Many different paths map to the same text, and training sums over all of them, so no manual frame labelling is needed. Because each frame's output is conditionally independent given the audio, the model has no internal sense of which word sequences are fluent, and it works well with a small encoder, runs fast and supports streaming. It leans on an external language model or lexicon when fluency or vocabulary matter.

def ctc_collapse(frames: list[str], blank: str = "_") -> str:
    out, prev = [], None
    for tok in frames:
        if tok != prev and tok != blank:   # merge repeats, then drop blanks
            out.append(tok)
        prev = tok
    return "".join(out)

ctc_collapse(list("ff_ii_fff_tt_yy"))  # -> "fifty"   (repeats merge: f i f t y)
ctc_collapse(list("cc_o_ff_f_ee_e"))   # -> "coffee"  (a blank keeps each doubled letter apart)

The figure below steps through the collapse rule on the first toy path.

Encoder-decoder (sequence-to-sequence)

An encoder turns the audio window into a sequence of hidden states; an autoregressive decoder, like the decoder of a translation model, attends to those states and emits tokens one at a time, each conditioned on the tokens already produced. The decoder is also an implicit language model, which is the source of fluent, punctuated, cased output and of a distinctive failure: when the audio is empty, unintelligible or clipped, the decoder can still continue plausibly and produce words the speaker never said, or repeat a phrase in a loop. Whisper (Radford et al., arXiv:2212.04356) is the best-known example of an encoder-decoder transformer trained on large weakly-supervised data and processed in fixed windows; the architecture needs the whole window before the decoder starts, so live use depends on application-level windowing around it. The attention mechanism and transformer lessons cover the shared machinery.

Transducers (RNN-T style) for streaming

A transducer keeps the per-frame streaming behaviour of CTC but removes the conditional-independence limit. A prediction network summarises the tokens emitted so far, much like a small language model, and a joint network combines that summary with the current audio-encoder frame to decide between emitting a token and advancing to the next frame. Because the audio encoder can be built to use only past context plus a few frames of lookahead, output can be emitted as the speech arrives, which is why this family is common in on-device and live captioning. Its costs are a more complex training recipe and an encoder that is deliberately given less future context than an offline model, which usually costs some accuracy.

Audio-language models

A newer pattern pairs an audio encoder with a language model: an adapter maps the audio encoder's output to embedding vectors the language model reads as if they were prompt tokens, so the same model can transcribe, translate, answer questions about the audio or follow instructions about format. This is a flexible interface and is where speech meets the unified-versus-modular choice discussed in the multimodal overview. It also moves text-model risks into the speech path: the transcription is whatever the language model generates, so verbatim fidelity, hallucinated fills and sensitivity to instructions embedded in the audio itself (a caller saying “ignore previous instructions”) become design concerns. Treat transcribed speech as untrusted data, a topic expanded in AI security.

Other representatives that are widely used as references include wav2vec 2.0 (arXiv:2006.11477), which learns speech representations from unlabelled audio before fine-tuning on labelled data. Whichever family you pick, the interface to the rest of your system should be the same transcript-with-metadata contract described later, so models can be swapped without touching consumers.

The acoustic model does not output words; it outputs probabilities. The decoder's job is to pick the best token sequence from a combinatorial space of candidates. In classical terms the goal is the sequence W that maximises P(W | audio), which Bayes' rule splits into an acoustic likelihood P(audio | W) and a language prior P(W). In practice most systems use a weighted log-linear combination:

score(W) = log P_acoustic(W | audio) + lambda * log P_lm(W | context) + beta * length(W)

lambda : how much to trust the language prior (tuned on held-out audio)
beta   : length bonus to counter the bias toward shorter outputs

Greedy decoding takes the single best token at each step; it is fast and can commit early to a locally likely but globally wrong choice. Beam search keeps the best B partial hypotheses at every step and extends them all, so an unlikely-looking token that leads to a much better continuation is not thrown away. Larger beams cost proportionally more compute and can also prefer fluent, generic text over faithful text, so beam width is a tuned parameter, not a free quality dial.

A language prior can be an n-gram model, a small neural model, the internal decoder of an encoder-decoder, or biasing lists that raise the probability of words you expect, such as product names, plan names, or the amount in an open invoice. Priors resolve real ambiguity (“recognize speech” versus “wreck a nice beach”) and they also cause the characteristic ASR error: plausible text that the audio does not contain. When the evidence is thin, the prior fills the gap with the most expected words, and the result reads perfectly. A fluent transcript is therefore not evidence of an accurate one.

Worked intuition: fifteen or fifty?

In the clip, the fan masks the end of the word. “Fifteen” stresses its second syllable and ends in a nasal; “fifty” stresses the first and ends in a vowel; with the last 100 ms buried in noise, the acoustic model scores them close. The numbers below are a toy to show how the pieces combine. They are not real ASR confidences, and no real system exposes values this tidy.

candidate   log P_acoustic   log P_lm("... charged twice ... X dollars")   score
fifteen        -2.1                -2.8                                    -2.1 + 0.5 * -2.8 = -3.5
fifty          -1.9                -2.6                                    -1.9 + 0.5 * -2.6 = -3.2   <- wins

with a context hint that an open invoice is for $15.00 (bias bonus +1.0 to "fifteen"):
fifteen                                                                    -3.5 + 1.0 = -2.5          <- wins
fifty                                                                      -3.2

Move the two sliders below to see when the ranking flips; the acoustic scores never change.

The ranking changed without the audio changing at all. That is precisely the value of contextual biasing when the hint is right, and the danger when it is wrong: if the customer really said “fifty” because she was charged for a different item, the hint now nudges the system toward the number we expected. Biasing lists are a way to improve recall for rare words, and they should come with a measurement of how often they change correct outputs into wrong ones. Where the hint comes from also matters: an invoice amount pulled through an authorised API is data you control; an amount the caller typed into a chat box is not.

Finally, raw decoder output is usually lower-case spoken form: “october twenty first fifteen dollars”. Post-processing restores punctuation and casing and applies inverse text normalisation (ITN) to produce “October 21, $15”. ITN is its own source of errors (“twenty first” as 20 1st, “four eight two one” as 4,821 instead of the card suffix 4821), so keep the spoken-form text alongside the formatted one for auditing.

Batch transcription reads the complete recording and returns one transcript; it can use all of the future audio as context, and the only latency that matters is total turnaround. A live assistant cannot wait, so a streaming recogniser consumes audio in small chunks (say 200 to 400 ms) and emits partial hypotheses that it later revises, then a final hypothesis when it decides an utterance is over. The distinction is not cosmetic: a partial result is a guess made with less right-hand context than the final one will have, so consumers must not treat it as settled.

Three mechanisms govern behaviour. Buffering and chunking: the front end needs a few frames of lookahead beyond the current chunk, so the first partial appears a few hundred milliseconds after the words are spoken. Endpointing: the system declares the utterance over after a configurable run of non-speech, such as 600 ms; a short threshold is responsive but cuts people off mid-thought, a long one adds dead time to every turn. Finalization: after endpointing, the model runs a last, usually higher-quality pass over the complete segment and emits the final text with stable timestamps.

Using the call from earlier, assume the customer is on a separate audio channel from the agent, so the recogniser sees only her voice. Speech runs from 1,200 ms to 4,800 ms. The sequence below is illustrative; the timing values are plausible rather than measured.

Note the correction. At 4,400 ms the partial read “twenty fifth”: the model had heard “twenty” and the start of a word whose beginning resembled “fifth”, and the final pass, with the noise-obscured end of the word and the silence that followed, revised it to “first”. This is the normal behaviour of a healthy streaming recogniser, and it dictates a contract: every event must say whether it is final, which earlier event it supersedes, and the consumer must not take irreversible action on a partial. An assistant that began checking October 25 on the account at 4,400 ms would have asked the wrong question. Another failure to watch for: a chunk boundary in the middle of a word produces split or duplicated words in naive stitching, which is why streaming models carry state across chunks rather than treating each chunk as an independent file.

An event contract that makes revision explicit

The payload below is an illustrative API contract, not any vendor's schema. A segment has a stable identifier; each hypothesis for that segment increments a revision and replaces the previous one, and only the event with is_final: true is durable. Timestamps are milliseconds from the start of the stream. The uncertainty object carries a model-reported score and is explicitly marked as not calibrated, because a decoder's token probability is not a probability that the word is correct.

{"type":"transcript","segment_id":"seg_0001","revision":2,"emitted_at_ms":3600,
 "start_ms":1200,"end_ms":3550,"speaker_label":"spk_0","language":"en",
 "text":"I was charged twice on October","is_final":false,
 "uncertainty":{"kind":"mean_token_logprob","value":-0.31,"calibrated":false}}

{"type":"transcript","segment_id":"seg_0001","revision":3,"emitted_at_ms":4400,
 "start_ms":1200,"end_ms":4380,"speaker_label":"spk_0","language":"en",
 "text":"I was charged twice on October twenty fifth","is_final":false,
 "uncertainty":{"kind":"mean_token_logprob","value":-0.58,"calibrated":false}}

{"type":"transcript","segment_id":"seg_0001","revision":4,"emitted_at_ms":5520,
 "start_ms":1200,"end_ms":4800,"speaker_label":"spk_0","language":"en",
 "text":"I was charged twice on October twenty first","is_final":true,
 "words":[{"w":"twenty","start_ms":4010,"end_ms":4330},{"w":"first","start_ms":4330,"end_ms":4800,"low_confidence":true}],
 "uncertainty":{"kind":"mean_token_logprob","value":-0.44,"calibrated":false}}

Two additions make this contract robust. First, derive segment_id from the stream and the segment's start offset, so a replayed stream after a disconnect produces the same identifier and the consumer's upsert is idempotent. Second, version the schema and keep unknown fields ignorable, because diarization, language detection and redaction metadata will be added later. This is the same discipline as any event-driven API; the audio-specific part is that revisions are the norm, not the exception.

Timestamps turn a transcript into something you can navigate, caption, redact and audit. Where they come from depends on the model family. A CTC or transducer model knows which frame it emitted each token on, which gives timing as a by-product, accurate to the frame (10 ms before encoder subsampling, 40 ms in the earlier example). Attention-based decoders do not have an explicit frame pointer, so word timing is either predicted as extra tokens or estimated afterwards, for example from cross-attention patterns, and is typically coarser.

Forced alignment is a separate task: given audio and a transcript you already trust, find where each word occurs. It is used for subtitles, for aligning a human-corrected transcript, and for redaction where you must bleep exactly the span containing a card number. It cannot repair a wrong transcript; given the wrong words it aligns them to the wrong sounds without complaint. Use it when the text is known, and use the recogniser's own timing otherwise.

Drift is the quiet failure. If each 30 s chunk is placed with a 50 ms systematic offset (a wrong sample-rate assumption, or an encoder-frame rounding), an hour of audio has 3,600 / 30 = 120 chunks and the last words are off by 120 × 50 ms = 6,000 ms. Compute offsets from sample counts at the original sample rate, not from chunk indices times a nominal duration, and spot-check timestamps near the end of long files.

Punctuation complicates segmentation because a recogniser's acoustic boundaries (pauses, endpoints) and a reader's sentence boundaries do not coincide. A caller who pauses mid-sentence produces two audio segments but one sentence; a run-on without pauses produces one audio segment containing several sentences. Decide what a segment means in your contract (an acoustic utterance, a sentence, a speaker turn) and keep word-level timestamps underneath so you can re-segment without re-running the model. Punctuation restoration models and inverse text normalisation also change the token count, so character offsets and word offsets must be recomputed together.

Diarization answers “who spoke when?” by partitioning the audio into speaker-homogeneous regions and labelling them spk_0, spk_1 and so on. It does not recognise words, and the labels are arbitrary within a recording: spk_0 in one call has no relation to spk_0 in another, and it does not establish who the person is. Speaker identification or verification is a different task that compares a voice against an enrolled voiceprint, is biometric processing with consent and legal implications, and should not be inferred from diarization output.

A common pipeline runs VAD to find speech, cuts it into short windows (around one to two seconds, sometimes with overlap), computes a speaker embedding per window with a model trained so that windows from the same voice are close and those from different voices are far, then clusters the embeddings without knowing the number of speakers in advance, and finally smooths the labels in time. End-to-end neural diarizers exist that predict per-speaker activity directly and handle overlap better. For contact-centre audio, if the platform records each party on its own channel, channel identity is a far more reliable speaker signal than any clustering, and you should use it.

Overlap is the hard case, and this is where the clip earlier is realistic. When two people talk at once, a single-label-per-frame system must pick one, and the speech of the other is lost to both diarization and recognition. Short backchannels (“mm-hm”), similar-sounding voices, telephone codecs and noise all push the embedding clusters together. Diarization quality is measured by the diarization error rate:

DER = (missed speech + false alarm + speaker confusion) / total reference speech time

reference speaker-time = 3,600 (customer) + 1,300 (agent) + 1,300 (customer) = 6,200 ms
error: customer speech 4,300 to 4,800 ms was labelled as agent only    -> missed speech 500 ms
DER = 500 / 6,200 = 0.0806 = 8.1 %

The practical consequence is attribution risk. In the mixed-mono transcript, the words the customer spoke during the overlap are either lost or attached to the wrong speaker, and “Sorry, which card was that?” could be displayed as the customer's own sentence. Where speaker attribution has consequences (who authorised a charge, who agreed to a term) record speakers on separate channels if at all possible, preserve the overlapped audio, and avoid presenting attribution as certain when it was inferred.

Accuracy is not one number across a population. Models trained on data that under-represents a regional accent, a non-native speaker, a child's voice or a domain's jargon will make more errors for those speakers, and aggregate accuracy hides this. Evaluate on slices that match your users, and treat a large gap between slices as a product defect rather than noise.

Language handling is three different tasks. Language identification picks the spoken language, often from the first seconds, and can be wrong on short or noisy clips. Transcription writes what was said in the language it was said in. Translation writes it in another language, introduces a second model's errors, and should be a separate, labelled step so the original-language transcript remains the evidence. A customer who switches mid-sentence (“I was charged twice, pero solo recibí un pedido”) is code-switching: a monolingual model may transliterate the foreign words into the wrong script or drop them, while multilingual models vary in how well they handle it, so test with real mixed utterances before promising support.

Domain vocabulary fails in predictable ways. Proper nouns, product names and acronyms are rare in generic training data and the language prior pulls them toward common words (“Acme Pay” becomes “acne pay”). Identifiers are worse: order numbers, IBAN fragments, SKUs and amounts are read digit by digit, with variable conventions (“oh” versus “zero”, “double five”), and a language prior has nothing to say about them. Remedies are contextual hints or biasing lists (as in the fifteen/fifty example), a pronunciation lexicon for names, a domain fine-tune when you have enough transcribed in-domain audio, and, for identifiers, a read-back confirmation step instead of trusting a single pass. Every one of these has a cost and should be justified by measured gains on your own call data.

Word error rate aligns the hypothesis to the reference with a minimum edit distance and counts three kinds of error:

WER = (S + D + I) / N

S = substituted words, D = deleted words, I = inserted words, N = number of words in the reference

Take a reference of exactly ten words, I was charged twice on October twenty first fifteen dollars, and a system output I charged twice on October twenty first fifty dollars please. The aligner finds one deletion (“was”), one substitution (“fifteen” becomes “fifty”) and one insertion (“please”), so WER = (1 + 1 + 1) / 10 = 0.3, or 30%. Note that WER can exceed 100% if the system inserts many words, and that it divides by the reference length, not the hypothesis length.

def align(ref: list[str], hyp: list[str]):
    """Word-level Levenshtein alignment; returns (S, D, I) counts."""
    n, m = len(ref), len(hyp)
    d = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1):
        d[i][0] = i
    for j in range(m + 1):
        d[0][j] = j
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            sub = d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1])
            d[i][j] = min(sub, d[i - 1][j] + 1, d[i][j - 1] + 1)
    i, j, s, dele, ins = n, m, 0, 0, 0
    while i > 0 or j > 0:
        if i > 0 and j > 0 and d[i][j] == d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]):
            s += ref[i - 1] != hyp[j - 1]
            i, j = i - 1, j - 1
        elif i > 0 and d[i][j] == d[i - 1][j] + 1:
            dele += 1
            i -= 1
        else:
            ins += 1
            j -= 1
    return s, dele, ins

def wer(reference: str, hypothesis: str) -> float:
    ref, hyp = reference.lower().split(), hypothesis.lower().split()
    s, d, i = align(ref, hyp)
    return (s + d + i) / len(ref)

ref = "I was charged twice on October twenty first fifteen dollars"
hyp = "I charged twice on October twenty first fifty dollars please"
print(align(ref.lower().split(), hyp.lower().split()), wer(ref, hyp))  # (1, 1, 1) 0.3

Normalisation policy is part of the metric. Is “twenty-first” equal to “twenty first” and to “21st”? Are casing and punctuation scored? Do “gonna” and “going to” match? Before comparing systems, apply the same documented normalisation to both reference and hypothesis, and report it; WER numbers computed under different policies are not comparable, and a reported improvement may come from the normaliser. Reference transcripts are themselves human work with disagreement, so quality-control them, and keep the test set separate from any data used for tuning or biasing lists.

Low WER, wrong where it matters

WER weights every word equally, but a business weights them very differently. Consider a 17-word reference, I did not authorize the second charge of fifteen dollars on October twenty first please refund it, and a hypothesis that drops one word: I did authorize the second charge .... That is one deletion, so WER = 1 / 17 = 0.059, about 5.9%, a number that looks excellent. But the customer said she did not authorize the charge, and the transcript says the opposite. Likewise, 11 of 12 words correct in the clip earlier still turned $15 into $50.

So report error rates at the level of the decision. For each high-impact entity class (amounts, dates, account or card fragments, negations, speaker attribution) measure entity-level precision and recall, or the fraction of calls where the extracted value exactly matches the reference. Set targets per class rather than from overall WER, collect a test set dense in those entities, and stratify by noise level, channel and accent. Report confidence intervals: with 200 reference amounts, one wrong digit is already a 0.5 percentage-point change.

Production handling follows from the same logic. Extract entities as structured fields with their spans and the model's uncertainty; read the value back to the customer (“I heard fifteen dollars on October twenty-first, is that right?”) when the value drives an action; and cross-check against the account system through an authorised API instead of trusting the transcript. The spoken date is a claim to be checked against the payment records; the checked records, not the transcript, justify a refund, and a model-proposed action still needs authorisation and, for consequential steps, a human, as covered under tool calling and human-in-the-loop guardrails. The false sense of certainty from a fluent transcript is the same problem hallucination describes for text.

The model is a small part of a speech service. Audio arrives either as a file upload or a live stream, and the two have different obligations: an upload can wait in a queue; a live stream is generated by a person talking in real time and will not slow down when your workers are busy.

Backpressure. Put a bounded queue between ingestion and workers and decide explicitly what happens when it fills. Uploads can be accepted and delayed. Live streams cannot be paused, so the options are to scale out, to degrade (switch to a smaller model, defer diarization, drop partial results but keep finals), or to refuse new streams with a clear error before a call starts. Silently dropping audio is the one choice to rule out, because it produces transcripts with holes that look complete. Serving concerns such as batching and autoscaling are covered in model serving and inference infrastructure.

Disconnect recovery. Mobile networks drop. Give each stream a session identifier, and have the server track the last acknowledged audio offset. On reconnect, the client resumes from that offset (it keeps a ring buffer of recent audio for this purpose) and the server continues the same segment numbering, so the consumer sees revisions of the same segment ids rather than duplicates. If the gap is longer than the buffer, emit an explicit gap event with start and end offsets instead of letting two segments run together.

Sensitive data, consent and retention. A call recording contains names, payment details and possibly health or identity information. Obtain and record consent or give notice as the applicable rules require (consult legal advice; requirements vary by jurisdiction), store raw audio and transcripts separately with independent retention timers, encrypt both, and restrict access. Detect and redact sensitive entities such as card numbers in the transcript before it reaches search indices or language-model prompts, and where the audio itself must be shared, redact the corresponding time spans using the word timestamps. Redaction is a model too and misses things, so measure its recall and design for the case where it fails. Log for operations without logging the content; the practices in AI observability apply. Finally, the assistant downstream receives text spoken by a stranger, so apply the same untrusted-input handling as for text from any other external source.

The real-time factor is processing time divided by audio duration. An illustrative job that takes 15 s to process 60 s of audio has RTF = 15 / 60 = 0.25. Below 1.0 the system keeps up with a live stream; above 1.0 it falls further behind every second. It is a throughput statement for a specific model, hardware, batch size and audio mix, not a guarantee about any model, and you have to measure it on your own infrastructure with your own audio length distribution, including the long tail.

RTF is also not user-visible latency. A streaming system with RTF 0.25 still has algorithmic delay from chunk size, lookahead and the endpointing rule discussed earlier. For instance, with a 400 ms chunk, 160 ms of lookahead and 80 ms of compute, the first partial for a word is available about 400 + 160 + 80 = 640 ms after it was spoken (an illustrative budget). The tradeoff is direct: smaller chunks reduce delay and raise overhead per second of audio; more lookahead raises accuracy and delay together.

Capacity follows from RTF. A worker that processes at RTF 0.25 can in principle sustain 1 / 0.25 = 4 real-time streams if it were fully utilised. Real systems run below full utilisation to absorb bursts, so with a target utilisation of 70%, plan for 4 × 0.7 = 2.8 streams per worker. For 200 concurrent calls the arithmetic is 200 × 0.25 = 50 worker-equivalents at full load, and 50 / 0.7 = 71.4, so 72 workers. Batching on GPUs raises throughput per device but makes streams wait for the batch to fill, which adds latency, so interactive and offline traffic are often served from separate pools.

Cost per hour of audio is the price of a worker-hour c divided by how many audio-hours it processes per hour: c × RTF / utilisation = c × 0.25 / 0.7 = 0.357 c, where c is whatever your worker-hour costs (not a quoted price). The levers are model size, quantisation, choosing CPU for light streaming models and GPU for large offline ones, dropping non-speech with VAD, and avoiding re-transcribing the same audio; the broader cost thinking is in cost optimization. Whenever you change a model or a preprocessing step to hit a latency target, re-measure entity-level accuracy, because the trade is rarely free.

The planner below recomputes the worker arithmetic above for any RTF, utilisation and call count.

A transcript contains words. Everything else that makes a call meaningful needs other components and its own evaluation, and bolting a language model onto the transcript does not remove that need.

• Sentiment and tone. “Fine.” can be agreement or fury; the information is in prosody, which the text does not carry. Speech emotion models exist, but they are unreliable across cultures and speakers, and decisions should not depend on them without careful validation.
• Intent. Deciding that the customer wants a refund, not an explanation, is a separate classification or extraction step over the text, evaluated on labelled intents.
• Background events. A doorbell, a crying child or a car horn can matter but ASR tends to ignore or mis-transcribe them; sound-event detection is a different model.
• Speaker verification. Confirming that the caller is the account holder is a biometric decision with its own error trade-offs (false accepts versus false rejects), spoofing risks and consent requirements, not a by-product of transcription or diarization.
• Live system facts. Whether the customer was charged twice is a fact in the payment ledger. The transcript is a claim about it, and the authorised API is the source of truth.

The screenshot of the payment error and the optional screen recording add further evidence of their own; how they are read is covered in image-to-text understanding and the speech component of recordings is picked up again in video understanding, where speech must be aligned with what is on screen. The assistant that reconciles those sources has to keep the provenance of each: what was heard, what was seen, what the system of record says, and where they disagree.

Audio is a signal that unfolds in time, and the problems of this chapter (alignment between a long signal and short output, finality of a hypothesis, drift, overlap) recur when the signal is video. A recording of the customer's screen carries both a soundtrack, handled by the pipeline above, and a sequence of frames, where a sampling rate again decides which events are seen at all. The next lesson shows how video systems represent time, locate events, keep audio and picture synchronised, and scale to long recordings without treating each frame as an unrelated image.