AI Observability

Watch a healthy AI system suddenly slow down — then follow engineers as they use dashboards, logs, and traces to find the exact bottleneck, fix it, and keep costs under control.

What is AI observability, and which metrics matter? It is the practice of tracing each request and measuring token-level latency, answer quality, cost and safety, not just uptime. LLM observability differs from general service monitoring because a response can be fast, error-free and still wrong, and because latency splits into queueing, prefill and decode. Logging must also be privacy-safe: user content and identifiers stay out of metric labels and logs by default. The worked trace and incident below show each signal in use.

An AI application is observable when, for any single bad answer or slow answer, you can reconstruct what the system did: which gateway, retriever, index version, prompt version, model and tool were involved, how long each step waited and ran, and what it cost. Three signal types make that possible. Metrics are cheap aggregates that tell you that something changed and for whom. Traces follow one request across services and tell you where the time or the error went. Structured logs record discrete events with enough context to explain why. AI systems add two things a conventional web service does not need: latency that is split into queue, prefill and decode phases, and answers that can be fast, successful and wrong, which means quality has to be measured as a signal in its own right.

This lesson builds the idea from a production incident, then goes through the signals, a request trace, the latency and capacity metrics, quality and cost signals, SLOs, privacy limits, evaluation pipelines, dashboards, instrumentation code and an incident workflow. The running example is a multi-tenant support assistant that answers policy questions from authorised documents and can optionally look up an order through a tool; every number below is illustrative unless it is a definition or arithmetic you can recompute.

On a Tuesday afternoon the support team for one enterprise tenant reports two things. Answers feel slow: agents are staring at a blank panel for several seconds. And a few answers cite the wrong policy version: an agent told a customer a refund window that the current policy no longer states. The on-call engineer opens the service dashboard. Request rate is normal. The error rate is 0.0%. Every request returned HTTP 200. The CPU graph of the application servers is flat.

Both complaints are invisible to that dashboard, for different reasons. The slowness is hidden by where the status code is produced. A streamed answer sends its status line and headers when the first bytes go out; whatever happens afterwards, including a stall, a truncated generation or a failure mid-stream, cannot change the 200 that is already on the wire. And the time is not being spent on the application servers' CPUs at all: it is spent waiting for a slot in a model server, a place the CPU graph never looks. The wrong citation is hidden because it is not a failure of the transport or the process. The retriever returned a document, the model cited it, the response was well-formed. Nothing threw. The answer was wrong because the evidence was out of date, and no infrastructure metric encodes that.

That is the first lesson of observability for AI: availability metrics measure whether the machinery ran, not whether it ran well, quickly or truthfully. A system can be 100% available, within its CPU budget and quietly useless. The rest of this chapter is about the extra signals needed to make those two complaints diagnosable, and we will come back to this exact incident twice: a latency diagnosis built from spans and a queue calculation, and a stale-evidence diagnosis built from a freshness signal that most teams do not initially have.

Each signal type is a different trade between cost and detail, so each answers a different question.

A metric is a number aggregated over time and a small set of labels: a counter of requests, a gauge of queue depth, a histogram of time to first token. Because aggregation happens at write time, the cost of a metric grows with the number of distinct label combinations (its cardinality), not with traffic. Metrics are what you alert on and what dashboards plot, because they are cheap enough to keep for months and fast to query. What they cannot tell you is anything about an individual request.

A trace is the record of one request as a tree of spans. Each span has a name, a start time, a duration, a status, a parent, and attributes. The trace ID ties spans from different services together, so the question "where did this particular 5-second request spend its time?" has a direct answer. Traces are expensive per request, so they are sampled. A structured log is a discrete event, written as machine-readable key-value pairs rather than prose, which carries the reason something happened: a policy filter dropped three chunks, a tool call was retried, an output failed schema validation. A log line that carries the trace ID can be joined to its trace.

Cardinality: why the cost lives in the labels

Suppose the request-latency histogram carries labels for tenant (200 values), model (4), route (6) and status class (5). The number of series is the product: 200 × 4 × 6 × 5 = 24,000. A histogram with 12 buckets plus a sum and count stores 14 values per series, so 24,000 × 14 = 336,000 stored time series. That is a reasonable load. Now add the end-user ID as a label with 50,000 distinct users. The worst case multiplies by 50,000: 24,000 × 50,000 = 1.2 billion label combinations, or 16.8 billion series with the histogram buckets. Even if most combinations never occur, the store pays for each one that does, and queries slow to a crawl. The rule of thumb that follows: a metric label must have a small, bounded set of values. Per-user, per-request, per-document and per-prompt identifiers belong on spans and logs, which scale with traffic rather than with the product of values, and which can carry access control.

Sampling: choosing which traces to keep

Keeping every trace is rarely affordable, and a trace for an LLM call can be large. Head samplingdecides at the start of the request: keep, say, 5% at random. It is cheap and predictable, but it throws away 99.9th-percentile requests with the same probability as ordinary ones, so the slow trace you need during an incident is probably not stored. Tail sampling decides after the request finishes, in a collector that has seen the whole trace, so it can keep every trace that errored, exceeded a latency threshold or failed a quality check, and a small random fraction of the rest. The price is that the collector must buffer spans until the trace completes. A common design is a hybrid: always keep errors and slow requests, sample the healthy baseline, and keep every trace for a tenant that has an open support ticket.

A trace is only as good as the context propagation that stitches it together. When the gateway receives a request it starts a root span and creates a trace ID. Each downstream call carries that ID, along with the parent span ID, in a header, so that the span created by the retriever knows it is a child of the gateway's span. The W3C traceparent header is the standard carrier for this. The places where context is silently lost in AI systems are exactly the places teams add last: the hop into a message queue, the hop into a background worker, and the boundary into a model server. A trace that ends at the gateway because the model server dropped the header looks like a fast request with an unexplained gap.

The tree of spans for one request in the incident, laid out on a time axis, is a latency waterfall. Reading it is the single most useful debugging skill in this lesson: each bar starts when its work started and ends when it ended, children sit inside their parents, and a gap between bars is time nobody accounted for.

The same request as exported data is below. Only identifiers, versions, sizes and timings appear; there is no prompt text, no document text and no customer identifier. The attribute names beginning with gen_ai. follow the OpenTelemetry generative-AI semantic conventions; those starting with app. are this system's own, and are the ones that make the stale-evidence incident diagnosable later.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spans": [
    { "span_id": "00f067aa0ba902b7", "parent": null, "name": "POST /answer",
      "start_ms": 0, "duration_ms": 5100, "status": "OK",
      "attributes": { "app.tenant": "tenant-17", "app.release": "2026.10.2" } },
    { "span_id": "a1b2c3d4e5f60001", "parent": "00f067aa0ba902b7", "name": "auth",
      "start_ms": 0, "duration_ms": 40, "status": "OK" },
    { "span_id": "a1b2c3d4e5f60002", "parent": "00f067aa0ba902b7", "name": "retrieval",
      "start_ms": 40, "duration_ms": 300, "status": "OK",
      "attributes": { "gen_ai.operation.name": "retrieval",
                      "app.index_version": "idx-0912", "app.chunks_returned": 8,
                      "app.chunks_after_acl": 6, "app.chunks_stale": 0 } },
    { "span_id": "a1b2c3d4e5f60003", "parent": "00f067aa0ba902b7", "name": "prompt_build",
      "start_ms": 340, "duration_ms": 30, "status": "OK",
      "attributes": { "app.prompt_template": "support_answer@v14", "app.input_tokens_est": 2800 } },
    { "span_id": "a1b2c3d4e5f60004", "parent": "00f067aa0ba902b7", "name": "model_queue",
      "start_ms": 370, "duration_ms": 2400, "status": "OK",
      "attributes": { "app.queue_depth_at_enqueue": 33 } },
    { "span_id": "a1b2c3d4e5f60005", "parent": "00f067aa0ba902b7", "name": "chat model-a",
      "start_ms": 2770, "duration_ms": 2300, "status": "OK",
      "attributes": { "gen_ai.operation.name": "chat", "gen_ai.request.model": "model-a",
                      "gen_ai.usage.input_tokens": 2810, "gen_ai.usage.output_tokens": 90,
                      "app.ttft_ms": 500, "app.model_deployment": "model-a-2026-09" } },
    { "span_id": "a1b2c3d4e5f60006", "parent": "00f067aa0ba902b7", "name": "validate",
      "start_ms": 5070, "duration_ms": 30, "status": "OK",
      "attributes": { "app.citations_checked": 3, "app.citations_valid": 3 } }
  ]
}

Notice what the trace carries that no metric can: the specific index version the retriever used, the prompt template version, the deployment of the model that served it, and the queue depth this request met when it arrived. During an incident, these are the attributes you group and filter by, which is why they should be on every request before an incident happens rather than being added during one.

"Latency" in an LLM service is not one quantity. A streamed answer has a point where the user first sees something and a longer period in which text keeps arriving, and each phase has a different cause and a different fix. The useful set is the following.

• Time to first token (TTFT): from the user's request to the first generated token reaching them. It includes everything before generation (gateway, retrieval, prompt build, queue) plus the model's prefill, the single pass over the whole input that must complete before the first output token. It is what makes a response feel responsive or dead. In the trace above, 40 + 300 + 30 + 2400 + 500 = 3270 ms.
• Queue time: how long an admitted request waits for a free slot in the model server. It is zero when there is spare capacity and grows sharply when there is not.
• Prefill time: grows with input length, including retrieved context. A prompt that is 3,000 tokens longer costs more here and also consumes more memory.
• Inter-token latency (ITL) or time per output token (TPOT): the interval between successive generated tokens. Decode time is roughly the number of output tokens multiplied by this: 90 tokens over 1800 ms is 20 ms per token (illustrative).
• Total latency: request to the last token. It depends on output length, so a long answer is slow even when everything is healthy. Compare latencies for similar output lengths, or normalise by tokens.
• Retrieval and tool latency: the time in the non-model dependencies. An order lookup that goes to an external system can dominate a request that never touches the model's capacity limit.

The OpenTelemetry GenAI conventions define a time-to-first-chunk attribute and client duration metrics, measured from the client issuing the request; the same idea of "measure from where the user's wait starts" applies even if you name the measurements yourself.

Why the mean is the wrong summary

Take 100 requests. Ninety finish in 1.2 s. Ten are stuck behind a queue and take 9.0 s. The mean is (90 × 1.2 + 10 × 9.0) / 100 = (108 + 90) / 100 = 1.98 s, which looks like a mildly slower service. The median (p50) is 1.2 s, exactly what it was before. The 95th percentile is 9.0 s, because in the sorted list the 91st to 100th values are 9.0 s and the 95th falls among them; so does p99. One in ten users waited more than seven times longer than the typical user, and neither the mean nor the median registers it. Tails matter more for LLM services than for typical APIs, since a user's whole session consists of many calls: if each call has a 10% chance of a long wait, a five-call task has a 1 − 0.9⁵ ≈ 41% chance of including at least one.

Move the slider below to see how many queued requests it takes before each summary notices: the mean moves with every request, while p95 stays at 1.2 s until more than 5 of the 100 are slow.

Two practical rules come with percentiles. First, store histograms, not precomputed percentiles: you cannot average the p95 of two instances to get the p95 of the fleet, but you can add their histogram buckets and compute a percentile from the sum. Second, do not mix populations: TTFT for a 200-token prompt and for a 12,000-token prompt are different distributions, so slice by input-length bucket, model and tenant tier before concluding anything. The distinction between a throughput peak and a tail latency requirement is also why an SLO is written on a percentile, as discussed below.

Latency tells you users are suffering; capacity signals tell you why and, ideally, warn you before. They fall into three groups. Demand: requests per second, input tokens per second, output tokens per second. These are different loads, since input tokens are mostly consumed in prefill and output tokens in decode, and a shift in the mix (longer retrieved context, longer answers) changes load without changing the request rate. Concurrency and queueing: concurrent sequences in the model server and the queue depth. Resources: GPU utilisation, GPU memory in use, the share of the KV cache occupied, network transfer, and out-of-memory events. The queueing and capacity mathematics are the subject of AI Infrastructure Scaling; here the point is which signals to watch.

The saturation signals that matter most lead the latency ones. Queue depth rises before queue time does, and queue time rises before TTFT does. An alert on queue depth fires while users are still fine. GPU utilisation is less useful than it looks: the common utilisation counter reports the fraction of time during a sample window in which some kernel was running, which says the GPU was busy, not that it was doing useful work efficiently, so a GPU can show a high figure while being memory-bound and delivering a low fraction of its theoretical peak. Treat it as a coarse flag and prefer delivered tokens per second at a given latency as the real measure. Likewise distinguish allocated from used memory: many inference engines reserve a large block of GPU memory at start-up for the KV cache, so "memory used" looks full from the first minute. The informative signal is the occupancy of the cache blocks inside that reservation, and whether requests are being evicted or rejected because it is full. Out-of-memory events and preemptions are rare but critical, and should be counted and traced rather than discovered in a log.

If you use a hosted API, the equivalents are the provider's rate-limit responses, the number of in-flight requests you hold open, your own retry counts, and token usage per minute against your quota. A client-side queue or concurrency limiter in front of the API is the place where your own "queue time" lives, and it should be a span like any other.

Return to the slow-answers complaint, and take the waterfall at face value. The durations are 40 ms (gateway and auth), 300 ms (retrieval, of which 180 ms is vector search and 120 ms reranking), 30 ms (prompt build), 2400 ms (queue), 500 ms (prefill), 1800 ms (decode) and 30 ms (validation). The total is 40 + 300 + 30 + 2400 + 500 + 1800 + 30 = 5100 ms. The four stages the user most often blames, retrieval, queue, prefill and decode, account for 300 + 2400 + 500 + 1800 = 5000 ms, and the queue alone is 2400 / 5100 = 47% of the request. Remove the queue and the same request takes 2700 ms; its TTFT would fall from 3270 ms to 870 ms. The model was not slow. The request waited to be served.

The next question is why it waited. Assume, illustratively, that the deployment can hold 32 sequences at once and that each request occupies a slot for prefill plus decode, 500 + 1800 = 2300 ms. Each slot completes 1 / 2.3 = 0.435 requests per second, so 32 slots complete 32 / 2.3 = 13.9 requests per second. At the normal arrival rate of 6 requests per second, Little's Law (average number in the system = arrival rate × time in the system) gives 6 × 2.3 = 13.8 occupied slots, about 43% of the 32, and the queue stays almost empty. A new tenant starts a bulk job and arrivals rise to 16 per second. Demand is 16 × 2.3 = 36.8 slots' worth of work against 32 slots; the service completes 13.9 per second but receives 16, so the backlog grows by about 2.1 requests per second. A queue of 33 requests ahead of a new arrival, served at 13.9 per second, means a wait of 33 / 13.9 ≈ 2.4 s, which is the span we observe, and which matches the queue_depth_at_enqueue: 33 attribute in the trace. The numbers are consistent: queue depth, queue time and arrival rate are three views of one cause.

The figure plots that backlog as arrivals vary: below 13.9 requests per second the queue stays empty, above it the queue grows without bound until arrivals fall or capacity is added.

1. 1

   Narrow by dimension

   Group TTFT by tenant, route, model deployment and release. If only one tenant's requests are slow, suspect that tenant's traffic or its data; if all are slow, suspect shared capacity or a release.
2. 2

   Open a slow trace and read the waterfall

   The longest bar is the first suspect. Here it is model_queue, not the model call, which rules out a slower model, a longer prompt and a retrieval regression in one step.
3. 3

   Corroborate with a metric

   Check the queue-depth gauge and requests-per-second by tenant. The arrival rate roughly 2.7 times higher than usual (16 against 6) from one tenant explains the saturation.
4. 4

   Mitigate, then fix capacity

   Apply a per-tenant concurrency or rate limit so the bulk job cannot consume every slot, shed or defer low-priority traffic, and add replicas. Once arrivals fall below the 13.9 requests per second the fleet can complete, the queue drains at the difference.
5. 5

   Validate with the same signals

   Queue depth returns near zero, queue-time p95 drops, and TTFT p95 returns to its baseline. Confirm on the metrics, not on the absence of complaints.

A different bar would have meant a different incident. If prefill had been the longest bar, with an input of 2,810 tokens grown to, say, 9,000 after a prompt change, the cause would be the context size and the fix would be in retrieval or prompt construction. A long retrieval bar points at the index or reranker. A decode bar with a high per-token time and many concurrent sequences points at batch pressure or memory bandwidth in the serving layer; see Inference Infrastructure for what those mean. The waterfall's value is that it sends you to the right chapter of the system on the first look.

Now the second complaint. Quality observability exists because a model can return a fluent, well-formed, cited answer that is wrong, and no latency or error metric will say so. The goal is not to measure "truth" directly, which is usually impossible online, but to track a set of proxies, each of which fails in a known way, and to watch them change.

On the retrieval side, useful proxies are the fraction of queries where at least one retrieved chunk passes a relevance threshold, the number of chunks surviving the access-control filter, and, for a labelled set of queries, recall at a fixed cutoff (see the Retrieval Pipeline and Reranking lessons). On the generation side: the share of answers whose citations point at a chunk that was actually in the prompt; citation support, whether the cited chunk entails the sentence that cites it; groundedness, the share of claims in the answer supported by the retrieved context; the abstention rate, how often the assistant says it cannot answer; tool success rate for order lookups; and task-level outcomes, such as thumbs up or down, whether the ticket was resolved without escalation, and results from sampled human review. Groundedness and citation support are often scored by another model acting as a judge. That is useful and scalable, and it is also imperfect: judges are biased towards fluent answers, are inconsistent across prompts and versions, and can be wrong in correlated ways with the model they grade. Calibrate any automated judge against a human-labelled sample, and report its agreement rate alongside its scores.

Worked diagnosis: a grounded answer that is wrong

An agent asked: "Can an EU customer return an item after the standard window?" The assistant answered, citing POL-EU-009 chunk 2, that the return window is 14 days. The current policy, POL-EU-009 version 4 (effective two weeks earlier, illustrative), says 30 days. The automated groundedness score for this answer was 0.97, and citation validity passed: the quoted sentence really is in the cited chunk. Every quality metric that checks the answer against its evidence was green. The defect is in the evidence.

The trace made it visible once the retrieval span carried the right attributes. The span records app.index_version: idx-0912, and the chunk's metadata shows it came from document version 3. A lookup against the document registry says the current version is 4. The retrieval service was returning evidence from an index built on 12 September, and the ingestion job that should have re-embedded the updated policy had been failing since a change to the source system's webhook format, with errors logged at a level nobody was paged on. The assistant was faithfully quoting a document that was true when it was indexed.

The lesson for instrumentation is specific. Groundedness is a relationship between an answer and the evidence; it cannot detect that the evidence is wrong. Add signals about the evidence itself: the fraction of retrieved chunks whose document version is older than the registry's current version (a stale-chunk ratio), and an ingestion lag, defined as now minus the time of the newest source change that has not yet been indexed. With these, the incident would have alerted on the lag soon after the webhook failed, rather than being reported by a customer. The remediation has three parts: fix and re-run ingestion, invalidate anything that cached answers built on the old chunk (see Caching in AI Systems), and add the stale-chunk ratio and ingestion-lag alerts so the next failure is caught. This is also where the model's tendency to produce plausible text is relevant: see Hallucinations for why a fluent answer is not evidence of correctness.

Cost has to be observable at the same granularity as latency, or it can only be explained after the invoice. The basic unit for a hosted model is the token, and input and output tokens are priced differently, so record them separately, along with the model name, tenant, feature (policy Q&A, order lookup, summarisation) and outcome. Cost is then a derived metric: tokens multiplied by a price table that is versioned alongside the data. For a self-hosted deployment the unit is GPU-hours, and the observable quantities are the hours provisioned, the hours in which the fleet was actually serving, and the idle remainder, plus any cache hits that avoided a model call.

A worked example with invented prices. Assume the model costs $1.00 per million input tokens and $4.00 per million output tokens (illustrative, not any provider's price). An average request uses 2,800 input and 220 output tokens: input 2,800 × $1.00 / 10⁶ = $0.0028, output 220 × $4.00 / 10⁶ = $0.00088, a total of $0.00368 per request. That is not the number that matters. If only 80% of requests end in a completed task, and the rest are abandoned, retried or escalated, the cost per completed task is $0.00368 / 0.80 = $0.0046. Failures and retries cost money without producing value, and a falling completion rate is a cost increase that a cost-per-request chart will not show. Budget alerts belong on rates (spend per hour by tenant against a ceiling) so that a runaway loop or a bulk tenant is noticed within hours. The economics of reducing these numbers are treated in Cost Optimization; observability's part is to make each of them measurable.

A service level objective states, in user terms, how often the service must be good enough. The word "good" needs a definition per request, a service level indicator. For the support assistant, a request is good if it is not an error, its TTFT is at most 2.5 s, and, for the quality objective, its citations pass validation. The objective is then a ratio over a window: 99% of requests in 30 days are good. Note that the indicator is per request and user-centred; "the GPU is below 80%" is a diagnostic metric, not an SLO.

The complement is the error budget: the fraction of requests allowed to be bad. At 99% the budget is 1%. With an illustrative 5 requests per second, a 30-day window contains 5 × 86,400 × 30 = 12,960,000 requests, so the budget is 129,600 bad requests. A budget makes the trade-off explicit: while it is mostly unspent the team can ship quickly; when it is nearly gone, reliability work wins over features.

The incident above shows the budget in motion. Suppose 40% of requests breach the TTFT objective for one hour. The burn rate is the bad fraction divided by the budget fraction: 0.40 / 0.01 = 40, meaning the budget is being consumed 40 times faster than the rate that would exactly exhaust it at the end of the window. One hour at that burn consumes 40 × 1 h / 720 h = 5.6% of the 30-day budget; at that rate the whole budget lasts 720 / 40 = 18 hours. Burn-rate alerts use this directly. A widely used pattern, described in Google's SRE Workbook chapter on alerting on SLOs, pairs a long and a short window: page when both the 1-hour and 5-minute burn exceed 14.4 (which consumes 14.4 / 720 = 2% of the budget in an hour) or when the 6-hour burn exceeds 6 (6 × 6 / 720 = 5%), and open a ticket for a slow burn of 1 over three days (3 × 24 / 720 = 10%). The short window makes the alert reset quickly once fixed; the long window stops a brief blip from paging anyone. Apply the same arithmetic to a quality SLO, for example 98% of sampled answers judged citation-supported, accepting that its indicator comes from sampling and a judge and therefore has wider uncertainty.

The figure turns the burn-rate arithmetic into something you can move: choose the share of bad requests and read off the burn rate, the alert tier it would reach if sustained, and the budget consumed over each window.

On threshold versus anomaly alerts: a fixed threshold on a symptom (burn rate, queue depth above a level that has historically preceded breaches) is predictable and easy to explain. Anomaly detection on seasonal metrics can catch the unforeseen, but it fires on harmless deviations and is hard to reason about at 3 a.m. The usual compromise is to page only on SLO burn, a symptom a user would feel, and to send cause-oriented signals such as queue depth, GPU memory and ingestion lag to a ticket or dashboard. Noisy alerts are prevented by giving every page an owner and a runbook, grouping alerts from one root cause, requiring a condition to persist, and deleting any alert that was repeatedly acknowledged without action.

The cardinality argument and the privacy argument lead to the same rule from two directions. User IDs, order numbers, raw prompts, document text and any unbounded identifier must never be metric labels: they explode the series count, and because metrics are widely readable and retained for long periods, they spread sensitive data to every reader of the dashboard system. Those identifiers have a legitimate home in traces and logs where access can be restricted per role, retention is shorter and individual records can be deleted.

Prompts and retrieved chunks are the hardest case. They are exactly what you want when debugging a wrong answer, and exactly what contains customer data, internal policy text or secrets a user pasted into the chat. A defensible default is to log references and shapes rather than content: template name and version, token counts, chunk IDs and document versions, the hash of a prompt, the categories of redaction applied. Content capture is then an explicit, time-limited and audited mode for specific tenants or sampled requests, stored under stricter access, with redaction applied before write, not after. The same rule appears in the OpenTelemetry GenAI conventions, which make message and document content opt-in. Security aspects of what logs may reveal are discussed in AI Security.

{
  "ts": "2026-10-27T14:03:11.482Z",
  "level": "WARN",
  "event": "tool_call_retry",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a1b2c3d4e5f60007",
  "tenant": "tenant-17",
  "user_ref": "u_9c1e5a",             // keyed hash, rotated; not the user id
  "tool": "lookup_order",
  "attempt": 2,
  "error_class": "UpstreamTimeout",
  "latency_ms": 1500,
  "args_shape": { "order_id": "[HASHED:3fa9b0]" },
  "customer_email": "[REDACTED:email]",
  "prompt_template": "support_answer@v14",
  "redactions_applied": ["email", "order_id"],
  "payload_capture": false
}

Redaction is itself a component that can fail, and it should be tested like one: feed known sensitive strings through the pipeline in CI and assert that nothing leaves intact, apply it in the collector as well as in the application so a missed code path is still covered, and treat a hashed identifier as sensitive when the input space is small enough to enumerate (an order number is, unless hashed with a secret key).

Online monitoring tells you a regression has happened; evaluation pipelines try to find it first. The core asset is a golden dataset: a set of representative questions, each with the documents that should be retrieved, a reference answer or rubric, and tags (tenant tier, topic, language, tool needed). It must be versioned, must include the hard cases found in production incidents (the stale policy question becomes a test), and should be refreshed from sampled and de-identified traffic so it does not drift away from what users ask.

An offline replay runs candidate versions (a new prompt template, a different model, a re-built index, a changed reranker) against that dataset and compares them to the current production version on the same inputs. Comparisons must account for sample size. With 300 questions and a baseline groundedness rate near 90%, the standard error is sqrt(0.9 × 0.1 / 300) = 1.7 percentage points, so a single 95% interval is about ±3.4 points, and for a difference between two independent runs about ±4.8 points. A drop from 90% to 86% is therefore within noise for independent samples. Pair the comparison (the same questions under both versions, looking at the questions whose outcome changed), enlarge the set for decisions that matter, and have a human read the changed cases before a release gate trusts a judge's score.

Shadow tests send a copy of live requests to the candidate and record, but do not return, its answers: real inputs, no user risk, though you pay for the traffic and must not let the shadow call trigger real side effects such as a tool that changes an order. Canary releases then route a small slice of users to the candidate with automatic rollback if SLO burn or a quality proxy crosses a threshold. A release gate is the set of conditions required to proceed: no drop beyond the tolerance on retrieval recall, citation support and tool success; TTFT p95 within budget; no increase in cost per completed task beyond an agreed limit; and zero failures on the security and privacy test cases. Drift monitoring compares the distribution of production inputs (topic mix, length, language), retrieval scores and answer outcomes with the baseline, so a quiet shift such as a new product launch or a changed document corpus shows up as a trend before it becomes a complaint.

A good dashboard is a decision tree rendered as panels: each panel answers a question that rules a class of cause in or out. Start from the user outcome and descend toward resources, so that the person on call can walk from a symptom to a cause in a few clicks without needing to know the system by heart.

Five views cover the system. The service health view shows request rate, error rate, SLO burn and p50/p95/p99 TTFT and total latency by tenant tier; it diagnoses whether users are affected and who. The inference performance view shows queue depth and time, prefill and per-token latency, running sequences, KV-cache occupancy, and tokens per second in and out; it separates capacity saturation from slow compute, and for a hosted model it shows your client-side concurrency and the provider's rate-limit responses. The retrieval quality view shows retrieval latency, chunks per query before and after the access filter, relevance-score distribution, stale-chunk ratio and ingestion lag; it diagnoses wrong or missing evidence. The cost view shows tokens by model and tenant, spend against budget, cache hit savings, idle GPU-hours and cost per completed task; it explains bills and finds runaway users. The safety and quality view shows citation validity, groundedness sample, abstention and refusal rates, guardrail triggers, tool failures and feedback. Every panel should link to example traces for its time window; a graph without a path to a request is a dead end. Put version markers (a release, a prompt change, an index rebuild) on every time series, since "what changed?" is the first question of almost every investigation.

The instrumentation below sketches the handler for one request. It uses the OpenTelemetry Python API: get_tracer, start_as_current_span, set_attribute, record_exception and set_status for spans, and a histogram from get_meter for latency. These API calls are real; with no SDK configured they are no-ops. The functions retrieve, call_model and check_citations are placeholders for your own code, as do RELEASE, registry_version and token_usage, so the block is a pattern, not a drop-in module. Notice what is and is not recorded: versions, counts, sizes and a correlation ID, never prompt text or user identifiers, and the tenant appears as a span attribute but as a metric label only because its cardinality is bounded.

import time
from opentelemetry import trace, metrics
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("support-assistant")
meter = metrics.get_meter("support-assistant")

ttft_ms = meter.create_histogram(
    "assistant.ttft", unit="ms", description="Request start to first token")
stale_ratio = meter.create_histogram(
    "assistant.retrieval.stale_chunk_ratio", unit="1",
    description="Share of retrieved chunks older than the registry version")

def answer(request, tenant, tier):               # tier is a bounded label: free / pro / enterprise
    t0 = time.monotonic()
    with tracer.start_as_current_span("POST /answer") as root:
        root.set_attribute("app.tenant", tenant)
        root.set_attribute("app.release", RELEASE)

        with tracer.start_as_current_span("retrieval") as span:
            span.set_attribute("gen_ai.operation.name", "retrieval")
            chunks = retrieve(request, tenant)   # placeholder: authorised search
            stale = [c for c in chunks if c.doc_version < registry_version(c.doc_id)]
            span.set_attribute("app.index_version", chunks.index_version)
            span.set_attribute("app.chunks_returned", len(chunks))
            span.set_attribute("app.chunks_stale", len(stale))
            stale_ratio.record(len(stale) / max(len(chunks), 1), {"tier": tier})

        with tracer.start_as_current_span("chat model-a") as span:
            span.set_attribute("gen_ai.operation.name", "chat")
            span.set_attribute("gen_ai.request.model", "model-a")
            first = None
            try:
                for token in call_model(chunks, request):    # placeholder: streaming call
                    if first is None:
                        first = (time.monotonic() - t0) * 1000
                        ttft_ms.record(first, {"tier": tier})
                        span.set_attribute("app.ttft_ms", first)
                span.set_attribute("gen_ai.usage.input_tokens", token_usage.input)
                span.set_attribute("gen_ai.usage.output_tokens", token_usage.output)
            except Exception as exc:             # a mid-stream failure is still an error here
                span.record_exception(exc)
                span.set_status(Status(StatusCode.ERROR))
                raise

        with tracer.start_as_current_span("validate") as span:
            ok, total = check_citations(chunks)  # placeholder
            span.set_attribute("app.citations_checked", total)
            span.set_attribute("app.citations_valid", ok)

The correlation ID is the trace ID. Put it in the response header returned to the client, in every log line (most logging integrations can inject it from the active span), and in the user-facing "report a problem" action, so that when an agent reports a wrong answer the engineer starts from the exact trace instead of from a description. The same discipline applies to tools: a retry of the order lookup should be visible as two spans with an attempt number, which is how a duplicated side effect is later explained.

The following runnable Python computes burn rate and budget consumed, so the numbers in the SLO section can be reproduced and the thresholds tuned against real traffic.

def error_budget(slo, rps, days):
    total = rps * 86_400 * days
    return total, total * (1 - slo)

def burn_rate(bad_fraction, slo):
    return bad_fraction / (1 - slo)

total, budget = error_budget(slo=0.99, rps=5, days=30)
print(total, budget)                     # 12960000 129600.0

b = burn_rate(bad_fraction=0.40, slo=0.99)
window_hours, period_hours = 1, 30 * 24
print(b)                                  # 40.0
print(b * window_hours / period_hours)    # 0.0555...  (5.6% of the budget in 1 h)
print(period_hours / b)                   # 18.0 hours until the budget is gone
print(14.4 * 1 / period_hours)            # 0.02 -> the 14.4 paging threshold means 2% per hour

The workflow that connects the signals is the same for both incidents in this chapter, and it is worth stating as a sequence because each step uses a different signal. Detect with an SLO burn alert or a quality alert. Scope by slicing on tenant, model deployment, release, index version and input-length bucket: who is affected and what do they share? Trace a representative slow or wrong request and read it. Isolate the regression to a change or a cause: a release, a traffic shift, a failed job, a dependency. Mitigate with the least risky reversible action: roll back the release or prompt, rate-limit the noisy tenant, fail over to another model, serve from a known-good index, or degrade gracefully (for example answering without the order tool). Validate on the same SLO and quality signals that raised the alarm. Prevent by turning what was learnt into a test in the golden set, a new alert, or a limit.

An observability stack has failure modes of its own, and several of them are specific to AI traffic. Head sampling can discard the slow trace you need; the fix is tail sampling for errors and outliers. A collector under a traffic spike drops data exactly when it is most needed, so give it back-pressure limits, monitor its drop counters and alert on them. Lost context at a queue or a model-server boundary produces traces that end early; test propagation end to end and alert on the share of traces lacking a model span. Clock skew between hosts makes children appear to start before parents, so rely on durations measured on one host and on a time-synchronised fleet. Streaming makes a naive status-code metric wrong, so record success at the end of the stream, with a distinct outcome for client disconnects. A judge model that changes version shifts every quality score at once; pin it and re-calibrate when it must change. Redaction bugs leak data; test them. Finally, dashboards that only ever show the mean, a single "healthy" colour, or the GPU graph alone repeat the original mistake of this chapter: they measure that the machinery ran.

For how these signals are combined with the security, scaling and deployment layers of a complete system, see Production AI Architecture; for the serving layer whose queue produced the 2.4 s wait, see Model Serving.

The cost signals above, tokens by tenant, GPU-hours and cost per completed task, are what make optimisation a measured activity. Without them, an optimisation that lowers cost by degrading answers looks like a win. With them, the quality and latency signals built in this chapter become the constraints within which cost is cut.

1. 1

   Availability dashboards show whether the machinery ran, not whether answers were fast or right; an AI service can be all green while users see slow or wrongly grounded answers.

2. 2

   Metrics, logs and traces answer different questions. Keep labels low-cardinality, and never use user IDs or prompts as labels or log raw prompts and retrieved chunks without redaction (and test the redactor).

3. 3

   Latency is several numbers: TTFT, queue time, prefill, inter-token latency, total and non-model dependency time. Store histograms rather than precomputed percentiles, because a mean can look healthy while a tail of queued requests waits several times longer.

4. 4

   Saturation signals such as queue depth and KV cache utilisation lead latency; read the trace waterfall to find the longest span before changing anything (a long queue span points to capacity, a long prefill span points to prompt length).

5. 5

   Quality is a signal too: log index version, prompt version and retrieval attributes on spans so that a fluent answer with the wrong citation can be traced to the retrieval step.

6. 6

   Define SLOs in user terms and alert on error-budget burn rate rather than raw thresholds; track cost per request at the same granularity as latency.

7. 7

   Catch regressions before release with offline replay and shadow tests of prompt, model and index changes, then confirm online.