Caching in AI Systems

Watch the same question get asked twice — once computed the expensive way, once served instantly from a cache — and see how prompt, embedding, and response caching combine at scale.

What can you cache in an LLM application? Responses, semantic (similarity-matched) answers, prompt prefixes and their KV state, embeddings, retrieval results and tool results, each with its own safety limits. An AI request is a pipeline of computations, and almost every stage can be cached: the final answer, the prompt prefix the model has already processed, the embedding of a piece of text, the list of chunks a search returned, the result of a tool call. Each of these caches stores a different object, is owned by a different component, and stays valid under different conditions. A cached value is safe to reuse only when every input that influenced it is the same as, or provably irrelevant to, the new request. That includes inputs that are easy to forget: the tenant, the caller's permissions, the model version, the prompt template, the state of the knowledge base, and the time.

So caching in AI is mostly a correctness problem with a performance reward, not the other way round. A conventional web cache can often key on a URL. An AI answer is a function of a prompt that was assembled from many sources, so the key has to describe that whole assembly. This chapter walks the request path layer by layer, shows what each layer's key and invalidation rule must contain, does the hit-rate, latency and cost arithmetic, and then deals with the failure modes (cross-tenant leaks, semantic false hits, stale policy answers, stampedes) that make a naive cache dangerous.

Our running system is an enterprise support assistant. It answers policy questions from retrieved, authorised documents and can optionally call an order-lookup tool. Two people type exactly the same sentence within a minute of each other: "How many days do I have to return an item?"

The first is an agent at tenant Acme, whose published policy gives 30 days. The second works at tenant Globex, whose policy gives 14 days for the same product class. The text is identical, the correct answers differ, and a cache that stores "question to answer" will hand one of them the other's policy. Change the scenario slightly and the same two employees at the same tenant can still differ: one is in the EU and sees the statutory withdrawal addendum, one is a contractor who is not allowed to see the internal exceptions document. Or the sentence is the same and the policy changed an hour ago. Repetition of the question is evidence that reuse might be possible. It is never proof.

A useful way to hold this: the model call, the retrieval and the prompt builder are together a function from inputs to an answer. Caching is memoisation of that function, and memoisation is correct exactly when the key captures all the arguments that matter. Anything that influenced the output but is missing from the key is a latent bug that fires when that hidden argument differs. Some hidden arguments are also stochastic: with a sampling temperature above zero, the model would have produced a slightly different wording each time, so a cache freezes one sample. That is usually acceptable for a policy answer and unacceptable for, say, a brainstorming feature where variety is the product.

The visual journey above showed the economic argument: recomputing an answer wastes tokens, GPU time and seconds. The rest of this chapter is about the cost side of the trade, which is the engineering needed to make sure the reused thing is still the right thing.

Follow one request through the assistant from left to right in the figure. It is authenticated at the API edge, where the tenant and the caller's permission scope become known and trusted. Before any AI work, the exact response cache is consulted. If that misses, a semantic cache may be consulted. If both miss, the orchestrator embeds the question, searches an index, optionally calls a tool, builds a prompt and sends it to the model gateway, which forwards it to a serving engine that may reuse a cached prompt prefix and always maintains a per-sequence KV cache while decoding.

Seven distinct objects are being stored, and keeping them apart in your head prevents most design mistakes.

The exact response cache stores a finished answer, keyed on a fingerprint of everything that determines it. It is usually owned by your application because only the application knows the tenant, locale and knowledge version. The semantic response cache also stores finished answers, but finds them by embedding similarity of the question instead of equality, so it can match differently-worded questions and also match ones that are not equivalent. The prefix (prompt) cache stores no answer at all: it stores the model's already-computed internal state for the beginning of a prompt so that the next request with the same beginning skips part of the prefill work. It is owned by whoever runs the model: a hosted provider, or your serving engine. The KV cache is the per-sequence attention state that makes token-by-token decoding affordable; it exists for every request regardless of any policy you configure, and its internals (sizing, paging, eviction under memory pressure) belong to Inference Infrastructure, where the memory arithmetic is done. This chapter does not repeat that arithmetic; it only asks when KV state may be shared across requests.

The embedding cache stores vectors for text, keyed by the text and the embedding model, and is owned by the retrieval or ingestion layer. The retrieval cache stores the outcome of a search: ranked chunk identifiers for a query, filters, permission scope and index version. The tool or API cache stores results of calls to systems outside the model. A hit at the far left saves the entire pipeline; a hit at the far right saves a smaller step, but the keys are simpler and the risks are lower. As a rule, the more work a layer lets you skip, the more inputs its key must cover, because it also skips more of the checks that would otherwise have caught a mismatch.

For quick reference once the reasoning above is clear, here is how the layers compare.

The simplest cache looks like this: normalise the user's question, hash it, look the hash up. It is tempting because it works in a demo and the hit rate looks wonderful. It is unsafe in production for the reason in the opening example, and it is worth watching the failure happen once, in sequence.

Nothing here is a model failure: the model answered correctly for Acme. The defect is a key that omitted the tenant. It is also a data-exposure incident, not merely a wrong answer, because the cache has served content derived from one customer's documents to another. The same mechanism produces smaller wrongs: an answer grounded in a document that only managers may read, served to a contractor; an answer about last year's policy, served after an update; a reply in German served to someone who asked in English.

The repair is to make the key a fingerprint of everything the answer depends on. The fields fall into four groups. Identity and authorisation: tenant, and a digest of the caller's access scope, derived on the server from the authenticated identity and never taken from the request body. Content versions: the knowledge epoch (which index and which policy set the answer was grounded on), and the prompt template version. Model and generation: the exact model identifier and any parameter that changes output (temperature, maximum output tokens, response format, tool set). The request itself: the normalised question, the locale, and any conversation context that is actually in the prompt; a follow-up like "what about the EU?" is meaningless without the previous turn, so either include a digest of that context or do not cache multi-turn requests at this layer.

UNSAFE   key = sha256("how many days do i have to return an item?")

SAFE     key = "resp:acme:" + sha256(canonical_json({
  "scope":    { "tenant": "acme", "acl_scope": "grp-digest:7f3a1c", "locale": "en-GB" },
  "versions": { "model": "support-llm-2025-06-01", "template": "answer-t7",
                "policy_epoch": 4, "index_epoch": 12 },
  "params":   { "temperature": 0, "max_output_tokens": 400, "tools": ["none"] },
  "input":    { "question": "how many days do i have to return an item?",
                "context_digest": null }
}))

Globex asking the same sentence produces a different key because tenant,
policy_epoch and acl_scope differ, so it simply misses and computes its own answer.

Three design choices in that key are worth explaining. First, the tenant appears twice: once as a readable prefix, so that purging, quota accounting and encryption-key selection can work on a tenant without scanning every entry, and once inside the hash so that two tenants can never collide even if the prefix handling has a bug. Second, the permission scope is a digest of what the caller may see, not the user id. Two staff members who share the same groups should share cache entries, which is where most of the hit rate lives; two users with different entitlements must not. The digest is computed from the groups or entitlements the retriever will actually filter on, so the cache and the retriever can never disagree about who is allowed to see what. Third, canonical JSON (sorted keys, no insignificant whitespace) matters: without it, the same logical key serialises two ways and the hit rate silently halves.

A cache key that is safe costs hit rate. Suppose the question-only key showed a 60% hit rate on a mixed multi-tenant workload, and an audit finds that 12 percentage points of those hits were served across tenants or scopes. The honest hit rate is 48%. Expected mean latency, using the illustrative numbers from the arithmetic section below, rises from 0.86 s to 0.48×0.1 + 0.52×2.0 = 1.088 s. That difference is the price of correctness, and it is a price to pay, not to engineer away.

LLM caching by semantic similarity returns a stored answer when a new question's embedding is close enough to a previously answered one.

An exact cache misses when users phrase things differently. A semantic cache embeds the incoming question (see Embeddings), searches a vector index of previously answered questions inside the same scope, and returns the stored answer if the nearest neighbour's similarity is above a threshold. The extra work is an embedding call plus a nearest-neighbour search on every request, hit or miss, which is cheap against an LLM call but not free.

The danger is that embedding similarity measures topical closeness, not whether two questions have the same answer. Embedding models are trained so that sentences about the same subject land near one another, and the words that flip a policy answer (a number, "before" versus "after", a country) barely move the vector. Here are illustrative pairs with cosine similarities of the sort a typical sentence-embedding model might produce. The numbers are invented to show the shape of the problem, not measured from any model.

A  "Can I get a refund on a digital download within 14 days?"
B  "Can I get a refund on a digital download after 14 days?"
   cosine(A,B) ~ 0.97   -> above a 0.95 threshold; answers differ by policy  => FALSE HIT

C  "What is the status of my refund?"   (user u-1042)
D  "What is the status of my refund?"   (user u-2217)
   cosine(C,D) = 1.00   -> identical text; answer depends on who is asking     => MUST NOT SHARE

E  "How long do refunds take?"
F  "How many days until my refund arrives?"
   cosine(E,F) ~ 0.93   -> same intent, same answer, below the 0.95 threshold  => safe reuse MISSED

Pair A and B is the false hit: a customer asking about a refund after the window receives the answer to one asking within it. Pair C and D shows that no similarity threshold can fix identity dependence, because the inputs are identical and the answer depends on data that is not in the text. Pair E and F is the cost of caution: any threshold high enough to reject A and B will also reject some genuinely equivalent questions. The trade-off is inherent, and it is why you measure it rather than guess.

Measure by building a labelled set: pairs of questions, with a human or a strong reference check marking whether the correct answers are the same. Sweep the threshold and read off, for each value, the fraction of true matches you recover (recall) and the fraction of matches you served that were truly equivalent (precision). The table below is an illustrative sweep over 2,000 pairs, half equivalent and half not, with deliberately hard negatives like the refund-window pair.

1,000 equivalent pairs + 1,000 non-equivalent pairs

threshold   served-as-hit (TP)   wrong hits (FP)   recall   precision
0.90        820                  52                82.0%    94.0%
0.95        610                   8                61.0%    98.7%
0.98        380                   1                38.0%    99.7%

precision = TP / (TP + FP);   recall = TP / 1000

Move the threshold in the figure to see why no single value keeps pair E/F while rejecting pair A/B.

What precision you need depends on what a wrong answer costs, and this should be decided per intent, not per system. For a low-risk, non-personal question ("What are your support hours?"), 98.7% may be fine. For anything involving money, eligibility or legal terms, a wrong answer served confidently is worse than the 2 seconds you saved, so use the strictest setting or no semantic cache at all. Remember too that this pair set is adversarial on purpose; the false-hit rate on real traffic will usually be lower per request but is not zero, and the only way to learn it is the shadow evaluation described later in this chapter.

Several controls reduce the risk beyond the threshold. Apply the exact cache's scope fields first: a semantic match is only searched among entries with the same tenant, access scope, locale and versions, so a semantic hit can never cross a boundary the exact key would have respected. Add guard checks that are cheap and deterministic: if the two questions contain different numbers, dates, product identifiers, currencies or negations, treat it as a miss whatever the similarity. Restrict semantic caching to intents classified as generic and non-personal, and never to a request that needs the order tool or contains an account reference. Finally, version the cached entries, because a policy change invalidates them just as it does exact entries, and similarity search happily returns an old answer that is still very close in vector space.

It also has a latency break-even. If the semantic lookup costs o seconds on every request, a miss costs o + L_m, so it only reduces mean latency when the hit rate h satisfies h > o ÷ (L_m − L_h). With the illustrative 2.0 s miss and 0.1 s hit from the arithmetic section and a 60 ms lookup, that is 0.06 ÷ 1.9 ≈ 3.2%: the lookup pays for itself quickly in latency terms. The binding constraint is not speed, it is whether you can afford its false hits.

The caches so far store outputs. A prefix cache stores work. When a model processes a prompt it first runs the prefill phase over all input tokens, producing for every token and every layer the key and value tensors that attention will later consult. During decoding, those tensors for the current sequence are held in the KV cache so each new token only computes its own contribution. That per-sequence KV cache is mandatory and automatic; it dies with the request.

Prefix caching keeps some of those tensors after the request ends, so that another request whose prompt begins with exactly the same tokens can start decoding from the saved state rather than recomputing the shared beginning. The model still runs, still generates a fresh answer for the new question, and still consults the new retrieved passages: only the prefill for the unchanged leading part is skipped. That is the key difference from a response cache: nothing about the answer is reused, only the cost of reading the unchanged instructions again. The sizing of KV memory and the effect of block reuse on a serving engine are covered in Inference Infrastructure.

Because the cached state is computed left to right, with each token's state depending on every token before it, reuse works only for a contiguous leading run of identical tokens. This has direct consequences for how you assemble prompts. Put the stable material first (instructions, tool definitions, tenant-level policy text) and the volatile material last (retrieved passages, history, the question). A timestamp, request id or user name placed in the system prompt changes the first tokens and destroys the whole prefix; the same text placed at the end costs nothing. In the figure, 2,200 of 3,000 tokens (73%) are a candidate prefix for requests from the same tenant. Order retrieved passages deterministically too if you want them to be reusable as a second-level prefix.

The savings depend entirely on the price or time the system assigns to reused tokens, which varies between providers and engines and changes over time. Illustratively, assume input tokens cost $2 per million, as in the cost model of the arithmetic section, and that reused prefix tokens are billed at one tenth of that. Then the 2,200-token prefix costs 2,200 × $2 ÷ 1,000,000 = $0.0044 uncached and $0.00044 reused, a saving of $0.00396 per request. Applied to the 40,000 daily misses of the later example, that is $158.40 a day on top of whatever the response cache saved, if every miss found a warm prefix. In practice it does not: whether a prefix is still warm depends on traffic, on retention limits, and, for self-hosted engines, on how much GPU memory is left after running sequences claim theirs.

Prefix state is valid only for the exact computation that produced it. The model weights must be identical, so a model version bump, a fine-tune, a different quantisation or a different adapter invalidates every entry. The tokenizer must produce the same token ids for the prefix, so even a whitespace or template change in the system prompt invalidates it. The position encodings must line up, which is why reuse is anchored at the start of the sequence. A well-designed engine keys its saved blocks by the token ids and the model identity, so a mismatch is a miss, not a wrong answer. The risks that remain are operational: a prompt template deploy that reorders two paragraphs quietly drops your reuse rate to zero, and a shared cache across tenants can reveal through timing whether another tenant sent a given prefix, which is why tenant-scoped or isolated prefix caches are the conservative choice when prompts contain confidential material. Treat the prefix hit rate as a metric you deploy-gate on, not as a hidden provider detail.

The expensive part of ingestion is usually computing embeddings, and the expensive part of answering is often the retrieval fan-out (vector search, keyword search, reranking). Both can be cached, with simpler validity rules than whole answers, because their outputs are narrower.

Embeddings: content-addressed and model-scoped

An embedding is a deterministic function of the text, the embedding model and the preprocessing (truncation, language handling, any instruction prefix). Key it on exactly that: a hash of the normalised text, the model id and dimensionality, and a preprocessing version. Because the key contains the content, document edits need no explicit invalidation: a changed chunk hashes differently and simply misses. Suppose a nightly job re-ingests a knowledge base of 1.2 million chunks and 3% of them changed. With a content-addressed cache the job embeds 36,000 chunks instead of 1.2 million, and it can embed them in batches sized for the embedding endpoint instead of one by one. A change of embedding model, by contrast, makes every entry useless: vectors from different models live in different spaces and cannot be compared. The model id in the key makes that automatic, and the reindexing (see Vector Databases) must rebuild queries and chunks with the same model.

Query embeddings are a smaller win: the same sentence rarely repeats exactly outside popular FAQs, so the hit rate is typically modest. Their main role is as an input to the semantic cache and retrieval cache, which would otherwise embed the same query twice in one request. As for privacy, embeddings are not anonymised data. Treat a vector as being as sensitive as the text it came from, scope the cache per tenant by default, and delete entries when the source text is deleted. Sharing identical-text vectors across tenants is defensible only for genuinely public corpora.

Retrieval: cache identifiers, then re-authorise on read

A retrieval result depends on the query, the filters, the index contents and the caller's permissions. The key therefore needs the tenant, the normalised query (or its embedding), the filter set, the digest of the caller's access scope, the index epoch (a counter that changes whenever the index is updated) and the retriever configuration (top-k, reranker model version, hybrid weights). If any of them changes, retrieval may rank different chunks, and reuse would be wrong.

A particularly useful design is to cache identifiers and scores only: the ordered list of chunk ids, not their text. On a hit, the retriever fetches current text for those ids and re-checks, per document, that it still exists and that the caller is still allowed to read it. This costs a key-value lookup per chunk, which is much cheaper than a vector search plus reranking, and it makes permission revocation effective immediately for every cached entry, rather than after a TTL. It cannot, however, notice that a newly added document would now outrank the cached ones; that is the job of the index epoch. The distinction between a revoked permission (checked on read) and a changed corpus (handled by versioning) will come up again in the invalidation section.

See Retrieval Pipeline for how queries are processed and filtered, and Context Injectionfor how the retrieved chunks end up in the prompt.

Tool and API results

Tool calls need the most care, because their results are often the live, personal facts that a model cannot otherwise know. A tool result is reusable only if the tool is a side-effect-free read, the key includes the tool version, the normalised arguments and the credential scope used to call it, and the data's volatility justifies a TTL. A carrier's published shipping-times table or a currency rate can be cached for minutes or hours and shared inside a tenant. An order-status lookup for a specific customer is volatile and personal: cache it at most for the lifetime of one request or one conversation turn, so that the model asking for the same order twice does not call the order system twice, and key it on tenant, user and order id. Never cache a failure or a timeout as if it were a result, and never cache the outcome of a tool that writes. See Tools and Tool Calling for the authorisation and idempotency rules that govern those calls.

There is a consequence for the layers above. An answer that incorporated a personal tool result inherits that result's scope. The response cache must either refuse to store it or key it on the user, with a lifetime no longer than the tool result's. The safest policy is the first: mark any answer assembled from user-scoped data as not cacheable at the response layers.

There are four ways a cached value stops being valid, and a robust system uses a combination of mechanisms aimed at each.

Time. A TTL bounds how long any entry can be wrong, even if you forget something. It is the backstop, not the design: it makes staleness bounded rather than absent. Versions in the key.Each of model id, template version, index epoch and policy epoch is part of the key, so a change in any one of them makes every old entry unreachable without scanning or deleting anything; the old entries simply age out by TTL or eviction. This is the cheapest and most reliable form of invalidation and should be the default for anything that changes in a controlled deploy. Events. When something changes outside your deploy process, such as a policy document being edited by the legal team, a publishing event bumps the relevant epoch or purges entries tagged with that document. Validation on read. Cheap checks at hit time (does the cited document still exist, is the caller still authorised) catch the changes that would otherwise take a TTL to expire. A fifth, stale-while-revalidate, serves an expired entry once while refreshing in the background; it suits low-stakes content and should never be used for policy or eligibility answers.

Consider the timeline of a real policy update, because the order of operations is subtler than "change the document, clear the cache".

The stale window is not zero even with events, and it cannot be: until the new document has been ingested into the index, a fresh answer would be wrong too. The most important rule follows from that. Bump the cache epoch only when the new knowledge is actually queryable, not when the document is merely published. If you bump at minute 1, the first misses recompute using the still-old index, store the old answers under the new epoch, and you have re-poisoned the cache with stale content that now lasts a full TTL. Making the index epoch the thing that the cache keys on removes the ordering problem: the epoch changes at the moment of the index swap, by construction.

A second subtle case is a request already in flight when the epoch changes. It read epoch 12 at the start, computes against the old index and finishes after the bump. Because its key was built from epoch 12 and not from the current value at write time, it stores under the old key, which no new request will ever look up. That is harmless and a good reason to capture versions once at the start of the request and use that captured value for both the read and the write.

Fine-grained invalidation has a limit. Tagging an entry with the ids of the documents it cited lets you purge precisely the entries affected when one document is edited or deleted, which is valuable for deletions and permission changes. But it cannot account for a document added that would have been retrieved: the cached answer cites other documents and carries no tag for it. For additions and re-rankings, use the coarse index epoch. The cost of a coarse bump is that the whole cache goes cold, so teams prewarm: replay the most frequent queries from the previous epoch's logs against the new index in the background, with a rate limit, to refill the popular entries before users ask for them.

Cache busting on model and template changes works the same way and deserves a deployment gate. Pin the model to a specific version identifier in the key. A hosted alias that quietly points to a newer model would change answers while your key stays the same, so either pin it or treat the provider's reported model identifier as part of the key. Include the prompt template version for the same reason. After a deploy, expect a cold-cache latency spike and check that the prefix hit rate has recovered.

All numbers in this section are illustrative assumptions for a worked example, not measurements of any system or provider price list.

Mean latency

Let the hit rate be h, the latency of a hit L_h = 0.1 s and the latency of a miss L_m = 2.0 s. Then

E[L] = h * Lh + (1 - h) * Lm

h = 0.60:  0.60 * 0.1 + 0.40 * 2.0 = 0.06 + 0.80 = 0.86 s   (excluding cache overhead)
h = 0.48:  0.48 * 0.1 + 0.52 * 2.0 = 0.048 + 1.04 = 1.088 s (the honest, scope-safe rate)

With a 5 ms lookup paid by every request (hit or miss):
h = 0.60:  0.005 + 0.86 = 0.865 s

A 60% hit rate reduces the mean from 2.0 s to 0.86 s, a 57% reduction. The mean, however, is not what a user at the slow end experiences, and the service-level objectives you commit to are normally percentiles.

The figure lets you vary the hit rate h and read the mean latency and daily cost from the formulas above.

Why p95 is still a miss latency

Sort every request by latency. With h = 0.60, the fastest 60% are hits at about 0.1 s, so the median (p50) is 0.1 s. The 95th percentile falls inside the miss population: it sits at position (0.95 − 0.60) ÷ 0.40 = 0.875 of the way through the misses, that is, at the 87.5th percentile of miss latency. If misses have a median of 2.0 s and an 87.5th percentile of, say, 3.0 s (illustrative), then p95 for the whole service is 3.0 s even though the mean is 0.86 s. Caching changed the mean and the median dramatically and the tail hardly at all. The tail moves with the hit rate only when h exceeds 95%, because only then is the 95th percentile a hit. In every other regime, the way to improve p95 is to make misses faster, through prefix caching, a better queue, a smaller model or less context, not to push the hit rate up a few more points. Report hit and miss latencies separately (see AI Observability), or a good mean will hide a bad tail.

Cost

Take an API-hosted model with illustrative prices of $2 per million input tokens and $8 per million output tokens, and a request of 3,000 input and 400 output tokens.

miss cost  = 3,000 * $2/1e6 + 400 * $8/1e6 = $0.0060 + $0.0032 = $0.0092

100,000 requests/day, no cache:      100,000 * $0.0092            = $920.00
h = 0.60, so 40,000 misses:           40,000 * $0.0092 = $368.00
  + cache lookups at $0.00002 each:  100,000 * $0.00002 =   $2.00
  total                                                     $370.00
saving: $920 - $370 = $550/day (59.8%)

add prefix cache on the 40,000 misses (2,200 tokens at 10% price):
  40,000 * ($0.0044 - $0.00044) = 40,000 * $0.00396 = $158.40
  total                                      $370.00 - $158.40 = $211.60/day

The savings do not scale in lock-step with the hit rate because the cache has costs: the lookup, the storage and, for a semantic cache, an embedding per request. They are small relative to a model call, which is why caching is so attractive, but they are not zero and they are paid on misses too. For a self-hosted model there is no per-token price: the saving is capacity. A hit does not occupy a GPU slot, so at the same hardware you can serve more distinct work. Capacity sizing is the subject of AI Infrastructure Scalingand the broader economics of Cost Optimization; the point to carry here is that a cache hit rate is an input to that capacity model, and a fragile one, so provision for the miss path and treat a hit rate as a bonus, not as a guarantee. When a deploy cold-starts the cache, the service sees the full miss load for a while.

A cache is a shared object under concurrent access, and the dangerous moment is the instant a popular entry expires. Suppose the answer to a common question is requested 250 times per second and takes 2 seconds to regenerate. When the entry expires, every request during the next 2 seconds sees a miss, and finds no result in progress, because nothing has been stored yet. That is 250 × 2 = 500 simultaneous model calls for the same answer. At the illustrative $0.0092 each, that is $4.60 of spend to produce one answer 500 times, and the model tier may meanwhile be queueing, which makes every one of those 500 calls slower than 2 seconds. This is a cache stampede, and a deploy that cold-starts a cache produces the same effect for many keys at once.

Request coalescing (singleflight) is the core fix: the first miss for a key becomes the leader and computes; later misses for the same key wait for that result instead of starting their own. Within one process, a map from key to an in-flight promise is enough, as in the code below. Across many replicas you need either a distributed lock or a short-lived "in progress" marker in the shared cache, with the lock's lifetime bounded so a crashed leader cannot block everyone: followers that time out compute for themselves (at some risk of duplicates) or are served a fallback. Coalescing introduces head-of-line coupling, and the followers share the leader's fate and its latency, so the leader's call needs its own timeout, and an error must be returned to the followers instead of being cached.

TTL jitter prevents entries created together from expiring together: randomise each TTL by about ±10%. Prewarming refills the popular entries after a deploy or epoch bump from log-derived top queries, at a controlled rate. Refresh-ahead regenerates a hot entry shortly before it expires, so users never see the miss; it spends a little extra model work to protect the tail. For content where a short period of staleness is acceptable, serving the old value while one background request refreshes it works too, but, as said earlier, not for policy answers.

Memory is the other concurrency constraint. A response cache holding answers of about 1.5 KB each and a million entries is on the order of a gigabyte (decimal) before overheads, which is easily affordable; a retrieval cache holding ranked lists is similar. An embedding cache of 1,536-dimension float32 vectors is 6,144 bytes each, so a million entries is about 6.1 GB decimal and a fraction of that if you quantise. Set explicit memory limits and an eviction policy (least-recently-used is a sensible default for response caches), and make per-tenant quotas or per-tenant caps part of the design so one tenant's burst of unique queries cannot evict everybody else's entries. Finally, define the failure behaviour: if the cache is down or slow, the request should bypass it after a tight timeout (tens of milliseconds) and go to the model, because a cache is an optimisation and not a dependency, with the caveat that the model tier must then be able to absorb the full miss load, or you shed load rather than fall over. Do not let a cache outage turn into a model-tier outage by unbounded retries.

A cache is a second copy of your data, with its own access path, its own lifetime and usually weaker controls than the database it derives from. That makes it the place where an otherwise-careful design leaks. The following rules fall out of the earlier sections.

Partition by tenant at the storage layer, not just in the key. A tenant prefix in key names allows per-tenant purge, per-tenant quotas and, if the cache supports it, per-tenant encryption keys, so that deleting a tenant's key destroys its entries cryptographically. The key hash is the second defence, not the only one. Check the tenant stored inside the entry against the request's tenant on every read, as the code below does: it costs nothing and turns a bug that would have been a leak into a miss and an alert.

Never trust a client-supplied scope. The tenant id, the access-scope digest and the user id that go into keys are taken from the verified session, not from request fields. The access-scope digest must be computed from the same entitlement data the retriever filters on, at request time, so a permission change takes effect on the next request automatically: the digest changes, the key changes, the old entry is never read. For permission changes that do not alter the caller's own digest, such as a document that was moved into a restricted collection, use per-document validation on read for retrieval results and tag-based purging for stored answers.

Personal data. Cached answers can contain names, order contents and addresses that the model picked up from user-scoped tool results; the safest rule is not to store such answers. Where something personal is legitimately cached (for instance a user's own conversation summary), scope it to the user as well as the tenant, encrypt it at rest, set a TTL that is no longer than the retention you promised, and make sure data-subject deletion can find it: that needs tags or an index from user id to entry keys, because the key is a hash and cannot be searched backwards. Keep raw prompts out of cache keys, cache logs and metrics labels; log the key hash and the reason for hit or miss.

Authorised invalidation. A purge endpoint is a powerful operation: it can degrade performance for every tenant if misused, and it can be used to probe for the existence of entries. Authenticate it as a privileged internal action, restrict tenant administrators to their own namespace, and audit every call. Rate limit epoch bumps; a buggy ingestion loop that bumps every minute destroys the hit rate quietly.

Tool results. As covered earlier, results from tools that act on a particular user's behalf belong to that user's scope. They are not cached across users, and an answer that depends on them does not enter the shared response cache. If you do cache a user-scoped result for the length of a conversation to avoid duplicate calls, key it on tenant, user, tool, arguments and credential scope, and give it a short TTL. Broader threat modelling, including prompt injection through cached content, belongs to AI Security; the cache-specific point is that anything a cache returns re-enters the prompt or the user's screen with the trust level of the original, so stored content should be written only from validated outputs, and cached documents are untrusted input just as the fresh ones are.

A raw hit ratio is the most reported cache metric and the least informative. It counts reuse, not correct reuse. For an AI cache you need at least these quantities, measured per layer, per tenant and per intent.

Hit ratio = hits ÷ lookups. Useful-hit ratio = (hits − stale hits − false hits) ÷ lookups, where a stale hit served content that a fresh computation would have changed in a meaningful way and a false hit is a semantic match that was not equivalent. Stale-hit rate = stale hits ÷ hits. False-match rate (semantic layers) = false hits ÷ hits. Then the outcomes: net cost saved = (hits × miss cost) − cache infrastructure and lookup cost − the cost of the wrong answers you can price; hit and miss latency at p50 and p95 separately; and the quality delta, the difference in task success, groundedness or user rating between cached and freshly generated answers on comparable queries.

Stale and false hits cannot be read off from logs, because the system thought it was right. You find them by shadow recomputation: for a small random sample of hits (say 1%), also run the full pipeline in the background, and compare the fresh result against what was served using a reference check: a deterministic comparison on cited documents and key facts where possible, a judge model or human review where not. Worked example, again with illustrative numbers: a day has 100,000 lookups and 60,000 hits, and you shadow-sample 600 hits. Six differ in meaning. The estimated false or stale rate is 6 ÷ 600 = 1.0%; because the sample is small, the 95% Wilson interval is roughly 0.46% to 2.16%, so the true rate could plausibly be a good deal lower or twice as high. Scaling up, about 1.0% of 60,000 = 600 bad hits a day, giving a useful-hit ratio of (60,000 − 600) ÷ 100,000 = 59.4%. Whether that is acceptable depends on what those 600 were about; sample more heavily on high-risk intents and set alerts on the rate, not only on the hit ratio.

Two operational practices make these numbers trustworthy. First, label every response with its provenance (cache layer, key hash, entry age, versions) in the trace, so that a user complaint can be traced to a particular cached entry and purged. Second, run controlled experiments when you change thresholds or TTLs: route a slice of traffic to the new setting and compare useful-hit ratio, quality and complaint rates, instead of judging from hit ratio alone. A hit ratio that rises after a threshold change is as likely to be an increase in false hits as a success.

The code below is a TypeScript sketch for one process, using an in-memory map as the store so that it is self-contained. It was checked by running it under Node.js with type stripping, but it is a teaching sketch rather than production code: a real deployment would put the store in a shared service, add metrics and make the distributed variant of singleflight explicit. It shows the rules of this chapter as code: canonical, scope-and-version-aware keys; a tenant check and version check on every read; jittered TTLs and a size bound; tag-based purging; coalesced misses; and a refusal to store personal answers.

import { createHash } from 'node:crypto';

interface Scope { tenantId: string; aclScope: string; locale: string }
interface Versions { model: string; template: string; policyEpoch: number; indexEpoch: number }
interface Entry<T> { value: T; tenantId: string; versions: Versions; expiresAt: number; tags: string[] }

const sha = (s: string) => createHash('sha256').update(s).digest('hex').slice(0, 32);

// Stable JSON: sorted keys, so field order never changes the key.
function canonical(v: unknown): string {
  if (Array.isArray(v)) return '[' + v.map(canonical).join(',') + ']';
  if (v && typeof v === 'object') {
    const o = v as Record<string, unknown>;
    return '{' + Object.keys(o).sort().map((k) => JSON.stringify(k) + ':' + canonical(o[k])).join(',') + '}';
  }
  return JSON.stringify(v);
}

function makeKey(layer: string, scope: Scope, versions: Versions, input: Record<string, unknown>): string {
  const normalized = { ...input, ...(typeof input.question === 'string' ? { question: input.question.trim().replace(/\s+/g, ' ').toLowerCase() } : {}) };
  // Tenant stays readable in the prefix so purge-by-tenant and per-tenant quotas are cheap.
  return layer + ':' + scope.tenantId + ':' + sha(canonical({ scope, versions, input: normalized }));
}

class ScopedCache<T> {
  private store = new Map<string, Entry<T>>();
  private maxEntries: number;
  private now: () => number;
  constructor(maxEntries: number, now: () => number = Date.now) {
    this.maxEntries = maxEntries;
    this.now = now;
  }

  get(key: string, tenantId: string, versions: Versions): T | undefined {
    const e = this.store.get(key);
    if (!e) return undefined;
    const stale = e.expiresAt <= this.now() || JSON.stringify(e.versions) !== JSON.stringify(versions);
    if (stale || e.tenantId !== tenantId) { this.store.delete(key); return undefined; } // defence in depth
    this.store.delete(key); this.store.set(key, e); // refresh recency (LRU order)
    return e.value;
  }

  set(key: string, value: T, tenantId: string, versions: Versions, ttlMs: number, tags: string[] = []): void {
    const jitter = 1 + (Math.random() - 0.5) * 0.2; // +/-10% so entries written together do not expire together
    this.store.set(key, { value, tenantId, versions, expiresAt: this.now() + ttlMs * jitter, tags });
    while (this.store.size > this.maxEntries) this.store.delete(this.store.keys().next().value as string);
  }

  purgeTag(tag: string): number {
    let n = 0;
    for (const [k, e] of this.store) if (e.tags.includes(tag)) { this.store.delete(k); n++; }
    return n;
  }
}

// Singleflight: concurrent misses on one key share a single computation.
const inflight = new Map<string, Promise<unknown>>();
function singleflight<T>(key: string, compute: () => Promise<T>): Promise<T> {
  const existing = inflight.get(key) as Promise<T> | undefined;
  if (existing) return existing;
  const p = compute().finally(() => inflight.delete(key));
  inflight.set(key, p);
  return p;
}

interface Answer { text: string; personal: boolean; citedDocs: string[] }

async function answerWithCache(
  cache: ScopedCache<Answer>, scope: Scope, versions: Versions, question: string,
  generate: () => Promise<Answer>,
): Promise<{ answer: Answer; source: 'cache' | 'model' }> {
  const key = makeKey('resp', scope, versions, { question });
  const hit = cache.get(key, scope.tenantId, versions);
  if (hit) return { answer: hit, source: 'cache' };

  const answer = await singleflight(key, generate);
  // Only shareable answers are stored; anything built from user-scoped tool data is not.
  if (!answer.personal && answer.citedDocs.length > 0) {
    cache.set(key, answer, scope.tenantId, versions, 15 * 60_000, answer.citedDocs.map((d) => 'doc:' + d));
  }
  return { answer, source: 'model' };
}

The calling pattern shows the behaviours that matter. Fifty concurrent identical requests produce one model call; a repeat is served from cache; bumping the policy epoch or changing tenant makes the same question a miss.

const v1  = { model: 'm-2025-01', template: 't7', policyEpoch: 4, indexEpoch: 12 };
const acme   = { tenantId: 'acme',   aclScope: 'staff', locale: 'en-GB' };
const globex = { tenantId: 'globex', aclScope: 'staff', locale: 'en-GB' };

// 50 concurrent identical requests       -> generate() called once
// the same request afterwards            -> source: 'cache'
// same question, policyEpoch 4 -> 5      -> source: 'model'  (new key, old entry unreachable)
// same question, tenant globex           -> source: 'model'  (different tenant, different key)

Several things in the sketch are deliberately minimal and are where a production version needs more thought. The version check on read duplicates what the key already provides; it is there to catch a bug in key construction. The tag index is a linear scan here and needs a reverse index (tag to keys) at scale. The inflight map is per process, so across replicas you need a shared lease. The decision of which answers are cacheable is a single boolean computed upstream, and the quality of that classification, which is a product and security decision more than a technical one, determines the safety of the whole system.

Having built the machinery, it is worth being explicit about where it should be switched off, because each cache adds a way to be wrong.

Personalised and high-risk answers. If the answer depends on who is asking and what is in their account, as with refund status, entitlement decisions or anything involving money or legal commitments, the response layers should not store it. Compute fresh, and keep only the sub-results that are safe to share, such as the retrieved policy passages for the generic part of the question. Volatile account state. Order status, balances and stock levels change in seconds, and the correctness of the answer is the whole value of calling the tool. A cache turns "the live status" into "the status a few minutes ago", which a customer can then be told is current.

Low repetition. If most prompts are unique (long free-form documents, code files, individual case histories), the exact and semantic response caches have hit rates near zero and only cost lookup time and memory. Check by measuring the repeat rate of normalised keys on real traffic before building the layer; the prefix cache may still help, since system instructions repeat even when questions do not. Strict freshness requirements. If the business rule says a policy change must apply to the next answer, TTL-based reuse is incompatible, and even event-driven invalidation has a window you must be able to state (the six minutes in the timeline above). Where that window is unacceptable, cache only things downstream of the freshness boundary.

Contexts where invalidation cannot be trusted. If the upstream system changes without emitting events, you do not control the model version because a provider alias floats, or you cannot compute a reliable scope digest, then you cannot construct a key that captures everything the answer depends on, and an incorrect cache is worse than a slow answer. In those situations either cache only with very short TTLs for content whose staleness is tolerable, or do not cache that layer. A good test is to ask, for each layer: can I name every input that affects this value, and can I detect when any of them changes? If either answer is no, the layer is not safe yet.

A cache that cannot be observed will drift: a template change drops prefix reuse, a new tenant has a different question mix, an index job bumps epochs too often, and nobody notices until the bill or the complaints arrive. The practical work after building the layers is to emit, for every request, the layer consulted, hit or miss, the miss reason (not found, expired, version mismatch, scope mismatch, not cacheable), the entry age, the versions involved and the time spent, as trace attributes rather than a single global counter. With those you can split the p95 into hit and miss populations, find which tenant is cold after a deploy, tie a bad answer to a specific cached entry, and alert on the stale and false-match rates from the previous section.

That instrumentation, and the way traces, metrics and quality signals combine into diagnosis, is the subject of the next module.

1. 1

   Caching in AI is mostly a correctness problem with a performance reward: the cache key must be a fingerprint of everything the answer depends on (tenant, access scope, model and template versions, normalised question, corpus epoch), or one tenant gets another tenant's right answer.

2. 2

   Normalise safely: lowercasing and collapsing whitespace is fine, but stripping numbers, punctuation or negation words can merge questions with different answers.

3. 3

   A semantic cache reuses answers by topical closeness, not equivalence of intent. No single similarity threshold both keeps true paraphrases and rejects near-miss pairs, so measure precision on a labelled pair set and set the bar per intent by the cost of a wrong answer.

4. 4

   Prompt (prefix) caching reuses computation, not answers: it only applies to an identical leading prefix produced by identical weights and tokenizer, so put stable content first and verify reuse with cached-token counts or time to first token.

5. 5

   Layer freshness: a TTL as the backstop, versions or epochs for model, template and corpus changes, and events for updates. The stale window is never zero, and answers that used personal tool results must not be shared.

6. 6

   Hit rate helps the mean far more than the tail: at 60% hits the median is a hit but p95 is still a miss. Cache costs (lookup, storage, invalidation) mean savings do not scale in lock-step with hit rate.

7. 7

   Protect against stampedes with request coalescing and TTL jitter, bound memory, partition by tenant at the storage layer, never trust client-supplied scope, and set PII retention for the cache as for any copy of data.