Cost Optimization

Watch a startup's AI bill grow with its user base — then see how caching, smaller models, batching, and autoscaling bring costs back down without sacrificing performance.

How can you reduce LLM costs without reducing answer quality? Measure cost per successful task first, then apply levers in order: fix retries and failures, trim context, reuse work with scoped caches, route easy requests to smaller models, batch and right-size capacity, and only then quantise. Gate each change on an evaluation of quality so a cheaper answer never becomes a worse one. AI cost optimization, including LLM inference optimization, starts from what drives the bill.

AI infrastructure cost is driven by four things: how many tokens each task consumes (input and output counted separately), how many model calls a task fans out into (tool use, retries, multi-step chains), how much provisioned capacity sits idle when you run your own accelerators, and how often the work is wasted because the task fails or must be redone. Optimisation means lowering cost per successful outcome while holding quality and latency at the levels the product promised. The levers are trimming context, reusing work safely, routing easy requests to cheaper models, batching and right-sizing, and scaling capacity to demand. None of them is free, and each has a failure mode that is cheaper to test for before launch than to discover in the bill.

cost per successful task = total spend / tasks that did what the user needed

fleet $12 per hour / 1,000 successful tasks per hour = $0.012 per successful task

This chapter is about economics, not mechanics. How batching, KV memory and quantisation work lives in Inference Infrastructure; how cache layers are keyed and invalidated lives in Caching in AI Systems; how to size a fleet from concurrency lives in AI Infrastructure Scaling. Here we ask what each of those choices is worth in dollars, what it risks, and how to prove it before shipping.

An enterprise support assistant launches as a prototype. It answers policy questions in a single model call: 2,000 input tokens (system prompt, tool descriptions, a few retrieved policy passages, the question) and 500 output tokens. Under the fictional rates used throughout this chapter ($1 per million input tokens, $4 per million output tokens) that call costs $0.004, and about 85% of cases end without a human. Six months later the assistant is multi-tenant, has an order-lookup tool, keeps conversation history, and serves ten times the traffic. Finance asks why cost per resolved case is not flat but nearly four times higher, when the price per token never changed.

The answer is that traffic was never the only thing that grew. Take one production case and account for every piece, using assumptions you would replace with your own traces:

• Longer prompts. A case now averages two turns. The second turn carries the first turn's question and answer, so its input is 2,550 tokens, not 2,000. Turn one costs 2,000 × $1/M + 500 × $4/M = $0.002 + $0.002 = $0.004; turn two costs 2,550 × $1/M + 500 × $4/M = $0.00255 + $0.002 = $0.00455. Answer generation alone is $0.00855 per case.
• Tool fan-out. In 30% of turns the model decides to call the order-lookup tool. That turn becomes two model calls: the first (2,000 in, 80 out) emits the tool call and costs $0.002 + $0.00032 = $0.00232; the second carries the tool result (2,480 in, 450 out) and costs $0.00248 + $0.0018 = $0.00428. The tool turn costs $0.0066 instead of $0.004, an extra $0.0026. Two turns at a 30% rate means 0.6 tool turns per case: 0.6 × $0.0026 = $0.00156.
• Retries. 6% of calls are retried after a timeout, a malformed tool call or a server error, and a retry re-bills the full input: 0.06 × ($0.00855 + $0.00156) = $0.000607.
• Retrieval and reranking infrastructure. Embedding and reranking run on a small GPU that costs a fictional $3 per hour and can process 7,500 queries per hour at full load. It is only 40% utilised, so the effective price is $3 ÷ (7,500 × 0.40) = $0.001 per query, $0.002 per two-turn case. Only the busy share, 0.40 × $0.002 = $0.0008, does work; the remaining $0.0012 buys idle headroom.

These add to 0.00855 + 0.00156 + 0.000607 + 0.0008 + 0.0012 = $0.012717 per case, already 3.2 times the prototype's $0.004. Then comes the denominator. The prototype resolved 85% of cases; the production assistant faces harder, messier questions and resolves 70% (illustrative). Spend on a case that ends in a human handoff is not refunded, so the cost of one resolved case is $0.012717 ÷ 0.70 = $0.018167, against $0.004 ÷ 0.85 = $0.004706 for the prototype: a factor of 3.86 with unchanged token prices.

Two lessons follow. First, a bill is the product of several multipliers: tokens per call, calls per task, attempts per call, and tasks per resolved outcome. A 10% saving on any one of them is worth the same on paper, but they are very different engineering projects, and only measurement tells you which is large. Second, almost a tenth of this production cost ($0.0012 of $0.012717) is pure idle capacity, a kind of cost that no token-trimming effort will ever touch. If you only track tokens, you will optimise the wrong thing.

A dollar figure means nothing without something to divide it by, and each denominator answers a different question. Cost per request is the easiest to compute and the least meaningful when a task takes several calls. Cost per output token is what vendors price and what GPU benchmarks report, but it ignores input tokens and everything that is not generation: in the case above the all-in cost is $0.012717 for about 1,000 answer output tokens, or roughly $12.72 per million output tokens, more than three times the list output rate of $4. Cost per successful task finally counts failures: it is total spend divided by tasks that did what the user needed, which is why it rose 3.86× in our example. Cost per active user (monthly spend ÷ monthly active users) is how product and finance reason about pricing plans: four cases per user per month at $0.012717 is about $0.051 per user per month, a number that tells you whether a per-seat price can cover the feature.

The most important refinement is quality-adjusted cost: what a failure costs the business. Suppose a wrong or unhelpful answer forces a human agent to take over, at a fictional $4 per escalated case. Compare two models on the same traffic (illustrative):

model   cost per call   resolution rate   escalation cost        cost per case incl. escalations
large   $0.0040         92%               (1 - 0.92) x $4 = 0.32   $0.0040 + $0.32   = $0.3240
small   $0.0006         70%               (1 - 0.70) x $4 = 1.20   $0.0006 + $1.20   = $1.2006

per resolved case, model cost only:  large 0.0040 / 0.92 = $0.00435    small 0.0006 / 0.70 = $0.00086

This calculation exposes a trap in the rest of the chapter. Human handling of a failed case ($4) can be a thousand times the model cost of a call ($0.004), so a one-percentage-point drop in resolution is worth $0.04 per case, ten times the whole model bill. Optimisations that save fractions of a cent are only worth shipping if they cost essentially nothing in resolution rate, which is why every lever below is paired with a quality guardrail and why the experiment section sets its acceptance criterion on quality first. The cheapest token is rarely the cheapest outcome; the cheapest outcome is the one where the cost-per-success numerator falls and the success denominator does not.

With a hosted model API you buy tokens, and the bill is a sum of per-token charges plus anything else the provider meters. For one call the structure is

call_cost = ( fresh_input x P_in  +  cached_input x P_cached  +  output x P_out ) / 1,000,000

fresh_input = input_tokens - cached_input_tokens
P_in, P_cached, P_out = price per million tokens for this model tier (input, cached input, output)
task_cost   = sum over the calls in the task of call_cost x expected_attempts

Input and output are priced separately because they cost the provider different amounts of work: input tokens are processed together in a parallel pass, while output tokens are generated one at a time. That is why output is typically priced above input, and why a verbose answer can cost more than a long prompt. Our fictional rates follow that shape: input $1 per million, output $4 per million.

The base arithmetic

With the fictional rates, a call with 2,000 input tokens and 500 output tokens costs

input  : 2,000 tokens x $1 / 1,000,000  = $0.002
output :   500 tokens x $4 / 1,000,000  = $0.002
call   : $0.002 + $0.002                = $0.004     (before caching, tools and retries)

at 1,000,000 calls per month: $4,000;   at 10,000,000 calls per month: $40,000

Note that although the prompt is four times longer than the answer, input and output contribute equally to the bill because output is priced four times higher. A reader who trims only the prompt is attacking half the cost.

What changes the base number

Cached input. Some providers offer a lower price for input tokens that match a recently processed prefix; whether it exists, its discount, its minimum prefix length and its lifetime vary by provider and model, so check the current documentation of the one you use. Suppose, fictionally, cached input is $0.25 per million and 800 of the 2,000 input tokens (500 of system prompt and 300 of tool descriptions) are an identical prefix on every call. If the cache hits, the call costs 1,200 × $1/M + 800 × $0.25/M + 500 × $4/M = $0.0012 + $0.0002 + $0.002 = $0.0034, a saving of $0.0006 or 15%. With a 90% hit rate the expected cost is 0.9 × $0.0034 + 0.1 × $0.004 = $0.00346. The prefix must be byte-identical and must come first in the prompt, which is a prompt-layout decision, not a switch.

Model tier. Providers sell several model sizes at different prices. The tier is usually the largest single multiplier on the bill and the one you control most directly, which is why routing gets its own section.

Tool charges and fan-out. If a tool call is a separate billed service (a search API, a code-execution sandbox, a retrieval service), it adds a per-call charge on top of tokens. More importantly, each tool round trip re-sends the conversation so far, as the tool-using turn above showed: $0.0066 against $0.004. See Tools & Tool Calling for why one user turn can become several model calls.

Retries and context expansion. A retried call re-bills its whole input; with a 5% retry rate and at most two retries the expected attempts per call are 1 + 0.05 + 0.05² = 1.0525, so $0.004 becomes $0.00421. Context expansion is the slow leak: conversation history, longer retrieved passages and bigger tool results each add a few hundred tokens that nobody budgeted. Whether a cancelled or timed-out request is billed for tokens already generated depends on the provider; assume it is until you have confirmed otherwise.

interface Rates { inPerM: number; cachedInPerM: number; outPerM: number }
interface Call { inputTokens: number; cachedTokens: number; outputTokens: number }

// Fictional rates in dollars per million tokens.
const ILLUSTRATIVE: Rates = { inPerM: 1, cachedInPerM: 0.25, outPerM: 4 };

function callCost(c: Call, r: Rates): number {
  const fresh = c.inputTokens - c.cachedTokens;
  return (fresh * r.inPerM + c.cachedTokens * r.cachedInPerM + c.outputTokens * r.outPerM) / 1e6;
}

// Each call is retried with probability p; at most two retries:
// expected billed attempts per call = 1 + p + p^2.
function taskCost(calls: Call[], retryRate: number, r: Rates): number {
  const attempts = 1 + retryRate + retryRate ** 2;
  return calls.reduce((sum, c) => sum + callCost(c, r) * attempts, 0);
}

const plain: Call = { inputTokens: 2000, cachedTokens: 0, outputTokens: 500 };
const toolCall1: Call = { inputTokens: 2000, cachedTokens: 0, outputTokens: 80 };
const toolCall2: Call = { inputTokens: 2480, cachedTokens: 0, outputTokens: 450 };

console.log(callCost(plain, ILLUSTRATIVE).toFixed(5));                         // 0.00400
console.log(taskCost([toolCall1, toolCall2], 0, ILLUSTRATIVE).toFixed(5));     // 0.00660
console.log(callCost({ ...plain, cachedTokens: 800 }, ILLUSTRATIVE).toFixed(5)); // 0.00340
console.log(taskCost([plain], 0.05, ILLUSTRATIVE).toFixed(5));                 // 0.00421

When you run the model yourself the economics invert. A hosted API is almost entirely a variable cost: no requests, no bill. A self-hosted deployment is mostly fixed: you rent or own accelerator hours whether or not anyone is asking, and the bill per task depends on how full you keep what you paid for.

What the cost model contains

The visible line is accelerator instance-hours: replicas × hourly price. Each replica also needs enough accelerator memory for the weights plus working space for in-flight requests (the reasoning is in Inference Infrastructure), which determines how many accelerators a replica needs and therefore its hourly price. Around it sit networking (load balancers, cross-zone and egress traffic), storage (model artefacts for every version you keep warm, logs, vector index), observability (metrics and trace ingestion at AI-sized payloads), and operations labour: the share of an engineer's time spent on driver upgrades, rollouts, capacity and incidents. Finally there is idle headroom: capacity deliberately kept free so a traffic spike or a lost replica does not breach the latency objective. Headroom is not waste; it is an insurance premium you should price, and shrinking it is a trade against reliability.

Fixed versus variable matters for decisions. Fixed cost has to be amortised over volume: it makes self-hosting attractive at high, steady volume and punishing at low or spiky volume. The marginal cost of one more request on an already-paid-for replica with spare capacity is nearly zero; the marginal cost of the request that forces a new replica is a full instance-hour step.

Worked arithmetic: compute cost per success

Suppose, fictionally, a fleet of generation replicas costs $12 per hour in total, and over a representative hour it completes 1,000 successful tasks. The compute cost per successful task is

fleet cost                = $12 / hour
successful tasks          = 1,000 / hour
compute cost per success  = 12 / 1,000 = $0.012     (before networking, storage, operations and failure)

That $0.012 is three times the $0.004 hosted call from the previous section, and the comparison is not apples to apples: the models, the task definition and the quality differ. But the interesting question is why it is that high, and the answer is utilisation. Suppose the fleet's sustained capacity at the latency objective (not the theoretical peak throughput of the hardware, and not the throughput you reach with the queue growing without bound) is 2,500 tasks per hour. At 1,000 tasks per hour it is 40% utilised, and the cost per task is simply fleet cost divided by useful work:

utilisation   tasks/hour   cost per task
   20%           500       12 / 500   = $0.0240
   40%         1,000       12 / 1,000 = $0.0120
   60%         1,500       12 / 1,500 = $0.0080
   70%         1,750       12 / 1,750 = $0.0069
   80%         2,000       12 / 2,000 = $0.0060
   90%         2,250       12 / 2,250 = $0.0053

A tempting but wrong conclusion is "drive utilisation to 100%". Utilisation near the capacity limit means a queue, and a queue is latency. The relevant ceiling is the utilisation at which your p95 still meets its objective, which is a property of the system (see AI Infrastructure Scaling), and it is why the realistic target is the shoulder of the curve rather than its end.

Now add what the headline figure leaves out. Say networking, storage and observability add a fictional $3 per hour, and operations labour is a fictional $8,000 per month, or $8,000 ÷ 730 hours = $10.96 per hour. The loaded fleet cost is $12 + $3 + $10.96 = $25.96 per hour, or $0.02596 per task at 1,000 tasks per hour. If 5% of tasks fail and are wasted, cost per successful task is $0.02596 ÷ 0.95 = $0.02733. The $0.012 headline has become $0.027 once everything is counted, 2.3 times higher, and the fixed overheads are the reason small deployments rarely make economic sense.

The break-even logic is a simple inequality. To match a $0.004 hosted price on compute alone, the $12 fleet must complete 12 ÷ 0.004 = 3,000 successful tasks per hour. The first configuration tops out at 2,500, so it can never win; a configuration with the same price and 4,000 tasks per hour of capacity breaks even at 3,000 ÷ 4,000 = 75% utilisation. Counting the loaded $25.96 per hour pushes break-even to 6,490 tasks per hour. This is why the usual self-hosting motivations are not "tokens are cheaper" but data residency, latency control, fine-tuned or custom models, or very high steady volume, and why the comparison should always be made at equal quality and with all overheads included.

For a hosted API the cheapest token is the one you do not send, and for a self-hosted model it is the one the GPU does not have to prefill. Context optimisation is therefore usually the first lever to measure because it needs no new infrastructure. Our 2,000-token prompt has a concrete composition you can read off a trace:

system prompt and policy rules      500   static, identical on every call
order-lookup tool schema            300   static
conversation history                450   grows every turn
retrieved passages (4 x 175)        700   varies, includes near-duplicates
user question                        50
total                             2,000

Remove redundant history. Re-sending every previous word is the easiest waste. Replace old turns with a structured state the application maintains (customer tier, issue type, entities mentioned, what has already been answered), perhaps 150 tokens instead of 450. This also helps quality: models are not better for reading stale filler. The risk is lossy summarisation that drops a constraint the user stated in turn one, so the state schema must be explicit about what it keeps.

Retrieve fewer but better passages. If the retrieval stage returns four passages and two are near-duplicates of each other (the same policy paragraph in two document versions, or overlapping chunks), the model pays for the same sentence twice. Deduplicate chunks, rerank, and cut to the number that actually influences answers. The Reranking lesson covers why putting the right passage first lets you send fewer, and Context Injection covers how to place them. Dropping one 175-token passage after deduplication is a saving of $0.000175 per call.

Put the static material first. Moving the 800 tokens of system prompt and tool schema to the very start and keeping them byte-identical is what makes prefix caching possible; a timestamp or user name at the top defeats it.

The arithmetic, and the regression risk

Summarising history saves 300 tokens (450 → 150) and dropping one duplicate passage saves 175. The prompt falls from 2,000 to 1,525 tokens, a 23.75% reduction in input, and the call falls from $0.004 to 1,525 × $1/M + 500 × $4/M = $0.001525 + $0.002 = $0.003525, an 11.9% reduction in total call cost because output is untouched. A 24% cut in prompt size is a 12% cut in the bill: always convert a token saving into a cost saving with both sides of the formula.

The figure shows the same composition before and after the two trims.

Whether $0.000475 per call is worth it depends on what the trimmed context did. Each of the removed tokens might have been the single sentence that grounds an answer. So context changes need a quality regression test that runs the old and new prompts on the same frozen set of real questions, scores groundedness and answer correctness, and compares them case by case. The experiment section shows exactly such a test with acceptance thresholds. A saving that is not verified against a quality metric is a guess.

Caching in AI Systems explains the layers (response, prompt/prefix, KV, embedding and retrieval caches) and how their keys must be scoped. The economic question is different: for a given layer, does the expected saving exceed the cost of lookups, invalidation and wrong answers served from a stale entry?

The saving per request from a response cache is

net_saving_per_request =
    h x C_miss                 avoided model work on hits (h = hit rate, C_miss = cost of a full call)
  - L                          lookup cost paid on every request, hit or miss
  - h x s x C_stale            expected cost of stale answers (s = fraction of hits that are stale,
                               C_stale = business cost of one stale/wrong answer)
  - invalidation_and_ops       cost of event-driven purges, key versioning, monitoring (amortised)

Take an exact-match cache over policy questions within one tenant and policy version. Fictional measurements: hit rate h = 12%, full call cost $0.004, lookup cost $0.00002 per request. The gross saving is 0.12 × $0.004 − $0.00002 = $0.00048 − $0.00002 = $0.00046 per request, 11.5% of the call cost, or $460 per million requests. That looks pleasant. Now price the stale answers. If a policy changes (a refund window moves from 30 to 14 days) and the cache keeps serving the old answer, each stale hit is a wrong answer; using the same fictional $4 escalation cost, and suppose 1.5% of hits are stale because entries expire only by TTL:

gross saving                          = 0.12 x 0.004 - 0.00002          = +$0.00046 / request
stale cost, TTL-only (s = 1.5%)       = 0.12 x 0.015  x 4               = -$0.00720 / request
net                                                                      = -$0.00674 / request   (loses money)

stale cost, event-invalidated (s = 0.05%) = 0.12 x 0.0005 x 4          = -$0.00024 / request
net                                                                      = +$0.00022 / request   (half the gross saving)

break-even stale fraction of hits     = 0.00046 / (0.12 x 4)             = 0.096%

Slide the stale fraction to see where the cache stops paying for itself.

The practical consequences are what the formula implies, not a general rule that caching is good. A cache is safest where answers depend only on content that has an invalidation event: public policy text keyed by tenant, policy-index version, prompt version and model. It is dangerous for anything involving a live order or account state, which must never be served from a response cache keyed only on the question text, and a personalised or authorisation-dependent answer must be keyed (or excluded) by the identity and permissions it depended on. A cache hit that leaks one tenant's answer to another costs far more than any saving.

Prefix caching has a different profile: no stale-answer term, because the model still generates a fresh answer; the only failure is a miss. Using the fictional cached-input rate above, a 90% hit rate on the 800-token prefix saves 0.9 × 800 × ($1 − $0.25)/M = $0.00054 per call (13.5%). The cost is a prompt layout discipline (static first), which is why it is usually the safest cache to enable first. Embedding and retrieval caches save the retrieval GPU's busy time, which only matters if that GPU's idle share can then be reduced; a cache that lowers utilisation of a fixed-size fleet saves nothing until the fleet shrinks.

A GPU serving one request at a time spends most of its time moving model weights from memory for a small amount of arithmetic. Serving several requests together amortises that movement, so throughput per device rises and the cost of each request falls. The mechanism, including continuous batching, is in Inference Infrastructure; the cost consequence is that batch size is a dial between cheap and fast, and the dial has a wrong end.

Take one illustrative replica at $6 per hour and measure it at different maximum batch sizes, using the load profile of the assistant. Cost per task is $6 divided by tasks per hour. Time to first token (TTFT) and the end-to-end latency grow because requests wait for a batch slot and share compute:

max batch   tasks/hour   cost/task            cost/1,000 tasks   p95 TTFT   p95 total
   1          600        6 / 600   = $0.0100     $10.00          0.4 s      1.2 s
   4        1,800        6 / 1,800 = $0.0033     $ 3.33          0.5 s      1.5 s
   8        2,800        6 / 2,800 = $0.0021     $ 2.14          0.7 s      2.0 s
  16        4,000        6 / 4,000 = $0.0015     $ 1.50          1.2 s      3.1 s
  32        4,800        6 / 4,800 = $0.0013     $ 1.25          2.4 s      5.6 s

The shape is the point. Going from batch 1 to 4 cuts the price of a task by 67% at a cost of 0.1 s of TTFT. Going from 16 to 32 saves only 17% (from $1.50 to $1.25 per thousand) and costs 1.2 s more TTFT. Returns on throughput diminish while latency penalties compound, because larger batches mean more memory pressure and longer queues. With a product objective of p95 TTFT ≤ 1.0 s (an illustrative requirement for a streaming chat UI, where the user perceives the pause before the first word), the best affordable point is batch 8: $2.14 per thousand tasks, 4.7 times cheaper than batch 1, and both larger settings are ruled out by the objective, not by taste. A misconfigured scheduler can put you off this frontier entirely, so confirm that your measured points actually trace a curve like this one before choosing among them.

Not every question needs the largest model. Asking "what is the refund window for EU customers?" and "my order was split across two shipments, one was charged twice, and the customer is disputing one charge; what can I do?" are different difficulties, yet a single-model design pays the large-model price for both. Routing sends a request to the cheapest model that is likely to answer it well, and falls back when that model is not enough.

The diagram shows where routing and caching sit on one request path. Notice the ordering: cheap deterministic checks come first, the expensive model last, and a verifier decides whether the cheap answer is acceptable.

Cost of the cascade. Use fictional rates for a small model at $0.15 input and $0.60 output per million tokens. A 2,000/500 call on it costs 2,000 × $0.15/M + 500 × $0.60/M = $0.0003 + $0.0003 = $0.0006. The router is a 300-token classification on the small model: 300 × $0.15/M = $0.000045. Suppose 60% of traffic is routed as easy and 10% of those answers fail the verifier and re-run on the large model at $0.004. Expected cost per request:

router              = $0.000045                                  (all requests)
easy path (60%)     = 0.6 x ( 0.0006 + 0.10 x 0.0040 )            = 0.6 x 0.0010 = $0.000600
hard path (40%)     = 0.4 x 0.0040                                                 = $0.001600
total               = 0.000045 + 0.000600 + 0.001600              = $0.002245     (56% of $0.004)

saving per request  = 0.004 - 0.002245 = $0.001755

That 44% saving is only real if quality is unchanged, and here the earlier unit-economics warning applies: break-even is a resolution loss of $0.001755 ÷ $4 = 0.044 percentage points when failures end in a $4 human escalation. That is below what almost any evaluation can detect. In practice this means a router must be judged by a paired comparison on a large frozen set, with a verifier that catches the small model's mistakes rather than letting them reach users, and routing rules written per question category (the FAQ lookup that the small model handles as well as the large one) instead of a global "use the small model". A cascade also changes latency: an escalated request pays the small model, the verifier and the large model in sequence, so its p95 is worse than a direct large call, and a router or verifier that is wrong is a new component to monitor.

Avoid unnecessary call chains. The most expensive routing mistake is not picking the wrong model but adding calls: a separate "planner" call, a "rewrite the question" call, a "check the answer" call and a "format the answer" call turn $0.004 into $0.02 before anything useful happens. Before adding a model call, ask whether code, a regular expression, a schema validator or a single better prompt can do the same step. Evaluate quality at each hop: a four-step chain that adds 0.5 points of accuracy for five times the cost has not passed the cost-per-success test, and chains also multiply latency. Fallback to a second provider or model on failure is a different thing: it is a reliability mechanism, discussed below, and should be rate-limited so that it cannot silently double spend.

If you self-host, the model must fit in accelerator memory, and what is left over after the weights decides how many requests you can serve at once. Quantisation stores weights in fewer bits, shrinking memory and often increasing speed (because less data moves per token) at some risk to accuracy. The mechanics are in Inference Infrastructure and AI Accelerators; here is the money.

Take a hypothetical 30-billion-parameter model (not any real model) on a hypothetical 80 GB accelerator (GB meaning decimal gigabytes, 10⁹ bytes, throughout). Weight memory is parameters × bytes per parameter:

weights, 16-bit : 30e9 x 2   bytes = 60 GB
weights,  8-bit : 30e9 x 1   byte  = 30 GB
weights,  4-bit : 30e9 x 0.5 byte  = 15 GB   (plus small overhead for scales; ignored here)

device memory 80 GB; reserve 4 GB for runtime/activations (assumption)
KV cache per token (assumed for this model): 0.5 MB

16-bit: 80 - 60 - 4 = 16 GB for KV  ->  16,000 MB / 0.5 = 32,000 tokens  ->  / 2,500 tokens per request = 12.8 -> 12 concurrent requests
 8-bit: 80 - 30 - 4 = 46 GB for KV  ->  46,000 MB / 0.5 = 92,000 tokens  ->  / 2,500 tokens per request = 36.8 -> 36 concurrent requests

At the same $6 per hour, three times the concurrency can mean substantially lower cost per task, and in the 16-bit case 60 GB of weights on a 40 GB device would not fit at all, forcing a second accelerator, a communication overhead and double the instance price. That is the real economic effect of quantisation and of right-sizing the accelerator: choose the cheapest hardware where the model fits with enough working room, not the fastest chip, and watch for a memory-bound (as opposed to compute-bound) workload where a cheaper device with similar memory bandwidth performs nearly as well.

The cost of this lever is that it can change answers. Lower precision can reduce accuracy unevenly: average benchmark scores can look unchanged while specific behaviours (long-context recall, rare terminology, numeric reasoning, structured-output formatting, tool-call arguments) degrade. So testing means running the quantised and original models on the same frozen evaluation set of your own support questions, including tool-call and citation cases, and comparing scores slice by slice, not only the average. Then measure speed on your real prompt and output lengths at the real concurrency: a quantised format only speeds things up if your runtime has efficient kernels for it on that hardware, so operational compatibility (serving framework support, driver versions, whether a smaller device is supported at all) belongs in the decision. A throughput win that needs a custom runtime to maintain is paid for in operations labour.

A fleet sized for the daily peak is idle most of the day. The support assistant's illustrative demand is 4,800 tasks per hour from 10:00 to 14:00, 2,400 for eight shoulder hours, and 600 for the twelve night hours. Each replica handles 1,200 tasks per hour within the latency objective and costs $6 per hour. To keep 25% headroom the peak needs 4,800 × 1.25 ÷ 1,200 = 5 replicas.

tasks per day         = 4,800 x 4 h + 2,400 x 8 h + 600 x 12 h = 19,200 + 19,200 + 7,200 = 45,600

fixed at peak         = 5 replicas x 24 h = 120 replica-hours  -> $720/day  -> $0.0158 per task   (utilisation 31.7%)
autoscaled            = 5 x 4 h + 3 x 8 h + 2 x 12 h = 20 + 24 + 24 = 68 replica-hours
                                                        -> $408/day  -> $0.0089 per task   (utilisation 55.9%)

saving                = (120 - 68) / 120 = 43.3%

The night floor is two replicas, not the one that 600 tasks per hour (0.5 of a replica) would suggest, and not zero, for reasons that are all failure modes rather than arithmetic. Cold start: a new replica must be scheduled, pull the model artefact, load weights into accelerator memory and warm up before it takes traffic; call it a fictional 90 seconds. Scaling from zero means the first request waits that long, an unacceptable user experience for chat. Redundancy: one replica is a single point of failure, so a replica restart takes the service down. Reaction lag: a reactive scaler that adds replicas when queue depth rises is always behind a sudden spike by the cold-start time, so the traffic that arrives in those 90 seconds is served by the headroom you kept, or not at all.

Tuning this is the interplay of three knobs. A lower queue-depth or utilisation target (scale out at 50% rather than 80%) buys headroom at the cost of more replica-hours; a higher one is cheaper and more fragile. Minimum replicas set the floor cost; scheduled scaling (the 10:00 ramp in the figure is predictable) removes the reaction lag for known patterns. Regional headroom is the same trade on a larger scale: keeping spare capacity in a second region to absorb a failover multiplies fixed cost, and whether the SLO warrants it is a business decision. Overprovisioning is expensive and visible on the bill; underprovisioning is cheaper on the bill and expensive everywhere else, which the reliability section prices. The capacity math itself (Little's Law, queueing) is in AI Infrastructure Scaling, and the request path, health checks and rollout behaviour in Model Serving.

Training and serving are different cost structures and mixing them produces bad decisions. Serving cost scales with every request. Training cost is a lump you pay up front (and again at each refresh) and recover by spreading it over future requests. The relevant question for most application teams is not "how much does it cost to train a model" but "is adapting a model cheaper than prompting or retrieval for this need?"

The components are compute (accelerator-hours for each run, including the runs that fail or are discarded), experiments (you rarely get the right data mix or hyperparameters on the first attempt, so the number of runs is the multiplier), checkpoints (large artefacts stored per run and per version, plus the cost of recomputing if they are lost), data pipelines (collection, cleaning, labelling, deduplication, and the people doing it, often the largest line item), and evaluation of every candidate. Operating the cluster is covered in Training Infrastructure.

Fine-tune, prompt or retrieve? A worked amortisation

Suppose the team considers fine-tuning a model so the assistant's tone and formatting rules no longer need 600 tokens of instructions in every prompt. All fictional:

one run                : 8 accelerators x 10 h x $2.50/accelerator-hour = $200
experiments            : 6 runs                                         = $1,200
labelling              : 3,000 examples x $2                            = $6,000
one-off total          : 1,200 + 6,000                                  = $7,200
annual refresh         : 4 x $500 (retrain + eval after policy/format changes) = $2,000

saving per request     : 600 tokens x $1 / 1,000,000                    = $0.0006
volume                 : 2,000,000 requests/month x 12                  = 24,000,000 / year
annual saving          : 24,000,000 x 0.0006                            = $14,400
net first year         : 14,400 - 7,200 - 2,000                         = +$5,200
break-even volume      : (7,200 + 2,000) / 0.0006                       = about 15.3 million requests

At 2 million requests a month the project pays back inside the first year; at a tenth of the volume it never would. But the arithmetic hides the decisive point: it covers only the instructions that never change. Policy facts (refund windows, eligibility, regional rules) change when the business changes, and baking them into weights means retraining to correct them, with a long lag during which the model is confidently wrong. Those belong in retrieval, where an updated document is available as soon as it is indexed. Fine-tuning is a reasonable way to buy style, format and task behaviour; it is a poor way to store changeable facts. If the fine-tuned model is also self-hosted, add its serving cost model from above; if it replaces a hosted frontier model with a smaller tuned one, count the quality tests as part of the project cost.

Reliability and cost are usually discussed as opposing forces. The numbers show that under-investing in reliability is often the most expensive option, because failures consume spend and then demand more.

Retries multiply spend and can amplify an outage. With a retry probability p per attempt and two retries, expected billed attempts are 1 + p + p². At a healthy p = 5% that is 1.0525, a 5% surcharge. During overload p can rise to 20%, giving 1.24, and to 40%, 1.56: the retries triggered by overload add load that creates more timeouts, a feedback loop. A timeout set shorter than the slow tail of the model (say 8 s when the p95 of a valid response is 9 s) turns 5% of healthy requests into retries that re-bill the full input. The discipline is bounded retries with jitter, retry budgets (no more than a percentage of traffic may be retried), and idempotent tool calls so that a retry cannot execute a refund twice.

Failed tasks are paid for twice. A task that fails after consuming tokens produces no value, and then costs a human handoff. If 3% of tasks fail after their full spend, cost per success is cost ÷ (1 − 0.03) = 1.031 × cost, and the handoff costs more than the model: 3% × $4 = $0.12 per task.

Provider outages and incidents. A hosted dependency will have incidents. A fallback to another provider or a self-hosted standby costs standby money and a prompt-compatibility testing effort, and a fallback that quietly routes everything to a pricier model during an incident can double the day's bill; set alerts on spend rate, not only on error rate. Incident response itself is cost: engineers' hours, partial refunds or credits under an SLA, and the trust lost by users who were told to wait.

The false economy of aggressive scale-down

From the autoscaling example, the night floor is 2 replicas. Cutting it to 1 saves 1 replica × 12 night hours × $6 = $72 per day. Now price what the cold start does to the 06:00–07:00 ramp, where a cold replica takes 90 seconds to arrive. If, in this scenario, the ramp hour has 1,200 tasks and a quarter of them (300) time out or fail during the gap, then at $4 per escalation the loss is 300 × $4 = $1,200, which is 16.7 times the $72 saving, before counting the retries those failures trigger. The same reasoning applies to an aggressive timeout: shaving the wait budget looks like protection against slow calls and in fact converts slow successes into paid failures. The assumptions (a quarter of the ramp lost, $4 per escalation) are illustrative; the lesson is to price both sides, and to treat the cost of a lost task as part of the capacity decision, not an afterthought.

Optimisation changes can be wrong in ways that are invisible at small scale. A disciplined experiment changes one lever at a time (so you know which change moved which number), freezes the inputs, and fixes the acceptance rule before looking at results. Our lever is the context trim from earlier: summarised history and one fewer duplicate passage. It is a paired experiment: the same 9,000 real (anonymised) questions are replayed through the baseline and the treatment, so each question has two outcomes to compare.

Baseline, acceptance and rollback thresholds

Write the thresholds down first. In this example (fictional) the team requires: cost per successful task down at least 8%; p95 end-to-end latency no worse than the baseline; and groundedness (the share of answers whose claims are supported by the retrieved sources) no more than 1.0 percentage point lower, with a 95% one-sided confidence bound, i.e. non-inferiority rather than "the mean looks fine". Guardrails during the canary: abort if online groundedness sampled by the evaluator falls more than 1.5 points, p95 latency rises more than 10%, or the human-escalation rate rises by more than 0.5 points.

                       baseline      treatment (context trim)
input tokens / call    2,000         1,525
call cost              $0.004000     $0.003525      (1,525 x 1/M + 500 x 4/M)
success rate           92.0%         91.5%          (answer correct and grounded and resolved)
cost per success       $0.004348     $0.003852      (0.004/0.920 ; 0.003525/0.915)   -11.4%
p95 latency            4.8 s         4.4 s          (shorter prefill)
groundedness           94.0%         93.5%

Cost per success fell 11.4%, which is better than the 8% bar, and latency improved. The decision hinges on the half-point of groundedness. Is a 0.5-point drop real or noise? With 9,000 paired cases, 288 were right on the baseline only and 243 right on the treatment only (the rest agree). The paired difference is (243 − 288) ÷ 9,000 = −0.5 points. Because the comparison is paired, the variance comes only from the disagreeing cases: SE = √(((288 + 243)/9,000 − 0.005²) ÷ 9,000) = 0.256 points, so the one-sided 95% lower bound is −0.5 − 1.645 × 0.256 = −0.92 points, which clears the −1.0 margin, narrowly. An unpaired comparison with the same counts would have a wider interval and fail; designing the experiment as a replay is what makes it sensitive enough to answer.

from math import sqrt

def paired_gate(n, base_only, treat_only, margin_pp=1.0, z=1.645):
    """One-sided non-inferiority test on paired pass/fail outcomes.
    base_only: baseline passed, treatment failed. treat_only: the reverse."""
    diff = (treat_only - base_only) / n
    var = ((base_only + treat_only) / n - diff ** 2) / n
    lower = diff - z * sqrt(var)
    return diff * 100, lower * 100, lower * 100 > -margin_pp

def cost_per_success(cost_per_task, success_rate):
    return cost_per_task / success_rate

diff, lower, ok = paired_gate(9000, 288, 243)
print(round(diff, 2), round(lower, 2), ok)          # -0.5 -0.92 True
b = cost_per_success(0.004, 0.920)
t = cost_per_success(0.003525, 0.915)
print(round(b, 6), round(t, 6), round((1 - t / b) * 100, 1))   # 0.004348 0.003852 11.4

With the offline gates passed, the change goes to a canary taking 5% of live traffic with the guardrails above and a hard rollback to the previous configuration, which must stay deployable (prompt, retrieval parameters and cache keys versioned, so the rollback does not leave stale cache entries behind). After the canary, you re-baseline and move on to the next single lever. If the gates had failed, the useful output would be the failing slice, not just "no". Observability for the online guardrails comes from AI Observability: cost per success cannot be a monthly spreadsheet; it must be a metric you can slice by tenant, route and prompt version.

There is no universal ranking of levers; there is a measured cost breakdown and a set of constraints. The flow below encodes the order in which questions are worth asking, after you have produced the decomposition from the first section for your own system. It deliberately starts with reliability, because waste from retries and failures is multiplied by every other lever.

Constraints override the order. A tenant with strict data-residency requirements cannot be moved to a cheaper provider no matter what the cost chart says; a regulated workflow may forbid a cache that could serve a stale answer; a product with a 1.0 s TTFT objective has already decided the batch size. Treat the objective, the compliance rules and the quality guardrail as fixed, and the levers as the search space.

There are, therefore, no blanket rules. "Always cache" ignores stale-answer cost and tenant leakage. "Always quantise" ignores slice regressions and operational compatibility. "Use the smallest model" ignores the $4 escalation that dwarfs the token bill. "Autoscale to zero" ignores cold starts. Each statement is true in some scenario with measured numbers and false in another, and the discipline of this chapter is to calculate before applying it.

1. 1

   Instrument cost per successful task

   Break it into tokens, calls per task, attempts per call, retrieval/tool charges, idle capacity and resolution rate, per tenant and route.
2. 2

   Rank drivers by share of spend

   The biggest slice of the decomposition gets the first experiment, subject to the quality guardrail and latency objective.
3. 3

   Test one lever with a frozen replay

   Pre-register acceptance and rollback thresholds, run the paired comparison, report by slice.
4. 4

   Canary, then re-baseline

   Ship small with live guardrails, keep the rollback path, and measure again before choosing the next lever.

Everything above assumed that traffic is honest. Token-priced systems have a property that traditional APIs do not: a single request can be made arbitrarily expensive by its content. An attacker, or just a runaway client, who sends very long inputs, induces long outputs, or tricks an agent into looping through tool calls, can turn your cost model into a denial-of-wallet attack. The cost controls in this chapter therefore double as security controls: per-tenant and per-user token budgets, hard caps on input size, output length and tool iterations per request, spend-rate alerts that fire before the invoice does, and rate limits that apply to cost, not only to request count. They also interact with the cache: a cache shared across tenants without scoped keys saves money and leaks data, which is the same mistake in a different currency.

Budget enforcement is a design choice with its own cost: a hard cap that cuts a legitimate long conversation mid-answer is a quality failure, so caps should degrade gracefully (summarise, shorten the answer, escalate to a human) and be measured like any other lever. The next module, AI Security, widens the lens from cost to the full set of threats against AI systems and the controls that contain them. For how these cost decisions fit into a deployable whole, see Production AI Architecture.