Load Shedding

Load shedding is the deliberate refusal of some work when a system cannot safely process everything it receives. Its goal is to keep a useful subset of requests fast and correct instead of letting overload make all of them slow or failed. It is an admission-control decision: the service admits that its current capacity is limited and picks what to serve.

Take a checkout platform that can reliably complete 5,000 requests per second given today's dependencies and resources, and a surge that offers 8,000. If it accepts all 8,000, queues grow, callers time out and retry, and the service spends its capacity on answers nobody is waiting for. If it rejects 3,000 cheaply at the door, the 5,000 it admitted complete inside their deadlines. The system serves less and delivers more.

The mental model is a bouncer with a headcount, not a bigger room. The name comes from electrical grids, which cut supply to some areas so the hospital stays lit, and the logic carries over: the decision about who goes dark is made in advance, on purpose, by value. Load shedding is also not scaling. Scaling raises sustainable capacity if the bottleneck scales and the new capacity has started; shedding is the protection for the interval when demand exceeds the capacity you have right now, including when a dependency degrades and no amount of extra instances would help.

A saturated service behaves nonlinearly. As concurrency rises, memory use, lock contention, garbage collection, context switching, and contention on downstream pools all grow. Queues add waiting time on top. A request that takes 50 ms when the service is healthy may sit 2 seconds in a queue before it starts, and by then its caller has given up and retried. The retry adds load; the abandoned original still gets processed. Eventually much of the work the service completes is work no caller needs.

The arithmetic is short enough to do by hand. Capacity is 5,000 rps, offered load is 8,000 rps, and clients time out after 1 second. The queue grows by 3,000 requests per second. A newly queued request waits queue-length ÷ 5,000 seconds, so waiting reaches 1 second when the queue holds 5,000 requests, after 5,000 ÷ 3,000 ≈ 1.67 s. From then on, in a first-in-first-out queue with no deadline check, every request is dequeued after its caller has already timed out. The server runs flat out at 5,000 rps and almost none of that output is useful. Throughput as measured by the server looks fine; goodput, meaning completions that arrived within the deadline, falls sharply.

That is why the right objective is not maximum accepted requests. It is to maximize valuable work completed within its deadline while keeping the system stable. Rejecting a request costs microseconds to milliseconds; admitting one that will miss its deadline costs its full service time and delivers nothing. This also argues for rejecting before expensive steps such as body parsing, database queries, or GPU execution, though never by skipping authentication or authorization for requests that are admitted.

Teams often respond to this by adding a big queue so that no request is ever dropped. A queue is a buffer for bursts, not a capacity strategy: when the average arrival rate exceeds the service rate, a longer queue only raises the latency at which everything fails, and eventually exhausts memory. Bounded queuing is fine when it is combined with rejection at the queue's admission point.

A shedder needs a measurement that says "admitting more work now will hurt." Candidates include in-flight request count, queue occupancy, queue age, latency, CPU saturation, memory pressure, connection pool exhaustion, and downstream saturation. Each fails in some situation:

No signal is universal. Pick the one tied to your bottleneck, smooth it so a single garbage-collection pause does not trigger a shed, and avoid rapid flipping between accepting and rejecting (covered below under hysteresis).

The concurrency limiter

The most direct signal is a cap on in-flight work. If a service admits at most 200 concurrent expensive operations, the 201st is rejected, queued briefly within a strict bound, or routed elsewhere. Little's law explains why this works: the average number of requests in the system equals arrival rate times time in the system, L = λ × W. Rearranged for a fixed limit L, the sustainable admission rate is λ = L ÷ W. With 200 slots and 50 ms per request, that is 4,000 requests per second. If a dependency slows and each request takes 200 ms, the same 200 slots admit 1,000 per second. Nothing was reconfigured; slots stay occupied longer, so fewer new requests fit.

This is the key difference from a rate limiter. A rate limit fixes requests per time and does not notice that each request has become four times more expensive. A concurrency limit tracks cost automatically, which makes it the better protection when the cause of overload is a slow dependency.

Adaptive limits

A static limit is only right for the conditions in which it was measured. Adaptive concurrency control adjusts the limit from observed latency and throughput, in the spirit of TCP congestion control: probe upward while latency stays near the lowest observed value, and back off when latency or errors say the system is queueing internally. Netflix's open-source concurrency-limits library is a well-known implementation of this family. The feedback loop needs care. Too aggressive and it oscillates; too slow and it reacts after damage is done; and latency-based schemes can mistake a dependency's slowness or a workload change for overload. A minimal additive-increase, multiplicative-decrease loop looks like this:

on completion(request):
  if request.dropped or request.latency > targetLatency:
    if now - lastDecrease > roundTripEstimate:      # at most one decrease per RTT
      limit = max(minLimit, limit * 0.9)            # multiplicative decrease
      lastDecrease = now
  else if inflight >= limit * 0.5:                  # only grow when the limit is being used
    limit = min(maxLimit, limit + 1 / limit)        # additive increase, ~+1 per limit completions

Queue age and deadlines

When requests do wait in a queue, shed on how long they have waited, not how many are waiting. Queue length depends on arrival and service rates; queue delay is what the caller experiences. Controlled-delay schemes such as CoDel (RFC 8289, originally for network queues) formalise dropping when the minimum delay stays above a target for an interval. Two simple rules capture most of the value. First, drop any request whose remaining deadline is shorter than its expected service time, because finishing it is pure waste. Second, under sustained overload serve newest-first (LIFO) or reject on dequeue, so the work you do finish is work whose caller is still waiting. Both depend on deadlines being propagated from the edge, for example through gRPC deadlines or an explicit timeout header, so every hop knows how much time is left.

The earlier a request is refused, the less it costs, so shed at the first place that can make the decision: a gateway or load balancer for coarse limits, then in-process admission on each service for the bottleneck it knows best. The constraint is that the decision itself must be cheap. Classifying by route and method needs only the request line and headers. Classifying by tenant requires identifying the caller, which needs authentication, and that has a cost. Do the cheapest classification first and keep security intact: admitted requests are authenticated and authorized as usual, and a shedder is never a reason to skip a check.

Admission can be written as one small function that composes with the rest of the request pipeline. Giving each class a fraction of the total concurrency makes lower priorities yield first as the service fills:

LIMIT = 200
CLASS_SHARE = { critical: 1.0, optional: 0.8, background: 0.6 }

function admit(req):
  cls = classify(req)                          # from route + trusted identity, never a client header
  if req.deadlineRemaining < expectedServiceTime(cls):
    return reject(req, reason = "deadline")
  if inflight >= LIMIT * CLASS_SHARE[cls]:
    return reject(req, reason = "overload:" + cls)
  inflight += 1                                # decrement in a finally block on completion
  return admit

With one shared limit, a flood of critical requests can still fill the pool and starve everything else. If lower classes need a floor, reserve a minimum share for them or put a maximum wait on how long they can be deferred; the next section returns to this. Critical paths need limits too, because admitting an unlimited number of "critical" requests collapses the service as surely as any other.

Not all requests carry equal value. During overload a commerce service should favour payment confirmation and order retrieval over recommendations, personalised banners, or analytics. A search service should keep interactive queries ahead of indexing and batch refresh. It helps to sort work into three kinds: critical correctness paths, user-visible optional features, and deferrable background work. A typical ordering is:

Authentication belongs in P0 because the critical paths depend on it; classifying login as low priority turns an overload into a lockout, including for the operators who need dashboards to fix it. The same goes for health checks: if an orchestrator probes a shedding instance and gets a rejection, it may restart or remove an instance that was merely busy, shrinking capacity at the worst moment. Keep liveness checks cheap and exempt, and keep them independent of deep dependency checks.

The figure shows the principle that you shed only as much as needed, lowest value first. Priority is not always business tier alone. Two other dimensions are worth encoding. Staleness: a request already past its deadline should be dropped before anything else. Cost: a request estimated at 50× the median is a natural candidate within a tier, and weighted admission, where an expensive request takes several slots, keeps a handful of 10-second requests from monopolising a pool that was tuned for 10 ms ones.

Keeping priorities honest

If every client labels its work critical, the system has no priorities. Derive the class on the service side from route, authenticated identity, and policy; if priority has to propagate between internal services, set it at the trusted edge and strip or ignore any client-supplied value. The opposite failure is a strict highest-first scheduler that starves lower classes indefinitely. Guard against it with minimum service shares, per-class quotas, and maximum waiting times, and decide explicitly whether starvation of background work is acceptable during an incident (usually yes, for a bounded time).

Tenant-based shedding

In multi-tenant products, plan tier can be a shedding dimension: free-tier traffic is reduced first while premium and contractual customers keep their service level. This differs from rate limiting. A rate limit applies per tenant all the time; tenant-aware shedding applies only when the platform is short of capacity and picks tenants by policy. Combine them: rate limits bound each tenant, and weighted shedding decides who yields when the sum of compliant tenants is still too much. Fairness among equals is approximate at best (per-tenant queues or token buckets get closer), so avoid promising perfect fairness.

Real systems combine several approaches. Reject excess is the simplest: above a global limit, return an overload response with retry guidance. It protects the service but treats all work alike. Priority-based shedding adds the value ordering above and is the most common choice at scale; it needs the priorities to be defined and tested beforehand. Adaptive shedding adjusts the limit or the shed fraction from live signals, which gives the best protection and the most tuning risk. Feature degradation turns optional capabilities off while the main workflow keeps running, so users get a lesser experience rather than none. Tenant-based shedding applies plan or SLA tiers during overload.

The trade-off runs along one axis. Shedding too little leaves the system in the collapse region; shedding too aggressively rejects legitimate users that capacity could have served, and every rejection is unmet demand. Start with a global concurrency limit, add priorities for the two or three classes whose distinction you can defend, and add adaptivity once you have the metrics to tune it.

Rate limiting enforces a predefined policy, usually per user, tenant, or key, even when the system is idle. Load shedding responds to measured overload and can refuse traffic that is within a caller's quota: a customer below 100 requests per minute can still get a temporary 503 because a dependency is saturated. Rate limiting cannot protect against 100,000 legitimate users arriving together; shedding can.

Backpressure tries to slow the producer so the excess is never sent; shedding refuses what arrives anyway. A circuit breaker stops calls to one unhealthy dependency, while shedding refuses at the entry because the whole service is saturated; when the breaker fails fast on a slow database, in-flight work drains and the shedder has less to do. Bulkheads partition capacity so one workload cannot consume everything; shedding decides what happens when a pool is full. Retries (see Retry & Backoff) sit on the caller side and can amplify everything else. These are complementary, and the capstone below puts them on one request path.

For temporary service overload, 503 Service Unavailable is the usual response, and the server may include Retry-After (a delay in seconds or an HTTP date) when it can give meaningful guidance. 429 Too Many Requests (RFC 6585) conventionally signals a rate-limit policy applied to that caller, although APIs differ in how they classify admission rejections. Choose one convention and document it. In gRPC the analogues are UNAVAILABLE for transient overload and RESOURCE_EXHAUSTED for quota-like exhaustion. The IETF RateLimit header fields are still an Internet-Draft rather than a published standard, so treat them as optional, evolving metadata. Give rejections a predictable machine-readable body:

HTTP/1.1 503 Service Unavailable
Retry-After: 2
Content-Type: application/problem+json

{"type":"urn:example:overloaded","title":"Service temporarily overloaded",
 "reason":"overload:optional","retryable":true}

If every rejected client retries immediately, shedding creates more traffic than it removes. Say 3,000 of 8,000 requests are refused and each rejected client resends once with no delay: the next second offers 8,000 plus 3,000 retries, and the service refuses 6,000. Pair rejection with retry guidance, exponential backoff with jitter, and caller-side retry budgets (for instance retries capped at roughly 10% of requests). A retry should also be abandoned when the caller's deadline has expired. Clients that honour Retry-After become part of the recovery.

What a rejection means for side effects depends on when it happened. A request refused at the door, before the handler, did not execute and is safe to retry. A request shed midway, or one whose response was lost, may have partly or fully succeeded, and the response must not claim it definitely did not. For operations with ambiguous completion, make them idempotent and reconcile rather than resubmit blindly. Pre-handler rejection is why the shedding check should be the first thing on the path.

Shedding does not have to mean an error page. A service can serve cached recommendations, omit optional enrichments, lower image resolution, defer nonessential writes, or return partial results, provided the API contract says so. It must not silently skip legally required checks, weaken authorization, or return fabricated success for a failed payment. Every degraded mode needs a correctness boundary and its own telemetry, so operators can tell full service from reduced service.

Some workloads cannot be shed without data loss. A telemetry pipeline may sample low-value events; a financial ledger cannot discard committed transactions. For durable asynchronous work, the usual approach is to stop accepting new jobs while preserving those already acknowledged (see Queues & Workers). "Drop the request" and "drop the data" are not equivalent: refusing a request at the edge loses nothing that was promised, whereas discarding an acknowledged job breaks a promise.

A naive controller sheds when CPU exceeds 80% and restores everything when it reads 79%. Shedding lowers CPU, the controller restores traffic, CPU rises, and the cycle repeats, producing alternating surges and collapses. The fixes are standard control practice: smooth the signal (an average over several seconds, not an instantaneous read), use hysteresis (a higher threshold to start shedding than to stop), and ramp admission back gradually instead of all at once.

ENTER, EXIT = 0.85, 0.70
ramp = 1.0                          # fraction of low-priority traffic admitted

every second:
  s = ewma(signal, alpha = 0.3)
  if shedding == false and s > ENTER:
    shedding, ramp = true, 0.0      # reject low-priority work
  else if shedding and s < EXIT:
    ramp = min(1.0, ramp + 0.1)     # re-admit 10% more each second
    if ramp == 1.0: shedding = false
  else if shedding and s >= EXIT:
    ramp = max(0.0, ramp - 0.2)     # signal rose again; pull back

Two further cautions. A system just recovered from a dependency outage may need a slower ramp because caches are cold and pools are empty. And shedding must always have an exit: if a rule is never relaxed, low-priority features stay dead for hours after the overload has passed and nobody notices until a user complains. Alert on shedding that remains active longer than the incident that caused it.

Finally, test the admission policy with realistic traffic mixes. A policy tuned on uniform 10 ms requests can fail when a few 10-second requests monopolise slots. Separate pools or weighted admission help when request costs differ widely, and a load test that drives offered load to two or three times capacity is the only way to see whether the limiter holds.

A checkout API normally receives 3,000 requests per second: roughly 1,500 critical (order creation, payment status), 900 optional (recommendations, banners), and 600 background (analytics events). Under normal conditions each request takes about 40 ms, so average concurrency is 3,000 × 0.04 = 120 in flight, and the service has a concurrency limit of 120 per instance group. These numbers are illustrative.

The database in one region slows down. Request time climbs from 40 ms to about 67 ms, and because only 120 slots exist, sustainable admission falls to 120 ÷ 0.067 ≈ 1,800 rps. The limiter and queue-age signal show rising in-flight counts. The shedding policy engages in stages: it defers analytics, which removes 600 and leaves 2,400; then it samples optional lookups down to one third (300 of 900), leaving exactly 1,800 (the figure above). If pressure persists, it limits new order-creation attempts while reserving capacity for payment-status reconciliation and already accepted orders. Rejected callers receive a controlled 503 with Retry-After; the retry path uses the idempotency key. Admitted requests keep finishing inside their deadlines.

The exact priorities are business decisions. For some systems order creation outranks status retrieval; for others, reconciliation must be preserved to prevent incorrect financial state. What matters is that the choice is written down, tested under load, and visible in dashboards before the incident, not improvised during it.

No shedding at all. At several times capacity the service tries to serve everything, queues fill, timeouts and retries pile on, and it enters the collapse region from the first section. The cure is to define tiers and test them well before a peak such as a sale event, not mid-incident.

Wrong priority rules. An authentication endpoint misclassified as low priority is rejected during overload; users cannot log in, and neither can the operators. Verify with synthetic load that P0 paths stay within their objective while lower tiers are being shed, and review classification whenever routes are added.

Shedding too late. A trigger at 99% CPU engages after queues are already full, and rejection cannot drain a backlog quickly. Trigger earlier, on a smoothed signal and with headroom for recovery. The right level comes from load testing the real bottleneck, not from a rule of thumb like "80%."

Rejection without guidance. A 503 with no Retry-After and clients that retry at once produces the amplification described earlier. Always give backoff guidance and enforce retry budgets on the caller side.

Shedding that never lifts. After the overload ends, the rule remains and low-priority work stays rejected for hours. Make restoration automatic, gradual, and alerting.

Shedding the health check. Probes rejected by the shedder cause the orchestrator to restart or evict healthy but busy instances, reducing capacity during the overload.

Trusting client-supplied priority. If a header decides the class, every client sets it to critical. Derive priority on the server.

Kubernetes' API Priority and Fairness lets the API server classify incoming requests into priority levels with their own concurrency shares and fair queuing, and reject requests with 429 when a queue is full, so a misbehaving controller cannot starve critical requests. That is request-level shedding almost exactly as described here. Kubelet node-pressure eviction is a related, coarser mechanism: under resource pressure it evicts pods, ranking by whether usage exceeds requests, then pod priority, which is shedding of running work rather than of requests.

Netflix has written publicly about prioritised load shedding at its API gateway, where requests are classified (for example, user-initiated playback versus prefetch or other background traffic) and the lower-priority classes are shed first. Google's SRE book and Amazon's Builders' Library both describe criticality-tagged requests and shedding by criticality as part of handling overload, along with client-side throttling so rejected work does not keep arriving at the same rate. The details vary by company; the shared structure is explicit classes, admission close to the front door, and degraded rather than binary behaviour.

Track offered load, admitted load, and completed useful work (within deadline) as three separate series; the gaps between them are the story. Add rejection rate by reason and by priority class, in-flight concurrency against the limit, queue waiting time, latency percentiles per class, downstream saturation, and retry rate or retry amplification as seen at the server (the ratio of requests with a retry-attempt marker to first attempts). Record the current value of the adaptive limit and how long shedding has been active.

Interpret rejection with care. A high rejection rate can mean the system is protecting itself exactly as designed, but it still represents unmet demand and needs capacity or product attention. Never call the system healthy because accepted-request latency looks good while most users are being rejected: measure success from the caller's side, including what fraction of offered load was served.

Use it for public or high-traffic APIs, and for any service where traffic spikes, partial dependency failure, or a retry storm can push demand above current capacity. It matters most where workloads have clearly different value: e-commerce checkout versus recommendations, playback versus discovery. In mission-critical systems it is essential and demands explicit priority contracts, because wrong tiers cause incidents rather than preventing them.

It is overkill for a small internal tool with predictable traffic and uniform work, where a simple concurrency cap and a clear error message are enough. It is also the wrong tool when the real problem is a standing capacity shortage: shedding converts overload into lost demand, which is acceptable for a spike and a failing strategy as a permanent state. If you are shedding daily, add capacity or reduce cost per request.

Overload and failure enter at different points of a request path, so a production API usually needs more than one control. Follow one checkout request through a gateway, an order service, a payment provider, an inventory service, and an event publisher:

1. Rate limiting applies customer or API-key policy at ingress, whether or not the platform is overloaded.
2. Load shedding rejects some new work when current capacity or dependency health cannot support it, preserving the most important operations.
3. Bulkheads reserve bounded resources for payment, inventory, and optional enrichment so one slow path does not consume every thread or connection.
4. Circuit breakers stop repeated calls to a dependency that is demonstrably unhealthy, subject to limited recovery probes.
5. Retries with backoff recover from selected transient failures when the operation is safe to repeat, the retry budget permits, and the deadline has room.
6. Backpressure propagates downstream capacity limits upstream, preventing unbounded queue growth in streaming and asynchronous pipelines.

These are complementary controls, not a mandatory stack for every service. Applying all of them indiscriminately adds latency, state, and new failure modes. Start from the service's actual bottleneck and failure history, describe the behaviour you want, and select the smallest set that enforces it.

Ordering and interactions

One workable arrangement puts the cheapest, most policy-driven checks outermost. Rate limiting runs first because it is per-identity and cheap; shedding comes next because it is service-wide; the breaker comes before the bulkhead gate so an open circuit consumes no permit (libraries differ); and a timeout bounds the call itself. Retries wrap the caller side. Details to get right across lessons: a retry must not retry a shed or breaker-open rejection immediately, and should honour Retry-After. Retries count against the client's own rate limit and against the retry budget, so a retry policy that ignores 429s can burn through a quota. A breaker should count timeouts and server-side failures (including 503 overload from the dependency) but not client errors or the caller's own 429s. A token bucket's burst capacity (for example refill 100/s, bucket 200) permits a brief spike that load shedding may still reject if capacity is short at that moment, which is correct, since the two answer different questions. And a bounded queue that drops or rejects when full is shedding; true backpressure slows or blocks the producer, as with credit-based or pull-based flow control.

Walk the layers in the order they engage. A timeout of 2 seconds limits each payment call. The payment bulkhead has 20 slots; at 100 payment calls per second, slow calls fill it within about 0.2 seconds, so order reads and inventory calls keep their own pools. When the first batch of timed-out calls returns at about 2 seconds, the breaker has enough slow-call evidence and opens. Further payment attempts fail in milliseconds instead of holding threads.

Because payment is critical, the application does not return a fake success. It returns an explicit pending or failed state according to its business contract. If an earlier attempt may have completed, the order is reconciled by stable payment identifiers and idempotency keys. Meanwhile, optional analytics events pass through a bounded queue whose consumers signal pressure upstream, and a service-wide admission controller sheds nonessential work if saturation stays high (here from about 1 to 8 seconds), then re-admits it gradually until about 20 seconds. Per-tenant rate limits keep enforcing fair use throughout, independently of the incident. When the breaker goes half-open it sends only a few probes; the first set fails because the provider is still slow, so the breaker reopens. The probe at the second attempt succeeds and the breaker closes.

The critical distinction is between protecting availability and preserving correctness. Fast rejection protects capacity. It cannot replace reconciliation, durable state, or correct transaction semantics, and none of these patterns turns a payment that failed into a payment that succeeded.

Read the table as a first question, not a final answer. Incidents usually involve two rows at once: a slow dependency fills a pool (bulkhead), retries amplify it (retry), and the service ends up over capacity (shedding). When symptoms overlap, ask which control acts closest to the cause and which one would have limited the damage if the first had not been in place.