Interactive guide · 8 chapters · ~22 min read
Load Balancer Production Engineering
How load balancers are actually configured, operated, and debugged in production — reverse proxies, TLS, health checks, session affinity, auto scaling, global traffic management, and safe deployments.
Mangesh MandlikLast updated:
01 · Edge layer
Reverse Proxy in Production
In production, the reverse-proxy layer often owns request-handling policy — caching, compression, buffering, and sanitization — while the load-balancing role keeps backend pools healthy and distributed.
Teams may run NGINX, HAProxy, or Envoy as a dedicated reverse proxy directly in front of application servers, separate from the component whose main job is distributing traffic across a server pool. In other stacks the same product does both jobs. Either way, production configuration usually covers upstream server pools, timeouts, buffering behaviour, and cache/header rules keyed on request path and headers. For the fuller conceptual distinction, see Reverse Proxy Fundamentals.
Response caching
Cache-Control headers and proxy-level cache rules (e.g., NGINX proxy_cache) let the proxy serve repeat GET requests without hitting a backend, dramatically cutting load for read-heavy endpoints.
Header sanitization
Strip internal headers (X-Internal-*), set X-Forwarded-For/X-Forwarded-Proto for backends to trust, and enforce a maximum header size and reject requests with ambiguous Content-Length / Transfer-Encoding framing to blunt request-smuggling style attacks.
02 · Encryption
SSL Offloading
Centralizing TLS at the load balancer is common, but you still need to choose between termination, passthrough, and re-encryption.
Certificate management
Use automated renewal (ACME/Let's Encrypt, or a cloud provider's managed certificates) so certificates never expire silently. Certificate expiry is one of the most common self-inflicted outages in production systems.
Cipher suite hygiene
Disable legacy protocols (SSLv3, TLS 1.0/1.1) and weak ciphers. Compliance frameworks such as PCI DSS prohibit SSL and early TLS and require strong cryptography, which in practice means TLS 1.2 or later (check the current requirement text).
Termination, passthrough, and re-encryption
With TLS termination, encryption ends at the load balancer unless it opens a second TLS connection to the backend (re-encryption). With TLS passthrough, the load balancer never decrypts in the first place, which preserves end-to-end TLS but removes most L7 routing and inspection features. Pick deliberately based on trust boundaries, compliance, and routing needs.
03 · Reliability
Health Checks in Production
Getting health checks right is one of the highest-leverage reliability investments you can make.
Deep vs shallow checks
A shallow check (/healthz returns 200) only proves the process is alive. A deep check also verifies critical dependencies (database connection, cache reachability) are healthy — but must stay fast and must not check things the server can't independently fix.
Graceful degradation
Prefer a health endpoint that reflects genuine inability to serve traffic, not transient dependency blips. Flapping a server in and out of rotation due to a noisy dependency check is often worse than leaving it in with a slightly elevated error rate.
Startup vs liveness vs readiness
Kubernetes-style deployments distinguish startup probes (has the app finished booting), liveness probes (should this process be restarted), and readiness probes (should traffic be routed to it right now). Load balancers should key off readiness, not liveness.
Worked example: detection time and a minimal HAProxy config (illustrative)
Assumptions: probe every 5 s, unhealthy after 3 consecutive failures, healthy again after 2 consecutive successes. If a server dies just after a passing probe, the failing probes land at +5 s, +10 s and +15 s, so detection ≈ interval × unhealthy threshold = 5 × 3 = 15 s (worst case; a failure just before a probe is noticed in about 10 s). Until then the balancer keeps sending it traffic, so passive checks (ejecting on real 5xx/timeouts) shorten the visible error window. Rejoin takes about 5 × 2 = 10 s. Per-probe timeouts add to this if probes hang instead of failing fast.
textIllustrative HAProxy excerpt (adapt names, addresses and timeouts to your deployment)
backend app
balance leastconn
option httpchk GET /healthz
server app1 10.0.0.11:8080 check inter 5s fall 3 rise 2
server app2 10.0.0.12:8080 check inter 5s fall 3 rise 2inter is the probe interval, fall the consecutive failures before removal and rise the consecutive successes before rejoining: the same interval × threshold arithmetic as above.
04 · State
Session Affinity in Production
Sticky sessions are a compatibility tool for stateful backends, not the production default you aim for.
Cookie-based affinity (the load balancer sets an AWSALB/srv_id-style cookie) is the most common approach for HTTP; IP hash is the fallback when cookies are not available (raw TCP, some mobile clients). Both reduce the load balancer's freedom to redistribute traffic — a server that is overloaded still keeps receiving requests from clients pinned to it, and each pinned client becomes partially dependent on that backend staying available.
Draining pinned clients
When you need to remove a server (deploy, scale-in, failure), pinned clients need a plan: either re-establish affinity elsewhere (losing their in-memory state) or move session state to a shared store first. This is why affinity is best treated as a transitional workaround, not the long-term architecture.
05 · Elasticity
Auto Scaling
The load balancer and the auto-scaling group are two systems that must agree, in real time, about which servers exist.
Registration on boot
A newly launched instance registers itself with the load balancer (or is registered by the orchestration layer) and starts receiving traffic only once it passes its first health checks — never immediately on boot.
Connection draining on scale-in
Before terminating an instance, stop sending it new requests and wait for in-flight requests to finish (a deregistration delay, commonly 30–300 seconds) before the actual shutdown. Killing an instance mid-request causes visible errors.
Scaling policies
Target-tracking (keep CPU, request concurrency, or latency near a target), step scaling (add N instances per threshold breached), and predictive scaling (pre-provision ahead of known traffic patterns) are the three common policy shapes.
06 · Multi-region
Global Load Balancing
At global scale, load balancing happens before a request even reaches a data center.
Swipe to pan
GeoDNS
DNS resolves to a different IP depending on the resolver's geographic location, routing users to their nearest region. Simple, but DNS caching (client-side and resolver-side) means failover isn't instant.
Anycast
The same IP address is announced from multiple locations; network routing (BGP) delivers each user's packets to the topologically nearest location. Failover is a routing-layer decision, not a DNS one, so it avoids DNS caching delays but still depends on BGP convergence (time varies with network and configuration) and can break in-flight connections — this is how large CDNs and DDoS-resilient services operate.
Regional failover
When a region's health checks fail (or its capacity is exhausted), global load balancing reroutes its traffic to the next-nearest healthy region — turning a regional outage into a latency increase instead of a full outage, at the cost of increased load on the failover region.
07 · Safe rollouts
Blue/Green & Canary Deployments
The load balancer is the mechanism that makes zero-downtime deployments possible.
Swipe to pan
Blue/Green
A full parallel environment (green) is deployed and verified while receiving zero production traffic, then the load balancer switches 100% of traffic to it in a single cutover (existing long-lived connections may take a moment to drain). Rollback is just as instant — point traffic back at blue. Costs double the infrastructure during the transition.
Canary
A small percentage of production traffic is routed to the new version while the rest continues on the stable version. Traffic is ramped up gradually (5% → 25% → 50% → 100%) as metrics confirm health, keeping the blast radius small if something is wrong.
08 · Operating it
Monitoring, Common Issues & Debugging
The signals a load balancer exposes are some of the highest-value metrics in the entire system, because every request passes through it.
Key metrics
Request rate, error rate (4xx/5xx) by backend, latency percentiles (P50/P95/P99), active connection count per backend, and healthy/unhealthy host count over time.
Common production issues
Uneven distribution (usually a sticky-session or IP-hash artifact), thundering herd on scale-out (new instances flooded before warmed up), health check flapping (thresholds too aggressive), certificate expiry outages, and connection pool exhaustion from missing timeouts.
Debugging playbook
When a load-balanced service misbehaves: check per-backend error rate first (is it one bad server or all of them?), then check the healthy host count over time (was a server silently removed?), then check for a recent deploy or scaling event, then check certificate expiry dates, and finally check for sticky-session or IP-hash induced hot-spotting.
Key takeaways
- 01Set explicit connect, read and idle timeouts on the proxy/LB layer; one hung backend without them can exhaust the balancer's own pool and take healthy traffic down too.
- 02Key health checks off readiness, not liveness, and size detection time deliberately: a 5 s interval × 3-failure threshold is about 15 s before a dead server leaves rotation.
- 03Drain connections on scale-in and deploys (deregistration delay), and let new instances take traffic only after they pass health checks.
- 04Plan capacity for N-1 (and for a lost region): the surviving servers or regions must absorb 100% of the lost share.
- 05Automate rollback for canaries and blue/green, and alert on rising removed-from-pool events and per-backend error rates.