Load Balancer Fundamentals

A load balancer is a system that sits between clients and a pool of backend servers, distributing incoming requests so no single server is overwhelmed.

From the client's perspective, a load balancer is invisible — it looks like a single server at a single address. Behind that address, the load balancer decides, request by request, which of several backend servers should actually handle the work. This decision is what makes horizontal scaling possible: instead of buying one increasingly expensive machine, you run many ordinary machines and let the load balancer spread traffic across them.

Load Balancer Architecture: Clients, Balancer, and Backend Pool

The diagram above shows the basic architecture. Clients send every request to one address; the load balancer picks a healthy backend server using a distribution algorithm; and the chosen server does the work. Health monitoring keeps failed servers out of the pool, so the pool can grow, shrink, or lose a member without clients noticing.

The core responsibilities

Every load balancer story starts the same way: one server, then too much traffic for one server.

The bottleneck

A single application server has a hard ceiling — CPU cores, memory, open file descriptors, network bandwidth. You can buy a bigger machine (vertical scaling), but that has diminishing returns and a very real dollar and physical ceiling. Eventually the only way to serve more traffic is to run more machines (horizontal scaling) — and the moment you have more than one machine, something has to decide which machine handles which request. That something is the load balancer.

Availability, not just throughput

The second motivation is just as important: a single server is a single point of failure. If it crashes, restarts, or gets overwhelmed, every user is affected. A pool of servers behind a load balancer means one server failing degrades capacity slightly instead of taking the whole application down — assuming the load balancer notices the failure and reacts (see health checks, next chapter).

Load balancers operate at different layers of the network stack — and the layer determines what they can see and decide on.

Layer 4 (transport layer)

A Layer 4 load balancer works with IP, TCP, and UDP metadata. It can choose a backend using source/destination IPs, ports, and connection state, but it does not parse HTTP or other application payloads. That means it cannot make path-based, header-based, or cookie-based routing decisions. It often preserves end-to-end TLS via passthrough because it is not decrypting the traffic to inspect it.

Layer 7 (application layer)

A Layer 7 load balancer understands the application protocol — commonly HTTP/HTTPS or gRPC over HTTP/2. It typically terminates TLS, reads request metadata such as host, path, headers, or cookies, and then routes accordingly. That visibility enables content-based routing, redirects or rewrites, caching, and canary rules, at the cost of more CPU and memory per request than L4.

A reverse proxy sits in front of one or more backend servers and forwards client requests to them. A load balancer's primary job is specifically to distribute traffic across multiple backends, so many real systems use one product for both roles.

The term "reverse proxy" describes architectural position and forwarding responsibility; "load balancer" describes the traffic-distribution responsibility. Sometimes a reverse proxy fronts a single origin and does no balancing at all; sometimes the same NGINX, HAProxy, or Envoy deployment also performs load balancing, TLS termination, caching, and header policy. For the full reverse-proxy treatment, see Reverse Proxy Fundamentals.

Once you have a pool of servers, you need a rule for picking one per request. Every algorithm is a trade-off between simplicity, fairness, and awareness of real server load.

Worked trace: round robin vs least connections (illustrative)

Assumptions: three equal servers A, B, C; one request arrives per second; R1 is slow (10 s) and R2–R6 each take 1.5 s; ties go to the first server.

Round robin ignores load, so its fairness depends on requests costing about the same. Least connections uses the live active count, which is why it copes better with a mix of slow and fast requests.

A load balancer is only as good as its knowledge of which servers are actually healthy.

A health check is a lightweight periodic probe — often a GET /healthz request every few seconds — that the load balancer sends to each backend server. If a server fails enough consecutive checks (the unhealthy threshold), it's removed from rotation immediately, without waiting for an actual user request to fail.

Load balancing works best when servers are stateless — but many real applications keep some state in memory, and that changes the routing rules.

If a server keeps session data (a shopping cart, an authenticated session, a WebSocket connection) purely in its own memory, then subsequent requests from that same user often need to reach that same server or the state appears to vanish. Sticky sessions (session affinity) are the workaround: they pin a client to a server, commonly via a cookie the load balancer sets or via IP hash. The trade-off is uneven load distribution and a soft dependency on that specific backend.

Two more responsibilities complete the fundamentals picture: encryption and elasticity.

SSL / TLS termination

In TLS termination, the load balancer holds the certificate, decrypts incoming HTTPS, and then either forwards plain HTTP internally or opens a new TLS connection to the backend (re-encryption). In TLS passthrough, the load balancer does not decrypt at all; it forwards encrypted traffic end-to-end, which fits L4 designs but gives up most L7 routing visibility.

Horizontal scaling & high availability

Because the load balancer is the single entry point, it's also the natural place to add and remove capacity. Auto-scaling groups register new servers with the load balancer as demand rises, and deregister them (after draining connections) as demand falls — all invisible to the client.