Load Balancing

A load balancer spreads incoming connections or requests across a pool of backend instances, so no single instance is overloaded and the pool behaves like one reliable service. It also hides failures, by health-checking targets and routing only to those that are ready; enables scaling, because instances can be added or removed without clients noticing; and makes deployments safe through connection draining and slow start. Load balancing happens at several layers: internet-facing balancers such as AWS ALB or NLB, ingress controllers and internal balancers inside the platform, and client-side balancing in gRPC clients or service mesh sidecars. The choice of algorithm and health policy matters as much as having a balancer at all.

L4 load balancing works on connections: it sees IP addresses and ports, forwards TCP or UDP flows, and picks a target once per connection. It is extremely fast, handles millions of connections, can preserve client IPs and offers static addresses, which is why AWS NLB suits extreme or non-HTTP traffic. L7 load balancing understands the application protocol: it terminates TLS, reads hosts, paths and headers, routes each request by rules, can retry, and can balance individual HTTP/2 or gRPC requests over shared connections. The cost is more processing per request. A common pattern combines both: an NLB at the edge for raw capacity, then an L7 ingress or mesh for routing.

Round robin assumes every target is equal and every request costs the same, and neither is true in production. If one pod sits on a noisy node or is stuck in long GC pauses, it still receives its full share of requests, which queue up behind it. At ShopNow, one pod answering in 800 ms received a fortieth of the traffic and pushed p99 to 820 ms for the whole service. Round robin also ignores mixed instance sizes and the difference between heavy and light requests. Load-aware algorithms such as least outstanding requests, power of two choices or latency-based EWMA send less traffic to slow targets automatically; with outlier ejection, p99 fell to 150 ms.

Connection draining, called deregistration delay on AWS, means the load balancer stops sending new requests to a target that is being removed but lets in-flight requests finish before the target is fully dropped. It matters during scale-in and on every rolling deploy. To work end to end, the application must cooperate: on SIGTERM it should fail its readiness check, keep serving existing requests and shut down gracefully within a timeout shorter than the drain period. In the ShopNow example, aligning a 30-second deregistration delay with Spring Boot graceful shutdown at 25 seconds and readiness set to false on SIGTERM eliminated about 300 errors with status 502 per deploy.

Outlier ejection is passive health checking: the load balancer or sidecar watches real traffic and temporarily removes a target that is misbehaving, even if its health endpoint still reports healthy. In Envoy or Istio you might eject a host after five consecutive 5xx responses, keep it out for 30 seconds and lengthen the ejection for repeat offenders. It catches problems that active checks miss, such as a pod with a broken downstream connection or a degraded node. The critical safety setting is the maximum ejection percentage, for example 20 percent, so a shared problem such as a failing database cannot make the balancer eject the entire pool and turn degradation into an outage.

Power of two choices picks two targets at random and sends the request to the less loaded of the two, measured by outstanding requests or a latency estimate. It sounds trivial, but it gives load distribution close to always choosing the least-loaded target, without the cost and herd behaviour of finding the global minimum. When many independent balancers, such as sidecars, all pick the single least-loaded host from slightly stale data, they stampede it at the same moment; random sampling prevents that. Envoy's LEAST_REQUEST policy uses it by default, and combining it with EWMA latency makes it react quickly to an instance that has become slow.

gRPC runs over long-lived HTTP/2 connections, and an L4 balancer picks a target only when a connection opens. After scale-out, existing clients keep their connections to the old pods, so new pods get almost no traffic while old ones stay hot. The fix is to balance per request, not per connection. Use an L7 proxy or service mesh that spreads individual gRPC streams across backends, or client-side balancing with xDS or a headless service so each client holds connections to many pods. Set a maximum connection age, such as five minutes with jitter, so connections are recycled, and send GOAWAY on scale-in so clients reconnect cleanly elsewhere.

Because if the backend closes an idle keep-alive connection before the load balancer does, the balancer may send a new request on a connection that is already closing, and the client receives a 502. It appears as a small, steady trickle of errors that is hard to reproduce. The rule is that each hop's idle timeout must be longer than that of the hop in front of it. With an ALB idle timeout of 60 seconds, the Tomcat or Netty keep-alive timeout should be longer, for example 75 seconds. The same rule applies through every proxy layer, including service meshes, NAT gateways and client connection pools.

When requests benefit from always reaching the same backend for a given key, usually because that backend holds state or a warm cache for it. Routing by cartId or userId to cache nodes, sharded in-memory stores or stateful stream processors keeps hit rates high and avoids duplicate work. Consistent hashing, such as ring hash or Maglev, means adding or removing a node remaps only about one nth of the keys instead of reshuffling everything. The trade-off is balance: hot keys create hot nodes, and the algorithm ignores current load. Mitigate with virtual nodes and bounded-load consistent hashing, and keep it away from stateless tiers, where least-request balancing is better.

Per-target metrics matter more than aggregate ones, because averages hide the single bad instance. I watch load imbalance, such as the ratio of maximum to mean requests or in-flight requests per target, alerting above about 1.5; 5xx rate and latency percentiles broken down by target and by zone; ejection counts and healthy host counts; and connection metrics such as active connections, new connection rate and rejections. At the balancer itself, track capacity units such as ALB LCUs before big events, TLS errors, and target response time against total time. Deploy-time 5xx is a key signal as well: the goal is zero errors during rolling deployments.