Horizontal Scaling

Horizontal scaling, or scaling out, adds more identical instances of a component and spreads work across them, usually behind a load balancer, so capacity grows with the number of instances rather than the size of each one. In Kubernetes that means more pods of a Deployment, managed by the HPA and with nodes added by Karpenter; in ECS, a higher desired task count. ShopNow's search runs 40 to 110 pods and serves 22,000 requests per second. Beyond capacity, it gives redundancy, because losing one of 60 pods costs only 1.7% of capacity, elasticity to scale in at night, and the ability to use cheaper spot or Graviton capacity.

The main prerequisite is stateless, interchangeable instances: any instance must be able to serve any request, so nothing a client needs may live only in one instance's memory or disk. Sessions, carts, rate-limit counters, uploaded files and scheduled-job ownership must move to shared stores such as Valkey, a database or S3, and identity should travel in tokens. If that is not done, the load balancer needs sticky sessions, scale-in loses user state and pod failures log people out. ShopNow's legacy seller portal kept sessions in memory and moved them to Spring Session with Valkey before it could scale. The other prerequisites are load balancing, scalable shared state, automation and idempotency.

A load balancer gives clients one stable address while the set of instances behind it changes constantly. It spreads requests across instances with an algorithm such as round robin or least outstanding requests, routes only to instances that pass health checks, and registers and deregisters instances as the autoscaler adds or removes them. During scale-in or deployments it drains connections so in-flight requests complete. Without it, clients would need to know every instance and react to failures themselves. At ShopNow an ALB fronts the services, with target groups fed by the AWS Load Balancer Controller, and inside the cluster Kubernetes Services do the same job for internal calls.

Horizontal scaling stops helping when the bottleneck moves to something all instances share. The usual culprits are the database writer, whose CPU and locks rise while throughput flattens; database connections, where pool size multiplied by pod count exceeds max_connections; hot cache keys that saturate one Valkey shard; partner API quotas; Kafka partition counts that leave extra consumers idle; and coordination such as distributed locks or leader election. The Universal Scalability Law describes this: contention and coherence costs make throughput per instance fall as you add instances. The signal is falling throughput per pod or flat p95 after scale-out, and the fix is to scale or partition the shared tier, not to add pods.

Make removal a coordinated sequence so the load balancer stops sending traffic before the application stops accepting it. When Kubernetes terminates a pod, the endpoint removal and the SIGTERM happen in parallel, so a preStop hook that sleeps about ten seconds gives the ALB and kube-proxy time to deregister the pod. Spring Boot's graceful shutdown then stops accepting new requests and lets in-flight ones finish within a phase timeout such as 30 seconds, and terminationGracePeriodSeconds must exceed the total. Align the ALB deregistration delay with it. At ShopNow, scaling from 60 to 24 pods under load produced 0.7% 502 and 504 errors before these fixes and 0.01% after.

Idempotency keys are needed because in a scaled-out system a retry may land on a different instance from the original request, so no instance can rely on its own memory to recognise a duplicate. A client times out, retries the order placement, and without protection two orders or two payment captures are created. With an Idempotency-Key header, the first instance records the key and the result in a shared store, and any instance that receives the retry returns the stored result instead of repeating the side effect. Store keys with a time-to-live matching the retry window, and make the record and the business change atomic, for example in the same database transaction.

Spread instances so that no single failure domain removes too many of them. Use topology spread constraints with the zone key and a maxSkew of one so replicas are balanced across three Availability Zones, and pod anti-affinity or a hostname spread so replicas do not share a node. A PodDisruptionBudget limits voluntary disruptions, such as node drains during upgrades or Karpenter consolidation, to, say, 10% of replicas at once. Size the fleet so that losing a whole zone still leaves enough capacity; ShopNow keeps an AZ reserve, which is why its checkout efficiency at 66 pods is 79%. Finally, run chaos tests that kill pods and simulate a zone loss under load.

Within a Kafka consumer group, each partition is assigned to exactly one consumer, so the number of active consumers can never exceed the number of partitions. Any extra consumer receives no assignment and sits idle while still costing memory and a slot. ShopNow scaled the inventory consumer group from 12 to 24 pods during a sale, but orders.events had 12 partitions, so half the pods did nothing and consumer lag kept growing. The fix is to provision partitions for future peak parallelism, here 48, ahead of time, because adding partitions later changes key-to-partition mapping and can break ordering for keyed events. Autoscalers such as KEDA should cap replicas at the partition count.

Plan it tier by tier, because each tier scales out in a different unit and has its own ceiling. For every tier, define the scale-out unit, the current range, the hard limit and the next step: checkout pods from 24 to 89 with a maximum of 110, search pods plus OpenSearch shards, Valkey shards from six to twelve, Aurora readers auto-scaling from two to five, Kafka partitions and consumers. Then budget what scale-out multiplies, especially connections through a proxy and memory per pod. Load test to measure scaling efficiency, find the next shared bottleneck and fix it before the sale, and check cloud quotas such as vCPU limits, subnet IPs and ALB targets.

The common culprits are in-memory HTTP sessions, local rate-limit counters, files written to local disk, in-process caches used as a source of truth, and scheduled jobs that run on every pod. Each works on one instance and breaks silently on many. ShopNow's rate limiter kept counters in pod memory, so scaling from 20 to 60 pods tripled the effective limit and partners exceeded their quotas until counters moved to Valkey. A @Scheduled cleanup ran on all 66 pods and multiplied database load 66 times until it used ShedLock or a dedicated CronJob. Sticky sessions on the load balancer often hide such state, so audit for them before scaling out.