Capacity

Capacity planning is the practice of ensuring a system can handle expected demand while still meeting its SLOs, before that demand arrives. It has four parts. Forecast demand with the business, including growth, campaigns and events such as Diwali. Measure capacity, per pod and at scale, through realistic load tests up to the breaking point. Plan resources with headroom and failure tolerance, including downstream limits such as database connections, partner rate limits and AWS quotas. Then define what happens beyond the plan, through autoscaling, pre-scaling and load shedding. It is a continuous cycle, revisited quarterly and before every major event, not a one-off sizing exercise.

Headroom is spare capacity deliberately kept above the expected peak, so that forecast errors, sudden spikes, autoscaling lag and partial failures do not push the system past its knee, where latency climbs sharply. It is expressed as a margin over forecast, ShopNow uses at least 30%, plus a separate allowance for losing an availability zone. Headroom exists in layers: running pods, warm spare node capacity from placeholder pods, and reserved limits in databases and downstream services. Too little and the first surprise becomes an incident; too much and you pay for idle resources. Tier-0 paths such as checkout justify more headroom than internal reporting.

Each test type answers a different question. A load test checks whether the system meets its SLOs at the expected peak with a realistic request mix. A stress or breakpoint test keeps increasing load until SLOs break, revealing where the knee is and how the system fails. A spike test applies sudden jumps, such as five times normal in 30 seconds, to see how autoscaling and shedding cope. A soak test runs normal load for 8 to 24 hours to expose leaks, connection exhaustion and slow degradation. A failover test removes an AZ, node group or cache under load to check that the remaining capacity holds.

Little's law says that the average number of requests in a system equals throughput multiplied by the average time each request spends there. It lets you size thread pools, connection pools and semaphores from measured numbers rather than guesses. For example, ShopNow checkout at 2,100 requests per second with 0.15 second latency has about 315 requests in flight across the fleet. Per pod, a target of 300 requests per second at 0.2 seconds means about 60 concurrent requests, which is fine for Tomcat or virtual threads. The same maths shows the risk: if latency doubles during an incident, concurrency doubles too and pools exhaust.

Autoscaling reacts to load after it arrives, and it has limits it cannot see. HPA takes 30 to 90 seconds to act and new nodes another one to two minutes, so a tenfold flash-sale spike in 30 seconds overwhelms the fleet before help arrives; ShopNow saw a p99 of 2 seconds for three minutes. Scaling pods does not scale shared dependencies: more pods can exhaust Aurora connections or a partner rate limit faster. Account quotas for vCPU, ENIs or IP addresses can stop scaling entirely. So combine autoscaling with pre-scaling for known events, warm placeholder capacity, downstream agreements, raised quotas and load shedding.

Run scale tests beyond single-pod capacity and look for where throughput stops growing linearly. First measure one pod's maximum rps at the SLO, for example 120. Then test at increasing pod counts: if 10 pods give 1,200 rps but 40 give only 3,000, something shared is saturating. Instrument the shared components during the test: database connections, CPU and lock waits, cache operations per shard, Kafka partitions and consumer lag, NAT gateway throughput and partner response times. At ShopNow, a test at 2.5 times peak showed 80 pods opening 1,600 connections and exhausting Aurora, which led to RDS Proxy and smaller pools before the sale.

Provision so that the remaining zones can carry full peak load without help from autoscaling. With three AZs, losing one leaves two thirds of capacity, so each AZ must handle half of peak, which means provisioning about 150% of peak across all three. With autoscaling you can provision less and rely on scale-out, but only if AZ failure does not also block scaling through capacity shortages or quotas, so tier-0 services normally keep static stability. Spread pods with topology spread constraints, keep databases Multi-AZ with replicas able to take writes, and test the assumption by removing an AZ during a load test rather than trusting the arithmetic.

Load shedding is deliberately rejecting some work when a service is near saturation so that the most important work keeps meeting its SLOs, instead of everything slowing down together. You classify requests by priority, for example checkout payment over recommendations, and when a saturation signal crosses a threshold, such as available permits, CPU or queue depth, low-priority requests get a fast 503 with Retry-After. Degradation goes with it: disable non-essential features and queue non-critical work such as emails. ShopNow disables recommendations above 85% CPU. Shedding must be fast and cheap, happen early in the request, and be paired with client retry budgets so rejected work does not return as a retry storm.

Pay for guaranteed capacity only where it is needed, and make the rest elastic. Provisioning permanently for peak would cost ShopNow about ₹48 lakh a month, against ₹21 lakh with autoscaling plus pre-scaling for known events. Headroom is set by tier: checkout and payments get generous headroom and AZ-loss tolerance, while reporting runs with little and is throttled or shed first. Buy the steady baseline with Savings Plans, run elastic stateless work on Spot, and reserve capacity for critical event days. Use ephemeral, production-sized test environments instead of permanent ones. Compare plan versus actual after each event, since Diwali's extra cost came in at ₹2.7 lakh against ₹3.1 lakh planned.

A capacity review brings product, marketing, engineering, SRE and finance together, quarterly and before every major event, because no single team holds all the inputs. Cover the demand forecast and the business events behind it, load-test results including per-pod capacity, breakpoints and failure modes, and current utilisation against capacity per service. Review shared limits: database connections, cache shards, partitions, partner agreements and AWS quotas. List risks and mitigations such as pre-scaling, shedding and degradation, and cost against budget. Finish with owners and dates for every action, then review the last event's plan against actuals so the next forecast improves. Clear owners matter more than slides.