Latency Components

The five components are network, CPU, database, serialization and queueing. Network time covers propagation, round trips, TCP and TLS handshakes and bandwidth. CPU time is business logic, framework overhead, garbage collection and crypto. Database time includes pool wait, query parsing and planning, execution and I/O, lock waits and result transfer. Serialization is encoding and decoding JSON or protobuf, plus compression. Queueing is time spent waiting for threads, connections, locks or partitions. End-to-end latency is the sum of these along the critical path, so the useful skill is not naming them but measuring which one dominates a specific request before changing anything.

RTT, or round-trip time, is the time for a message to travel from one endpoint to a peer and for the reply to come back, excluding processing on the far side. It is the basic unit of network cost because protocols need several of them: a new TCP connection costs one RTT, a full TLS 1.2 handshake two more and TLS 1.3 one, before the first request byte is sent. Typical values are around 0.3 ms within an AZ, about 1 ms across AZs, tens of milliseconds across regions and 60 to 80 ms on 4G. Ten sequential calls at 1 ms RTT cost 10 ms before any work happens.

Trace first. Instrument every service with the OpenTelemetry Java agent so each hop produces spans, then group span time by type: HTTP and gRPC clients for network and dependencies, JDBC for the database, messaging spans and the gaps between spans. Identify the critical path, because parallel calls only count for the slowest branch. For CPU-heavy spans, capture a JFR or async-profiler profile to see hot methods and GC. Then compare with pool and executor metrics, such as hikaricp.connections.pending, to separate waiting from working. The result is a latency budget table with an owner per component, and you attack the largest critical-path slice first.

Through queueing. Waiting time depends on utilisation, not on how fast the code is: for a simple queue, waiting is roughly service time multiplied by ρ/(1−ρ). A 100 ms operation responds in about 200 ms at 50% utilisation, 330 ms at 70% and around a second at 90%, with identical code. So more traffic, a smaller connection pool, a noisy neighbour stealing CPU, a slower dependency holding threads longer, or a hot row creating lock waits can all raise latency without a deployment. Other causes include data growth changing query plans and cache hit ratios falling. Pool, executor and lock-wait metrics reveal this hidden component.

Remove round trips first, because they dominate. Replace chatty call sequences with coarser APIs or batch requests, for example one Valkey MGET instead of fourteen GETs, and run independent calls in parallel so you pay only for the slowest. Reuse connections with keep-alive and pooling, prefer HTTP/2 or gRPC multiplexing, and use TLS 1.3 with session resumption to avoid repeated handshakes. Keep chatty paths in the same AZ with zone-aware routing, since each cross-AZ hop adds around a millisecond, and never put synchronous cross-region calls on a user request path. Finally, shrink payloads with smaller DTOs and compression where bandwidth matters.

Garbage collection consumes CPU and, in its pause phases, stops application threads, so any request in flight during a pause waits for it. The effect lands mostly in the tail: average latency barely moves, but p99 and p99.9 spike whenever a pause coincides with requests. Allocation rate is the main driver, because more allocation means more frequent collections. ShopNow's checkout allocated 2.4 MB per request and saw 180 ms G1 pauses every 20 seconds. Reducing allocation and moving to generational ZGC, the only ZGC mode in Java 25, cut pauses below a millisecond. Monitor jvm.gc.pause and allocation rate, and size heaps and container memory together.

Protobuf, usually with gRPC, encodes and decodes several times faster than JSON and produces much smaller messages, which cuts both serialization CPU and network transfer on high-volume internal paths. It also gives a typed schema with clear rules for backward-compatible evolution, which suits contracts between teams. The costs are real: messages are not human-readable, debugging needs tooling, browsers and many partners expect JSON, and schema files must be shared and versioned. That is why the usual split is protobuf between services, where volume and latency matter, and JSON at the edge for public APIs. Measure first, because on low-volume calls the gain may be negligible.

Database latency is several components, not one number. First is pool wait, the time to borrow a connection, visible as HikariCP acquire time and pending threads. Then come the network round trip to the database, parse and plan time, and execution, which includes CPU, buffer cache misses and storage I/O. Lock waits follow, often the hidden part under contention, and finally transferring and mapping the result set. Each needs a different fix: pool sizing, batching to cut round trips, indexes, shorter transactions or optimistic locking, and projections or pagination. Aurora Performance Insights and pg_stat_statements show where database time actually goes, for example lock waits rather than CPU.

Present it as a decision, not a data dump. Start with the business symptom, for example checkout p95 at 690 ms against an 800 ms target for the next sale. Show one component table built from traces: each critical-path slice with its duration, owner and evidence, so nobody argues from opinion. Highlight the critical path and the largest slices, then propose the top two or three changes with the expected saving, effort and risk of each. For ShopNow that was pool tuning and batching, taking p95 from 690 to 480 ms. Finish with how you will verify the result, through load tests and dashboards, and who owns each action.

The structural choices remove latency rather than tune it. Cut synchronous hops by giving services coarser APIs, letting them own the data they need, or composing in a BFF with parallel fan-out. Keep data close to compute with caches, same-AZ reads and CDN edges, and serve users from a nearby region, because distance cannot be optimised away. Move side effects such as emails, analytics and notifications off the request path using events or an outbox. Use efficient internal formats such as protobuf. Finally, design shared resources for headroom, keeping utilisation of pools, partitions and hot rows low, because queueing grows non-linearly near saturation.