Distributed System Complexity

Four intertwined sources of complexity that do not exist inside one process. The network is unreliable: messages can be lost, delayed, duplicated, reordered or blocked by partitions, and you can never be sure a message arrived. Timing is unpredictable: there is no global clock, clocks drift and jump, and pauses such as garbage collection make a slow node look dead. Partial failure means some components fail while others keep working, and a caller often cannot tell whether an operation happened. Consistency becomes a choice, because replicas and caches diverge. These are inherent properties, not bugs to fix once, so the architect's job is to choose the guarantees needed and pay for them knowingly.

A timeout only tells you that no response arrived in time; it says nothing about what happened on the other side. The request may have been lost before reaching the server, the server may still be processing it, for example during a GC pause or behind a slow database, or it may have completed successfully with the response lost on the way back. From the caller's side these cases look identical. That is why treating a timeout as a definite failure is dangerous: in the note's double-charge example, a payment retried after two seconds had in fact succeeded. Safe handling means idempotent retries, status queries or reconciliation rather than blind compensation.

Because wall-clock time can jump, so durations computed from it are unreliable. System.currentTimeMillis and Instant.now follow the system clock, which NTP, Chrony or an operator can move forwards or backwards, and virtual machine migrations can make it leap. If the clock moves back during a request, a timeout computed from wall-clock differences may fire late or never; a forward jump fires it early. For elapsed time use a monotonic clock, System.nanoTime in Java, which only moves forwards within a JVM, as the note's example shows. Keep wall-clock time for business timestamps, inject a java.time.Clock for testability, and never use it to order events across nodes.

Make the operation idempotent, so performing it twice has the same effect as performing it once. For calls with side effects, such as charging a card, the client generates an idempotency key tied to the business operation, for example the order id plus charge, and sends it with every attempt. The server records the key with the result and, on a repeated key, returns the stored result instead of charging again. Message consumers do the same with an event id or entity version table. Combine this with bounded retries, exponential backoff with jitter and a retry budget, so retries do not amplify an outage, and only retry operations known to be safe.

The Two Generals problem is a thought experiment showing that two parties cannot be certain they agree when communicating over a channel that may lose messages. One general sends an attack time by messenger; the other replies with an acknowledgement, but the sender cannot know the acknowledgement arrived, so it needs an acknowledgement of the acknowledgement, and so on forever. Whatever the last message is, it can be lost. The practical lesson is that no protocol can guarantee agreement over a lossy network, so real systems settle for practical certainty instead: retries with idempotency, timeouts and reconciliation. It is the same reason an HTTP timeout leaves the outcome of a request unknown.

It is a system model that sits between the synchronous model, where message delays and clock drift have known bounds, and the asynchronous model, which assumes no timing bounds at all. In a partially synchronous system the bounds exist but hold only most of the time, or eventually after periods of instability. This matters because FLP shows deterministic consensus is impossible in a purely asynchronous system with even one faulty process. Raft and Paxos assume partial synchrony: they always stay safe, never deciding two different values, but only make progress when timing behaves, for example electing a leader once heartbeats arrive reliably. Real networks, and clusters such as etcd, approximate this model.

A paused process does not know it was paused, so after waking it acts on assumptions that are no longer true. A stop-the-world GC, VM migration or heavy swapping can freeze a JVM for seconds. In the note's example a scheduler held a 10-second lock lease, paused for 12 seconds, and another instance took the lock, so both ran the nightly settlement. The same happens with leader leases and cached tokens. Checking the lease before writing does not help, because the pause can happen right after the check. The robust fix is a fencing token: the lock service issues an increasing number with each lease, and storage rejects writes carrying an older token.

I avoid theory and start with incidents they recognise. Each of the four sources gets one concrete story: a lost response that double-charged customers, a GC pause that ran a batch job twice, a partial outage where checkout failed because a minor dependency was down, and replica lag that showed sold-out stock as available. Then I show what each guarantee costs, such as idempotency work, extra latency for strong consistency, or cross-region coordination at around 150 ms per commit. Finally I tie choices to business requirements: strong consistency for payments and stock reservation, eventual consistency for listings and analytics. Framing it as risk, cost and customer impact lets stakeholders choose guarantees knowingly.