What Is a Distributed System?

1Concept / Definition

A distributed system is a collection of independent computing nodes that cooperate by exchanging messages over a network to provide a service, where each node can fail independently of the others — yet users should experience it as one coherent system.

2Why It Exists

We distribute to scale, to survive failures, to be close to users and to respect organisational boundaries (Topic 1.2). The price is that the comfortable guarantees of a single process — shared memory, one clock, all-or-nothing failure — disappear.

3Core Concepts

3.1 Multiple nodes

3.2 Network communication

// A remote call looks like a method call — but it isn't one
RestClient pricing = RestClient.builder()
        .baseUrl("http://pricing-svc")
        .requestFactory(new JdkClientHttpRequestFactory(HttpClient.newBuilder()
                .connectTimeout(Duration.ofMillis(300)).build()))
        .build();
 
PriceQuote quote = pricing.post().uri("/quotes")
        .body(new QuoteRequest(sku, qty))
        .retrieve()
        .body(PriceQuote.class);  // may be slow, fail, or never return without timeouts

3.3 Independent failures

4How It Works Internally — Single Machine vs Distributed

4ADeep Dive — The Fallacies as a Design Checklist

5APIs / Features / Syntax

6Practical Examples

6.1 "It's just a Spring Boot app"

A team insisted their system wasn't distributed: one Spring Boot service with PostgreSQL. A network blip between the app and the database left transactions in unknown state and exhausted the connection pool. The app + DB pair is a distributed system.

6.2 Hidden node

An internal feature-flag service nobody considered critical went down; every service's startup blocked waiting for flags. Lamport's definition in action — the failure of a computer they didn't know mattered.

6.3 Worked example — inventory nodes and links

Scenario. ShopNow's checkout service failed in a different way during each incident, and the on-call engineers had no shared picture of what it depended on.

Walkthrough. The team listed every network link from checkout, with its protocol and the user-visible impact if it failed. Pricing over HTTP: checkout cannot quote prices. PostgreSQL over JDBC: orders cannot be saved. Kafka: events are not published, so downstream systems fall behind. Each row then got an owner, a timeout and a planned behaviour, such as using cached prices or buffering events in an outbox.

Result. The inventory became the basis of the service's runbook, and the next incident was diagnosed by checking the failing link in the table rather than by guesswork.

Takeaway. Start reasoning about a distributed system by inventorying its nodes and links and the impact of each failure. You cannot design for failures you have not listed.

6.4 Worked example — local vs remote latency

Scenario. A developer split a hot code path into two services and was surprised that a page which used to load in 50 ms now took over a second.

Walkthrough. The numbers explain it. An in-memory method call takes around 10 nanoseconds; an HTTP call within the same availability zone takes around a millisecond, about 100,000 times longer; a cross-region call takes around 120 ms. The page made 20 calls that used to be in-memory, so even at the best case it added 20 ms of pure network time, and with serialisation, connection set-up and retries it added far more.

Result. The team merged the chatty calls back into one coarse-grained API returning everything the page needed, and latency fell to 70 ms.

Takeaway. Remote calls are orders of magnitude slower than local calls. Design remote interfaces to be coarse-grained, and keep chatty interactions inside one process.

6.5 Worked example — timeouts everywhere

spring:
  http:
    client:
      connect-timeout: 300ms
      read-timeout: 1s

Scenario. ShopNow's checkout used Spring's HTTP client with default settings. When the pricing service hung during a deployment, every checkout thread waited indefinitely and the checkout service stopped responding.

Walkthrough. Every remote call needs explicit limits. The configuration sets a connection timeout of 300 ms, so unreachable hosts fail quickly, and a read timeout of 1 second, so hung responses are abandoned. Callers then handle the failure with a fallback, a retry with back-off or an error, instead of waiting forever. The same discipline applies to database, cache and broker clients.

Result. During the next pricing deployment problem, checkout returned cached prices within a second and stayed available, while the pricing team fixed their release.

Takeaway. Never rely on default timeouts. Set connection and read timeouts on every client based on the caller's latency budget, and define what happens when they fire.

6.6 Worked example — serialisation boundary

Scenario. A developer extracted pricing from the monolith but kept passing Order objects to it, expecting changes pricing made to the order to appear in checkout, as they had in-process.

Walkthrough. Across a network there are no shared object references. Checkout sends a representation of the order, such as JSON or Avro, and pricing works on its own copy. Any change pricing makes stays in its copy unless it is returned explicitly. The interface was redesigned to send exactly the data pricing needs and to return a quote, which checkout applies to its own order.

Result. Bugs caused by expecting shared state disappeared, and the contract between the services became explicit and much smaller.

Takeaway. Remote interfaces exchange copies, not references. Design them around explicit requests and results, and never assume that a callee's changes are visible to the caller.

6.7 Worked example — version skew

Scenario. During a rolling deployment of pricing from version 17 to 18, about half of checkout's calls failed for a few minutes, and then everything recovered.

Walkthrough. In a distributed system, different versions run side by side during every rollout. Checkout version 42 was calling both pricing v17 and v18. Version 18 had renamed a response field, so calls routed to new pods failed until the rollout finished. The fix was to make every change backward compatible: add the new field, keep the old one until all consumers migrate, and test compatibility in the pipeline.

// Tolerant reader on the consumer side
@JsonIgnoreProperties(ignoreUnknown = true)
record QuoteResponse(BigDecimal total, @JsonAlias("amount") BigDecimal price) { }

Result. Later rollouts caused no errors, because providers stayed compatible and consumers ignored unknown fields.

Takeaway. Assume mixed versions are always running. Contracts must tolerate both old and new versions during rollouts, and consumers should be tolerant readers.

6.8 Worked example — partial state

Scenario. A support engineer asked which version of the order service was running in production. The honest answer during a deployment was "both".

Walkthrough. During a rolling deployment, three pods run the old version and three the new one, so requests may be handled by either. Database migrations, caches and messages in flight may also reflect either version. There is never a single system version during change, and sometimes not between changes either, if a rollout stalls.

Result. The team added the application version to logs, traces and response headers, so every incident report showed which version handled each request, and investigations stopped assuming a single version.

Takeaway. Design and operate for partial state: mixed versions, half-applied migrations and in-flight messages are normal. Make versions visible in telemetry.

6.9 Worked example — node identity

Scenario. A batch job called the inventory service using a pod's IP address copied from a dashboard. It worked until the pod was rescheduled the next morning.

Walkthrough. Pod IP addresses are ephemeral: pods are replaced on every deployment, scale-in or node failure. Service DNS names, such as inventory.shop.svc.cluster.local, are stable and resolve to the current healthy pods behind a Kubernetes Service. Clients should address services, not individual nodes, and let the platform handle load balancing and discovery.

Result. The job was changed to use the service name and has run without failures since, through dozens of redeployments.

Takeaway. Address services by stable names and let the platform route to current instances. Anything that hard-codes node identities will break in a dynamic environment.

6.10 Worked example — one request, many nodes

Scenario. A product manager asked why clicking "Place order" could fail when "the server" was up.

Walkthrough. Tracing one click showed it touched five nodes: the API gateway, a checkout pod, a pricing pod, the PostgreSQL primary and a Kafka broker. Each hop is a separate process on separate infrastructure with its own failure modes, and the request succeeds only if every required hop succeeds within its timeout.

Result. Seeing the full path changed the discussion from "is the server up?" to which hops are essential, which can degrade gracefully and which need redundancy.

Takeaway. A single user action usually spans many nodes. Map the path of key requests so availability and latency can be reasoned about hop by hop.

6.11 Worked example — tracing the path

Scenario. Checkout latency rose from 300 to 900 ms at p99, and three teams each believed their service was not the cause.

Walkthrough. OpenTelemetry instrumentation produces a span for each hop with its timing, linked by a trace ID carried in the traceparent header. Viewing a slow trace showed that the pricing span took 600 ms, of which most was spent waiting for a database connection from an undersized pool.

management:
  tracing:
    sampling:
      probability: 0.1     # 10% of requests traced
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces

Result. Pricing increased its pool size and the p99 returned to normal the same day, without further debate.

Takeaway. Make distribution visible with distributed tracing. In a system of many nodes, traces replace guesswork with evidence.

6.12 Worked example — independent failure matrix

Scenario. ShopNow's leadership asked what would happen to customers if each service went down, but nobody could answer for every component.

Walkthrough. The team built a matrix of components and user impact. If recommendations is down, there is no impact because the product page falls back to no recommendations. If payments is down, checkout is blocked, which is critical. Building the matrix also exposed a surprise: the review service, considered non-critical, was called synchronously during checkout and could block orders.

Result. The review call was moved off the checkout path, and the matrix now drives which services need the highest availability targets and which need fallbacks.

Takeaway. Know how each dependency's failure affects users. Independent failures are only harmless if callers are designed to tolerate them.

6.13 Worked example — the database is remote

Scenario. Under load, ShopNow's order service occasionally failed with "connection reset" errors from the database, and the code treated these as fatal, losing customer orders.

Walkthrough. The database is a remote node, and connections can break mid-transaction because of network blips, failovers or idle-connection limits. For idempotent operations and transient error codes, the application should retry the whole transaction. The service was changed to retry transactions on specific transient SQL states, with a limit and back-off.

@Retryable(retryFor = TransientDataAccessException.class,
           maxAttempts = 3, backoff = @Backoff(delay = 100, multiplier = 2))
@Transactional
public OrderId place(PlaceOrder cmd) { ... }   // idempotent via cmd.requestId()

Result. Database failovers during maintenance now cause a short latency spike instead of lost orders.

Takeaway. Treat the database as a remote dependency that can fail transiently. Retry whole transactions for transient errors, but only when the operation is idempotent.

6.14 Worked example — clocks

Scenario. An audit report ordered events by timestamp across services and showed a payment captured before the order was placed, which alarmed the auditors.

Walkthrough. Each node has its own clock, and even with NTP, clocks differ by milliseconds and drift. Pod A was 12 ms ahead and pod B 35 ms behind, so events a few milliseconds apart could appear in the wrong order. Timestamps from different nodes are only approximately comparable.

Result. The report was changed to order events by their causal relationship, using the order's version and event sequence numbers, while timestamps are shown for information only.

Takeaway. Do not rely on wall-clock timestamps from different nodes to determine order. Use sequence numbers, versions or causal links when order matters.

6.15 Worked example — message loss handling

Scenario. A webhook to a logistics partner was sometimes received twice, and sometimes not at all, even though the network uses TCP, which "guarantees delivery".

Walkthrough. TCP retransmits lost packets within one connection, but it cannot help when a connection fails after the request was sent and before the response arrived. The caller does not know whether the request was processed. Recovery must therefore also happen at the application level: retry with the same idempotency key, and have the receiver ignore duplicates.

Result. With idempotency keys and retries, missing deliveries disappeared, and duplicates were harmlessly ignored by the partner.

Takeaway. Network reliability exists at two levels. TCP handles packets; applications must handle lost or uncertain requests with retries and idempotency.

6.16 Worked example — egress cost

Scenario. ShopNow's cloud bill showed a large and growing item for data transfer between availability zones, which nobody had budgeted for.

Walkthrough. Chatty services spread across three zones exchanged about 50 TB per month across zone boundaries, and cloud providers charge for cross-zone traffic. The biggest contributors were services fetching full product objects repeatedly instead of caching them, and logging pipelines shipping verbose logs across zones.

Result. Caching, smaller payloads and zone-aware routing for the heaviest flows reduced cross-zone traffic by about 60%, cutting the bill significantly.

Takeaway. Transport is not free. Measure network volume between zones and regions, and treat bandwidth as a cost to design for.

6.17 Worked example — secure links

Scenario. A security review found that ShopNow's services talked to each other over plain HTTP inside the cluster, on the assumption that the internal network was safe.

Walkthrough. The network is not secure by default: a compromised pod could read or alter traffic between services. The team introduced a service mesh that gives every workload a cryptographic identity and encrypts all service-to-service traffic with mutual TLS, with authorisation policies that state which services may call which.

Result. Traffic between services is now encrypted and authenticated without application code changes, and the policies document allowed communication paths.

Takeaway. Do not trust the internal network. Encrypt and authenticate service-to-service traffic, and restrict which services may talk to each other.

6.18 Worked example — design review question

Scenario. ShopNow's design reviews often approved integrations that later failed in production because nobody had considered what happens when a call hangs.

Walkthrough. The architecture group added a standard prompt to every review: "What if this call never returns?" For each remote call, the team must state the timeout, what the user sees, whether the operation is retried, and whether it is idempotent.

Result. The prompt surfaced missing timeouts and fallbacks in most new designs, and incidents caused by hanging calls fell sharply within two quarters.

Takeaway. Use a few standard review questions to apply distributed-systems thinking consistently. Simple prompts catch the most common and costly mistakes.

6.19 Worked example — count the fallacies

Scenario. A new team's design for a product page looked reasonable, but the reviewer quickly found several problems by checking it against the fallacies of distributed computing.

Walkthrough. Each design flaw maps to a fallacy. No timeout on the HTTP client assumes the network is reliable. Twenty sequential calls per page assume latency is zero. Large uncompressed payloads assume bandwidth is infinite. Plain HTTP between services assumes the network is secure. Naming the fallacy makes the flaw easy to explain and fix.

Result. The team added timeouts, aggregated the calls into two, compressed responses and enabled mutual TLS before going live.

Takeaway. Use the fallacies of distributed computing as a review checklist. Most distributed design flaws are one of them in disguise.

6.20 Worked example — minimum distributed setup

Scenario. A team argued that their small application was not a distributed system, because it was "just a Spring Boot app with a database".

Walkthrough. Even the minimal production set-up has two application pods, a database primary and a replica: four nodes connected by a network, each able to fail independently. Connections can break, the replica can lag, and pods can be replaced at any time. The same concerns as in larger systems already apply.

Result. The team added timeouts, retries for transient database errors and readiness checks, which prevented several outages during later database maintenance.

Takeaway. Almost every production system is already distributed. Apply basic distributed-systems practices from the start, even to small applications.

6.21 Worked example — interview answer

Scenario. In interviews, ShopNow asks candidates to explain what makes a system distributed.

How to answer. A strong answer covers three properties. Nodes: the system consists of independent processes on separate machines. Network: they cooperate only by passing messages over an unreliable network with latency, loss and limited bandwidth. Failures: nodes and links fail independently, so parts of the system can fail while others keep running, creating partial failures.

How to finish. Linking these properties to design consequences, such as timeouts, retries, idempotency and observability, and giving a real example shows that the candidate can apply the theory.

Takeaway. Answer with properties and their consequences, illustrated by a real system. That demonstrates understanding beyond a textbook definition.

7Common Problems & Mistakes

• Treating remote calls like local calls.
• No timeouts on clients.
• Ignoring "small" dependencies (config, flags, DNS).
• Assuming one system version.
• Comparing timestamps across nodes naively.
• Believing the fallacies.

8Performance & Production Considerations

9Architecture & Design Considerations

• Recognise every system with a network hop as distributed.
• Design each call for slow, lost and unknown outcomes.
• Map nodes, links and failure modes explicitly.
• Use the fallacies as a review checklist.

10Interview Questions & Answers