Messaging Systems

Kafka is a distributed, partitioned commit log, while RabbitMQ is a message broker built around queues. In Kafka, producers append records to topic partitions, records are retained by time or size whether or not anyone has read them, and each consumer group tracks its own offset, so consumers can replay history and many groups can read the same data independently. Ordering is guaranteed per partition, which in practice means per key. In RabbitMQ, producers publish to exchanges, which route messages to queues by binding rules; consumers acknowledge each message, after which it is deleted, and failures can be redelivered or dead-lettered. Kafka suits event streams and replay, RabbitMQ suits work distribution and flexible routing.

JMS, now Jakarta Messaging 3.1, is a Java API standard rather than a broker. It defines how Java code connects, sends and receives through a ConnectionFactory, sessions, producers and consumers, with two destination types: queues for point-to-point delivery and topics for publish-subscribe, including durable subscriptions. Brokers such as ActiveMQ Artemis and IBM MQ implement the API, but their wire protocols differ, so JMS gives code portability, not interoperability between brokers. It supports local transacted sessions and, historically, XA transactions through JTA. Spring wraps it with JmsTemplate, the new fluent JmsClient in Framework 7, and @JmsListener with DefaultMessageListenerContainer. It remains common in banking and mainframe integration with existing MQ estates.

With Spring AMQP and Boot's spring-boot-starter-amqp, you send through RabbitTemplate, typically convertAndSend(exchange, routingKey, payload), which converts the object with the configured MessageConverter, usually a JSON converter, and publishes it to the exchange. You receive by annotating a bean method with @RabbitListener(queues = "billing"); a listener container consumes from the queue, converts the body to the method's parameter type, invokes the method and acknowledges on success. Topology such as a TopicExchange, a durable quorum Queue and a Binding is declared as beans, and RabbitAdmin creates it at start-up. For failures, configure retries and a dead-letter exchange, and enable publisher confirms when the producer must know that the broker accepted the message.

Spring Cloud Stream is a framework for event-driven microservices that binds plain functional beans to messaging destinations. You write a Supplier, Function or Consumer containing business logic, and the framework, through a binder for Kafka, RabbitMQ or a cloud service, connects its inputs and outputs to topics or exchanges named in configuration, such as ship-in-0 bound to orders.paid with a consumer group. It handles serialisation, consumer groups, partitioning, retries and dead-letter destinations consistently, and StreamBridge sends messages imperatively when needed. The benefit is uniform conventions and portable business code; the limit is that broker-specific behaviour still leaks through, so it does not remove the need to choose a broker deliberately.

I choose RabbitMQ when the workload is distributing work rather than publishing a replayable stream of facts. Typical signs are task or command queues where each message is processed once and then discarded, per-message acknowledgement and redelivery so that one slow message does not block others, priorities, per-message or per-queue TTLs, delayed retries, and routing by topic patterns or headers through exchanges. RabbitMQ also offers low latency at moderate volume and supports request-reply patterns. Kafka wins when consumers need replay, several independent consumer groups, long retention, very high throughput or strict per-key ordering at scale. When both needs exist, many organisations run Kafka for events and RabbitMQ or SQS for work queues.

Design for at-least-once delivery with idempotent consumers. Every mainstream broker redelivers a message if the consumer crashes before acknowledging or committing, and producers retry when an acknowledgement is lost, so duplicates are normal rather than exceptional. At-most-once loses data, and exactly-once holds only inside specific boundaries: Kafka transactions give exactly-once for read-process-write within Kafka, and SQS FIFO queues deduplicate within a five-minute window. As soon as the consumer writes to a database or calls an external API, those guarantees end. So consumers record processed message ids or use upserts and natural keys, and producers use the outbox pattern so the database change and the event cannot diverge.

Managed services such as SQS and SNS, Amazon MSK, Azure Service Bus or Google Pub/Sub remove broker operations: no patching, cluster sizing, disk management or failover drills. They scale elastically, integrate with cloud identity, encryption and monitoring, and often bill per use, which suits spiky or small workloads. The costs are semantics and limits that differ from self-managed brokers, such as SQS standard queues having no ordering, message-size and throughput quotas, visibility-timeout behaviour, and provider-specific APIs that tie code to one cloud. Bills can also grow sharply at high volume. I accept those costs when operations capacity is the bottleneck, and keep application code behind Spring templates or Spring Cloud Stream to contain lock-in.

I would start from purposes rather than products: event streaming for facts that many consumers need and may replay, and work queues for commands that are processed once. Pick one technology per purpose, often Kafka or a managed Kafka for events and RabbitMQ or a cloud queue for work, and publish platform conventions for topic and queue naming, partitioning keys, schemas in a registry with compatibility rules, security, retries and dead-letter handling. Mandate the outbox pattern for publishing from database transactions, and idempotent consumers. Provide an internal Spring starter that configures templates, listeners, tracing and consumer-lag metrics consistently. Finally, govern new broker requests through ADRs so the estate does not fragment team by team.