Message Queues v/s Message Brokers
This article intends to find the fundamental difference between Message Queues vs Message Brokers.
As is tradition with my newsletter topics, I selected this topic out of curiosity to study the difference between Message Queues and Message Brokers. Without further ado, let’s get started.
Problem Statement
Before deep diving in, let us understand why they even exist in our architecture.
Imagine you have two services: Service A and Service B. Now, Service A performs any operation(let’s say processed a payment, or completed a booking) and should update this information in a way that all other services (whoever needs it) should be able to consume this information almost immediately.
One way to solve this problem is for Service A to make a blocking call to Service B and update it in real time. But, with an increase in the number of services, Service A would have to call N number of services and it would have to consider all the edge cases: what if any downstream service is down, or the request gets timed out, etc. It’s simply not scalable.
That’s why we introduce asynchronous behavior in our architecture which works on a simple principle: decouple the producer and the consumer. Producer services produce the data in the form of a message in a queue-like structure and consumer services consume the data from these queues.
Message Queues
Message Queues are a single-queue data structure that stores messages temporarily in memory or on disk depending upon the configuration. These messages are produced by producers on one side and consumed by consumers on the other side. The key things to note here are:
FIFO order: First in first out, the order in which the messages are consumed from the queue is the same as the order in which they are produced in the queue.
One-Time Consumption: If a producer produces a message in the queue, then the message can only be consumed and processed by one consumer. It means that even if there are multiple consumers of a single queue, only one will be able to get the message and process it. Once it’s processed successfully, it will be gone from the queue and no other consumer can process it. The work distribution is such that each consumer gets a different set of items to process.
For reasons listed above, this type of messaging pattern is also called as one to one or point to point communications.
Message Queues usage: When you send an email, it goes into the recipient's inbox, where it waits until they choose to open and read it. This acts like a message queue, as the email is stored until the recipient is ready to access it.
Message Brokers
As per Wikipedia, “message broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. It is used for storing and delivering the message to the required destination.”
Message Brokers or Message Queues are both essentially used for asynchronous communication where the sender sends the message without caring about the fact if the receiver is ready to receive the message or not.
The main difference is Message Brokers consist of multiple Message Queues as highlighted in the above diagram. Each message queue could represent one type or category of data that consumer services can subscribe to and process.
For example: In a payments company, the message broker could consist of multiple message queues. One message queue could contain data about all the successful/failed transactions. Another message queue could contain data about promotions or offers. The consumers can listen to whatever data they are interested in. This is called the traditional pub/sub model.
Do note that a message queue doesn’t inherently handle routing, filtering, or broadcasting to multiple consumers – these are functions managed by the broker around the queue. Thus, can you imagine, that managing a broker that consists of multiple queues is a tedious task and has operational complexity involved in it.
Examples
All Message Brokers are Message Queues but all Message Queues are not Message Brokers.
The full-fledged Message Brokers provide more richer feature set compared to Message Queues like routing, load balancing, persistence of messages, scalability, delivering in a fan-out fashion, etc. That’s why nearly all of the popular technologies out there that are used for large-scale distributed systems can be used as a Message Broker with some configuration because they use Multiple Queues.
Kafka: Goes without saying, that it is probably one of the most popular tools used for large-scale distributed systems. Kafka can act both as a Message Queue(single consumer group) and a Message Broker(multiple consumer groups) as well.
RabbitMQ: The official website says: “RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine. It is currently used by millions worldwide.”
ActiveMQ: Apache ActiveMQ is an open-source message broker written in Java together with a full Java Message Service (JMS) client. The JMS is a middleware written for solving the producer-consumer problem and helps you with most of the boilerplate code.
Amazon SQS: Amazon Simple Queue Service (Amazon SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
If you want to use anyone of them as a Message Queue, then that’s fine as well.
When to use each?
Use a message queue when you need straightforward, point-to-point communication for task distribution or workload balancing, where messages are only consumed by one worker and are removed after processing.
This is ideal for simple, background job processing with basic reliability needs, as queues keep the system responsive by allowing consumers to handle tasks at their own pace.
In contrast, a message broker is best for complex, multi-pattern communication in systems that need flexible routing, like event-driven or microservices architectures.
Brokers allow for publish-subscribe models, making it easy to send the same message to multiple consumers and support real-time data processing and analytics. They often come with advanced features like replication, fault tolerance, and clustering, making them suitable for large-scale, highly reliable, and scalable systems
Hope you liked this edition.
Please consider liking 👍 and sharing with your friends as it motivates me to bring you good content. If you think I am doing a decent job, share this article in a nice summary with your network. Connect with me on Linkedin or Twitter for more technical posts in the future!
Resources
Topic or Queue: What exactly is Apache Kafka?
Kafka: Is it a Topic or a Queue?
Introduction to Message Brokers
Common Problems in Message Queues with Examples
Nice article Vivek! It's an important difference that sometimes is not clear enough.
Your visuals help a lot to understand the difference!
well explained.Thanks