Messaging Systems: RabbitMQ - Part 2

It’s already been quite some time since I have written about MQTT and ZeroMQ. Let’s continue the overview over messaging systems with a look into RabbitMQ now. I am currently using RabbitMQ for my tutorial series on the development of a simple cloud computing platform.

RabbitMQ is a message broker supporting multiple protocols. The main supported protocol is AMQP, but STOMP and MQTT are possible with plugins.

A message brokers sits in the middle of your application and distributes messages between producers and consumers. It can thus be used to decouple your application components, e.g. in a microservice architecture.

RabbitMQ works with exchanges and queues. A producer sends a message to an exchange, which leads to the distribution of the messages into one or many queues. An exchange is basically a container for a set of rules that define into which queues messages should go. The exchange can be the default exchange in which case the message will just be routed to a queue with the specified name.

With RabbitMQ the message queues should ideally be short, otherwise the performance will become worse. This is because with big queues elements might get moved to the hard disk to free up RAM, synchronisation in a cluster takes longer and restarts on broker failure take longer.

Exchange Types

There are four different exchange types in RabbitMQ:

• Direct Exchange
• Fanout Exchange
• Topic Exchange
• Headers Exchange

A direct exchange sends a message from the exchange into queues based on a routing key it receives from the producer. On the other end we can register queues on the exchange with a binding key. A message with routing key R will be sent to a queue with binding key B if R and B are equal. The default exchange is a special form of direct exchange that has all queues bound to the queue name.

With fanout exchanges we can implement the publish-subscribe pattern easily. Fanout exchanges will route a message to all connected queues, which means that we can have many consumers connected to a fanout exchange and all of them will receive the same message.

Topic exchanges are an extension of direct exchanges. Messages are routed to queues based on a topic key and when binding a queue to an exchange we can specify wildcard matches for the topic key. However, this scheme only works well if you can establish an ordered naming scheme for your topics. The RabbitMQ tutorials give the example of log messages in which case topics could be kernel.error, kernel.info, user.error and so on. Consumers can then filter on things like kernel.* or *.error.

If the keywords on which you want to filter are not so well structured you can use the header exchange. This routes messages based on the header values of messages. You can then filter on the values of multiple headers.

There is also a plugin for a consistent hash exchange. This can be used for scaling when strict in-order processing of some messages is required. When using standard competing consumer pattern the problem is that parallel consumers pull messages from the same queue. Since each worker prefetches one or many messages, it can happen that worker 1 pulls message A and worker 2 pulls message B. Now let’s assume message A was earlier than message B, it could still happen that worker 2 finishes message B earlier.

With the consistent hash exchange messages with the same hash of the routing key will be sent to the same queue. If there is one consumer attached to each queue we can have scalability (by using multiple queues), but within each queue the messages are processed in order.

Queues

There are different features a queue in RabbitMQ can have.

• Automatically Named
• Exclusiveness
• Durability
• Auto deletion

Queue names in RabbitMQ are up to 255-character UTF-8 strings. It’s possible to specify a queue name on ones own or let the RabbitMQ server define the name. Having RabbitMQ specify a name is a useful feature if only one consumer will be using a queue. Named queues on the other hand are important if multiple consumers will use the same queue or if the queue name is important for routing (e.g. with the default exchange).

An exclusive queue can only be used by one consumer and it will be deleted when the connection with this consumer is closed. This is quite useful, because RabbitMQ will do all the house-keeping. On the other hand, if the consumer dies and has to be restarted all messages will be lost.

Durable queues are persisted and will also be available after a broker restart. This does not automatically include the messages though! Messages themselves can also be marked as persisted or transient. Messages within durable queues will only be recovered on a broker restart if they have been marked as persisted.

Queues can also be marked to be auto-deleted. This is a bit similar to exclusive queues, but with important distinctions. An auto-delete queue can be used by many consumers. It will be auto-deleted once the last consumer stops consuming. Queues that never had a consumer attached will not get deleted and queues on which the consumers only used get (RabbitMQ calls this the Pull API) instead of basic_consume won’t be deleted, either.

Consumers can acknowledge messages either immediately on receipt or after having fully completed a job. This is an extremely useful behaviour, because if your consumer crashes while it’s working on a job the message will not be lost. It can still be picked up later. However unlike for example Amazons SQS there does not seem to be a processing-timeout in RabbitMQ. If your consumer crashes on a job the message will only be free for other consumers once your consumer completely disconnects. This means: If your process crashes completely everything works fine. If on the other hand you have an exception handler, you have to make sure to nack the message.

Conclusion

With this extremely short overview over RabbitMQ I want to end. I personally like it very much, it’s extremely easy to setup, quite simple to understand and gives a lot of possibilities with different exchanges and queues. I have also tried Apacke Kafka some years ago for another use case and Kafka was a whole different beast regarding setup and maintenance. Of course, they serve different use-cases (I recommend the linked blog, the author has some nice articles about messaging systems and it seem’s he also worked on RabbitMQ), but I would prefer RabbitMQ over Kafka if possible for the use-case.