Extending a RabbitMQ Cluster Across a WAN

Introduction

In a recent post on RabbitMQ, I mentioned that Rabbit clusters cannot span a WAN boundary. This post will dig into a few more details around the best ways to extend a cluster across a WAN.

When we build a cluster (of any type of software really), we are typically looking for enhanced scalability, performance, disaster recovery, or some combination of these factors. In the case of RabbitMQ, a cluster with mirrored queues really only provides the ability to allow producers and consumers to continue operating as if nothing happened if one or more nodes becomes unavailable. There are no performance or scalability enhancements. And in the case that a data center (or Amazon AWS availability zone/region) failure occurs, the cluster will be wiped out. There are other ways to handle these types of concerns if they are essential for your application, but Rabbit offers two approaches for building additional resiliency and/or scalability and performance into the messaging layer.

Federation

The concept of Federation is simple in RabbitMQ. An exchange or a queue can federate itself to other upstream exchanges or queues. This means that a message published to an exchange, or delivered to a queue, can be forwarded to another exchange or queue (whether within or across WAN boundaries). This provides many benefits, but the behavior of both needs to be understood or it can have unintended consequences. In particular, there are subtle differences between the two types of federated assets.

Federated Exchanges

In a federated exchange, an exchange federates to one or more other exchanges. Messages delivered to these “upstream” exchanges are then copied to the federated exchange. From the upstream exchange (the exchange that the producer initially published to), the message is then delivered to whatever queues are attached to that exchange locally. The message that was copied to the federated exchange is then ALSO delivered to the queues that are attached locally to that exchange. So the message will be delivered on both exchanges according to the rules of each exchange independently. It is similar in concept to a Topic with pub-sub semantics. Note that this will often result in a message being processed more than once, especially if this approach is used to “extend” the cluster across a WAN boundary. If the consumers are idempotent this will not be a problem. The following diagram illustrates federated exchanges.

Federated Queues

Federated queues behave quite differently. A queue can federate with other queues, in which case messages that get published to an upstream queue can be moved to the federated queue. RabbitMQ decides on which queue is more likely to have spare capacity (based on size of queue, average time to process, number of free consumers, and similar heuristics). Federated queues therefore allow for an increase in capacity and throughput via load balancing since Rabbit can offload processing to another node and/or cluster. The flow is also different than in a federated exchange. A message is published to an exchange, the exchange delivers it to the upstream queue, at which point Rabbit realizes is is federated and determines whether to move the message to the federated queue.

Shovel

Shovel is a related, but lower level, concept to federation. A shovel retrieves messages from a source queue and publishes them reliably to a destination exchange. The exchange is then free to send that message to the queues that have bound to it. This allows for the queue bindings in the downstream node to be applied and utilized for routing, whereas federated queues define completely the routing logic. This concept is illustrated below.

Shovels therefore behave more like federated exchanges in that they do not load balance but rather allow messages to be forwarded across a WAN. A shovel is literally a consumer and a producer connected to a source queue and a target exchange. The behavior of a Shovel is very often implemented using various Spring Integration components or hard coded Java, .NET, or Nodejs clients. Because they are so commonly seen, the RabbitMQ team added a plugin to allow for this use case to be implemented within a RabbitMQ configuration.

Summary

All three of the options listed above provide a useful and relatively simple way to connect RabbitMQ components across a WAN boundary. While they are all similar, they have subtle differences that should be considered when designing a cross-WAN link in RabbitMQ.