It seems like the worker pattern, fanout, and filtered topics can all be implemented with topic exchanges. Why would I ever use a direct or fanout exchange instead?
We would like to codify common patterns found in our org in a library that abstracts the infinite flexibility of amqp (naming conventions, defaulting to durable, sending common headers, expirations etc.). Should we leverage the different exchange types or implement all patterns with topics; why?
(We have consumers/publishers in Java via spring boot, in golang, and in php)
Why shouldn't I use rabbitmq topic exchanges for everything?
nothing says you shouldn't. if it works for you, have fun with it!
From my RabbitMQ: Layout ebook:
The truth about exchange types is that there is no “master” type - not one to be used as a default, or most of the time. Sure, a given application may have its needs served by a single exchange or exchange type, but this will not always be the case. Even with in a single system, there may be a need to route messages in different ways and have them end up in the same queue.
If you find yourself in a situation where choosing one of the above exchange types will preclude a needed set of routing behaviors for your messages, use more than one exchange. You can route from any number of exchanges to a single queue, or from a single exchange to any number of queues.
Don’t limit your systems routing needs to a single exchange type for any given message or destination. Take advantage of each one, as needed.
On the different exchange types (again, from my ebook)
Direct:
A direct exchange allows you to bind a queue to an exchange with a routing key that is matched, case sensitively. This may be the most straight-forward exchange of them all, as there is no pattern matching or other behavior to track and consider. If a routing key from a message matches the routing key of a binding in the exchange, the message is routed.
Fanout:
Fanout exchanges allow you to broadcast a message to every queue bound to an exchange, with no way to filter which queues receive the message. If a queue is bound to a fanout exchange, it will receive any message published through that exchange.
and topic exchanges:
A topic exchange is similar to a direct exchange in that it uses routing keys. Unlike a direct exchange, though, the routing keys do not have to match exactly for a message to be routed. Topic exchanges allow you to specify wild-card matching of “topics” (routing keys) in your bindings. This lets you receive messages from more than one routing key and provides a level of flexibility not found in the other exchange types.
Related
I understand that Topics have the additional features of subscriptions and filters which Queues do not.
In which case, when would I absolutely need to use a queue over a topic?
For consistency, could I use a topics everywhere including as a replacement for queues?
A topic is not a replacement for a queue. The combination of a topic and a subscription is. A topic is allowing to “replicate” the same message to multiple subscriptions. A subscription what actually holds messages. A subscription is identical to a queue in its attributes and behaviour. You could replace a queue with a topic+subscription combo if you’d like, generating 2 entities per use case instead of a single queue. Just keep in mind there’s a finite number of entities per namespace.
When sending a message, MassTransit wraps that payload with an envelope which has a field called destinationAddress. What purpose does this field have?
I found this because I have a number of C# microservices communicating with some node and java based services - so I've been using the minimum payload defined here:
http://masstransit-project.com/MassTransit/advanced/interoperability.html
I've had no problem integrating the two services together I was just wondering what the point was of having the destinationAddress as part of the message itself? Is it just a belts and braces kind of thing to make sure messages don't go on the wrong queue by mistake?
I would have thought that all of this information can be derived since it is literally just built up of a) the message bus host and b) the queue name used when actually sending the message?
Transports have a variety of ways to delivering messages. For instance, publishing a message to a topic would set the destination address to (URI of topic) but it may be delivered to a queue (via a subscription, forwarded by the transport) with a different address. In this case, the envelope has the original destinationAddress, whereas the queue would have a different address.
There are also cases where messages may be scheduled, redelivered, faulted, etc., and having that information helps in troubleshooting production systems in cases where the original destination may not be known otherwise.
So, yeah, in the simplest case it seems superfluous, however, it comes in useful down the road when trying to figure out why something doesn't work.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
Is it possible to consume multiple messages in one call from a Tibco EMS queue. I am using the Receive method of the MessageConsumer class to consume the data currently but this just returns one Message. I'm wondering if there's something that returns an array of Message objects?
Thanks
A queue should not be treated as an inbound array object... mostly because the number of such objects could be massive... and such behaviors would be in direct contradiction with the basic "atomic information piece" notions of messaging. Queues should really be seen as an input "faucet" providing a flux of information.
That said : You might be looking for the javax.jms.QueueBrowser facility object. It IS in contradiction of typical messaging patterns, but can be useful. (Rules are meant to be broken sometimes, are they not ?)
Here is a link to many related examples.
EMS is a JMS provider, so these examples can be used with it.
To retrieve specific messages (to put in an array ?), you could then use a "receive" with message selectors (ex: on the message ID)
UPDATE : There is also this non-jms response : Use the Native EMS API to purge.
I have a design query regarding queues. My scenario is as follows:
I have to use a messaging system, with single producer and multiple consumers (asynchronous). The producer pushes different types of messages into the messaging system. Depending upon the message type, that particular consumer has to consume that message. (Each consumer is running on a different server). If one consumer is down and a message comes for that consumer, it will be in the messaging system only. If I use a message queue, the message in the queue will block the next messages that can be consumed by the other consumers. Are queues suitable for handling this kind of situation? Or do we need to go for a topic?
Whether you use a queue or a topic should depend on whether there an instance where multiple consumers must process the same message. If that is the case then a topic is required do generate that one-to-many pattern.
On the other hand, if any one message will only ever be consumed by one consumer, then you can use a queue or topic and the consumers specify the message type as a JMS selector. In this way, all consumers can listen on the same queue and each selects a different subset of messages. In the event one application is not there, it's messages do not "block the next messages that can be consumed by the other consumers" but rather they just stack up in the queue and other consumers still receive their messages based on selection criteria.
Please also realize that queues are lightweight constructions and you can easily have one queue per consumer. Typically, things providing a service listen on a well-known queue and each queue represents a different function of the service or a different service. Thus there may be many service input queues. Similarly, reply messages are generally uniquely addressed to the application instance that made the request and go to a unique, often dynamic, reply-to queue. Both of these implementations I have described lead to a separation of traffic across queues rather than pooling different message types into the same queue. Since JMS selectors always impart an additional processing cost, using more queues is generally more performant than selecting many types of message from the same queue.
I am responding to your question about selectors in the comment section here since I have more space and can put links in...
Section 3.8.1 of the JMS 1.1 spec states:
A JMS message selector allows a client to specify, by message header, the
messages it’s interested in. Only messages whose headers and properties
match the selector are delivered. The semantics of not delivered differ a bit
depending on the MessageConsumer being used. See Section 5.8,
“QueueReceiver,” and Section 6.11, “TopicSubscriber,” for more details.
Message selectors cannot reference message body values.
A message selector matches a message if the selector evaluates to true when
the message’s header field and property values are substituted for their
corresponding identifiers in the selector.
As noted above, selectors can be on fields that are implicit in the message such as MsgID or CorrelationID or thsey can be on fields specifically set by the message producer such as a message property. Either way, the client must specify the value of any selectors used by the message consumer.