I'am trying to understand the topology of queues and exchanges MT creates in RabbitMQ.
I cannot get these two statements:
we generate an exchange for each queue so that we can do direct sends
to the queue. it is bound as a fanout exchange (is it about sending vs publishing?)
control queues are exclusive and auto-delete - they go away when you
go away and are not shared.
What for does MT need to send direct messages? Does this relate to control queues used by MT internally?
There is also no mentioning about dead letter queue, does that imply MT does not support one out of the box?
Oops, looked in a wrong place. It's here.
Michael Aldworth has described this in details in his excellent MassTransit Send vs. Publish blog post.
Essentially, for each MT endpoint, you get a exchange-queue pair, which is named after the queueName parameter value in the endpoint configuration. Topics, at the other hand, are RabbitMQ exchanges named after the message contract full type name.
Publish
You send a message to the topic, meaning to the message type exchange. MassTransit creates binding between the message type exchange and queue-name exchange on start up. In such way subscriptions work on the RabbitMQ levels. Publisher never knows who will be receiving the published message, if anyone at all.
Send
When sending, however, you need to specify the receiver address. By doing this you instruct MassTransit to deliver the message directly to the queue-name exchange. There is no binding between message-type exchange and queue-name exchange involved here. Therefore, the message will be delivered even if there is no consumer for this type of message at the target service. In such case the message will be moved to the deadletter queue (queue-name_Skipped).
Related
When a consumer connects to a receive endpoint, a uniquely named exchange and queue is created such as:
bus-{MachineName}-{ApplicationName}-{Identifier}
What is the purpose of this queue / exchange? Are there any docs detailing this?
This a non-durable queue and exchange per endpoint, used to handle request-response. Responses are sent to that address, as well as faults.
Let me start by describing the system. There are 2 applications, let's call them Client and Server. There are also 2 queues, request queue and reply queue. The Client publishes to the request queue, and the server listens for that request to process it. After the Server processes the message, it publishes it to the reply queue, which the Client is subscribed to. The Server application always publishes the reply to the predefined reply queue, not a queue that the Client application determines.
I cannot make updates to the Server application. I can only update the Client application. The queues are created and managed by the Server application.
I am trying to implement request/reply pattern from Client, such that the reply from the Server is synchronously returned. I am aware of the "sendAndReceive" approach with spring, and how it works with a temporary queue for reply purposes, and also with a fixed reply queue.
Spring AMQP - 3.1.9 Request/Reply Messaging
Here are the questions I have:
Can I utilize this approach with existing queues, which are managed and created by the Server application? If yes, please elaborate.
If my Client application is a scaled app (multiple instances of it are running at the same time), then how do I also implement it in such a way, that the wrong instance (one in which the request did not originate) does not read the reply from the queue?
Am I able to use the "Default" exchange to my advantage here, in addition to a routing key?
Thanks for your time and your responses.
Yes; simply use a Reply Listener Container wired into the RabbitTemplate.
IMPORTANT: the server must echo the correlationId message property set by the client, so that the reply can be correlated to the request in the client.
You can't. Unlike JMS, RabbitMQ has no notion of message selection; each consumer (in this case, reply container) needs its own queue. Otherwise, the instances will get random replies and it is possible (highly likely) that the reply will go to the wrong instance.
...it publishes it to the reply queue...
With RabbitMQ, publishers don't publish to queues, they publish to exchanges with a routing key. It is bad practice to tightly couple publishers to queues. If you can't change the server to publish the reply to an exchange, with a routing key that contains something from the request message (or use the replyTo property), you are out of luck.
Using the default exchange encourages the bad practice I mentioned in 2 (tightly coupling producers to queues). So, no, it doesn't help.
EDIT
If there's something in the reply that allows you to correlate it to a request; one possibility would be to add a delegating consumer on the server's reply queue. Receive the reply, perform the correlation, route the reply to the proper replyTo.
Is it possible to configure the topic to store a copy of just the last message and send this to new connections without knowing client identifiers or other info?
Update:
From the info provided by Shashi I found this two pages where they describe a use case similar to mine (applied over stock prices) by using retroactive consumer and a subscription recovery policy. How ever I'm not getting the desired behaviour. What I currently do is:
Include in the activemq the folowing lines in the policyEntry for topic=">"
<subscriptionRecoveryPolicy>
<fixedCountSubscriptionRecoveryPolicy maximumSize="1"/>
</subscriptionRecoveryPolicy>
Add to the URL used to connect to the brocker (using activemq-cpp) consumer.retroactive=true.
Set the consumer has durable. (But I strongly think this is not want since I only need the last one, but without it I didn't get any message when starting the consumer for the second time)
Start up the broker.
Start the consumer.
Send a message to the topic using the activemq web admin console. (I receive it in the consumer, as expected)
Stop consumer.
Send another message to the topic.
Start consumer. I receive the message, also as expected.
However, if the consumer receives a message, then it goes offline (stop process) and then I restart it, it doesn't get the last message back.
The goal is to whenever the consumer starts get the last message, no mater what (obviously, except when there weren't messages sent to the topic).
Any ideas on what I'm missing?
Background:
I have a device which publishes his data to a topic when ever its data changes. A variable number of consumer may be connected to this topic, from 0 to less than 10. There is only one publisher in the topic and always publish all of his data as a single message (little data, just a couple of fields of a sensor reading). The publication rate of this information is variable, not necessarily time based, when something changes a new updated message is sent to the broker.
The problem is that when a new consumer connects to the topic it has no data of the device readings until a new message is send to the topic by the device. This could be solve by creating an additional queue so new connections can subscribe to the topic and then request the device for the current reading through the queue (the device would consume the queue message which would be a request for data, and then response in the same queue).
But Since the messages send to the topic are always information complete I was wondering if is it possible to configure the topic to store a copy of just the last message and send this to new connections without know client identifiers or other info?
Current broker in use is ActiveMQ.
What you want is to have retroactive consumers and to set the lastImageSubscriptionRecoveryPolicy subscription recovery policy on the topic. Shashi is correct in saying that the following syntax for setting a consumer to be retroactive works only with Openwire
topic = new ActiveMQTopic("TEST.Topic?consumer.retroactive=true");
In your case, what you can do is to configure all consumers to be retroactive in broker config with alwaysRetroactive="true". I tested that this works even for the AMQP protocol (library qpid-jms-client) and I suspect it will work for all protocols.
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="FOO.>" alwaysRetroactive="true">
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy />
</subscriptionRecoveryPolicy>
</policyEntry>
The configuration example is taken from https://github.com/apache/activemq/blob/master/activemq-unit-tests/src/test/resources/org/apache/activemq/test/retroactive/activemq-message-query.xml
Messaging providers (WebSphere MQ for example) have a feature called Retained Publication. With this feature the last published message on a topic is retained by the messaging provider and delivered to a new consumer who comes in after a message has been published on a given topic.
Retained Publication may be supported by Active MQ in it's native interface. This link talks about consumer.retroactive which is available for OpenWire only.
A publisher will tell the messaging provider to retain a publication by setting a property on the message before publishing. Below is how it is done using WebSphere MQ.
// set as a retained publication
msg.setIntProperty(JmsConstants.JMS_IBM_RETAIN, JmsConstants.RETAIN_PUBLICATION)
I am following example as
https://svn.apache.org/repos/asf/qpid/trunk/qpid/java/client/example/src/main/java/org/apache/qpid/example/Hello.java
Here they have one exchange and one queue. How to send message to different queue in same exchange ? How to specify routing information in message ?
I am using java / JMS api
You can use topic exchange to deliver messages.
Consumer specifies routing key to a outgoing message.
Producer binds a queue to a topic type exchange with a binding key. Of course, you can bind two or more queues to one topic exchange with same binding key.
When a message with a routing key arrives in broker, the topic exchange will deliver it according to the match between routing key and binding key.
The above is the basic principle from AMQP. The rest is to implement your code in Java way.
I am trying to find a message bus provider that supports Durable Subscribers and allows me to replay, in order, based on the message timestamp, all messages for a given topic. Futhermore, I would like the message bus to reset each durable consumer's checkpoint when a message arrives late. E.g.
Client subscribes to topic 1 at 2009-12-22 12:00:00
Message 1 arrives, Timestamped 2009-12-22
Client receives Message 1
Client disconnects
Message 2 arrives, Timestamped 2009-12-21 18:00:00
Client connects
Client receives Message 2, then Message 1
I would strongly prefer an open source provider. Does anyone know of a message bus provider that supports this? I've tried to read the intro documentation for ActiveMQ, Mass Transit, etc but I have to admit that I am behind the curve on message bus terminology, so a lot of it went over my head.
AMQP (implemented by RabbitMQ, et al) lets you define durable queues and attach them to the same exchange. Each client that wants to receive messages first sets up its own durable queue, which will hold messages received from the exchange even while the client is disconnected.
The only limitation of this is that clients that have never connected, and which arrive on the scene unexpectedly, cannot belatedly setup a queue and request a dump of all previous messages. AMQP 1.0 might allow such universal persistence, but I don't know the new model that well, so I can't say for sure.
you may want to look at the spring integration project.
http://www.springsource.org/spring-integration