Is there any way we can know when a consumer disconnects from a queue or when a queue is deleted?
The requirement is as follows:
I'm building a system in which multiple clients can subscribe to certain events from the system. All clients create their own queue and registers themselves with the system using some sort of authentication. The system, as the events are generated, filters the events and forwards them to clients who are eligible for them.
I have implemented a POC for most part of it and it works well. An issue that I'm not able to fix is that, if a client just disconnects from the queue (due to program termination or so), the registration still exists and the system keeps trying to push messages to that client.
So we would like to be notified when a client disconnects or a queue gets deleted so that we can remove that client's registration data and no longer push messages to him.
Let your publisher utilize Confirms (aka Publisher Acknowledgements) and make client queue be exclusive and transient, so only one client at a time will be consuming from one queue and after it disconnection it will be deleted.
If you publish message that get routed to only one queue and that queue gone (assume you utilize publisher confirms and publish message with mandatory flag set) publisher will be notified that message cannot be routed with that message returned back to it, so you can stop publishing messages.
For details see How Confirms Work section in RabbitMQ blog post "Introducing Publisher Confirms" and Confirms (aka Publisher Acknowledgements) official docs.
Related
I am integrating several .Net modules using pub/sub messaging using RabbitMQ and MassTransit. Its important not to loose any message. When a module publishes a message it is supposed to be fire and forget action. In the moment the broker takes responsibility the publisher can fail and message will be delivered even much later.
I am trying to achieve the same behavior even when connection to broker is lost. I imagine a local persistent queue that will store the message(s) till they can be successfully delivered to the broker. The module can continue its operation meanwhile. If the module fails before it is able to deliver messages to the broker they stay stored locally. Once the module starts again and the connection to the broker is available, the stored messages are published to the broker. I am looking for an in-proc solution.
Does MassTransit support such a scenario out-of-the-box? Is there some 3rd party MT plugin that can do this?
Both the Solace queue and my program are subscribing to #LOG/INFO/SUB_ADD/DEVICE/ID/123.
When a new device which connects and subscribe to DEVICE/ID/123, both of them can successfully receive the subscribe event.
But, if I set subscription as #LOG/INFO/SUB_ADD/DEVICE/ID/>, only the solace queue receives the event.
Why can't my program receive the subscription event?
The problem persists even after my program sets it's subscription to #LOG/>.
One possible problem here is that there is an ACL rule preventing your application to subscribe to one or more topics in #LOG/INFO/SUB_ADD/DEVICE/ID/>. This can be easily verified with show log acl subscribe-topic.
If this is not the case, please revert back with the CLI outputs of show client <client-name> stats detail and show client <client-name> subscriptions to look further into the issue.
I built a small server (golang) to grab messages from a RabbitMQ and deliver them to connected browsers via a Websocket.
It works quite well, though has one caveat: the messages are acknowledged when delivered to the browser via the websocket. For most messages that is ok but some messages might be very important. If the user's browser received those but the user didn't SEE the message, it would be lost if the browser was closed or reloaded.
Is there a way to ack a message at a later time based on its message id (from the Delivery struct)?
The use case would be that some messages are acked when the user acknowledges them explicitly and at that point the message id is sent back to the tool to be acknowledged with RabbitMQ.
Even if you can do this, it's bad design.
What happens to the message if the user doesn't see it? Does your web server infinitely hang on to it? Does it "nack" the message back to the queue?
Neither of these options are good.
Hang on to every message, and RabbitMQ will start having issues with thousands of unacknowledged messages from a lot of users. Nack the message back to the queue and you'll thrash the message round in circles, spiking CPU resources on the web server and the RMQ server, as well as network traffic between the two.
The better solution to this problem is to store the message in a database, after pulling it out of RabbitMQ. When it gets sent to / viewed by the browser, update the database to reflect that.
From a yet-unpublished article I've written:
Store the message in a database.
Add a field to the database record that says who this message belongs
to. When the user reconnects later, query the database for any
messages that this user needs to see and send them along at that time.
The full process started above, then becomes this:
User's browser connects to SignalR/Socket.io/Pusher/websockets on web
server
Web server checks a queue for updates that happen during a long
running process
When a message for a logged in user comes in
If the
user is logged in, broadcast the message through the websocket to the
user
If the user is not logged in, store the message in a database
When the user logs in again, query the database and send all waiting
messages
It's what you would have done before the idea of a message
queue came in to play, right? It should be what you would do now that
you have a message queue, as well.
Is it possible to configure the topic to store a copy of just the last message and send this to new connections without knowing client identifiers or other info?
Update:
From the info provided by Shashi I found this two pages where they describe a use case similar to mine (applied over stock prices) by using retroactive consumer and a subscription recovery policy. How ever I'm not getting the desired behaviour. What I currently do is:
Include in the activemq the folowing lines in the policyEntry for topic=">"
<subscriptionRecoveryPolicy>
<fixedCountSubscriptionRecoveryPolicy maximumSize="1"/>
</subscriptionRecoveryPolicy>
Add to the URL used to connect to the brocker (using activemq-cpp) consumer.retroactive=true.
Set the consumer has durable. (But I strongly think this is not want since I only need the last one, but without it I didn't get any message when starting the consumer for the second time)
Start up the broker.
Start the consumer.
Send a message to the topic using the activemq web admin console. (I receive it in the consumer, as expected)
Stop consumer.
Send another message to the topic.
Start consumer. I receive the message, also as expected.
However, if the consumer receives a message, then it goes offline (stop process) and then I restart it, it doesn't get the last message back.
The goal is to whenever the consumer starts get the last message, no mater what (obviously, except when there weren't messages sent to the topic).
Any ideas on what I'm missing?
Background:
I have a device which publishes his data to a topic when ever its data changes. A variable number of consumer may be connected to this topic, from 0 to less than 10. There is only one publisher in the topic and always publish all of his data as a single message (little data, just a couple of fields of a sensor reading). The publication rate of this information is variable, not necessarily time based, when something changes a new updated message is sent to the broker.
The problem is that when a new consumer connects to the topic it has no data of the device readings until a new message is send to the topic by the device. This could be solve by creating an additional queue so new connections can subscribe to the topic and then request the device for the current reading through the queue (the device would consume the queue message which would be a request for data, and then response in the same queue).
But Since the messages send to the topic are always information complete I was wondering if is it possible to configure the topic to store a copy of just the last message and send this to new connections without know client identifiers or other info?
Current broker in use is ActiveMQ.
What you want is to have retroactive consumers and to set the lastImageSubscriptionRecoveryPolicy subscription recovery policy on the topic. Shashi is correct in saying that the following syntax for setting a consumer to be retroactive works only with Openwire
topic = new ActiveMQTopic("TEST.Topic?consumer.retroactive=true");
In your case, what you can do is to configure all consumers to be retroactive in broker config with alwaysRetroactive="true". I tested that this works even for the AMQP protocol (library qpid-jms-client) and I suspect it will work for all protocols.
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="FOO.>" alwaysRetroactive="true">
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy />
</subscriptionRecoveryPolicy>
</policyEntry>
The configuration example is taken from https://github.com/apache/activemq/blob/master/activemq-unit-tests/src/test/resources/org/apache/activemq/test/retroactive/activemq-message-query.xml
Messaging providers (WebSphere MQ for example) have a feature called Retained Publication. With this feature the last published message on a topic is retained by the messaging provider and delivered to a new consumer who comes in after a message has been published on a given topic.
Retained Publication may be supported by Active MQ in it's native interface. This link talks about consumer.retroactive which is available for OpenWire only.
A publisher will tell the messaging provider to retain a publication by setting a property on the message before publishing. Below is how it is done using WebSphere MQ.
// set as a retained publication
msg.setIntProperty(JmsConstants.JMS_IBM_RETAIN, JmsConstants.RETAIN_PUBLICATION)
I have a single publisher application (PUB) which has N number of subscribers (SUB)
These subscribers need to be able to catch up if they are restarted, or fall down and miss messages.
We have implemented a simple event store that the publisher writes to.
We have implemented a CatchupService which can query the event store and send missed messages to the subscriber.
We have implemented in the subscriber a PUSH socket which sends a request for missed messages.
The subscriber also has a PULL socket which listens for missed messages on a seperate port.
The subscriber will:
Detect a gap
Send a request for missed messages to our CatchupService, the request also contains the address on which to send the results to.
The catchup service has a PULL socket on which it listens for requests
When the CatchupService receives a request it starts a worker thread which:
Gets the missed messages
Opens a PUSH socket connecting to the subscribers PULL socket
Sends the missed messages to the subscriber.
This seems to work quite well however we are unsure if we are using the right socket types for this sort of application. Are these correct or should be using a different pattern.
Sounds okay. Otherwise 0MQ is able to recovery from message loss when peers go offline for a short time. Take a look at the Socket Options and specifically option ZMQ_SNDHWM.
I don't know just how guaranteed the 0MQ recovery mechanisms are so maybe you're best to stay with what you've got, but it is something to be aware of.