I am using Nifi to get data from IBM MQ. It is working fine. My question is once the message is read from an MQ queue, does it get deleted from the queue? How to just read messages from the queue without deleting them from the queue?
My question is once the message is read from an MQ queue, does it get
deleted from the queue?
Yes, that is the default behavior.
How to just read messages from the queue without deleting them from
the queue?
You use the option: MQGMO_BROWSE_FIRST followed by MQGMO_BROWSE_NEXT on the MQGET API calls.
You can also open the queue for browse only. i.e. MQOO_BROWSE option for MQOPEN API call.
It sounds as if you would like to use a "publish/subscribe" model rather than a "point-to-point" model.
From ActiveMQ:
Topics In JMS a Topic implements publish and subscribe semantics. When
you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues A JMS Queue implements load balancer semantics. A single
message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
If you have a queue, when a consumer consumes that message, it is removed from the queue so that the next consumer consumes the next message. With a topic, multiple consumers can be subscribed to that topic and retrieve the same message without being exclusive.
If neither of these work for you, I'm not sure what semantics you're looking for -- a "queue" which doesn't delete the message when it is consumed will never let a consumer access any but the first message.
Related
I don't quite understand what this means:
Each message may have multiple subscribers. There are time
dependencies that exist between publishers and subscribers;
subscribers to a topic may only consume messages
published after a subscription was created.
What does it mean that there exists "time dependencies"? The above says that "subscribers may only consume messages after a subscription was created", I think I'm missing something vital here because I understood that as a subscriber cannot consume a message that is not yet created (but isn't this common sense).
I couldn't find any explanation on Google, so thank you for any help!
Edit: I found an excellent channel on youtube explaining spring boot and other technologies. In particular, this video helped me understand the difference between Queue and Topic (also what durable subscriber is).
What does it mean that there exists "time dependencies"?
To understand this better, compare it with a JMS Queue in which the message broker preserves all the messages (till TimeToLive expiration) published by the producer even if there are NO consumers CREATED.
Now, coming to a JMS topic, the message broker DOES NOT preserve the messages published by the producer if there are NO subscribers CREATED.
During the creation of topic subscription, you can tell the Message Broker that I don't want to loose the messages for this topic published FROM NOW ON by creating the Topic subscription as non-durable (using topicSession.createDurableSubscriber() method). Again, even in the case of non-durable subscription, the broker preserves the messages published after (subscription creation time) till TimeToLive expiration.
Simply that if you publish messages in to the topic before the consumer has subscribed, they will not see the messages.
Similarly, this is related to persistent subscribers. Normal subscribers only see messages while they are connected to the broker. Any messages before or after the subscriber has disconnected, will not be see by the subscriber.
However, with a persistent subscription, a subscriber can disconnect, yet the broker will continue to keep messages destined for that subscriber. When the subscriber reconnects, it will fetch all of those stored messages as well as any new ones.
I have a Tibco EMS Message Queue on a production system that routes messages from a single producer to a single consumer application.
We are scaling the application vertically, but due to financial constraints, we cannot scale the two applications in pairs.
A single producer will route messages to the message broker and the message broker will need to bridge the message to 1 of 3 consumers based on a message selector.
I have set up a queue bridge and selector to route messages on the producer queue to consumer queues. This is a 1 to many queue bridge.
I noticed that the bridged consumer queues have consumers attached to them and they are receiving the messages correctly based on the selector, however the producer queue retains copies of the messages which must be manually purged at the end of day.
What is the best way to handle this scenario using bridges and selectors so that the message is retained on the broker until it is consumed (durable) but once a message is consumed by a consumer queue, the message is removed by the broker.
The easiest way to dispose those messages of the original queue is by introducing MaxMsgs and MaxBytes on the queue.
As for you requirement that the message on the original queue can only be disposes if it was consumed on one of the bridged queues, this is not possible.
That said, it also it not needed, since the bridged queue keep its own message copy, regardless of what happens to the message in the source queue. So expiring messages in the original queue has no effect on the already bridged messages
I am using ActiveMQ v5.10.0 and having an issue almost every weekend where my ActiveMQ instance stops delivering persistent messages sent on queues to the consumers. I have not been able to figure out what could be causing this.
While the issue was happening I tried following things:
I added a new consumer on the affected queue but it didn't receive
any messages.
I restarted the original consumer but it didn't receive any messages after the restart.
I purged the messages that were held on the queue but then messages started accumulating again and broker didn't deliver any of the new messages. When I purged the expiry count didn't increase neither the dequeue and dispatch counters.
I sent 100 non-persistent messages on the affected queue, surprisingly it received those messages.
I tried sending 100 persistent messages on that queue, it didn't deliver anyone of them, all the messages were held by broker.
I created a fresh new queue and sent 100 persistent messages and none of them was delivered to the consumer whereas all the non-persistent messages were delivered.
The same things happen if I send persistent or non-persistent messages from STOMP producers. Surprisingly all this happened only for queues, topic consumers were able to receive persistent as well as non-persistent messages.
I have already posted this on ActiveMQ user forum: http://activemq.2283324.n4.nabble.com/Broker-not-delivering-persistent-messages-to-consumer-on-queue-td4691245.html but no one from ActiveMQ has suggested anything.
The jstack output also isn't very helping.
More details:
1. I am not using any selectors, message groups feature
2. I have disabled producer flow control in my setup
I want some suggestions as to what configuration values might cause this issue- memory limits, message TTL etc.
I am still learning about this activemq and jms stuff.
I already tried some example and now I can produce and consuming message from the queue/topic.
Now I have a problem, when my client/consumer lost the connection, the message in queue/topic still send out that message, that message become lost and not kept in the queue/topic. So my question is how I can keep that failed message and how to make the broker resend that message again?
thanks
You are mixing up terminology a bit.
Queues will hold messages until consumed or the broker is restarted, unless the message has been marked as persistent, in which case they will stick around even after a broker restart.
Topics only deliver the current message to any current subscriber. However there are several methods you can use to persist messages published to a topic:
Durable subscribers.
Virtual Destinations .
Virtual Topics tend to be popular for many reasons over durable subscribers, but it really depends on the use-case.
How you create a durable subscriber depends on what you are using to create the subscriber (Spring, POJO, some other API?). All methods will at some point call the Session.createDurableSubscriber method, but I suggest reading up on how they behave before choosing this over Virtual Topic or Composite Queues.
The thing which you are looking for might be Durable subscription
You can find documentation for same at http://activemq.apache.org/how-do-durable-queues-and-topics-work.html
Let's suppose I have several subscribers consuming from a topic. After a message has been delivered to all the subscribers I'd like to trigger a job that would use this message in input.
So the easy way to do that would be to move messages that have been succesfully delivered to all the sucscribers to a queue from which my job would consume messages.
Is it part of JMS?
Is there any message broker able to do that directly?
If not is there a simple solution to solve this problem?
You should be able to do this using activemq's advisories.
See here for more about advisory messages: http://activemq.apache.org/advisory-message.html
So what you want to do, for the topic in question, is track:
the number of consumers
when a message is dispatched to them
when the message has been ack'd by each of the consumers
to get the number of consumers, listen to the "ActiveMQ.Advisory.Consumer.Topic." advisory topic
to get when a message is dispatched, listen to the "ActiveMQ.Advisory.MessageDelivered.Topic."
to get when a message has been ack'd, listen to "ActiveMQ.Advisory.MessageConsumed.Topic."
you could easily use Apache Camel to help out with this (listening to the topics) and aggregating whether or not all consumers have processed (ack'd) the message.. then that could kick off your further processing..
You could just create another durable subscription to route the message from the topic to queue directly. From that queue your job can consume messages. This is much easier than creating a trigger to route the messages to a queue.
So the easy way to do that would be to move messages that have been
succesfully delivered to all the sucscribers to a queue from which my
job would consume messages. Is it part of JMS?
No, this is not part of JMS specification.