If a publisher publish some messages to a WMQ topic, but the subsciber didn't take it, then where the messages are saved? is there any way to know the message count?
As MQ is JMS compliant, the answer is mostly a JMS answer.
If the subscription is not durable and no subscription is registered, the messages for that subscriber are discarded.
If the subscription is durable, MQ creates a queue (or uses a predefined one if specified by the subscriber) to deliver the messages. The messages will collect there if the subscriber is not consuming them.
The 3rd case as Dave points out int he comments is that the non-durable subscriber is holding the subscription open but not consuming the messages. Since a queue is created to receive these that queue depth can be queried to determine if there's a back-up.
Based on there being a queue for every subscription (durable or otherwise) just look in the durable subscriber's queue to determine the number of messages outstanding.
Please also see Publish/subscribe lifecycles in the MQ Knowledge Center for more description of the behavior and specification of durable subscriber queues.
Of course, if that queue fills up the behavior changes. Depending on the settings either the publishers block or the publications continue but the messages are routed to an exception queue (if specified), the DLQ, or discarded.
Thanks Dave Ware for the comments about non-durable subscriptions.
I'm wondering from the question if you're asking if MQ keeps a store of all the messages published to a topic, independent of any registered subscriptions?
If that's the question, then no, it doesn't. When messages are published they are matched to each existing subscription and a copy is sent to each of their associated queues as T.Rob describes.
So the only queue depths to worry about are those of the subscriptions.
(There is a caveat in that MQ supports "retained publications", - it means MQ keeps just the most recent publication on that topic string for late subscriptions if you choose to do that).
I try to explain all this here (slides/video), which may help... http://www.slideshare.net/DavidWare1/ame-2271-mq-publish-subscribe-pdf
Related
I am familiar with JMS and novice with Google Pub/Sub.
In JMS there are 2 options:
Queue: only one consumer can accept message.
Topic: each consumer accepts each message from the topic
I believe that Google Pub/Sub should support something like this, but a quick Googling didn't help me to answer that question.
Please point me out to the corresponding documentation part.
As the name "Pub/Sub" indicates, Google Pub/Sub supports publish/subscribes semantics which correspond to JMS topics. It doesn't support point-to-point semantics which correspond to JMS queues, at least not directly.
You can see an overview of the semantics in the documentation. The "Publisher-subscriber relationships" section may be helpful. To be clear, this documentation does use the word queue in two places:
In the "Pub/Sub message flow" section: "When a message is acknowledged by the subscriber, it is removed from the subscription's message queue."
In the "Common use cases" section: "For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances."
The term queue here is being used to refer to the actual subscription on the topic (i.e. where the messages are placed for subscribers to consume). Furthermore, the architectural overview includes this diagram:
This diagram demonstrates how multiple subscribers can receive messages from the same subscription (e.g. for balancing workloads). This would be akin to the "shared subscription" functionality added for topics in JMS 2.
I was looking for the answer to this question, and I've found this documentation that describe the behaviour of a queue rather than a topic :
https://cloud.google.com/pubsub/docs/subscriber
While a message is outstanding to a subscriber, however, Pub/Sub tries
not to deliver it to any other subscriber on the same subscription.
So my understanding is that if you want
a topic behaviour (one to many): you create many subscriptions, each will get a copy of the message
a queue behaviour : you create one subscription
Then on each subscription you can run one or more subscribers for load balancing, each message is delivered to only one subscriber
I didn't see what kind of distribution among subscribers of a subscription. likely to be round robin.
I don't quite understand what this means:
Each message may have multiple subscribers. There are time
dependencies that exist between publishers and subscribers;
subscribers to a topic may only consume messages
published after a subscription was created.
What does it mean that there exists "time dependencies"? The above says that "subscribers may only consume messages after a subscription was created", I think I'm missing something vital here because I understood that as a subscriber cannot consume a message that is not yet created (but isn't this common sense).
I couldn't find any explanation on Google, so thank you for any help!
Edit: I found an excellent channel on youtube explaining spring boot and other technologies. In particular, this video helped me understand the difference between Queue and Topic (also what durable subscriber is).
What does it mean that there exists "time dependencies"?
To understand this better, compare it with a JMS Queue in which the message broker preserves all the messages (till TimeToLive expiration) published by the producer even if there are NO consumers CREATED.
Now, coming to a JMS topic, the message broker DOES NOT preserve the messages published by the producer if there are NO subscribers CREATED.
During the creation of topic subscription, you can tell the Message Broker that I don't want to loose the messages for this topic published FROM NOW ON by creating the Topic subscription as non-durable (using topicSession.createDurableSubscriber() method). Again, even in the case of non-durable subscription, the broker preserves the messages published after (subscription creation time) till TimeToLive expiration.
Simply that if you publish messages in to the topic before the consumer has subscribed, they will not see the messages.
Similarly, this is related to persistent subscribers. Normal subscribers only see messages while they are connected to the broker. Any messages before or after the subscriber has disconnected, will not be see by the subscriber.
However, with a persistent subscription, a subscriber can disconnect, yet the broker will continue to keep messages destined for that subscriber. When the subscriber reconnects, it will fetch all of those stored messages as well as any new ones.
I am still learning about this activemq and jms stuff.
I already tried some example and now I can produce and consuming message from the queue/topic.
Now I have a problem, when my client/consumer lost the connection, the message in queue/topic still send out that message, that message become lost and not kept in the queue/topic. So my question is how I can keep that failed message and how to make the broker resend that message again?
thanks
You are mixing up terminology a bit.
Queues will hold messages until consumed or the broker is restarted, unless the message has been marked as persistent, in which case they will stick around even after a broker restart.
Topics only deliver the current message to any current subscriber. However there are several methods you can use to persist messages published to a topic:
Durable subscribers.
Virtual Destinations .
Virtual Topics tend to be popular for many reasons over durable subscribers, but it really depends on the use-case.
How you create a durable subscriber depends on what you are using to create the subscriber (Spring, POJO, some other API?). All methods will at some point call the Session.createDurableSubscriber method, but I suggest reading up on how they behave before choosing this over Virtual Topic or Composite Queues.
The thing which you are looking for might be Durable subscription
You can find documentation for same at http://activemq.apache.org/how-do-durable-queues-and-topics-work.html
I'm using ActiveMQ JMS implementation with Spring, and just switched from queue model to topic model, because my JMS clients increased from one to many. Topic subscribers should be durable, because clients may become unavailable in some circumstances.
I don't understand when does a persistent message get removed from a topic. Underlying engine is not aware how many subscribers may receive from that topic, so when does it remove the message from its internal database? Is it done in a time-based manner?
If you register a new durable subscriber to a topic, the broker server will keep track of that subscription and keep the messages around until every single subscriber (with a unique subscriber id) has successfully consumed the message.
It's pretty similar to have the message copied to a unique queue per receiver.
Underlying engine is not aware how many subscribers may receive from that topic
-- Yes it is aware, since the subscriptions are durable
I used the Subscriber SYSTEM.JMS.D.SUBSCRIBER.QUEUE and Client ID as setClientID("USER1") and used topicSubscriber = topicSession.createDurableSubscriber(topic,"SUB1");
The topicSubscriber is created and while trying to receive using this topicSubscriber.receive(); , it is not receiving the messages from topic , but there are messages in topic.
Can any one say why its not receiving messages and whether i need to chek any queue configurations.
Any help is appreciated.
Thanks in advance.
I already had topicConnection.Start() in my coding , also i checked in TopicSession there is no Start() Method.
The same code with Non durable subscriber method topicSession.createSubscriber(topic); is working , but for durable it is not working.
Thanks
Sorry, yes I meant topicConnection.Start(). It was worth a check.
I got the answer for durable subscriber not working ,
My queue depth has reached the max queue depth , so the subscriber is not able to subscribe the message.
Eg . my max queue depth for queue SYSTEM.JMS.D.SUBSCRIBER.QUEUE is set to 100 , and if we check our current queue depth and if it reaches 100 the subscriber will not work.
As an alternative way iam trying to create with Temporary Topic , here iam getting an error while creating the durable subscriber topicSession.createDurableSubscriber(topic,"SUB1");
JMS Exception :: javax.jms.InvalidDestinationException: MQJMS0003:
Destination not understood or no longer valid
Can anyone help to solve this error.
Thanks in Advance.
The problem seems to be how you are using SYSTEM.JMS.D.SUBSCRIBER.QUEUE. You appear to be directing messages and subscribers to this queue as the destination for a durable subscription. IBM MQ uses that queue to manage durable subscriptions.
As a general rule, queues whose names begin with SYSTEM are for internal system use by MQ. Some of them, such as SYSTEM.ADMIN.*.EVENT are OK to get messages from but you would not use these as a subscription destination for unrelated messages either.
Many tutorials use SYSTEM.DEFAULT.LOCAL.QUEUE as a destination for messages but this is only because the queue is known to exist on all versions of MQ and MQ uses only the definition of the queue and never the content of that queue. It is easier for the tutorial writer (and IBM is just as guilty here) to point at SYSTEM.DEFAULT.LOCAL.QUEUE then to walk the student through the need for and means to create their own queue. So although best practices suggest it should not be an exception to the "do not use SYSTEM objects" rule, common usage makes SYSTEM.DEFAULT.LOCAL.QUEUE the de facto exception.
The other exceptions are, of course, the command queues for MQ, MFT and IIB. These are also names SYSTEM.* but are designed for users to communicate with the software listening on the queue.
Note that all the exceptions "do not use SYSTEM objects" rule are interfaces between applications and MQ system resources. The event queues are MQ sending information to the user. The command queues are the user sending information to the system components. A subscription is neither of these categories. A destination for a subscription is considered an application-owned object, even when the system manages it on behalf of the subscriber.
When you want a durable subscription, either let the system assign a permanent queue and use that, or else pre-define a queue (that is not named SYSTEM.*) and use that. Whatever else you do, please do NOT try to hijack MQ's internal system queues for application-level purposes.