I'm currently faced with a use case where I need to process multiple messages in parallel, but related messages should only be processed by one JMS consumer at a time.
As an example, consider the messages ABCDEF and DEFGHI, followed by a sequence number:
ABCDEF-1
ABCDEF-2
ABCDEF-3
DEFGHI-1
DEFGHI-2
DEFGHI-3
If possible, I'd like to have JMS consumers process ABCDEF and DEFGHI messages in parallel, but never two ABCDEF or DEFGHI messages at the same time across two or more consumers. In my use case, the ordering of messages is irrelevant. I cannot use JMS filters because I would not know the group name ahead of time, and having a static list of group names is not feasible.. Messages are sent via a system which is not under my control, and the group name always consists of 6 letters.
ActiveMQ seems to have implemented this via their message groups feature, but I can't find equivalent functionality in IBM MQ. My understanding is that this behaviour is driven by JMSXGroupId and JMSXGroupSeq headers, which are only defined in an optional part of the JMS specification.
As a workaround, I could always have a staging ground (a database perhaps), where all messages are placed and then have a fast poll on this database, but adding an extra piece of infrastructure seems overkill here. Moreover, it would also not allow me to use JMS transactions for reprocessing in case of failures.
To me this seems like a common use case in messaging architecture, but I can't find a simple yes/no answer anywhere online, and the IBM MQ documentation isn't very clear about whether this functionality is supported or not.
Thanks in advance
IBM MQ also has the concept of message groups.
The IBM MQ message header, called the Message Descriptor (MQMD) has a couple of fields in it that the JMS Group fields map to:-
JMSXGroupID -> MQMD GroupID field
JMSXGroupSeq -> MQMD MsgSeqNumber field
Read more here
IBM MQ docs: Mapping JMS property fields
IBM MQ docs: Message groups
Related
I am trying to implement an application(Java) which will subscribe to different message types (XMLs) from other different applications via TIBCO EMS. Each of these message types will have a specific purpose. I am of the opinion that I should have multiple queues with multiple subscribers in my application, however, the TIBCO guy is adamant that there should be only one queue where all of these messages will be published and I will have one subscriber and the subscriber then should have logic to different tasks based on the XML received.
Which approach is better? One with multiple queues and subscribers OR the one queue and one subscriber? Please let me know reasons for the choice.
Thanks!
-Naveen
In general, if the same application is reading all the messages, it is much cleaner for that application to have a single input queue instead of multiple input queues. With multiple then the application will need to have logic to know which order to process the queues and so on. With one input queue, the messaging system can deal with the order of the messages - whether FIFO or by priority etc, and the application can just read the next message and process it.
Use unique message header for each type of xml while sending the message. And use message selectors / filters while receiving the same, so that it can be routed / delegated to the respective handler based on the header value. This way, you will be able to handle different type of xml messages by single queue as well.
The JMS 2.0 specification says
The JMSMessageID header field contains a value that uniquely
identifies each message sent by a provider.
...and...
The exact scope of uniqueness is provider defined. It should at least
cover all messages for a specific installation of a provider where an
installation is some connected set of message routers.
The specification does not explicitly state that the JMSMessageID returned from the publish API call must match the one present in the message when it is consumed. The discussion in the spec about moving the JMSMessageID to the JMSCorrelationID when replying to a request implies that the two would be the same. If the message ID was changed between publication and consumption, this style of request/reply would fail.
Certainly in the unified domain model of JMS 1.1 and now 2.0, it would not make sense for the behavior of the JMSMessageID to change depending on whether the destination is a queue or a topic. Under the unified model, one would expect all destinations to act alike in this regard.
Also, if "provider" as used in the first paragraph refers to the thing that is sending messages, then a publication that fanned out to 10 identical messages, with identical JMSMessageID values, would meet the spec since uniqueness is measured at the sending side.
Unfortunately, the specification liberally switches between using the term "provider" to describe the thing sending messages versus using it to describe the vendor of the JMS transport. This is evident in the two quoted passages above. This ambiguity doesn't help matters any.
At least one implementation (IBM's MQ) takes the approach that a publication fanning out to 10 messages has created 10 unique, new messages, and therefore each of these has a unique JMSMessageID value. This is arguably consistent with the second quoted passage which requires uniqueness scoped to the provider, where "provider" appears to refer to the vendor implementation and not the thing sending messages.
It is my belief that when a published message fans out to multiple subscribers the correct behavior would be that the JMSMessageID would be preserved in each instance of the message so that replies can be correlated as expected. In other words, I believe IBM's implementation to be non-compliant. Since the specification is ambiguous on the matter, I'm looking for an authoritative source which either states outright or strongly implies the behavior as intended by the spec, one way or the other. Depending on the response, I'll either stand down, or else raise the issue with IBM as a compliance defect.
The term "provider" here is simply a reference to the specific messaging product being used, and covers both client-side and server-side components. To avoid confusion, I'll use the word JMS product vendor here.
The purpose of the JMS specification is to define a Java API implemented by that messaging product. It uses loose terms like "provider" because the JMS spec does not define how the product is architected and is trying to avoid suggesting how the implementation should be shared between client-side and server-side components, or even whether there is a server (or cluster of servers) at all. You'll notice the spec never (well, almost never) says "the server does this" or "the server does that".
The sentence about the "exact scope of uniqueness" is there to make it easy for the JMS product vendor to implement the code that generates JMSMessageID values. It's saying that the code that generates JMSMessageID values doesn't need to worry about ensuring that the values generated are unique across the entire universe. It's sufficient to ensure that they are unique to that particular product installation.
You say that "The specification does not explicitly state that the JMSMessageID returned from the publish API call must match the one present in the message when it is consumed."
I think this is stated in Section 4.4.11 "How message header values are set". This states that the JMSMessageID is set by the "JMS provider send method". The same section goes on to say that "Message header fields that are defined as being set by the 'JMS provider send method' will be available on the sending client as well as on the receiving client."
This means that after the call to send() or publish() has returned, the sending application can use the method getJMSMessageID() to find the message ID that was assigned to that message. When this message is received, the receiving application can use the same method, and get the same value.
Each message sent to a topic is delivered to every subscriber on that topic. These subscribers will receive a separate copy of the same message, with the same body, properties and headers, including JMSMessageID value.
Feel free to argue; the JMS spec is not free of ambiguities.
I think the issue here is less about when the JMSMessageID field is set on a published message, and more about what happens to that message when it is processed within the JMS provider.
As stated in T.Rob's and Nigel's posts, section 3.4.3 of the JMS 2.0 specification states:
"The JMSMessageID header field contains a value that uniquely
identifies each message sent by a provider."
and also:
"A JMSMessageID is a String value which should function as a unique
key for identifying messages in a historical repository. The exact
scope of uniqueness is provider defined. It should at least cover all
messages for a specific installation of a provider where an
installation is some connected set of message routers."
That is to say, two or more messages, even if they contain the same data, ought to have different JMSMessageID values if they constitute different messages within a repository.
The spec also states, in section 4.2.1 that,
"A topic can be thought of as a mini message broker that gathers and
distributes messages addressed to it. By relying on the topic as an
intermediary, message publishers are kept independent of subscribers
and vice versa."
This would imply that the intention of the spec is that, when a message is sent to a Topic, the Topic can do some work on the message, including creating multiple copies of the message (or, more specifically, creating multiple messages with the same data that are considered separate within the provider's repository.
Finally, section 4.2.2 states:
"A subscription will receive a copy of every message that is sent to
the topic after the subscription is created, ... Each copy of the
message is treated as a completely separate message. Work done on one
copy has no effect on any other; acknowledging one does not
acknowledge any other; one message may be delivered immediately, while
another waits for its consumer to process messages ahead of it."
Putting these passages together, the spec can be read as saying
When a message is sent to a Topic, that Topic can create a copy of the message for each current subscription.
The copies of the message created when sending to a Topic can be considered as completely separate messages.
Because separate JMS messages are uniquely identified by their JMSMessageID field, each separate subscription message should have a different JMSMessageID
To pick up Nigel's last sentence the JMS specification isn't free of ambiguities. This is very true and vendors and customers have previously worked around issues, and work in the expert group does take place to clarify these and provide guidance as well as make suggestions for improving the compliance tests. Based on the understanding outlined above, and the tests within the JMS 2.0 Compliance Test Suite that IBM MQ v8 passes, the IBM MQ v8 implementation is JMS2.0 compliant (and likewise earlier IBM MQ versions are JMS1.1 compliant; the JMS 1.1 specification has the same ambiguity).
The request-response paradigm is a common one, though with a pub-sub based distribution model the sending application does potentially have to cope with multiple responses not just the one that would be more likely with a point-point architecture. We acknowledge that there are messaging scenarios where the capability for a message id to have a different 'value of uniqueness' from the one currently implemented by IBM MQ would provide value to some IBM MQ customers
For the above reasons IBM strongly believes that its MQ JMS solution is compliant, so a PMR will not be accepted. However, we do acknowledge that there are a number of use cases where maintaining the message ID would be beneficial to you. For that reason we will make RFE 35062 an uncommitted candidate, which means it has the highest probability of being addressed and we promise that we're actively working to provide the solution that best fits the needs as quickly as possible. But to do this we'd appreciate additional feedback on the RFE with descriptions of what the actual problems our users are trying to solve here. For example is this for audit purposes, request-reply, message flows, etc, and what it is you need replicated? The more information we have, the more likely the solution is to satisfy the need.
The JMS 2.0 specification says
The JMSMessageID header field contains a value that uniquely
identifies each message sent by a provider.
...and...
The exact scope of uniqueness is provider defined. It should at least
cover all messages for a specific installation of a provider where an
installation is some connected set of message routers.
The specification does not explicitly state that the JMSMessageID returned from the publish API call must match the one present in the message when it is consumed. The discussion in the spec about moving the JMSMessageID to the JMSCorrelationID when replying to a request implies that the two would be the same. If the message ID was changed between publication and consumption, this style of request/reply would fail.
Certainly in the unified domain model of JMS 1.1 and now 2.0, it would not make sense for the behavior of the JMSMessageID to change depending on whether the destination is a queue or a topic. Under the unified model, one would expect all destinations to act alike in this regard.
Also, if "provider" as used in the first paragraph refers to the thing that is sending messages, then a publication that fanned out to 10 identical messages, with identical JMSMessageID values, would meet the spec since uniqueness is measured at the sending side.
Unfortunately, the specification liberally switches between using the term "provider" to describe the thing sending messages versus using it to describe the vendor of the JMS transport. This is evident in the two quoted passages above. This ambiguity doesn't help matters any.
At least one implementation (IBM's MQ) takes the approach that a publication fanning out to 10 messages has created 10 unique, new messages, and therefore each of these has a unique JMSMessageID value. This is arguably consistent with the second quoted passage which requires uniqueness scoped to the provider, where "provider" appears to refer to the vendor implementation and not the thing sending messages.
It is my belief that when a published message fans out to multiple subscribers the correct behavior would be that the JMSMessageID would be preserved in each instance of the message so that replies can be correlated as expected. In other words, I believe IBM's implementation to be non-compliant. Since the specification is ambiguous on the matter, I'm looking for an authoritative source which either states outright or strongly implies the behavior as intended by the spec, one way or the other. Depending on the response, I'll either stand down, or else raise the issue with IBM as a compliance defect.
The term "provider" here is simply a reference to the specific messaging product being used, and covers both client-side and server-side components. To avoid confusion, I'll use the word JMS product vendor here.
The purpose of the JMS specification is to define a Java API implemented by that messaging product. It uses loose terms like "provider" because the JMS spec does not define how the product is architected and is trying to avoid suggesting how the implementation should be shared between client-side and server-side components, or even whether there is a server (or cluster of servers) at all. You'll notice the spec never (well, almost never) says "the server does this" or "the server does that".
The sentence about the "exact scope of uniqueness" is there to make it easy for the JMS product vendor to implement the code that generates JMSMessageID values. It's saying that the code that generates JMSMessageID values doesn't need to worry about ensuring that the values generated are unique across the entire universe. It's sufficient to ensure that they are unique to that particular product installation.
You say that "The specification does not explicitly state that the JMSMessageID returned from the publish API call must match the one present in the message when it is consumed."
I think this is stated in Section 4.4.11 "How message header values are set". This states that the JMSMessageID is set by the "JMS provider send method". The same section goes on to say that "Message header fields that are defined as being set by the 'JMS provider send method' will be available on the sending client as well as on the receiving client."
This means that after the call to send() or publish() has returned, the sending application can use the method getJMSMessageID() to find the message ID that was assigned to that message. When this message is received, the receiving application can use the same method, and get the same value.
Each message sent to a topic is delivered to every subscriber on that topic. These subscribers will receive a separate copy of the same message, with the same body, properties and headers, including JMSMessageID value.
Feel free to argue; the JMS spec is not free of ambiguities.
I think the issue here is less about when the JMSMessageID field is set on a published message, and more about what happens to that message when it is processed within the JMS provider.
As stated in T.Rob's and Nigel's posts, section 3.4.3 of the JMS 2.0 specification states:
"The JMSMessageID header field contains a value that uniquely
identifies each message sent by a provider."
and also:
"A JMSMessageID is a String value which should function as a unique
key for identifying messages in a historical repository. The exact
scope of uniqueness is provider defined. It should at least cover all
messages for a specific installation of a provider where an
installation is some connected set of message routers."
That is to say, two or more messages, even if they contain the same data, ought to have different JMSMessageID values if they constitute different messages within a repository.
The spec also states, in section 4.2.1 that,
"A topic can be thought of as a mini message broker that gathers and
distributes messages addressed to it. By relying on the topic as an
intermediary, message publishers are kept independent of subscribers
and vice versa."
This would imply that the intention of the spec is that, when a message is sent to a Topic, the Topic can do some work on the message, including creating multiple copies of the message (or, more specifically, creating multiple messages with the same data that are considered separate within the provider's repository.
Finally, section 4.2.2 states:
"A subscription will receive a copy of every message that is sent to
the topic after the subscription is created, ... Each copy of the
message is treated as a completely separate message. Work done on one
copy has no effect on any other; acknowledging one does not
acknowledge any other; one message may be delivered immediately, while
another waits for its consumer to process messages ahead of it."
Putting these passages together, the spec can be read as saying
When a message is sent to a Topic, that Topic can create a copy of the message for each current subscription.
The copies of the message created when sending to a Topic can be considered as completely separate messages.
Because separate JMS messages are uniquely identified by their JMSMessageID field, each separate subscription message should have a different JMSMessageID
To pick up Nigel's last sentence the JMS specification isn't free of ambiguities. This is very true and vendors and customers have previously worked around issues, and work in the expert group does take place to clarify these and provide guidance as well as make suggestions for improving the compliance tests. Based on the understanding outlined above, and the tests within the JMS 2.0 Compliance Test Suite that IBM MQ v8 passes, the IBM MQ v8 implementation is JMS2.0 compliant (and likewise earlier IBM MQ versions are JMS1.1 compliant; the JMS 1.1 specification has the same ambiguity).
The request-response paradigm is a common one, though with a pub-sub based distribution model the sending application does potentially have to cope with multiple responses not just the one that would be more likely with a point-point architecture. We acknowledge that there are messaging scenarios where the capability for a message id to have a different 'value of uniqueness' from the one currently implemented by IBM MQ would provide value to some IBM MQ customers
For the above reasons IBM strongly believes that its MQ JMS solution is compliant, so a PMR will not be accepted. However, we do acknowledge that there are a number of use cases where maintaining the message ID would be beneficial to you. For that reason we will make RFE 35062 an uncommitted candidate, which means it has the highest probability of being addressed and we promise that we're actively working to provide the solution that best fits the needs as quickly as possible. But to do this we'd appreciate additional feedback on the RFE with descriptions of what the actual problems our users are trying to solve here. For example is this for audit purposes, request-reply, message flows, etc, and what it is you need replicated? The more information we have, the more likely the solution is to satisfy the need.
We are using IBM WebSphere MQ as JMS provider with Spring MDP (Message Driven POJO).
Is there any way in JMS where we can configure time related properties in message so that message can be consumed at particular defined time only?
For example, if I am sending three messages into queue M1, M2 and M3. Where, I can configure M2 message property let say 3 AM. And consumer side, consumer can only pick this message # 3 AM only. If time is not defined, messages should be consumed in a way that JMS Receiver does.
JMS 2.0 specification has defines Delivery Delay. With this feature a message producer can specify that a message must not be delivered until after a specified time interval. The message will be available for delivery after the specified time. But this may not help you as you want to a message to be consumed at a specified time. Typically messaging applications are designed to consume messages as soon as they are made available by the messaging provider.
If you want to process messages at a specified time only, you could create another queue "queue_3am", and schedule a reader to run exactly at 3am.
A variation is to set the timestamp as a message property. So one queue could contain messages to be processed at different points in time. The reader could use message selectors to get relevant messages only.
But you should use a "message pickup timeframe" by adding two timestamps as message properties, for eaxmple set the window to 1 or 5 minutes.
The receiver can use a message selector: A selector is a condition using message properties.
Have a look at this
I have a design query regarding queues. My scenario is as follows:
I have to use a messaging system, with single producer and multiple consumers (asynchronous). The producer pushes different types of messages into the messaging system. Depending upon the message type, that particular consumer has to consume that message. (Each consumer is running on a different server). If one consumer is down and a message comes for that consumer, it will be in the messaging system only. If I use a message queue, the message in the queue will block the next messages that can be consumed by the other consumers. Are queues suitable for handling this kind of situation? Or do we need to go for a topic?
Whether you use a queue or a topic should depend on whether there an instance where multiple consumers must process the same message. If that is the case then a topic is required do generate that one-to-many pattern.
On the other hand, if any one message will only ever be consumed by one consumer, then you can use a queue or topic and the consumers specify the message type as a JMS selector. In this way, all consumers can listen on the same queue and each selects a different subset of messages. In the event one application is not there, it's messages do not "block the next messages that can be consumed by the other consumers" but rather they just stack up in the queue and other consumers still receive their messages based on selection criteria.
Please also realize that queues are lightweight constructions and you can easily have one queue per consumer. Typically, things providing a service listen on a well-known queue and each queue represents a different function of the service or a different service. Thus there may be many service input queues. Similarly, reply messages are generally uniquely addressed to the application instance that made the request and go to a unique, often dynamic, reply-to queue. Both of these implementations I have described lead to a separation of traffic across queues rather than pooling different message types into the same queue. Since JMS selectors always impart an additional processing cost, using more queues is generally more performant than selecting many types of message from the same queue.
I am responding to your question about selectors in the comment section here since I have more space and can put links in...
Section 3.8.1 of the JMS 1.1 spec states:
A JMS message selector allows a client to specify, by message header, the
messages it’s interested in. Only messages whose headers and properties
match the selector are delivered. The semantics of not delivered differ a bit
depending on the MessageConsumer being used. See Section 5.8,
“QueueReceiver,” and Section 6.11, “TopicSubscriber,” for more details.
Message selectors cannot reference message body values.
A message selector matches a message if the selector evaluates to true when
the message’s header field and property values are substituted for their
corresponding identifiers in the selector.
As noted above, selectors can be on fields that are implicit in the message such as MsgID or CorrelationID or thsey can be on fields specifically set by the message producer such as a message property. Either way, the client must specify the value of any selectors used by the message consumer.