I'm an IBM MQ novice, however have used other messaging systems in the past (Solace, RabbitMQ, BizTalk). I apologize if some of my MQ terminology is incorrect when it comes to local/remote/transmission queues etc.
I am integrating with a 3rd Party using MQ. I have no control over the use of MQ when it comes to this 3rd party. They define the set integration pattern.
I have my own local Queue Manager.
The 3rd party supports many 100s of different message types.
Typically each message sent to them will result in a response
Responses on overage will be delivered back to my QM within 0.5s (agreed SLA)
The basic model is as follows;
For all outbound messaging I publish to a single queue
All responses come to a single inbound queue on my QM
I do not believe they respect/use the ReplyToQ or ReplyToQmgr header properties
High level overview of the use case;
The MQ usage will be very active. Perhaps 500-1000 outbound messages per second
For the vast majority of these - there will be a user waiting for a response
The current design approach is to put a message onto the outbound queue and then create a subscription to the inbound queue with a JMS selector using the correlation ID.
Here is my challenge;
I am not sure if having so many concurrently subscriptions all with a unique filter on the single inbound queue will perform well. I would appreciate insights into this. I have a 2 node cluster running in a docker container. I don't have details on the spec yet.
My preference would be have messages arrive into the inbound queue and then be routed to many "function" specific local queues on the QM.
I would still use JMS selectors on the message ID in this model - however I would hope the load would be spread from a single deep queue to many shorter queues
It does not seem that I could route messages natively within MQ without an addon such as MQ Integration Broker or ESB (some other products within the WebSphere suite).
It is on this last point I could use some guidance. Can I route within my QM? If so - what options do I have. Alternatively - perhaps there are much better approaches that I have not considered?
Any guidance would be heartily appreciated!!
Related
I am really curious about this topic.
I will create a communication mechanism for internal systems and may also need connection to some external clients too. The internal modules are also distributed systems.
I need to create a ESB between that modules. The system should provide high performence over millions of subscribers.
publish subscribe or p2p communications are both needed,
When I first started to thinking about that implementation , I was planed to make a REST api on front and the REST api will communicate with a JMS bus .The JMS bus has an ability to provide communication between internal systems.
Unfortunately as per my investigation, using JMS can be caused so musch critical problems : performance,scalability... and looks like JMS is needless, I can create some adapters over internal modules and both can communicate with REST services.
Does anyone have any idea why should I use JMS for internal communication ?
Both REST and JMS/MQ enable communicate between remote systems (and local). You can get help based on the scenarios below:
Some Reasons for using JMS in your case:
If your producer is spitting messages at a very high rate than the consumer then the persistent messaging will help. This may also mean you are fine with the transaction/message to be processed later.
All systems are not up all the time.
You need a publish subscribe mechanism (topic).
Messages are not critical and discard old messages when load is high.
Reasons for using REST API (without any jms connected):
1. You want an immediate response that transaction is completed. Example, hotel booking etc.
2. All systems should be up all the time for processing to complete.
You would want to use JMS (or enterprise messaging) when don't have to rely on all the systems being available. So if one of your internal systems was down for some reason, then a REST api interface would fail when communicating to that system, but a JMS interface would not as you are communicating to the MOM.
For some MOM you don't have to just communicate using JMS, so you can have different runtimes communicate to the MOM.
I'm looking at swapping out ActiveMQ with RabbitMQ for a few reasons. I currently have multiple services which are each capable of publishing events (and they publish those events to a specific VirtualTopic in AMQ). Each of the services is also capable of consuming messages from the other services. Consumers are set up such that they subscribe as a consumer to a queue on the VirtualTopic.
This buys me the ability to fan messages out to multiple queues (topic-like functionality) while keeping the benefits of queues (load balancing and persistence).
It seems like this is roughly equivalent to RabbitMQ's fanout exchange. However, the part that I found very useful in ActiveMQ is that the producer doesn't need to have any knowledge of the consumers. It simply publishes to the virtual topic. It seems that in RabbitMQ, when the exchange is created, I need a definitive of queues to publish that message to.
tl;dr
Is there any routing scheme in RabbitMQ that is equivalent to ActiveMQ's Virtual Topic, such that I can produce messages to a topic that are distributed to any queue that has been created off of that Virtual Topic, without requiring a hard-coded routing scheme somewhere in RMQ?
I realized after posting this question that it is pretty trivial to do this (not sure why I never thought of it before).
I was looking at it from the wrong direction, wondering how I could automatically have the publisher configure queues for the recipients - which isn't the right way to approach this question.
Instead, I have the subscribers, when they start up, bind themselves to the exchange that the publisher users, which provides in the inversion of control I'm looking for (publishers need not know anything about their consumers).
Ref: Official GlassFish 4.0 docs/javaee-tutorial Java EE 7
Firstly, let us start with the destination-type of: topic.
As per GlassFish 4.0 tutorial, section “46.4 Writing High Performance and Scalable JMS Applications”:
This section describes how to use the JMS API to write applications
that can handle high volumes of messages robustly.
In the subsection “46.4.2 Using Shared Durable Subscriptions”:
The SharedDurableSubscriberExample.java client shows how to use shared
durable subscriptions. It shows how shared durable subscriptions
combine the advantages of durable subscriptions (the subscription
remains active when the client is not) with those of shared consumers
(the message load can be divided among multiple clients).
When we run this example as per “46.4.2.1 To Run the ShareDurableSubscriberExample and Producer Clients”, it gives us the same effect/functionality as previous example on destination-type of queue: if we follow “46.2.6.2 To Run the AsynchConsumer and Producer Clients”, points 5 onwards – and modify it slightly using 2 consumer terminal-windows and 1 producer terminal-window.
Yes, section “45.2.2.2 Publish/Subscribe Messaging Style” does mention:
The JMS API relaxes this requirement to some extent by allowing
applications to create durable subscriptions, which receive messages
sent while the consumers are not active. Durable subscriptions provide
the flexibility and reliability of queues but still allow clients to
send messages to many recipients.
.. and anyway section “46.4 Writing High Performance and Scalable ..” examples are queue style – one message per consumer:
Each message added to the topic subscription is received by only one
consumer, similarly to the way in which each message added to a queue
is received by only one consumer.
What is the precise technical answer for: why, in this example, the use of Shared-Durable-Consumer on Topic is supposed to be, and mentioned under, “High Performance and Scalable JMS Application” vs. use of Asynchronous-Consumer on Queue?
I was wonderign about the same issue, so I found out the following link. I understand that John Ament gave you the right reponse, maybe it was just too short to get a full understand.
Basically, when you create a topic you are assuming that only the subscribed consumers will receive its messages. However processing such a message may requires a heavy processing; in such a cases you can create a shared topic using as much threads as you want.
Why not use a queue? The answer is quite simple, if you use a queue only one consumer will be able to handle such a message.
In order to clarify I will give you an example. Let's say a federal court publishes thousand of sentences every day and you have three distinct applications that depends on it.
Application A just copy the sentences to a database.
Application B parse the sentence and try to find out all relation between people around all previously saved sentences.
Application C parse the sentence and try to find out all relation between companies around all previously saved sentences.
You could use a Topic for the sentences, where Application A, B and C would be subscribed. However it easy to see that Application A can process the message very quicly while Application B and C may take some time. An available solution would consist of create a shared subscription for application B and another one to application C, so multiple threads could act on each of them simultaneouly...
...Of course there are other solutions, you could for example use a unshared topic (i.e. a regular one) and post all received messages on a ArrayBlockingQueue that would be handled by a pool of threads some time later; howecer in such a decision the developer would be the one to worry about queue handling.
Hope this can help.
The idea is that you can have multiple readers on a subscription. This allows you to read more messages faster, assuming you have threads available.
JMS Queue :
queued messages are persisted
each message is guaranteed to be delivered once-and-only-once, even no consumer running when the messages are sent.
JMS Shared Subscription :
subscription could have zero to many consumers
if messages sent when there is no subscriber (durable or not), message will never be received.
I have a TIBCO BusinessWorks process that is publishing to a JMS topic -- lets call it TOPIC.A -- there is one process that is subscribing to that topic with a name SUBSCRIBE.A.
The problem I have is that the first server that starts listening to SUBSCRIBE.A hooks in fine. The other 3 servers, running the exact same process get an error of "WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2429' ('MQRC_SUBSCRIPTION_IN_USE')"
This can't be reasonable behavior for enterprise software, and I know WebsphereMQ, JMS and TIBCO Businessworks all scale well, so I must be missing something. I only want each event processed once, but a single box just isn't going to do it, both for fail-over reasons and shear volume reasons.
What do I have to do to let all 4 servers in the cluster to service the subscription SUBSCRIBE.A?
I agree it does not seem like reasonable behavior for enterprise software - but this restriction is imposed by the JMS specification. The JMS 1.1 spec, section 6.66.1 says "Only one session at a time can have a TopicSubscriber for a particular durable subscription".
That said, WebSphere MQ does provide a vendor-specific option that allows you to do what you want: see the CLONESUPP Connection Factory property. This is documented in the Infocenter at the CLONESUPP properties page.
While it's specific to MQ, if you specify this using administered objects you code will not need to use any vendor-specific methods.
The reason for MQRC 2429 is because all of your 4 subscribers are using the same client ID and all are trying to use the same durable subscription. When a subscriber is already actively listening on a durable subscription, no other subscribers can listen on the same. If you want all subscribers to listen concurrently, then have separate client ids for each of your subscribers.
But you must note that all subscribers will get a copy of the same message published by a publisher (TIBCO BusinessWorks in your case). So if all subscribers are active, all will process the event.
For fail-over you can look at using WebSphere MQ Multi-instance queue manager feature or any other HA solutions. For load sharing you can look at WebSphere MQ clustering feature.
I'm queuing messages to a WebSphere MQ queue (NB: A point-to-point queue -- not a topic) using a stored procedure in my Oracle database. Is there a way to publish each message to multiple queues with a single call? What I would like is to find a solution that would incur zero additional latency on my database compared to sending the message to a single queue.
Solutions that involve changing my WebSphere MQ settings are certainly welcome! What I had in mind was somehow creating a "clone" queue that got all the same messages as the original one, but I've been unable to locate anything like this in the documentation.
Thanks,
Jeff
With WMQ v7 you can do this easily and with administration only. You would create a topic object and then an alias over the topic. The Oracle app writes to the alias and does not know that it is actually publishing.
Meanwhile, you make two administrative subscriptions on the topic so that publications are delivered to your two destination queues. The apps consuming them have no idea that the messages were published as opposed to delivered through point-to-point queues.
If you are not familiar with the new WMQ v7 features, take a look at the Infocenter. In particular, the "What's New in V7" section and the sections on Pub/Sub.
You can accomplish this using "Distribution Lists" in WebsphereMQ. These have to be configured on your queue manager.
Take a look at the Wesbphere MQ Application Programming Guide for more info.