I have a question regarding sharing data in distributed system environemnt.
Usually I am using scenario A, ie. I am pushing messages from system A to a remote Queue such as Amazon SNS/SQS and I am expecting system B to pick the message up when system B is up and running.
I have thought about it and I came with an alternative that I am not sure is good enough. Basically, System A now pushes message to local queue and local queue pushes message to system A. Prerequisites for this are that both systems A and B are up and running and that queue is specific only for system A, ie. there is no system C, D, E... that are pushing message to the same (duplicate) local queue somewhere on C, D, E.
Is this a good alternative?
I was also considering scenario B as a backup plan for scenario A, when remote queue is unavailable and I dont want to lose a message. In that case system B would be a remote queue.
Well the scenario C is actually the way distributed messaging should be and is actually done . This is called outbox pattern in Async communication. There kind of patterns are used for resiliently publishing to the event bus . Its very much like mail box where your mail box keeps your email in the outbox until delivered. The bit of a tweak is that instead of local queue use persistent storage.
But you need to have worker which will be looking for pending messages and will publish them as soon as queue is available.
You can find more about this pattern here.
Outbox pattern
Resiliently publishing to the event bus
Related
Question:
Is it possible to copy MQ Messages from one Queue Manager/Queue to a different Queue Manager/Queue?
Scenario:
I have a "PROD" Queue Manager and when it receives a Message on it's Queues I would like to "copy" the Message to a queue on a "TEST" Queue Manager.
Requirements
The original message must be left on the PROD queue to be processed as normal.
This must be an automated process (lots of messages during a day). I could not intervene on a Message by Message basis.
If at all possible I would like this to be implemented by some native MQ functionality rather than an ad hoc program/script.
The copying must be as near to real time as possible
Must work with MQ version is 7.0.2.1(!). This cannot be changed.
Must run on Red Hat Enterprise Linux Server release 5.11 (Tikanga). Again, can't be changed.
I'm no MQ expert so use small words please
Thanks in advance
The only problem with the technote pointed out by gouda is that MQ will modify/changed the MsgId and CorrelId of each message replicated.
If the MsgId and/or CorrelId fields are important then the only other option is an MQ API Exit that replicates the message. You may need a commercial product like MQ Message Replication.
The next question is how are you going to move the message from a PROD queue to a TEST queue? You definitely do NOT want to create channels between a PROD queue manager and a TEST queue manager.
There are lots of tools that can off offload PROD messages to a file then you can move the file to your TEST environment and then load the messages into a TEST queue. Here is a list of MQ tools that can do it. The 2 tools you should try out are: MQ Batch Toolkit and QLoad.
Personally, I would create a schedule task (CRONTAB) to be run every night at midnight to off the messages and I would make the filename include the date and time. The last steps of the script would be to zip/compress the file and delete the original file (because the data you offload could be massive).
Hence, any time you want a particular day's PROD messages, just copy the file to your TEST server and unzip/uncompress it and load it into the queue.
all you need is mqadmin staff and this technote
I have 2 queues say Q1 and Q2. When I insert messages using MQPUT into Q1, is there any way that this message be replicated into Q2?
Does WMQ have support for queue copy?
Note : The queues reside on different queue managers.
Thanks,
Hudson
If you have multiple consumers(Queues), which needs the same message, then Publish-Subscribe is made exactly for this purpose.
You can refer here for details about pub-sub.
If you are using Websphere MQ 7.XX, then you can use an Alias queue to publish the message.
You can follow below steps:
Create a new Topic(Lets say "MyTopic" is your topic).
Create an Alias queue(Lets call it as "MyPublisher"), with Base Type as Topic and give Base Object as "MyTopic"(Name of your topic).
Now say Q1, Q2, Q3 have to get the message. Create subscriptions for these queues for Topic "MyTopic".
Now, the messages posted in queue "MyPublisher", will reach to all queues subscribing to the topic "MyTopic"(Q1,Q2,Q3).
Even remote queues can be given in destination while creating subscriptions. So, different queue managers will not be a problem as far as you have connectivity between them.
If you don't want to bother with all of the Pub/Sub stuff, then have a look at the free open source project Message Multiplexer (MMX) at http://www.capitalware.biz/mmx_overview.html
Its purpose is to read messages from a source queue and put the exact same message to 'n' target queues (up to 99).
The download contains builds for AIX, HP-UX, Linux, IBM i (OS/400), Solaris, Windows and z/OS (mainframe).
MMX is a very simple and straight-forward program to use.
How would I set up MQ so that every message received is immediately written to file system?
I have the "redbooks", but at least need someone at least point me to a chapter or heading in the book to figure it out.
We are a .NET shop. I have written C# via API to read the queue, and we currently use BizTalk MQ adapter. Our ultimate goal is to write same message to multiple directories in file system to "clone" the feed for our various test environments (DEV, STAGE, TRAINING, etc..). The problem with BizTalk is that when we consume the message, we map it at the same time to a new message, so the message is already changed, and we want the original raw message to be cloned, not the morphed one. Our vendors don't offer multiple copies of the feed, for example, they offer DEV and PROD, but we have 4 systems internally.
I suppose I could do a C# Windows Service to do it, but I would rather use built-in features of MQ if possible.
There is no configuration required. If the message is persistent, WMQ writes it to disk. However, I don't think that's going to help you because they are not written as discrete messages. There's no disk file to copy and replication only works if the replicated QMgr is identical to the primary and is offline during the replication.
There are a number of solutions to this problem but as of WMQ V7, the easiest one is to use the built-in Pub/Sub functionality. This assumes that the messages are arriving over a QMgr-to-QMgr channel and landing on a queue where you then consume them.
In that case, it is possible to delete the queue and create an alias of the same name over a topic. You then create a new queue and define an administrative subscription that delivers messages on the topic into the new queue. Your app consumes from the new queue.
When you need to send a feed to another QMgr or application, define a new subscription and point it at the new destination queue. Since this is Pub/Sub, MQ will replicate the original message as many times as there are subscriptions and the first application and its messages are not affected. If the destination you need to send to isn't accessible over MQ channels (perhaps DEV and QA are not connected, for example), you can deliver the messages to the new queue, use QLoad from SupportPac MO03 to write them to a file and then use another instance of QLoad to load them onto a different QMgr. If you wanted to move them in real time, you could set up the Q program from SupportPac MA01 to move them direct from the new subscription queue on QMgr1 to the destination queue on QMgr2. And you can replicate across as many systems as you need.
The SupportPacs main page is here.
If all you are using is the Redbooks, you might want to have a look at the Infocenters. Be sure to use the Infocenter matching the version of WMQ you are using.
WMQ V7.0 Infocenter
WMQ V7.1 Infocenter
WMQ V7.5 Infocenter
In JMS there are Queues and Topics. As I understand it so far queues are best used for producer/consumer scenarios, where as topics can be used for publish/subscribe. However in my scenario I need a way to combine both approaches and create a producer-consumer-observer architecture.
Particularly I have producers which write to some queues and workers, which read from these queues and process the messages in those queues, then write it to a different queue (or topic). Whenever a worker has done a job my GUI should be notified and update its representation of the current system state. Since workers and GUI are different processes I cannot apply a simple observer pattern or notify the GUI directly.
What is the best way to realize this using a combination of queues and/or topics? The GUI should always be notified, but it should never consume anything from a queue?
I would like to solve this with JMS directly and not use any additional technology such as RMI to implement the observer part.
To give a more concrete example:
I have a queue with packages (PACKAGEQUEUE), produced by machine (PackageProducer)
I have a worker which takes a package from the PACKAGEQUEUE adds an address and then writes it to a MAILQUEUE (AddressWorker)
Another worker processes the MAILQUEUE and sends the packages out by mail (MailWorker).
After step 2. when a message is written to the MAILQUEUE, I want to notify the GUI and update the status of the package. Of course the GUI should not consume the messages in the MAILQUEUE, only the MailWorker must consume them.
You can use a combination of queue and topic for your solution.
Your GUI application can subscribe to a topic, say MAILQUEUE_NOTIFICATION. Every time (i.e at step 2) PackageProducer writes message to MAILQUEUE, a copy of that message should be published to MAILQUEUE_NOTIFICATION topic. Since the GUI application has subscribed to the topic, it will get that publication containing information on status of the package. GUI can be updated with the contents of that publication.
HTH
I'm building a simple message delegation application. Messages are being send on both ends via JMS. I'm using a MDB to process incoming messages, transform them and send them to a target queue. Unfortunately the same messages can be send to the incoming queue more than once but it is not allowed to forward duplicates.
So what is the best way to accomplish that?
Since there can be multiple MDBs listening on the incoming queue a need a single cache where I can store the unique message uuids of the incoming messages for at least an hour. How should this cache be accessed? Via a singleton/ static class (I'm running Java EE 5 and thus don't have the singleton annotation)?
In addition I think all operations must be synchronized, right? Does that harm performance too much?
#Ingo: are you OK with database solution. You can full fledged DB server or simple apache derby solution for this..
If so, you can have a simple table where you can store message unique UId and can check against it for uniqueness....this solution will have following benefits:
Simple code
No need of time bound cache(1 hour). You can check for uniqueness of a message forever.
Persistent record of what messages came in.
No need of expensive synchronized, you can rely on DB isolation level to have consistency.
centralized solution for your possibly many deployments of application.