Question:
Is it possible to copy MQ Messages from one Queue Manager/Queue to a different Queue Manager/Queue?
Scenario:
I have a "PROD" Queue Manager and when it receives a Message on it's Queues I would like to "copy" the Message to a queue on a "TEST" Queue Manager.
Requirements
The original message must be left on the PROD queue to be processed as normal.
This must be an automated process (lots of messages during a day). I could not intervene on a Message by Message basis.
If at all possible I would like this to be implemented by some native MQ functionality rather than an ad hoc program/script.
The copying must be as near to real time as possible
Must work with MQ version is 7.0.2.1(!). This cannot be changed.
Must run on Red Hat Enterprise Linux Server release 5.11 (Tikanga). Again, can't be changed.
I'm no MQ expert so use small words please
Thanks in advance
The only problem with the technote pointed out by gouda is that MQ will modify/changed the MsgId and CorrelId of each message replicated.
If the MsgId and/or CorrelId fields are important then the only other option is an MQ API Exit that replicates the message. You may need a commercial product like MQ Message Replication.
The next question is how are you going to move the message from a PROD queue to a TEST queue? You definitely do NOT want to create channels between a PROD queue manager and a TEST queue manager.
There are lots of tools that can off offload PROD messages to a file then you can move the file to your TEST environment and then load the messages into a TEST queue. Here is a list of MQ tools that can do it. The 2 tools you should try out are: MQ Batch Toolkit and QLoad.
Personally, I would create a schedule task (CRONTAB) to be run every night at midnight to off the messages and I would make the filename include the date and time. The last steps of the script would be to zip/compress the file and delete the original file (because the data you offload could be massive).
Hence, any time you want a particular day's PROD messages, just copy the file to your TEST server and unzip/uncompress it and load it into the queue.
all you need is mqadmin staff and this technote
Related
I have a application where I am using IBM MQ in a Pseudo-Synchronous way. I have request Q and Response Q. This will called through service. Now there is a time out period for my service to show the response.
But the actual response can be pushed to the response Q after the timeout also. Now I want to move such all delayed messages(timed out) to a separate message Q, such that they can be consumed by a different process.
Is this is possible through setting by time to live? Is any approach available in IBM MQ for such movement.
Please refer below image for the scenario.
Don't use time to live because the message will expire and be deleted - unless that is what you want. From your description/picture, it sounds like you want another application to process late responses.
There is nothing in MQ that will move messages older then "x" seconds/minutes to another queue. You will need to write a program to do that. If you write a program, make sure you do the move under a unit of work.
There are several commercial programs that can do it: (1) GUI program: MQ Visual Edit and (2) Command Line (shell): MQ Batch Toolkit.
We are currently implementing mq fte solution
One of the projects need to executive file to queue function because the target system reads only from mq .
We are looking for a way not only upload the files to queue but to keep the order of the files too.
We need that the oldest file will be uploaded first ( by modification or creation date ) and the the oldest file after him in the folder
Someone had this request on fte ? How did you handle it ?
The source system is windows .
Thanks for the assistance .
That depends on your setup. Is there a single queue manager in your scenario? Does the source system share the same local queue manager with the target system?
The order of messages might be guaranteed by default, as the MQ v7 Infocenter states in chapter Priority, in these cases:
If an application puts a sequence of messages on a queue, another
application can retrieve those messages in the same order that they
were put, provided:
The messages all have the same priority
The messages were all put within the same unit of work, or all put outside a unit of work
The queue is local to the putting application
If these conditions are not met, and the applications depend on the messages
being retrieved in a certain order, the applications must either
include sequencing information in the message data, or establish a
means of acknowledging receipt of a message before the next one is
sent.
If you do not meet these requirements (for example when the communication spans multiple queue managers), you can meet the requirements by:
ensuring that next message is put if and only if the recipient confirmed getting the previous one (for example by a MQ reply message)
using Message Groups to retrieve messages in logical order - that requires setting GroupId and MsgSeqNumber in the MQMD by the putting application and subsequently using MQGMO_LOGICAL_ORDER option by getting application (see chapter Logical and Physical ordering)
I have two MQ queue manager with same queue names configured. Both are configured to send data to different servers. Currently queue manager(QM1) is stopped(status Ended Immediately) and QM2 is running
Now my program opens the queue and sends data. It doesnot specify queue manager name. When I execute the program, MQ connection request returns error 2059.
My questions are:
What happens when multiple queue managers have same queue name?
How to tackle situation without changing the code?
Please forgive if the description is vague. It would be helpful if anyone provide links so that newbie like me can learn something.
Thanks
It would be helpful if could provide details on your application. Whether it's using server bindings or client mode connection to queue manager. What version of MQ are you using?
The below information is valid for MQ v7.x:
If you are using client mode then you can use multiple CONNNAMEs to connect. If one queue manager is down, your application will connect to next queue manager in CONNAME list. One of the simplest way to do when using client mode connection is to define MQSERVER environment variable and specify multiple CONNNAMEs.
SET MQSERVER=<channel name>/TCP/host1(port1), host2(port2)
For example when both queue managers are on local host:
SET MQSERVER=MYSVRCONCHN/TCP/localhost(1414),localhost(1415)
In server bindings mode if queue manager name is not specified, then application will attempt to connect to the default queue manager. If the default queue manager is down, then 2059 is thrown.
Your explaination doesn't provide clarity about your requirements.
You wrote:
My questions are 1. What happens when multiple queue managers have same queue name.
Nothing. Its a normal scenario. Different queue managers may have queues with same name and it doesn't create any ambiguity. Although, scenario will be a little different when the queue managers are in same cluster and the queue is also a cluster queue. Then everything will depend on requirements and design.
You wrote:
2. How to tackle situation without changing the code
Run the queue manager which is stopped.
You wrote:
Now my program opens the queue and sends data. It doesnot specify
queue manager name.
What application are you using?For a client application, you access a queue using an object of queue manager.
I am asssuming that you are using an application(client) which doesn't take queue manager details from you, only takes queue details. And may be the queue manager is hard coded within the code. And it sends the message first to the queue of Queue manager 1 and then to queue manager 2. But, in your case queue manager 1 is down.
If above is the case, then the application's code needs to be changed. You should have exception handling in such a way that it executes the code for sending the message to the second queue manager even though the first lines of code throws error.
How would I set up MQ so that every message received is immediately written to file system?
I have the "redbooks", but at least need someone at least point me to a chapter or heading in the book to figure it out.
We are a .NET shop. I have written C# via API to read the queue, and we currently use BizTalk MQ adapter. Our ultimate goal is to write same message to multiple directories in file system to "clone" the feed for our various test environments (DEV, STAGE, TRAINING, etc..). The problem with BizTalk is that when we consume the message, we map it at the same time to a new message, so the message is already changed, and we want the original raw message to be cloned, not the morphed one. Our vendors don't offer multiple copies of the feed, for example, they offer DEV and PROD, but we have 4 systems internally.
I suppose I could do a C# Windows Service to do it, but I would rather use built-in features of MQ if possible.
There is no configuration required. If the message is persistent, WMQ writes it to disk. However, I don't think that's going to help you because they are not written as discrete messages. There's no disk file to copy and replication only works if the replicated QMgr is identical to the primary and is offline during the replication.
There are a number of solutions to this problem but as of WMQ V7, the easiest one is to use the built-in Pub/Sub functionality. This assumes that the messages are arriving over a QMgr-to-QMgr channel and landing on a queue where you then consume them.
In that case, it is possible to delete the queue and create an alias of the same name over a topic. You then create a new queue and define an administrative subscription that delivers messages on the topic into the new queue. Your app consumes from the new queue.
When you need to send a feed to another QMgr or application, define a new subscription and point it at the new destination queue. Since this is Pub/Sub, MQ will replicate the original message as many times as there are subscriptions and the first application and its messages are not affected. If the destination you need to send to isn't accessible over MQ channels (perhaps DEV and QA are not connected, for example), you can deliver the messages to the new queue, use QLoad from SupportPac MO03 to write them to a file and then use another instance of QLoad to load them onto a different QMgr. If you wanted to move them in real time, you could set up the Q program from SupportPac MA01 to move them direct from the new subscription queue on QMgr1 to the destination queue on QMgr2. And you can replicate across as many systems as you need.
The SupportPacs main page is here.
If all you are using is the Redbooks, you might want to have a look at the Infocenters. Be sure to use the Infocenter matching the version of WMQ you are using.
WMQ V7.0 Infocenter
WMQ V7.1 Infocenter
WMQ V7.5 Infocenter
I was wondering if there is a way for you to configure a queue to automatically clear messages? We are striving to partially implement a component of our architecture and want to be able to send to the queue, but have the queue automatically remove the messages that are being sent so that we don't have to run scripts, etc to perform the clean-up.
So far the only thing I have been able to find is to run CLEAR QLOCAL or set the messages to expire from the publishing application.
For you use case there are a few options in IBM MQ:
Create a QALIAS that points to a TOPIC object which has a topic string with no subscribers, messages put to the QA will just disappear.
Have the sending application set message expiry.
Use the IBM MQ CAPEXPRY feature to administratively force message expiry at the queue level.
Run a script to issue CLEAR QLOCAL against the queue. There can not be open handles on the queue for this to work.
Programmatically issue the equivalent PCF command to CLEAR QLOCAL against the queue. There cannot be open handles on the queue for this to work.
Run the IBM MQ dmpmqmsg utility against the queue to read and discard the messages.