This is in Cluster Environment. Queue Manager lost its identity in cluster and it is unable to connect to other servers. All channels to repository and others were retrying state.
CPU usage is optimal in this server. This is a UNIX box.
When I checked the logs below is it,
AMQ9532: Program cannot set queue
attributes.
EXPLANATION: The attempt to set the
attributes of queue
'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on
queue manager 'QMGR.SERVER6A' failed
with reason code 2102.
ACTION: Ensure
that the queue is available and retry
the operation.
----- amqrmssa.c : 690 --------------------------------------------------------
AMQ9999: Channel program ended
abnormally.
EXPLANATION: Channel program
'Channel.Coord00' ended abnormally.
ACTION: Look at previous error
messages for channel program
'Channel.Coord00' in the error files to
determine the cause of the failure.
----- amqrccca.c : 883 --------------------------------------------------------
03/06/11 08:24:26 AMQ9544: Messages
not put to destination queue.
EXPLANATION: During the processing of
channel 'Channel.Server6A' one or more
messages could not be put to the
destination queue and attempts were
made to put them to a dead-letter
queue. The location of the queue is
1, where 1 is the local dead-letter
queue and 2 is the remote dead-letter
queue.
ACTION: Examine the contents of
the dead-letter queue. Each message
is contained in a structure that
describes why the message was put to
the queue, and to where it was
originally addressed. Also look at
previous error messages to see if the
attempt to put messages to a
dead-letter queue failed. The program
identifier (PID) of the processing
program was '1372200'.
----- amqrmrca.c : 1318 -------------------------------------------------------
Then I did recycled the queue manager it is now fine?
My question here is how did the MQ resource problem occurr? CPU usage of this server is not more than 15%. Please advise.
There are three different and unrelated problems shown in the log.
AMQ9532: Program cannot set queue
attributes.
EXPLANATION: The attempt to set the
attributes of queue
'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on
queue manager 'QMGR.SERVER6A' failed
with reason code 2102.
The 2102 is MQRC_RESOURCE_PROBLEM and presumably the resource issue referred to in the posting. The 2102 can be any kind of scarce resource, including semaphores, user processes, queue handles, etc. Since the QMgr was attempting to set an attribute of the queue, it would have already had a thread instantiated but it would have required additional queue handles. When something like this occurs, use your admin tool (WMQ Explorer, mqmon or one of the many 3rd party tools) to look into the number of open queue handles, open channels, etc. Note that for a resource error, it will be necessary to maintain an open connection to the QMgr or else the tool will be unable to make a new connection when the resource shortage occurs.
AMQ9999: Channel program ended
abnormally.
EXPLANATION: Channel program
'Channel.Coord00' ended abnormally.
ACTION: Look at previous error
messages for channel program
'C00.US.MP00' in the error files to
determine the cause of the failure.
This error appears to actually be two different errors since it references two different channels. One of these appears to be an outbound cluster channel and the other appears to be a point-to-point channel. Neither channel mentioned in this error are associated with the first and last error message.
03/06/11 08:24:26 AMQ9544: Messages
not put to destination queue.
EXPLANATION: During the processing of
channel 'Channel.Server6A' one or more
messages could not be put to the
destination queue and attempts were
made to put them to a dead-letter
queue. The location of the queue is 1,
where 1 is the local dead-letter queue
and 2 is the remote dead-letter queue.
ACTION: Examine the contents of the
dead-letter queue. Each message is
contained in a structure that
describes why the message was put to
the queue, and to where it was
originally addressed. Also look at
previous error messages to see if the
attempt to put messages to a
dead-letter queue failed. The program
identifier (PID) of the processing
program was '1372200'.
The last error appears to be an inbound cluster channel. Since the first error was trying to set attributes of the cluster transmit queue, it could only have been associated with an outbound channel. Therefore the first and last error messages are unrelated. This error message appears to show an inbound message that was destined for a queue and that queue was full, PUT-disabled, or otherwise unable to accept the message. The message was therefore routed to the dead letter queue.
For the resource error, I would suggest reviewing the performance report appropriate to your platform. Go to the SupportPacs page and look for those SupportPacs named MP* and then look for the one for your platform. The Performance Reports give you specific tuning advice.
You may also want to review the Problem Determination chapter in the System Administration manual for additional advice on how to identify resource issues.
The WebSphere MQ cluster design and operation article in the developerWorks Mission:Messaging series has specific advice about keeping clusters healthy.
Last but not least, the WebSphere MQ MustGather page has sections on troubleshooting for all major platforms and categorized by problem area.
To increase the MAXMSGL to 100 MB in IBMMQ,
(Reason code-2102 - MQRC_RESOURCE_PROBLEM) after setting the MAXMSGL to 100 mb
Category: IBM WebSphere MQ
If you are receiving error Reason code:2102 - MQRC_RESOURCE_PROBLEM, then try
Queue manager->properties->Extended->Increase Log->Log primary files and Log->Log secondary files->value to 20
Related
I have a Normal IBM MQ developer edition 924 installed on a ubuntu virtual machine. And have cpp code files for Publisher and Subscriber model. I created a topic and started sending messages on the topic.
The issue happens when I send a larger message on the topic, the error code of 2031(MQRC_MSG_TOO_BIG_FOR_Q_MGR) is outputted. (https://www.ibm.com/docs/en/ibm-mq/9.1?topic=arc-2031-07ef-rc2031-mqrc-msg-too-big-q-mgr)
I changed the maxmsgl value for the Queue manager and the SVR channel as well. I ran the following commands for this:
runmqsc QM1
ALTER QMGR MAXMSGL(104857600)
ALTER CHANNEL(SYSTEM.AUTO.SVRCONN) CHLTYPE(SVRCONN) maxmsgl(104857600)
But still the error persists, could someone reflect on what needs to be done regarding this error code. And also as message segmentation is not allowed on pubsub what is the maximum limit of message size it can handle.
When using a topic for publish/subscribe, the messages sent to subscribers are still put onto a queue for safe storage until they are consumed by the subscribing/getting application. These queues must also have MAXMSGL set appropriately.
If you do not know which queue to adjust, follow these steps:-
Run the subscriber application - assuming that it will wait around for a while
Issue the following MQSC command (e.g. using runmqsc):-
DISPLAY SUB(*) DEST DESTCLAS TOPICSTR DURABLE
The name shown in the DEST field in the output is the queue your subscription is using. If it's name starts with SYSTEM.MANAGED and it also shows DESTCLAS(MANAGED) then your subscription is a managed subscription. You say that your subscription is not managed, but if you don't know which queue is in use by that subscriber then it is most likely that it IS managed, since that means you don't have to worry about the queue.
A managed subscription queue is built from a MODEL queue referenced on the topic. If you are unsure which MODEL queue would be used, issue the following command using the TOPICSTR value from the output of the above command.
DISPLAY TPSTATUS('topic-str-value') MNDURMDL MDURMDL
If the output from the DISPLAY SUB command earlier showed that your subscription was DURABLE(YES) then the MODEL queue in use is the one shown in the field MDURMDL (Managed DURable MoDeL). If it showed DURABLE(NO) then the MODEL queue in use is the one shown in the field MNDURMDL (Managed Non-DURable MoDeL).
Using the name determined in step 4 if it is not managed, or in step 5 if it is managed, issue the following MQSC command:-
ALTER QLOCAL('q-name-from-step-4') MAXMSGL(104857600)
or
ALTER QMODEL('q-name-from-step-5') MAXMSGL(104857600)
Now re-make your subscription (so that it deletes the previous temporary queue it created and makes a new one from the model) and re-run your publisher application.
We are using IBM MQ8.0. Activitiy logs are getting logged for outgoing messages which we are sending to external system. But there is no log available for the messages which are from external system to our MQ Manager.
Is it problem with client channel configuration ?
Or MQ logging configuration issue ?
IBM describes these "activity logs" as recover logs in the Knowledge center page "Making sure that messages are not lost (logging)"
IBM MQ records all significant changes to the persistent data controlled by the queue manager in a recovery log.
This includes creating and deleting objects, persistent message updates, transaction states, changes to object attributes, and channel activities. The log contains the information you need to recover all updates to message queues by:
Keeping records of queue manager changes
Keeping records of queue updates for use by the restart process
Enabling you to restore data after a hardware or software failure
Please note that non-persistent messages are not logged to the recover log.
Based on your question it is likely that the messages you are sending to the external system are persistent messages and the messages you are receiving from the external system are non-persistent messages, this would explain why they are not logged to the recover log files.
Persistence is determined at the time the message is first PUT.
IBM has a good Technote "Message persistence FAQs" about this subject.
Q3. What is the best way to be certain that messages are persistent?
A3. Set MQMD message persistence to persistent (MQPER_PERSISTENT), or nonpersistent (MQPER_NOT_PERSISTENT) and your message will always retain that value.
Note: MQPER_PERSISTENCE_AS_Q_DEF is the default setting for the persistence value in the MQMD. See the persistence values listed below.
...
Additional information
MQPER_PERSISTENCE_AS_Q_DEF can lead to unexpected results. If there is more than one definition in the queue-name resolution path, the default persistence attribute is taken from first queue definition in the path at the time of the MQPUT or MQPUT1 call. This queue could be an:
alias queue
local queue
local definition of a remote queue
queue-manager alias
transmission queue
cluster queue
The external system will need to make sure the messages they send you are set as persistent messages if you want them to be logged.
I have an application listening to messages on an IBM Websphere MQ queue.
Once a message is consumed, the application performs some processing logic.
If the processing completed OK, I would like the application to acknowledge the message and have it removed from the queue.
If an error occurred while processing, I would like the message to remain in the queue.
How is this implemented? (I'm using the .NET API)
Thanks.
MQ supports a single-phase commit protocol. You specify syncpoint when you get the message, then issue COMMIT or ROLLBACK as required. The default action if the connection is lost is ROLLBACK and if the program deliberately ends without resolving the transaction a COMMIT is assumed. (This is platform dependent so the customary advice is to explicitly call COMMIT and not rely on the class destructors to do it for you.)
This works whether the message is persistent or not. However if the message has an expiry specified and expires after being rolled back there's a chance it won't be seen again.
Of course, if the program issues a ROLLBACK the message will normally be seen again since it goes back to the same spot int he queue and for a FIFO queue that's the top. If the problem with the message is not transient then this causes a poison message loop of read/rollback/repeat. To avoid that the app can check the backout count and if it exceeds some threshold requeue the message to an exception queue.
When using JMS or XMS this is done for you by the class libraries. If the input queue's BOQNAME and BOQTHRESH attributes are set the requeue is to the queue names in BOQNAME. Otherwise a requeue to the Dead Queue is attempted. IF that fails (as it should if the system is properly secured) the listener will stop receiving messages.
The usual advice is to always specify a backout queue and either let the classes use it or code the app to use it.
Please see Usage Notes for MQGET in the MQAPI Reference and the MQGetMessageOptions.NET page in the .Net class reference.
You may want to look at the MQ Reporting Options.
Expiry, Confirmation of Arrival and Confirmation of Delivery can be requested and sent via a response queue back to the sending application by the receiving Queue Manager.
Positive and Negative Acknowledgements can also be generated by the receiving application provided they use the related reporting attributes found in the Message Descriptor.
Exception can be requested and sent via a response queue back to the sending application by any Queue Manager in the transmission chain or generated by the receiving application.
1 Read the message using MQC.MQGMO_SYNCPOINT,
2 process it
3 call MQQueueManager.Commit()
If Commit() is not called explicitly, or implicitly (eg exception is thrown), all messages that have been de-queued will be re-enqueued.
I am using circular logging. Because of human intervention, one of the
queue files is corrupted.
Since the circular logging is not having the ability of recovering the
corrupted queue files, what will be the next steps it will take?
Will queue manager create an empty queue file for that queue and start
enrolling the messages to it? Else, it will just show the pending
messages in the queue but not allow the applications to process?
As you correctly note, MQ cannot recover from a damaged queue file when it is configured for circular logging.
Will queue manager create an empty queue file for that queue and start enrolling the messages to it? Else, it will just show the pending messages in the queue but not allow the applications to process?
None of the above. The queue manager will return an error to any process attempting to access that queue.
When a queue file is damaged, it may or may not have had messages in it. There is no automatic recovery possible that would correctly reconcile the state of any messages that may have been enqueued, therefore no further processing is done on that queue and any access returns an error. Human intervention is required in that case and the fix is to delete and redefine the queue using runmqsc.
If additional queue recovery is required to make sure messages are not lost in such cases, linear logging is mandatory.
The queue manager is not going to create an new queue file automatically. If you truly have a corrupted queue then you may have to delete and recreate it. It would be helpful if you can provide more info about the error you see indicating the queue is corrupt. Also, what version of MQ are you using?
I have two MQ queue manager with same queue names configured. Both are configured to send data to different servers. Currently queue manager(QM1) is stopped(status Ended Immediately) and QM2 is running
Now my program opens the queue and sends data. It doesnot specify queue manager name. When I execute the program, MQ connection request returns error 2059.
My questions are:
What happens when multiple queue managers have same queue name?
How to tackle situation without changing the code?
Please forgive if the description is vague. It would be helpful if anyone provide links so that newbie like me can learn something.
Thanks
It would be helpful if could provide details on your application. Whether it's using server bindings or client mode connection to queue manager. What version of MQ are you using?
The below information is valid for MQ v7.x:
If you are using client mode then you can use multiple CONNNAMEs to connect. If one queue manager is down, your application will connect to next queue manager in CONNAME list. One of the simplest way to do when using client mode connection is to define MQSERVER environment variable and specify multiple CONNNAMEs.
SET MQSERVER=<channel name>/TCP/host1(port1), host2(port2)
For example when both queue managers are on local host:
SET MQSERVER=MYSVRCONCHN/TCP/localhost(1414),localhost(1415)
In server bindings mode if queue manager name is not specified, then application will attempt to connect to the default queue manager. If the default queue manager is down, then 2059 is thrown.
Your explaination doesn't provide clarity about your requirements.
You wrote:
My questions are 1. What happens when multiple queue managers have same queue name.
Nothing. Its a normal scenario. Different queue managers may have queues with same name and it doesn't create any ambiguity. Although, scenario will be a little different when the queue managers are in same cluster and the queue is also a cluster queue. Then everything will depend on requirements and design.
You wrote:
2. How to tackle situation without changing the code
Run the queue manager which is stopped.
You wrote:
Now my program opens the queue and sends data. It doesnot specify
queue manager name.
What application are you using?For a client application, you access a queue using an object of queue manager.
I am asssuming that you are using an application(client) which doesn't take queue manager details from you, only takes queue details. And may be the queue manager is hard coded within the code. And it sends the message first to the queue of Queue manager 1 and then to queue manager 2. But, in your case queue manager 1 is down.
If above is the case, then the application's code needs to be changed. You should have exception handling in such a way that it executes the code for sending the message to the second queue manager even though the first lines of code throws error.