I am using circular logging. Because of human intervention, one of the
queue files is corrupted.
Since the circular logging is not having the ability of recovering the
corrupted queue files, what will be the next steps it will take?
Will queue manager create an empty queue file for that queue and start
enrolling the messages to it? Else, it will just show the pending
messages in the queue but not allow the applications to process?
As you correctly note, MQ cannot recover from a damaged queue file when it is configured for circular logging.
Will queue manager create an empty queue file for that queue and start enrolling the messages to it? Else, it will just show the pending messages in the queue but not allow the applications to process?
None of the above. The queue manager will return an error to any process attempting to access that queue.
When a queue file is damaged, it may or may not have had messages in it. There is no automatic recovery possible that would correctly reconcile the state of any messages that may have been enqueued, therefore no further processing is done on that queue and any access returns an error. Human intervention is required in that case and the fix is to delete and redefine the queue using runmqsc.
If additional queue recovery is required to make sure messages are not lost in such cases, linear logging is mandatory.
The queue manager is not going to create an new queue file automatically. If you truly have a corrupted queue then you may have to delete and recreate it. It would be helpful if you can provide more info about the error you see indicating the queue is corrupt. Also, what version of MQ are you using?
Related
We are using IBM MQ and recently we faced an issue where some messages that were declared as sent to the MQ server by our client application were not consumed by our MQ consumer.
We lacked logging produced/consumed messages so we tried to check messages in MQ server log/data.
We found that messages are stored in /var/mqm/qmgrs/MQ_MANAGER/queues/ but we didn't find there all messages in the queue file (old messages were not found)
What is the rollover policy of IBM MQ and where does old queues files go?
That's not how the queue files work. They are not rollover logs. The same space is continually overwritten as needed to store messages, but messages may not be written there at all if they can be processed through memory caches etc.
PERSISTENT messages are usually logged in files under /var/mqm/log, but there are circumstances where even that can be avoided. Your qmgr's recovery logfile configuration (circular/linear etc) will determine whether historic information about PERSISTENT messages remains available.
NONPERSISTENT messages are never logged in those files.
In IBM MQ messages can be either persistent or non-persistent.
If a message is persistent it will normally be written to the transactional logs (usually under /var/mqm/log/MQ_MANAGER/active) before a commit completes or before the PUT completes if not done under a unit of work.
If a message is non-persistent it will not be written to the transactional logs.
At this point either type of message may reside only in memory and will only be written to the queue file (usually under /var/mqm/qmgrs/MQ_MANAGER/queues) if it needs to offload memory or if it is persistent and a check point is taken.
If the message is consumed in a timely manner it may never be written to the queue file.
The queue file will shrink in size if space taken up by messages that are no longer needed is in use, this happens automatically and is not configurable or documented by IBM as far as I know.
Non-persistent messages generally do not survive a queue manager restart.
Transactional logs can be configured as circular or linear. If circular the logs will be reused once they are no longer needed. If linear with automatic log management (introduced in 9.0.2) they will work similarly to circular. If linear without automatic log management, what happens to logs that are no longer needed would be based on your own log management.
If the message is still in the transactional log you may be able to view it as described in "Where's my message? Tool and instructions to use the MQ recovery log to find out what happened to your persistent MQ messages on distributed platforms".
I have got started with ActiveMQ and able to move forward, but I am not able to figure out how I can view all messages that are sent to a particular queue.
I can see the message in the queue as long as it is not consumed by a consumer but as soon as it is consumed by a consumer, I can no longer see the message in the queue.
In my project I used to use "MQJ Explorer" and connect with a queue manager and there I could view all the messages that were sent to a particular queue, so I am expecting similar thing with ActiveMQ.
I know while sending I can have the message marked as "persistent" but still it will be there in the queue only till it is not consumed, once it is consumed I cannot see it any more.
Please let me know if things work differently with ActiveMQ.
The Queue holds messages until they are consumed at which point they are discarded. So you cannot browse all messages that were sent to the queue and then consumed. The only solution to this would be to use a Camel route or similar to mirror the Queue to some other Audit type Queue but you'd need to use a lot of care as this could easily lead to filling the message store as the mirrored messages will continue to grow and take space unless they are purged periodically.
We are using IBM MQ8.0. Activitiy logs are getting logged for outgoing messages which we are sending to external system. But there is no log available for the messages which are from external system to our MQ Manager.
Is it problem with client channel configuration ?
Or MQ logging configuration issue ?
IBM describes these "activity logs" as recover logs in the Knowledge center page "Making sure that messages are not lost (logging)"
IBM MQ records all significant changes to the persistent data controlled by the queue manager in a recovery log.
This includes creating and deleting objects, persistent message updates, transaction states, changes to object attributes, and channel activities. The log contains the information you need to recover all updates to message queues by:
Keeping records of queue manager changes
Keeping records of queue updates for use by the restart process
Enabling you to restore data after a hardware or software failure
Please note that non-persistent messages are not logged to the recover log.
Based on your question it is likely that the messages you are sending to the external system are persistent messages and the messages you are receiving from the external system are non-persistent messages, this would explain why they are not logged to the recover log files.
Persistence is determined at the time the message is first PUT.
IBM has a good Technote "Message persistence FAQs" about this subject.
Q3. What is the best way to be certain that messages are persistent?
A3. Set MQMD message persistence to persistent (MQPER_PERSISTENT), or nonpersistent (MQPER_NOT_PERSISTENT) and your message will always retain that value.
Note: MQPER_PERSISTENCE_AS_Q_DEF is the default setting for the persistence value in the MQMD. See the persistence values listed below.
...
Additional information
MQPER_PERSISTENCE_AS_Q_DEF can lead to unexpected results. If there is more than one definition in the queue-name resolution path, the default persistence attribute is taken from first queue definition in the path at the time of the MQPUT or MQPUT1 call. This queue could be an:
alias queue
local queue
local definition of a remote queue
queue-manager alias
transmission queue
cluster queue
The external system will need to make sure the messages they send you are set as persistent messages if you want them to be logged.
According to the information center from IBM MQ, we can backup queue manager data in order to Backup and restore QMGR. One of the step is to take the copy of qmgr data and log file directories.
My question is what the data and log file directories specially mean? is it my below understanding correct?
data directory ---- /var/mqm/qmgrs/QMGR01/
log directory ---- /var/mqm/log/QMGR01/
Another one is MQ has the non-persistent and persistent message type. As for the non-persistent, is the message only being stored in memory? Once whatever crash, it can not be recovered.Rgt? However, the persistent message can survive that crash. But where is the persistent message stored normally?
Please help me out. Thanks very much
Yes, you have the directories correct. Just make sure that if you take a filesystem backup that the QMgr is shut down at the time.
Be aware that point-in-time backups are usually not a good strategy for backing up a QMgr. Whatever messages are on the QMgr at the time will be redelivered when the QMgr is restored, unless you take measures to stop that from happening. If the QMgr is in a cluster, it will be out of synch with the cluster when restored.
Generally the approach to backing up a QMgr is to save the object definitions, the access control lists, any exits and their parm files. Restoring the QMgr is a matter of using crtmqm to create a new instance and running in all the definitions.
Non-persistent messages are stored in memory until they overflow memory and then they are stored to the queue file on disk. If the queue is marked as NPMCLASS(HIGH) then the QMgr will attempt to save and restore non-persistent messages through an orderly shutdown and restart but will discard them if the QMgr crashes.
Persistent messages are hardened to both the queue and log files before control is returned to the calling program if written out of syncpoint. If persistent messages are written under syncpoint, WMQ allows lazy cached writes of the messages but insures they are all flushed before returning control from the COMMIT command.
This is in Cluster Environment. Queue Manager lost its identity in cluster and it is unable to connect to other servers. All channels to repository and others were retrying state.
CPU usage is optimal in this server. This is a UNIX box.
When I checked the logs below is it,
AMQ9532: Program cannot set queue
attributes.
EXPLANATION: The attempt to set the
attributes of queue
'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on
queue manager 'QMGR.SERVER6A' failed
with reason code 2102.
ACTION: Ensure
that the queue is available and retry
the operation.
----- amqrmssa.c : 690 --------------------------------------------------------
AMQ9999: Channel program ended
abnormally.
EXPLANATION: Channel program
'Channel.Coord00' ended abnormally.
ACTION: Look at previous error
messages for channel program
'Channel.Coord00' in the error files to
determine the cause of the failure.
----- amqrccca.c : 883 --------------------------------------------------------
03/06/11 08:24:26 AMQ9544: Messages
not put to destination queue.
EXPLANATION: During the processing of
channel 'Channel.Server6A' one or more
messages could not be put to the
destination queue and attempts were
made to put them to a dead-letter
queue. The location of the queue is
1, where 1 is the local dead-letter
queue and 2 is the remote dead-letter
queue.
ACTION: Examine the contents of
the dead-letter queue. Each message
is contained in a structure that
describes why the message was put to
the queue, and to where it was
originally addressed. Also look at
previous error messages to see if the
attempt to put messages to a
dead-letter queue failed. The program
identifier (PID) of the processing
program was '1372200'.
----- amqrmrca.c : 1318 -------------------------------------------------------
Then I did recycled the queue manager it is now fine?
My question here is how did the MQ resource problem occurr? CPU usage of this server is not more than 15%. Please advise.
There are three different and unrelated problems shown in the log.
AMQ9532: Program cannot set queue
attributes.
EXPLANATION: The attempt to set the
attributes of queue
'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on
queue manager 'QMGR.SERVER6A' failed
with reason code 2102.
The 2102 is MQRC_RESOURCE_PROBLEM and presumably the resource issue referred to in the posting. The 2102 can be any kind of scarce resource, including semaphores, user processes, queue handles, etc. Since the QMgr was attempting to set an attribute of the queue, it would have already had a thread instantiated but it would have required additional queue handles. When something like this occurs, use your admin tool (WMQ Explorer, mqmon or one of the many 3rd party tools) to look into the number of open queue handles, open channels, etc. Note that for a resource error, it will be necessary to maintain an open connection to the QMgr or else the tool will be unable to make a new connection when the resource shortage occurs.
AMQ9999: Channel program ended
abnormally.
EXPLANATION: Channel program
'Channel.Coord00' ended abnormally.
ACTION: Look at previous error
messages for channel program
'C00.US.MP00' in the error files to
determine the cause of the failure.
This error appears to actually be two different errors since it references two different channels. One of these appears to be an outbound cluster channel and the other appears to be a point-to-point channel. Neither channel mentioned in this error are associated with the first and last error message.
03/06/11 08:24:26 AMQ9544: Messages
not put to destination queue.
EXPLANATION: During the processing of
channel 'Channel.Server6A' one or more
messages could not be put to the
destination queue and attempts were
made to put them to a dead-letter
queue. The location of the queue is 1,
where 1 is the local dead-letter queue
and 2 is the remote dead-letter queue.
ACTION: Examine the contents of the
dead-letter queue. Each message is
contained in a structure that
describes why the message was put to
the queue, and to where it was
originally addressed. Also look at
previous error messages to see if the
attempt to put messages to a
dead-letter queue failed. The program
identifier (PID) of the processing
program was '1372200'.
The last error appears to be an inbound cluster channel. Since the first error was trying to set attributes of the cluster transmit queue, it could only have been associated with an outbound channel. Therefore the first and last error messages are unrelated. This error message appears to show an inbound message that was destined for a queue and that queue was full, PUT-disabled, or otherwise unable to accept the message. The message was therefore routed to the dead letter queue.
For the resource error, I would suggest reviewing the performance report appropriate to your platform. Go to the SupportPacs page and look for those SupportPacs named MP* and then look for the one for your platform. The Performance Reports give you specific tuning advice.
You may also want to review the Problem Determination chapter in the System Administration manual for additional advice on how to identify resource issues.
The WebSphere MQ cluster design and operation article in the developerWorks Mission:Messaging series has specific advice about keeping clusters healthy.
Last but not least, the WebSphere MQ MustGather page has sections on troubleshooting for all major platforms and categorized by problem area.
To increase the MAXMSGL to 100 MB in IBMMQ,
(Reason code-2102 - MQRC_RESOURCE_PROBLEM) after setting the MAXMSGL to 100 mb
Category: IBM WebSphere MQ
If you are receiving error Reason code:2102 - MQRC_RESOURCE_PROBLEM, then try
Queue manager->properties->Extended->Increase Log->Log primary files and Log->Log secondary files->value to 20