Queue manager ended unexpectedly - ibm-mq

Question

We are using MQ 9.0.0.1 version on linux machine and it is active/passive configuration.
We have faced issue in production server issue is "queue manager ended unexpectedly" around 10:30 AM IST, we have taken queue manager restart manually post restart everything is normal issue got resolved.
From the Queue manager logs we have observed below error stacks. Request to all,
please provide your analysis from the below queue manager logs why the queue manager ended unexpectedly?
-------------------------------------------------------------------------------
03/07/2019 07:21:10 AM - Process(5082.15855) User(tsg) Program(amqzlaa0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ7234: 20000 messages from queue 'NEFT_SMS_INQUIRY' loaded on queue manager
'QMGR_NEFT'.
EXPLANATION:
20000 messages from queue NEFT_SMS_INQUIRY have been loaded on queue manager
QMGR_NEFT.
ACTION:
None.
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5003.1) User(tsg) Program(amqzxma0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5008: An essential IBM MQ process 5016 (zllCRIT) cannot be found and is
assumed to be terminated.
EXPLANATION:
1) A user has inadvertently terminated the process. 2) The system is low on
resources. Some operating systems terminate processes to free resources. If
your system is low on resources, it is possible it has terminated the process
so that a new process can be created.
ACTION:
IBM MQ will stop all MQ processes. Inform your systems administrator. When
the problem is rectified IBM MQ can be restarted.
----- amqzxmb0.c : 10095 ------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5080.1) User(tsg) Program(amqpcsea)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ8506: Command server MQGET failed with reason code 2009.
EXPLANATION:
An MQGET request by the command server, for the IBM MQ queue
SYSTEM.ADMIN.COMMAND.QUEUE , failed with reason code 2009.
ACTION:
None.
----- amqphrea.c : 86 ---------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5069.1) User(tsg) Program(amqzmgr0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5025: The command server has ended. ProcessId(5080).
EXPLANATION:
The command server process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5051.1) User(tsg) Program(amqrrmfa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9542: Queue manager is ending.
EXPLANATION:
The program will end because the queue manager is quiescing.
ACTION:
None.
----- amqrrmfa.c : 3011 -------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5051.1) User(tsg) Program(amqrrmfa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9411: Repository manager ended normally.
EXPLANATION:
The repository manager ended normally.
ACTION:
None.
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5047.9) User(tsg) Program(amqzmuf0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5976: 'IBM MQ Distributed Pub/Sub Command Task' has ended.
EXPLANATION:
'IBM MQ Distributed Pub/Sub Command Task' has ended.
ACTION:
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5047.10) User(tsg) Program(amqzmuf0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5976: 'IBM MQ Distributed Pub/Sub Publish Task' has ended.
EXPLANATION:
'IBM MQ Distributed Pub/Sub Publish Task' has ended.
ACTION:
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5047.8) User(tsg) Program(amqzmuf0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5976: 'IBM MQ Distributed Pub/Sub Fan Out Task' has ended.
EXPLANATION:
'IBM MQ Distributed Pub/Sub Fan Out Task' has ended.
ACTION:
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5047.7) User(tsg) Program(amqzmuf0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5976: 'IBM MQ Distributed Pub/Sub Controller' has ended.
EXPLANATION:
'IBM MQ Distributed Pub/Sub Controller' has ended.
ACTION:
-------------------------------------------------------------------------------
03/07/2019 10:30:15 AM - Process(31712.199) User(tsg) Program(amqrmppa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9508: Program cannot connect to the queue manager.
EXPLANATION:
The connection attempt to queue manager 'QMGR_NEFT' failed with reason code
2059.
ACTION:
Ensure that the queue manager is available and operational.
----- cmqxrmsa.c : 6146 -------------------------------------------------------
03/07/2019 10:30:15 AM - Process(31712.199) User(tsg) Program(amqrmppa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9772: MQCTL failed with MQRC=2009.
EXPLANATION:
The indicated IBM MQ API call failed for the specified reason code.
ACTION:
Refer to the Application Programming Reference manual for information about
Reason Code 2009.
----- cmqxrstf.c : 2663 -------------------------------------------------------
03/07/2019 10:30:15 AM - Process(31712.199) User(tsg) Program(amqrmppa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9999: Channel 'GPSSVRCONN' to host 'XXXXXX' ended abnormally.
EXPLANATION:
The channel program running under process ID 31712 for channel 'GPSSVRCONN'
ended abnormally. The host name is 'XXXXXX'; in some cases the host name
cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 938 --------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5081.5) User(tsg) Program(runmqlsr)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9508: Program cannot connect to the queue manager.
EXPLANATION:
The connection attempt to queue manager 'QMGR_NEFT' failed with reason code
2059.
ACTION:
Ensure that the queue manager is available and operational.
----- cmqxrmsa.c : 471 --------------------------------------------------------
03/07/2019 10:30:15 AM - Process(5081.5) User(tsg) Program(runmqlsr)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9999: Channel 'GPSSVRCONN' to host 'XXXXXX' ended abnormally.
EXPLANATION:
The channel program running under process ID 5081 for channel 'GPSSVRCONN''
ended abnormally. The host name is 'XXXXXX'; in some cases the host name
cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 938 --------------------------------------------------------
03/07/2019 10:30:17 AM - Process(5078.1) User(tsg) Program(runmqchi)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9510: Messages cannot be retrieved from a queue.
EXPLANATION:
The attempt to get messages from queue 'SYSTEM.CHANNEL.INITQ' on queue manager
'QMGR_NEFT' failed with reason code 2009.
ACTION:
If the reason code indicates a conversion problem, for example
MQRC_SOURCE_CCSID_ERROR, remove the message(s) from the queue. Otherwise,
ensure that the required queue is available and operational.
----- amqrimna.c : 1085 -------------------------------------------------------
03/07/2019 10:30:18 AM - Process(5069.1) User(tsg) Program(amqzmgr0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5023: The channel initiator has ended. ProcessId(5078).
EXPLANATION:
The channel initiator process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
03/07/2019 10:30:24 AM - Process(31712.192) User(tsg) Program(amqrmppa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9540: Commit failed.
EXPLANATION:
The program ended because return code 2009 was received when an attempt was
made to commit change to the resource managers. The commit ID was
'AMQRMRSASFMS.TO.IPAYNEFT XXXXXX E怜͇'.
ACTION:
Tell the systems administrator.
----- amqrmrca.c : 2977 -------------------------------------------------------
03/07/2019 10:30:24 AM - Process(31712.192) User(tsg) Program(amqrmppa)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ9999: Channel 'SFMS.TO.IPAYNEFT' to host 'XXXXXX' ended abnormally.
EXPLANATION:
The channel program running under process ID 31712 for channel
'SFMS.TO.IPAYNEFT' ended abnormally. The host name is 'XXXXXX'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 938 --------------------------------------------------------
03/07/2019 10:30:24 AM - Process(5071.1) User(tsg) Program(amqfqpub)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5807: Queued Publish/Subscribe Daemon for queue manager QMGR_NEFT ended.
EXPLANATION:
The Queued Publish/Subscribe Daemon on queue manager QMGR_NEFT has ended.
ACTION:
None.
-------------------------------------------------------------------------------
03/07/2019 10:30:24 AM - Process(5003.1) User(tsg) Program(amqzxma0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5050: An essential IBM MQ process 5071 (amqfqpub) cannot be found and is
assumed to be terminated.
EXPLANATION:
1) A user has inadvertently terminated the process. 2) The system is low on
resources. Some operating systems terminate processes to free resources. If
your system is low on resources, it is possible it has terminated the process
so that a new process can be created. 3) MQ has encountered an unexpected
error. Check for possible errors reported in the MQ error logs and for any
FFSTs that have been generated.
ACTION:
IBM MQ will attempt to restart the terminated process.
----- amqzxmb0.c : 10095 ------------------------------------------------------
03/07/2019 10:30:24 AM - Process(5003.1) User(tsg) Program(amqzxma0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ8004: IBM MQ queue manager 'QMGR_NEFT' ended.
EXPLANATION:
IBM MQ queue manager 'QMGR_NEFT' ended.
ACTION:
None.
-------------------------------------------------------------------------------
03/07/2019 10:30:25 AM - Process(5069.1) User(tsg) Program(amqzmgr0)
Host(xxxxxxx) Installation(Installation1)
VRMF(9.0.0.1) QMgr(QMGR_NEFT)
AMQ5027: The listener 'LISTENER.TCP' has ended. ProcessId(5081).
EXPLANATION:
The listener process has ended.
ACTION:
None.
-------------------------------------------------------------------------------