Parse IBM MQ v9.1 Error Logs using Splunk - ibm-mq

Question

I'm forwarding my IBM MQ v9.1 error logs using splunk forwarder to a centralized cluster to see trends on common error occurring across my distributed messaging systems.
However I'm unable to parse the required fields, since the format of MQ error logs are varying i.e. the severity of the messages could be error, warning, informational, severe and termination and each have different set of fields in itself and are not consistent.
Please let me know if anyone have used regex in splunk for parsing the fields of IBM MQ error logs for v9.1.
I have tried few regex patterns but it wasn't parsing as expected.
I have already referred below link, but that is for v8 and there is a different in format of error logs for v9,
https://t-rob.net/2017/12/18/parsing-mq-error-logs-in-splunk/
Also the splunk user is unable to access the error logs. I have updated below stanza in qm.ini
Filesystem:
ValidateAuth=No
also set chmod -R 755 to /var/mqm/qmgrs/qmName/errors folder.
Though the permissions for the ERROR logs doesn't change whenever it gets updated, when the logs rotate the permissions are revoked and splunk user is not able to read the logs.
Please let me know how to overcome this without adding splunk user to mqm group

Answer 1

I would suggest enabling JSON logging and forward those logs to Splunk which should be able to parse this format.
In IBM MQ v9.0.4 CDS release IBM added the ability to log out to a JSON formatted log, MQ will always log to the original AMQERR0x.LOG files even if you enable the JSON logging. This is included in all MQ 9.1 LTS and CSD releases.
The IBM MQ v9.1 Knowledge Center Page IBM MQ>Configuring>Changing IBM MQ and queue manager configuration information>Attributes for changing queue manager configuration information>Diagnostic message logging>Diagnostic message service stanzas>Diagnostic message services has information on the topic. You can add the following to your qm.ini to have it output the log information to a JSON formatted file called AMQERR0x.json in the standard queue manager errors directory:
DiagnosticMessages:
Service = File
Name = JSONLogs
Format = json
FilePrefix = AMQERR
As noted by the OP the JSON formatted logs do not contain the EXPLANATION or ACTION portion that you see in the normal logs.
In IBM MQ v9.1 you can use the mqrc command to convert the JSON format to the familiar format you see in AMQERR01.LOG.
One simple example is below:
cat <<EOL |mqrc -i json -o text -
{"ibm_messageId":"AMQ9209E","ibm_arithInsert1":0,"ibm_arithInsert2":0,"ibm_commentInsert1":"localhost (127.0.0.1)","ibm_commentInsert2":"TCP/IP","ibm_commentInsert3":"SYSTEM.DEF.SVRCONN","ibm_datetime":"2018-02-22T06:54:53.942Z","ibm_serverName":"QM1","type":"mq_log","host":"0df0ce19c711","loglevel":"ERROR","module":"amqccita.c:4214","ibm_sequence":"1519282493_947814358","ibm_remoteHost":"127.0.0.1","ibm_qmgrId":"QM1_2018-02-13_10.49.57","ibm_processId":4927,"ibm_threadId":4,"ibm_version":"9.1.0.5","ibm_processName":"amqrmppa","ibm_userName":"johndoe","ibm_installationName":"Installation1","ibm_installationDir":"/opt/mqm","message":"AMQ9209E: Connection to host 'localhost (127.0.0.1)' for channel 'SYSTEM.DEF.SVRCONN' closed."}
EOL
The output will be:
02/22/2018 06:54:53 AM - User(johndoe) Program(amqrmppa)
Host(0df0ce19c711) Installation(Installation1)
VRMF(9.1.0.5) QMgr(QM1)
Time(2018-02-22T11:54:53.942Z)
RemoteHost(127.0.0.1)
CommentInsert1(localhost (127.0.0.1))
CommentInsert2(TCP/IP)
CommentInsert3(SYSTEM.DEF.SVRCONN)
AMQ9209E: Connection to host 'localhost (127.0.0.1)' for channel
'SYSTEM.DEF.SVRCONN' closed.
EXPLANATION:
An error occurred receiving data from 'localhost (127.0.0.1)' over TCP/IP. The
connection to the remote host has unexpectedly terminated.
The channel name is 'SYSTEM.DEF.SVRCONN'; in some cases it cannot be determined
and so is shown as '????'.
ACTION:
Tell the systems administrator.
----- amqccita.c : 4214 -------------------------------------------------------
You can also use mqrc with just the error message from the JSON, for example AMQ9209E, you can run the command like this:
mqrc AMQ9209E
The output will be:
536908297 0x20009209 rrcE_CONNECTION_CLOSED
536908297 0x20009209 urcMS_CONN_CLOSED
MESSAGE:
Connection to host '<insert one>' for channel '<insert three>' closed.
EXPLANATION:
An error occurred receiving data from '<insert one>' over <insert two>. The
connection to the remote host has unexpectedly terminated.
The channel name is '<insert three>'; in some cases it cannot be determined and
so is shown as '????'.
ACTION:
Tell the systems administrator.
You could take it further and specify the inserts from the JSON:
Exmple portion of the JSON log:
"ibm_messageId":"AMQ9209E","ibm_arithInsert1":0,"ibm_arithInsert2":0,"ibm_commentInsert1":"localhost (127.0.0.1)","ibm_commentInsert2":"TCP/IP","ibm_commentInsert3":"SYSTEM.DEF.SVRCONN"
In the command below each ibm_arthInsert is specified with a proceeding -n flag in order following by each ibm_commentInsert with a proceeding -c flag:
mqrc AMQ9209E -n 0 -n 0 -c "localhost (127.0.0.1)" -c "TCP/IP" -c "SYSTEM.DEF.SVRCONN"
The output is below:
536908297 0x20009209 rrcE_CONNECTION_CLOSED
536908297 0x20009209 urcMS_CONN_CLOSED
MESSAGE:
Connection to host 'localhost (127.0.0.1)' for channel 'SYSTEM.DEF.SVRCONN'
closed.
EXPLANATION:
An error occurred receiving data from 'localhost (127.0.0.1)' over TCP/IP. The
connection to the remote host has unexpectedly terminated.
The channel name is 'SYSTEM.DEF.SVRCONN'; in some cases it cannot be determined
and so is shown as '????'.
ACTION:
Tell the systems administrator.