I want to load files in the file system to a WebSphere MQ Queue. There are couple of support pacs - Q Program and MO03: WebSphere MQ Queue Load / Unload Utility
that come close but they mandate the files to be in a specific format. I have the messages which are XML files and want a quick way to load them to a queue. The number of files run into a few hundred so looking for an utility to do this job instead of having to write an application to achieve this.
I could not locate some general purpose application to achieve this. So looking for some help here
Thanks
Why do you believe that the Q program requires a specific file format? According to the README.TXT file, the following options are available:
-f<filename>
Input file.
Each line of the file will be put to output queue as a different
message.
See "Z/OS FILE NAME FORMAT EXAMPLES" for specific z/OS details.
-F[+]<filename>
Input/output file.
Entire file will be put to the output queue as a single message.
If '+' is specified the dataset attributes will be retained if
the output dataset exists - z/OS only.
See "Z/OS FILE NAME FORMAT EXAMPLES" for specific z/OS details.
So if you specify -F (without the +) all lines in the the XML file are loaded into a single message. You can also specify the message options using the -a parameter:
-a<Opts> Sets message attributes when put to the output queue
n - forces non-persistence
p - forces persistence
q - uses queue default persistence
d - put a datagram message type
r - put a reply message type
R - put a request message type
t - put a report message type
x - don't treat lines starting with '#' as special
Although the Q program will interpret files by default, note that the -ax option above tells it to ignore lines with # which it would ordinarily interpret as commands. This allows you to load XML files or source code with comments or even binary files such as a PDF or JPG.
Was there a specific limitation in Q that you are unable to work with? If so, it would be helpful to know what that is so we might point you to something that would better fit your purpose.
UPDATE
Responding to Spyro's comments, Q is not limited to 1000 characters. Here's an example where the README file from the Q distribution is written to a single message and read back.
D:\WMQ\MA01>q -m JMSDEMO -OSYSTEM.DEFAULT.LOCAL.QUEUE -FREADME
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 1 2012 ]
Connecting ...connected to 'JMSDEMO'.
D:\WMQ\MA01>echo dis q(SYSTEM.DEFAULT.LOCAL.QUEUE) curdepth | runmqsc JMSDEMO
5724-H72 (C) Copyright IBM Corp. 1994, 2011. ALL RIGHTS RESERVED.
Starting MQSC for queue manager JMSDEMO.
1 : dis q(SYSTEM.DEFAULT.LOCAL.QUEUE) curdepth
AMQ8409: Display Queue details.
QUEUE(SYSTEM.DEFAULT.LOCAL.QUEUE) TYPE(QLOCAL)
CURDEPTH(1)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
D:\WMQ\MA01>q -m JMSDEMO -dl -iSYSTEM.DEFAULT.LOCAL.QUEUE
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 1 2012 ]
Connecting ...connected to 'JMSDEMO'.
MQGET 24309 bytes
============================================================================
Message Descriptor (MQMD)
Report :00000000
Message Type :8 (Datagram)
Format :'MQSTR '
Priority :0
Persistence :0 (Not Persistent)
Message Id :A M Q J M S D E M O . . . R . * .
414D51204A4D5344454D4F20202020201DDEA052200B2A02
'AMQ JMSDEMO ...R .*.'
ReplyToQ :' '
ReplyToQMgr :'JMSDEMO '
----------------------------------------------------------------------
| |
| |
| DESCRIPTIVE NAME WebSphere MQ Q Program |
| |
------- 8><-------------------------------------------------------------
REMAINDER OF MSG OUTPUT OMITTED FOR BREVITY. PRINT-OUT RESUMES...
------- 8><-------------------------------------------------------------
No more messages.
D:\WMQ\MA01>
Note the header lines where the message was printed. The -dl option tells Q to print the message length which, in this case, was 24309 bytes. I downloaded the current version to perform this test so this is accurate as of 7 December 2013.
If you are looking for loading the file to queue.. Its easy to work with RFHUtil s/w or application.
In RFHUtil you can easily load the file to MQ and clear the Queue, purge ect...
Many more options are provided .
Related
I am trying to use the cmo.deleteMessages to clean up messages that are older than 30 days.
connect(...)
domainRuntime()
print 'Cleaning Message from QUEUE:myqueue'
try:
cd('ServerRuntimes/myserver/JMSRuntime/myserver.jms/JMSServers/myserver/Destinations/JMSMODULE!JMSmyserver#myqueue')
cmo.deleteMessages("JMSTimestamp > 5200000000")
except:
pass
However Weblogic doesnt recognize the attribute "JMSTimestamp > 5200000000". It deletes all the messages.
When I put the entry "JMSTimestamp > 5200000000" in the Message Selector [in wl console], it shows up all messages instead of messages that are only 30 days old [5200000000 milliseconds is 30 days].
The problem is the format "JMSTimestamp > 5200000000" is either not recognized by Weblogic or the python script. Any idea what I am missing.
I was able to create the timestamp in milliseconds using a modified date command tool in Linux.
$ date +%s%N | cut -b1-13
1617374452236
JMS time stamp parameter accepted this format and was able to perform the task.
I was executing two mqsc commands on IBM MQ v8 to find out the number of connections made by App1 on queue manager QMGR and found a discrepancy between the output values on one of the attribute called CONNAME
echo 'dis chs('APP1.SVRCONN.CHL')'| runmqsc QMGR | grep CONNAME
This time I got 14 IP's as CONNAME
echo 'dis qs(APP1.QUEUE) type(handle) All where(CHANNEL eq 'APP1.SVRCONN.CHL')'| runmqsc QMGR | grep CONNAME
When I checked the number of handles on the only App1 queue connected by App1 channel I got only 7 IP's as CONNAME
So there are 7 handles on the App1 queue but there are 14 connections by the App1 channel on queue manager. How can I relate these two values? Is this multi-threaded connections on each queue handles and if so how can I find a relationship between them?
The DIS CHS command will show you how many channel instances are running which is really how many TCP sessions there are established between your client and the queue manager.
MQ v7.0.1 and higher supports shared conversations within each channel. Each time the app makes a connection to the queue manager this can be another conversation in the same channel instance. The client and the server can set a maximum number of shared connections. On the queue manager's SVRCONN the attribute is SHARECNV, on the client it could be in a CCDT or programatically. When the channel starts up the client and server will negotiate to the lowest value between them. On a DIS CHS you can add MAXSHCNV to see what the running channel instances negotiated the maximum to and CURSHCNV to see how many conversations are running in each channel instance.
The DIS QS command with TYPE(HANDLE) will only show you connections where the application has a handle open to that queue at the point in time you ran the command.
An application can connect to the queue manager without having a queue open, or it may open and close the queue, if you don't catch it with the queue open you won't see a handle.
A more accurate command to show what the channel is doing is DIS CONN which will show all connections, each connection represents a conversation, so the number of connections should match the number total of CURSHCNV from all channel instances.
The following command will show you all connections running through a channel with the name specified.
echo "DIS CONN(*) TYPE(ALL) WHERE(CHANNEL EQ APP1.SVRCONN.CHL)"|runmqsc QMGR
The output may look like this:
AMQ8276: Display Connection details.
CONN(ABABABABABABABAB)
EXTCONN(ABABABABABABABABABABABABABABABAB)
TYPE(*)
PID(99998) TID(998)
APPLDESC(WebSphere MQ Channel) APPLTAG(App1)
APPLTYPE(USER) ASTATE(NONE)
CHANNEL(APP1.SVRCONN.CHL) CLIENTID( )
CONNAME(10.10.10.10)
CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
USERID(appuser) UOWLOG( )
UOWSTDA( ) UOWSTTI( )
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR)
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.0) UOWSTATE(NONE)
AMQ8276: Display Connection details.
CONN(BABABABABABABABA)
EXTCONN(BABABABABABABABABABABABABABABABA)
TYPE(*)
PID(99999) TID(999)
APPLDESC(WebSphere MQ Channel) APPLTAG(App1)
APPLTYPE(USER) ASTATE(STARTED)
CHANNEL(APP1.SVRCONN.CHL) CLIENTID( )
CONNAME(10.10.10.10)
CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
USERID(appuser) UOWLOG( )
UOWSTDA(2018-05-21) UOWSTTI(10.11.27)
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR)
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.99999) UOWSTATE(ACTIVE)
OBJNAME(APP1.QUEUE) OBJTYPE(QUEUE)
ASTATE(ACTIVE) HSTATE(INACTIVE)
OPENOPTS(MQOO_INPUT_SHARED,MQOO_BROWSE,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
In the above output CONN(ABABABABABABABAB) only has a connection to the queue manager and no objects open while CONN(BABABABABABABABA) had a connection to the queue manager and has the queue APP1.QUEUE open.
If you were to add to the above command |grep 'APP1.QUEUE' the count should match the number of handles from the DIS QS command.
You may have a app that browses the queue on one connection to look for messages and then dispatches to other threads to actually process the messages. In your case 7 channel instances may have the queue open for BROWSE and the other 7 would be waiting to open the queue for INPUT to read a message off.
Below is a command I use on Linux that will spit out a CSV format of the DIS CONN command along with each object that is open.
echo "DIS CONN(*) TYPE(ALL) WHERE(CHANNEL EQ CHL_NAME)"|runmqsc QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["APPLTAG"], p["USERID"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'
The fields in the CSV output are:
"CONN","CHANNEL","APPLTAG","USERID","CONNAME","OBJNAME","OBJTYPE","OPENOPTS"
Objects can be the QMGR itself, a QUEUE, TOPIC, etc. The CONN id will be repeated in lines of the output if the same CONN has multiple objects open. If the CONN has no objects open you will see a single entry with no objects listed at the end of the line, if at least one object is open there will not be a line representing no objects:
I'm sending binary .gz files from Linux to z/OS via ftps. The file transfers seem to be fine, but when the mainframe folks pkunzip the file, they get a warning:
PEX013W Record(s) being truncated to lrecl= 996. Record# 1 is 1000 bytes.
Currently I’m sending the site commands:
SITE TRAIL
200 SITE command was accepted
SITE CYLINDERS PRIMARY=50 SECONDARY=50
200 SITE command was accepted
SITE RECFM=VB LRECL=1000 BLKSIZE=32000
200 SITE command was accepted
SITE CONDDISP=delete
200 SITE command was accepted
TYPE I
200 Representation type is Image
...
250 Transfer completed successfully.
QUIT
221 Quit command received. Goodbye.
They could read the file after the pkunzip, but having a warning is not a good thing.
Output from pkunzip:
SDSF OUTPUT DISPLAY RMD0063A JOB22093 DSID 103 LINE 25 COLUMNS 02- 81
COMMAND INPUT ===> SCROLL ===> CSR
PCM123I Authorized services are unavailable.
PAM030I INPUT Archive opened: TEST.FTP.SOA5021.GZ
PAM560I ARCHIVE FASTSEEK processing is disabled.
PDA000I DDNAME=SYS00001,DISP_STATUS=MOD,DISP_NORMAL=CATALOG,DISP_ABNORMAL=
PDA000I SPACE_TYPE=TRK,SPACE_TYPE=CYL,SPACE_TYPE=BLK
PDA000I SPACE_PRIMARY=4194304,SPACE_DIRBLKS=5767182,INFO_ALCFMT=00
PDA000I VOLUMES=DPPT71,INFO_CNTL=,INFO_STORCLASS=,INFO_MGMTCLASS=
PDA000I INFO_DATACLASS=,INFO_VSAMRECORG=00,INFO_VSAMKEYOFF=0
PDA000I INFO_COPYDD=,INFO_COPYMDL=,INFO_AVGRECU=00,INFO_DSTYPE=00
PEX013W Record(s) being truncated to lrecl= 996. Record# 1 is 1000 bytes.
PEX002I TEST.FTP.SOA5021
PEX003I Extracted to TEST.FTP.SOA5021I.TXT
PAM140I FILES: EXTRACTED EXCLUDED BYPASSED IN ERROR
PAM140I 1 0 0 0
PMT002I PKUNZIP processing complete. RC=00000004 4(Dec) Start: 12:59:48.86 End
Is there a better set of site commands to transfer a .gz file from Linux to z/OS to avoid this error?
**** Update ****
Using SaggingRufus's answer below, it turns out it doesn't much matter how you send the .gz file, as long as it's binary. His suggestion pointed us to the parameters sent to the pkunzip for the output file, which was VB and was truncating 4 bytes off the record.
Because it is a variable block file, there are 4 bytes allocated to the record attributes. Allocate the file with an LRECL of 1004 and it will be fine.
Rather than generating a .zip file, perhaps generate a .tar.gz file and transfer it to z/OS UNIX? Tar is shipped with z/OS by default, and Rocket Software provides a port of gzip that is optimized for z/OS.
As part of the installation of our application, I need to put a bunch of xml messages into an MQ queue. To make it more complicated, the messages need to have the usr-folder of the RFH2-header set.
I found that mqput2.exe from the IBM RFHUtil is an ideal tool for this task. Unfortunately, I have been unable to make setting of the usr-folder work, the documentation is too vague. What am I doing wrong? Or, is there another command-line tool available that works better?
My prop. file for mqput2 looks like this:
[header]
.....
*
* message type
*
* Allowed values for message type
* 1 - request
* 2 - reply
* 4 - report
* 8 - datagram
* 112 - MQE fields from MQE
* 113 - MQE fields
* 65536 to 999,999,999 - user
msgtype=8
.....
*
* rfh usage
* rfh = N for No rfh
* rfh = A for Automatic (look for RFH at beginning of data file)
* rfh = 1 or V1 for Version 1 rfh from parameters in parm file
* rfh = 2 or V2 for Version 2 rfh from parameters in parm file
* rfh = X for special V2 rfh with fixed portion only and format=xml
*
* only first character checked, except for V when second character is also checked
*
rfh=X
<usr>
<NotificationType>RDS.Codebook.Changes</NotificationType>
</usr>
.....
[filelist]
file1.xml
mqput2 simply ignores the <usr> section and doesn't include it with the message. I've also tried rfh=A and put the <usr> section at the beginning of the file1.xml file, but then I get MQPUT error reason=2142 (invalid MQ header structure).
Also dmpmqmsg utility, which is part of the WMQ8 installation, can be used for sending the messages from command-line. A message can be exported from a queue including the headers via
dmpmqmsg -m queue_manager> -I <queue> -f <msg_file> -a
and then submitted again via (I think, consult the docs)
dmpmqmsg -m queue_manager> -o <queue> -f <msg_file>
This utility is however not part of WMQ 7.5.
I have it set up this way, and it works fine:
[header]
qname=QName
qmgr=QMName
msgcount=1
format= "MQSTR "
codepage=1208
rfh=2
<usr>
<test>test</test>
</usr>
[filelist]
test.xml
I am trying to transfer a 60mb file to queue, but Websphere MQ fte stall the transfer and keep recovering. I am running WebSphere MQ FTE on default configuration.
I have tested following scenario with different results according to configuration changes I made.
These commands were issued to create monitor:
fteCreateTransfer -sa AGENT1 -sm TQM.FTE -da AGENT2 -dm QM.FTE -dq FTE.TEST.Q -p QM.FTE -de overwrite -sd delete -gt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml D:\\rvs\\tstusrdat\\ALZtoSIP\\INC\\*.zip
fteCreateMonitor -ma AGENT1 -mn TEST_MONITOR -md D:\\rvs\\tstusrdat\\ALZtoSIP\\INC -mt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml -tr match,*.zip
Test was performed on files: 53MB and 30MB
Default configuration (just enableQueueInputOutput=true added to AGENT2.properties)
1) all default
no success, transfer status: "recovering"
for both files
2) added maxInputOutputMessageLength=60000000, destination queue max message length changed to 103809024
result transfer status: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
for both files
After reading this: http://pic.dhe.ibm.com/infocenter/wmqfte/v7r0/topic/com.ibm.wmqfte.doc/message_length.htm I came with working settings:
3) maxInputOutputMessageLength=34603008 (its maximum value), destination queue max message length still to 103809024
result for file with size 30MB: succcess
result for file with size 53MB: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
So according to this I am afraid one can't transfer larger then 34603008 bytes.
If you are transferring file to queue you definitely can't use the default settings. You have to add "enableQueueInputOutput=true" to agent.properties for agent thet uses queue as source or destination.