Discrepancy between total CONNAME values on DIS QSTATUS Vs DIS CHSTATUS - ibm-mq

I was executing two mqsc commands on IBM MQ v8 to find out the number of connections made by App1 on queue manager QMGR and found a discrepancy between the output values on one of the attribute called CONNAME
echo 'dis chs('APP1.SVRCONN.CHL')'| runmqsc QMGR | grep CONNAME
This time I got 14 IP's as CONNAME
echo 'dis qs(APP1.QUEUE) type(handle) All where(CHANNEL eq 'APP1.SVRCONN.CHL')'| runmqsc QMGR | grep CONNAME
When I checked the number of handles on the only App1 queue connected by App1 channel I got only 7 IP's as CONNAME
So there are 7 handles on the App1 queue but there are 14 connections by the App1 channel on queue manager. How can I relate these two values? Is this multi-threaded connections on each queue handles and if so how can I find a relationship between them?

The DIS CHS command will show you how many channel instances are running which is really how many TCP sessions there are established between your client and the queue manager.
MQ v7.0.1 and higher supports shared conversations within each channel. Each time the app makes a connection to the queue manager this can be another conversation in the same channel instance. The client and the server can set a maximum number of shared connections. On the queue manager's SVRCONN the attribute is SHARECNV, on the client it could be in a CCDT or programatically. When the channel starts up the client and server will negotiate to the lowest value between them. On a DIS CHS you can add MAXSHCNV to see what the running channel instances negotiated the maximum to and CURSHCNV to see how many conversations are running in each channel instance.
The DIS QS command with TYPE(HANDLE) will only show you connections where the application has a handle open to that queue at the point in time you ran the command.
An application can connect to the queue manager without having a queue open, or it may open and close the queue, if you don't catch it with the queue open you won't see a handle.
A more accurate command to show what the channel is doing is DIS CONN which will show all connections, each connection represents a conversation, so the number of connections should match the number total of CURSHCNV from all channel instances.
The following command will show you all connections running through a channel with the name specified.
echo "DIS CONN(*) TYPE(ALL) WHERE(CHANNEL EQ APP1.SVRCONN.CHL)"|runmqsc QMGR
The output may look like this:
AMQ8276: Display Connection details.
CONN(ABABABABABABABAB)
EXTCONN(ABABABABABABABABABABABABABABABAB)
TYPE(*)
PID(99998) TID(998)
APPLDESC(WebSphere MQ Channel) APPLTAG(App1)
APPLTYPE(USER) ASTATE(NONE)
CHANNEL(APP1.SVRCONN.CHL) CLIENTID( )
CONNAME(10.10.10.10)
CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
USERID(appuser) UOWLOG( )
UOWSTDA( ) UOWSTTI( )
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR)
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.0) UOWSTATE(NONE)
AMQ8276: Display Connection details.
CONN(BABABABABABABABA)
EXTCONN(BABABABABABABABABABABABABABABABA)
TYPE(*)
PID(99999) TID(999)
APPLDESC(WebSphere MQ Channel) APPLTAG(App1)
APPLTYPE(USER) ASTATE(STARTED)
CHANNEL(APP1.SVRCONN.CHL) CLIENTID( )
CONNAME(10.10.10.10)
CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
USERID(appuser) UOWLOG( )
UOWSTDA(2018-05-21) UOWSTTI(10.11.27)
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR)
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.99999) UOWSTATE(ACTIVE)
OBJNAME(APP1.QUEUE) OBJTYPE(QUEUE)
ASTATE(ACTIVE) HSTATE(INACTIVE)
OPENOPTS(MQOO_INPUT_SHARED,MQOO_BROWSE,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
In the above output CONN(ABABABABABABABAB) only has a connection to the queue manager and no objects open while CONN(BABABABABABABABA) had a connection to the queue manager and has the queue APP1.QUEUE open.
If you were to add to the above command |grep 'APP1.QUEUE' the count should match the number of handles from the DIS QS command.
You may have a app that browses the queue on one connection to look for messages and then dispatches to other threads to actually process the messages. In your case 7 channel instances may have the queue open for BROWSE and the other 7 would be waiting to open the queue for INPUT to read a message off.
Below is a command I use on Linux that will spit out a CSV format of the DIS CONN command along with each object that is open.
echo "DIS CONN(*) TYPE(ALL) WHERE(CHANNEL EQ CHL_NAME)"|runmqsc QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["APPLTAG"], p["USERID"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'
The fields in the CSV output are:
"CONN","CHANNEL","APPLTAG","USERID","CONNAME","OBJNAME","OBJTYPE","OPENOPTS"
Objects can be the QMGR itself, a QUEUE, TOPIC, etc. The CONN id will be repeated in lines of the output if the same CONN has multiple objects open. If the CONN has no objects open you will see a single entry with no objects listed at the end of the line, if at least one object is open there will not be a line representing no objects:

Related

What circumstances will messages not be consumed or delay consumed in rocketmq cluster?

I have more than five hundred topic in one cluster with six nodes, I create the topic like this :
./mqadmin updateTopic -c MyCluster -n 192.168.1.100:9876 -r 16 -w 16 -t topic1;
./mqadmin updateTopic -c MyCluster -n 192.168.1.100:9876 -r 16 -w 16 -t topic500;
I can send the message without delay ,but I find sometime that the client can not consume the message because the broker do not have consume client.How to fix this bug?
do following check
more than 1 consumer instance started in one machine.
if so , make every consumer instance client id or client name different
consumer instances in this c_fy_core consumer group sub different topic and tags.
if so , make sure all consumer instances in one consumer group sub same topic-and-tags

Asyncio RuntimeError: readuntil() called while another coroutine is already waiting for incoming data

Using python3.6.8:
I'm attempting to script the initial configuration of network devices on boot. My script opens telnet connection on "ip_addr:port". Once connected, script stimulates the network device it's connecting to with "\n\n" (simulating two 'Enter' input from an admin).
connection = asyncio.open_connection(
ip_addr,
port,
)
try:
reader, writer = await asyncio.wait_for(connection, 5)
print(f"successfully connected to {ip_addr}:{port}")
writer.write(b'\n\n\n')
Some devices are already configured and I except " login: " to show up in the read buffer upon entering '\n'. However if the device is not configured yet, " login: " will not show up in the buffer. Therefore I thought I could use wait_for and timeout option to have this cancelled and move on with another reader.readuntil(...) expecting another output.
try:
await asyncio.wait_for(reader.readuntil(b' login: '), 3)
print(f"{ip_addr}:{port} alredy booted")
break
except (asyncio.TimeoutError, OSError):
print('Nope, moving forward')
await reader.readuntil(b'normal setup ?(yes/no)[n]: ')
However this raises a RuntimeError. Reading the documentation I excepted the task to be cancelled if the timeout is reached, so why can't it have another coroutine readuntil() ?

Efficient way of sending the same data to multiple dynamic processes

I have a stream of line-buffered data, and many readers from other processes
The readers need to attach to the system dynamically, they are not known to the process writing the stream
First i tried to read every line and simply send them to a lot of pipes
#writer
command | while read -r line; do
printf '%s\n' "$line" | tee listeners/*
done
#reader
mkfifo listeners/1
cat listeners/1
But that's consume a lot of CPU
So i though about writing to a file and cleaning it repeatedly
#writer
command >> file &
while true; do
: > file
sleep 1
done
#reader
tail -f -n0 file
But sometimes, a line is not read by one or more readers before truncation, making a race condition
Is there a better way on how i could implement this?
Sounds like pub/sub to me - see Wikipedia.
Basically, new interested parties come along whenever they like and "subscribe" to your channel. The process receiving the data then "publishes" it, line by line, to that channel.
You can do it with MQTT using mosquitto or with Redis. Both have command-line interfaces/bindings, as well as Python, C/C++, Ruby, PHP etc. Client and server need not be on same machine, some clients could be elsewhere on the network.
Mosquitto example here.
I did a few tests on my Mac with Redis pub/sub. The client code in Terminal to subscribe to a channel called myStream looks like this:
redis-cli SUBSCRIBE myStream
I then ran a process to synthesise 10,000 lines like this:
time seq 10000 | while read a ; do redis-cli PUBLISH myStream "$a" >/dev/null 2>&1 ; done
And that takes 40s, so it does around 250 lines per second, but it has to start a whole new process for each line and create and tear down the connection to Redis... and we don't want to send your CPU mad.
More appropriately for your situation then, here is how you can create a file with 100,000 lines, and read them one at a time, and send them to all your subscribers in Python:
# Make a "BigFile" with 100,000 lines
seq 100000 > BigFile
and read the lines and publish them with:
#!/usr/bin/env python3
import redis
if __name__ == '__main__':
# Redis connection
r = redis.Redis(host='localhost', port=6379, db=0)
# Read file line by line...
with open('BigFile', 'r') as infile:
for line in infile:
# Publish the current line to subscribers
r.publish('myStream', line)
The entire 100,000 lines were sent and received in 4s, so 25,000 lines per second. Here is a little recording of it in action. At the top you can see the CPU is not unduly troubled by it. The second window from the top is a client, receiving 100,000 lines and the next window down is a second client. The bottom window shows the server running the Python code above and sending all 100,000 lines in 4s.
Keywords: Redis, mosquitto, pub/sub, publish, subscribe.

Orphaned SYSTEM.MANAGED.DURABLE.* queue in Websphere MQ

I have a queue 'SYSTEM.MANAGED.DURABLE.ABCD***109' getting messages all the time and no one to consume it.
I tried to get its subscription but got the following result ,
dis sub(*) where (DEST LK 'SYSTEM.MANAGED.DURABLE.ABCD***109')
AMQ8096: IBM MQ subscription inquired.
SUBID(414D5120******************44A0109)
SUB(false)
DEST(SYSTEM.MANAGED.DURABLE.ABCD***44A0108)
then i tried to view the subscription via the subscription id listed,
dis sbstatus(*) where ( SUBID EQ '414D5120***44A0109')
AMQ8099: IBM MQ subscription status inquired.
SUB(false)
SUBID(414D5120***44A0109)
I don't have a subscription named "false" . I'm unable to clear or delete this queue as it is opened. I'm unable to view the open connection as well.
dis conn(*) where (objname eq 'SYSTEM.MANAGED.DURABLE.ABCD***44A0108')
AMQ8461: Connection identifier not found.
I need to cleanup & delete this queue to avoid disk space issue.
You can remove SUB objects with only the SUBID, try to remove it with this command:
DELETE SUB SUBID('414D5120***44A0109')
Note that the command is not specifying the SUB name, just the SUB keyword.
Before you delete it, if you are interested in seeing what the sub name actually is, you may want to try running the following command to dump the subscriptions:
amqldmpa -m <QueueManager> -c T -f /var/mqm/errors/amqldmpa_topic.out
Inside of the file /var/mqm/errors/amqldmpa_topic.out search for the SUBID in question and look for text similar to this:
Subscriber entry
{
SubId ( 414D5120***44A0109)
SubNameString ( SUBNAME_HERE )
TopicString ( TOPIC/STRING/HERE )
<more lines of information go here>
}
What does it show for the SubNameString field? Note that in the 8.0.0.6 version I ran this against it seems to pad each field with a leading and trailing space with the exception of SubId which did not have a trailing space.

Utility to load files to a MQ Queue

I want to load files in the file system to a WebSphere MQ Queue. There are couple of support pacs - Q Program and MO03: WebSphere MQ Queue Load / Unload Utility
that come close but they mandate the files to be in a specific format. I have the messages which are XML files and want a quick way to load them to a queue. The number of files run into a few hundred so looking for an utility to do this job instead of having to write an application to achieve this.
I could not locate some general purpose application to achieve this. So looking for some help here
Thanks
Why do you believe that the Q program requires a specific file format? According to the README.TXT file, the following options are available:
-f<filename>
Input file.
Each line of the file will be put to output queue as a different
message.
See "Z/OS FILE NAME FORMAT EXAMPLES" for specific z/OS details.
-F[+]<filename>
Input/output file.
Entire file will be put to the output queue as a single message.
If '+' is specified the dataset attributes will be retained if
the output dataset exists - z/OS only.
See "Z/OS FILE NAME FORMAT EXAMPLES" for specific z/OS details.
So if you specify -F (without the +) all lines in the the XML file are loaded into a single message. You can also specify the message options using the -a parameter:
-a<Opts> Sets message attributes when put to the output queue
n - forces non-persistence
p - forces persistence
q - uses queue default persistence
d - put a datagram message type
r - put a reply message type
R - put a request message type
t - put a report message type
x - don't treat lines starting with '#' as special
Although the Q program will interpret files by default, note that the -ax option above tells it to ignore lines with # which it would ordinarily interpret as commands. This allows you to load XML files or source code with comments or even binary files such as a PDF or JPG.
Was there a specific limitation in Q that you are unable to work with? If so, it would be helpful to know what that is so we might point you to something that would better fit your purpose.
UPDATE
Responding to Spyro's comments, Q is not limited to 1000 characters. Here's an example where the README file from the Q distribution is written to a single message and read back.
D:\WMQ\MA01>q -m JMSDEMO -OSYSTEM.DEFAULT.LOCAL.QUEUE -FREADME
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 1 2012 ]
Connecting ...connected to 'JMSDEMO'.
D:\WMQ\MA01>echo dis q(SYSTEM.DEFAULT.LOCAL.QUEUE) curdepth | runmqsc JMSDEMO
5724-H72 (C) Copyright IBM Corp. 1994, 2011. ALL RIGHTS RESERVED.
Starting MQSC for queue manager JMSDEMO.
1 : dis q(SYSTEM.DEFAULT.LOCAL.QUEUE) curdepth
AMQ8409: Display Queue details.
QUEUE(SYSTEM.DEFAULT.LOCAL.QUEUE) TYPE(QLOCAL)
CURDEPTH(1)
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
D:\WMQ\MA01>q -m JMSDEMO -dl -iSYSTEM.DEFAULT.LOCAL.QUEUE
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 1 2012 ]
Connecting ...connected to 'JMSDEMO'.
MQGET 24309 bytes
============================================================================
Message Descriptor (MQMD)
Report :00000000
Message Type :8 (Datagram)
Format :'MQSTR '
Priority :0
Persistence :0 (Not Persistent)
Message Id :A M Q J M S D E M O . . . R . * .
414D51204A4D5344454D4F20202020201DDEA052200B2A02
'AMQ JMSDEMO ...R .*.'
ReplyToQ :' '
ReplyToQMgr :'JMSDEMO '
----------------------------------------------------------------------
| |
| |
| DESCRIPTIVE NAME WebSphere MQ Q Program |
| |
------- 8><-------------------------------------------------------------
REMAINDER OF MSG OUTPUT OMITTED FOR BREVITY. PRINT-OUT RESUMES...
------- 8><-------------------------------------------------------------
No more messages.
D:\WMQ\MA01>
Note the header lines where the message was printed. The -dl option tells Q to print the message length which, in this case, was 24309 bytes. I downloaded the current version to perform this test so this is accurate as of 7 December 2013.
If you are looking for loading the file to queue.. Its easy to work with RFHUtil s/w or application.
In RFHUtil you can easily load the file to MQ and clear the Queue, purge ect...
Many more options are provided .

Resources