I have jmeter , where a single thread contains two mqtt gateway connection sampler & each sampler have three publishers connected to iothub.
Jmeter reference:
When I run the thread in loop 6frames / second for 10 seconds, I could see all 60 frames published successful in JMeter.
But when I check data count at iothub, first gate way point have received only 6 frames ( some data get missed it seems, problem with jmeter I assume ) & second gateway have received 42 frames. Second part led to major confusion, when it have to receive maximum of 30 frames, but received 42.
Diagram reference:
Each gateway (A &B) include the Connection panel with :
Iothub URL
Mqtt v 3.1.1
Username: iothuburl/device ID
Pwd: SAS token ( generated SAS from connection string available at iothubowner page from azure portal).
Each Gateway (A&B) include
three publishers & Each includes 200 JSON objects and size doesn't exceed 55kb.
Publisher QoS: 0
Operation:
For every one second, each gateway publish 3 frames ( total 600 JSON objects).
As I have mentioned 2 gateway, so total 6 frames with 1200 JSON object get published successful in JMeter.
But the data is missing at iothub.
note: while running two gateway in single thread, i could both gateway connection sharing the common connection string ID.
Any clue, where did I miss the major configuration, any help would be greatly appreciated. Thanks.
Change the QoS=1 in publisher panel. Though we have few latency time to wait for acknowledgement, but the simulation works fine without any loss of connection/data.
Related
We have been facing an issue where message rate of a xmitq is very slow comparing with what should be a normal performance.
We have many other Qmgrs with bigger MQ flows that don't have the same issue.
Our HUB qmgr connects to business line in the same company HUB qmgr, and even the destination queues on their side being empty the flow is really slow.
At OS and Network level they say nothing can be done. At MQ we have changed the Buffersizes so it matches the OS level and uses the system tcp windows.
Now at MQ level we have the channel SDR setup with BATCHSZ to 100, but seems the receiver is configured with 30.
We noticed that because we see messages flow in batches fof 30 messages. Also not sure if that is related but we see the XMITQ havs always 30 uncommited messages.
Our questiong for advice.
Would increase the BATCHSZ parameter on SDR/RCVR help the perfomance?
Is there any other parameter at MQ level that could help it?
DIS CHS(NAME) ALL
AMQ8417: Display Channel Status details.
CHANNEL(QMGRA.QMGRB.T7) CHLTYPE(SDR)
BATCHES(234) BATCHSZ(30)
BUFSRCVD(235) BUFSSENT(6391)
BYTSRCVD(6996) BYTSSENT(14396692)
CHSTADA(2020-04-16) CHSTATI(14.38.17)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(159.50.69.38(48702)) CURLUWID(398F3E5EEA43381C)
CURMSGS(30) CURRENT
CURSEQNO(43488865) EXITTIME(0,0)
HBINT(300) INDOUBT(YES)
JOBNAME(000051FC00000001) LOCLADDR(10.185.8.122(54908))
LONGRTS(999999999) LSTLUWID(398F3E5EE943381C)
LSTMSGDA(2020-04-16) LSTMSGTI(14.49.46)
LSTSEQNO(43488835) MCASTAT(RUNNING)
MONCHL(HIGH) MSGS(6386)
NETTIME(2789746,3087573) NPMSPEED(NORMAL)
RQMNAME(QMGRB) SHORTRTS(10)
SSLCERTI(*******************)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(*******************)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
XBATCHSZ(23,7) XMITQ(QMGRB.X7)
XQTIME(215757414,214033427) RVERSION(08000008)
RPRODUCT(MQMM)
qm.ini:
Log:
LogPrimaryFiles=10
LogSecondaryFiles=10
LogFilePages=16384
LogType=LINEAR
LogBufferPages=4096
LogPath=/apps/wmq/QMGR/log/QMGR/
LogWriteIntegrity=SingleWrite
Service:
Name=AuthorizationService
EntryPoints=13
TCP:
SvrSndBuffSize=0
SvrRcvBuffSize=0
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/opt/mqm75/lib64/amqzfu
ComponentDataSize=0
Channels:
MaxChannels=500
UPDATED: 15:41 GMT
Just to update the information, both sides are now with BATCHSZ 100 and seems slightly.
AMQ8417: Display Channel Status details.
CHANNEL(QMGRA.QMGRB.T7) CHLTYPE(SDR)
BATCHES(403) BATCHSZ(100)
BUFSRCVD(405) BUFSSENT(23525)
BYTSRCVD(11756) BYTSSENT(53751066)
CHSTADA(2020-04-17) CHSTATI(15.13.51)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(159.50.69.38(48702)) CURLUWID(6D66985E94343410)
CURMSGS(0) CURRENT
CURSEQNO(44115897) EXITTIME(0,0)
HBINT(300) INDOUBT(NO)
JOBNAME(0000172A00000001) LOCLADDR(10.185.8.122(2223))
LONGRTS(999999999) LSTLUWID(6D66985E93343410)
LSTMSGDA(2020-04-17) LSTMSGTI(15.30.06)
LSTSEQNO(44115897) MCASTAT(RUNNING)
MONCHL(HIGH) MSGS(23505)
NETTIME(101563,480206) NPMSPEED(NORMAL)
RQMNAME(QMGRB) SHORTRTS(10)
SSLCERTI(*************************************)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(****************************)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(MQGET)
XBATCHSZ(1,1) XMITQ(QMGRB.X7)
XQTIME(191225,794134) RVERSION(08000008)
RPRODUCT(MQMM)
AMQ8450: Display queue status details.
QUEUE(QMGRB.X7) TYPE(QUEUE)
CURDEPTH(0) IPPROCS(1)
LGETDATE(2020-04-17) LGETTIME(15.30.06)
LPUTDATE(2020-04-17) LPUTTIME(15.30.06)
MEDIALOG(S2488154.LOG) MONQ(LOW)
MSGAGE(0) OPPROCS(9)
QTIME(794134, 191225) UNCOM(NO)
I'll put a few observations in this answer, but based on any further feedback I may add more.
You are running a very old version of the software on the sender side, MQ 7.5 went out of support almost two years ago (April 30 2018). IBM for a cost will provide extended support for an additional three years, so maybe you fall in that group. The 7.5.0.2 maintenance release itself came out in July 11th 2013, so it is almost seven years old at this point. I would strongly suggest you move to a newer version.
Note that MQ v8.0 goes out of support April 30 2020, and IBM just announced a few days ago that MQ v9.0 goes out of support September 30 2021. When you do migrate you should go with either 9.1 which has no announced end of support (they give five years minimum so it could be 2023) or go with the next version of MQ that should be out some time this year.
You mention setting the following:
TCP:
SvrSndBuffSize=0
SvrRcvBuffSize=0
The above setting apply to the SVRCONN end of a client connection. You can see this in the MQ v7.5 Knowledge Center page WebSphere MQ>Configuring>Changing configuration information>Changing queue manager configuration information>TCP, LU62, NETBIOS, and SPX:
SvrSndBuffSize=32768|number
The size in bytes of the TCP/IP send buffer used by the server end of a client-connection
server-connection channel.
SvrRcvBuffSize=32768|number
The size in bytes of the TCP/IP receive buffer used by the server end of a client-connection
server-connection channel.
At IBM MQ v7.5.0.2 APAR IV58073 introduced the concept of setting various buffer settings to a value to 0 which means that it will allow the operating system defaults to be used. Unfortunately like many things in the Knowledge Center it does not look like IBM documented this correctly for 7.5.
You can however review the IBM MQ v8.0 Knowledge Center to get the full picture regarding these settings at the page Configuring>Changing configuration information>Changing queue manager configuration information>TCP, LU62, and NETBIOS, specifically you would want to set these two settings to have any impact on your Sender Channel:
SndBuffSize=number| 0
The size in bytes of the TCP/IP send buffer used by the sending end of
channels. This stanza value can be overridden by a stanza more
specific to the channel type, for example RcvSndBuffSize. If the
value is set as zero, the operating system defaults are used. If no
value is set, then the IBM MQ default, 32768, is used.
RcvSndBuffSize=number| 0
The size in bytes of the TCP/IP send buffer used by the sender end of
a receiver channel. If the value is set as zero, the operating system
defaults are used. If no value is set, then the IBM MQ default, 32768,
is used.
Starting at IBM MQ v8.0 any newly created queue manager will have all of the following in the qm.ini:
TCP:
SndBuffSize=0
RcvBuffSize=0
RcvSndBuffSize=0
RcvRcvBuffSize=0
ClntSndBuffSize=0
ClntRcvBuffSize=0
SvrSndBuffSize=0
SvrRcvBuffSize=0
However, any queue manager that is upgraded will not by default get those settings, meaning if those are not present they will not be added, if they are present they will remain the same. If the setting is not present then as it says above "the IBM MQ default, 32768, is used."
I had extensive discussions with IBM support on this topic and came to the conclusion that they did not see any reason to not set it to 0, they only saw benefit in doing so, but with an abundance of caution they do not change it to 0 for you.
I would recommend you add all of those to your qm.ini, but at minimum add the two I highlighted above.
These are good setting to implement but may not solve your problem if nothing changed recently on either end. If however something did change, for example a network difference, or MQ was upgraded to 8.0.0.8 on the remote side, then this setting just might solve your problem.
In the channel status output two values are interesting:
NETTIME(2789746,3087573)
XQTIME(215757414,214033427)
NETTIME means that based on recent activity it took 2.7 seconds to receive a response from the RCVR channel, over a longer period of time it took 3.1 seconds to receive a response from the RCVR channel. Can you compare this to a TCP ping from the sender channel server to the receive channel server, 2.7 seconds for a response over the network seems excessive. In the presentation Keeping MQ Channels Up and Running given at Capitalware's MQ Technical Conference v2.0.1.4, Paul Clarke who used to work for IBM states "NETTIME only measures network time, and does not include
the MQCMIT for example".
XQTIME means that based on recent activity and over a longer period of time it took ~215 seconds for a message on the XMITQ to be picked up by the SDR channel to be sent.
See below for how IBM documents these:
NETTIME
Amount of time, displayed in microseconds, to send a request to the remote end of the channel and receive a response. This time only measures the network time for such an operation. Two values are displayed:
A value based on recent activity over a short period.
A value based on activity over a longer period.
XQTIME
The time, in microseconds, that messages remained on the transmission queue before being retrieved. The time is measured from when the message is put onto the transmission queue until it is retrieved to be sent on the channel and, therefore, includes any interval caused by a delay in the putting application.
Two values are displayed:
A value based on recent activity over a short period.
A value based on activity over a longer period.
Information on the BATCHSZ channel parameter can be found in the IBM MQ v8.0 Knowledge Center page Reference>Configuration reference>Channel attributes>Channel attributes in alphabetical order>Batch size (BATCHSZ). I have quoted it and highlighted a few areas in bold.
This attribute is the maximum number of messages to be sent before a sync point is taken.
The batch size does not affect the way the channel transfers messages; messages are always transferred individually, but are committed or backed out as a batch.
To improve performance, you can set a batch size to define the maximum number of messages to be transferred between two sync points. The batch size to be used is negotiated when a channel starts, and the lower of the two channel definitions is taken. On some implementations, the batch size is calculated from the lowest of the two channel definitions and the two queue manager MAXUMSGS values. The actual size of a batch can be less; for example, a batch completes when there are no messages left on the transmission queue or the batch interval expires.
A large value for the batch size increases throughput, but recovery times are increased because there are more messages to back out and send again. The default BATCHSZ is 50, and you are advised to try that value first. You might choose a lower value for BATCHSZ if your communications are unreliable, making the need to recover more likely.
This attribute is valid for channel types of:
Sender
Server
Receiver
Requester
Cluster sender
Cluster receiver
Follow up questions:
Are the messages that are PUT to this XMITQ persistent?
Answer: Yes, in our PROD env all messages are pesistent.
Have you had a recent increase in volume going to this XMITQ?
Answer: No, we use a monitoring tools, we extracted a report that show very similar message rate during the period. The same rate over the last 2 weeks.
Do the putting applications set MQPMO_SYNCPOINT and then commit after 1 or more messages are PUT to the queue?
Answer: I will check with the application team.
A couple of things..
You have XBATCHSZ(1,1) so your recent batch size is 1 message per batch.
Total messages 23505 batches 403, so an average of 58 messages per batch. If your recent batch size is 1, then you must have had some larger (100?) batch sizes
XQTIME 191225 is number of microseconds messages were on the xmit queue before being sent. This is 0.1 second!
Nettime 101563 microseconds. This is a long time ( 0.1 seconds) 10,000 would be a good value. Compare this with a "TCP PING"
BUFSSENT 23525 is similar to number of messages - so message size is typically under 32K. Bytessent. messages gives 2286 so small messages.
Things to check
The queue at the remote end. Has it filled up? This would cause the sender queue to get more messages
The nettime seems very long. Compare this with TCP Ping. Nettime can include slow IO at the remote end - or a queue full at the remote end
XQTIME is high. This could be caused by sending applications not committing, or slow disk IO
I wrote "Why is my xmit queue filling up" in this blog
*Search for the title
have a read.
Capture these metrics over a day and see if they are typical
regards
Colin Paice
I set up an AMQP 1.0 link with just the path and a filter using the Apache Qpid Electron Go wrapper for Qpid Proton like this:
amqpConnection.Receiver(
// the path containing the consumer group
// and the partition Id
electron.Source("<EVENTHUB_PATH>"),
// the filter map contains some annotations filters
// for the Event Hub offset
electron.Filter(filterMap),
)
I followed this doc to setup the AMQP link options: https://godoc.org/qpid.apache.org/electron#LinkOption
However while running the Go app I've realised it was very slow in fetching messages, so I've added 2 more link options like this:
amqpConnection.Receiver(
electron.Source("<EVENTHUB_PATH>"),
electron.Capacity(100),
electron.Prefetch(true),
electron.Filter(filterMap),
)
but after adding the capacity and the prefetch link options I don't see any improvement in the performance.
I keep receiving approximately 10 messages every ~5 seconds from 4 parallel links (one link per partition).
I've tried to run the app with the environment variable PN_TRACE_RAW=true for the verbose output from Qpid Proton (cf. this: https://qpid.apache.org/releases/qpid-proton-0.18.0/proton/c/api/group__transport.html), but I am not sure
on what should I look for to troubleshoot this issue.
I don't think there is any issue with the Qpid settings, but anyway this is what I see on the terminal:
[0x9fd490]:0 -> #attach(18) [name="<MY_CUSTOM_NAME>",
handle=1, role=true, snd-settle-mode=0, rcv-settle-mode=0, source=#source(40) [address="<MY_CUSTOM_PATH>",
durable=0, expiry-policy=:"link-detach", timeout=0, dynamic=false, filter={:string=#:"apache.org:selector-filter:string"
"amqp.annotation.x-opt-offset > '<MY_CUSTOM_OFFSET>'"}], target=#target(41) [address="",
durable=0, expiry-policy=:"link-detach", timeout=0, dynamic=false], initial-delivery-count=0,
max-message-size=0]
[0x9fd490]:0 -> #flow(19) [next-incoming-id=1, incoming-window=2147483647, next-outgoing-id=1,
outgoing-window=0, handle=1, delivery-count=0, link-credit=100, drain=false]
I also tried to run the Go app in a Azure VM in the same Azure Location as the Event Hub, but no improvement in the performance.
How could I fetch many messages at the same time in the same "round trip"?
I need to process thousands of messages per seconds.
You are correct that you need a prefetch window but the electron client can do a LOT better than that.
I did a quick test with the electron examples from https://github.com/apache/qpid-proton/tree/master/examples/go/electron
I get 3000 msg/sec even without prefetch, and nearly 10000 msgs/sec with.
$ ./broker -qsize 100000 &
Listening on [::]:5672
$ ./send -count 10000 /x ; time ./receive -count 10000 /x
Received all 10000 acknowledgements
Listening on 1 connections
Received 10000 messages
real 0m2.612s
user 0m1.611s
sys 0m0.510s
$ ./send -count 10000 /x ; time ./receive -count 10000 -prefetch 1000 /x
Received all 10000 acknowledgements
Listening on 1 connections
Received 10000 messages
real 0m1.053s
user 0m1.272s
sys 0m0.277s
There is clearly something funny going on - I'd like to help you get to the bottom of it.
PN_TRACE_RAW is a bit too verbose to be helpful, try PN_TRACE_FRM=1 which will give you a more readable summary.
I'm happy to continue the conversation either here or on users#qpid.apache.org if it turns into more of a support case than a question/answer.
Standard squid config only logs one CONNECT line for any https transaction. What is being counted/timed by the reported bytes and duration fields in that line?
Got an answer via the squid-users mailing list [1]:
Unless you are using SSL-Bump or such to process the contents specially.
The duration is from the CONNECT message arriving to the time TCP close
is used to end the tunnel. The size should be the bytes sent to the
client (excluding the 200 reply message itself) during that time.
[1] http://lists.squid-cache.org/pipermail/squid-users/2016-July/011714.html
We are using Apache JMeter 2.12 in order to measure the response time of our JMS queue. However, we would like to see how many of those requests take less than a certain time. This, according to the official site of JMeter (http://jmeter.apache.org/usermanual/component_reference.html) should be set by the Timeout property. You can see in the photo below how our configuration looks like:
However, setting the timeout does not result in an error after sending 100 requests. We can see that some of them take apparently more than that amount of time:
Is there some other setting I am missing or is there a way to achieve my goal?
Thanks!
The JMeter documentation for JMS Point-to-Point describes the timeout as
The timeout in milliseconds for the reply-messages. If a reply has not been received within the specified time, the specific testcase failes and the specific reply message received after the timeout is discarded. Default value is 2000 ms.
This is timing not the actual sending the message but receipt of a response.
The source for the JMeter Point to Point will determine if you have a 'Receive Queue' Configured. If you do it will go through the executor path and use the timeout value, otherwise it does not use time timeout value.
if (useTemporyQueue()) {
executor = new TemporaryQueueExecutor(session, sendQueue);
} else {
producer = session.createSender(sendQueue);
executor = new FixedQueueExecutor(producer, getTimeoutAsInt(), isUseReqMsgIdAsCorrelId());
}
In your screen shot JNDI name Receive Queue is not defined, thus it uses temporary queue, and does not use the timeout. Should or should not timeout be supported in this case, that is best discussed in JMeter forum.
Alternately if you want to see request times in percentiles/buckets please read this stack overflow Q/A -
I want to find out the percentage of HTTPS requests that take less than a second in JMeter
Using Andrew Rapp's XBee-API, how can I sample I/O data via a coordinator from more than two endpoints?
I have 17 Series 1 XBees. I have programmed one to be a coordinator (API mode = 2) and the rest to be endpoints. Using XBee-API I am sending a Force I/O Sample ("IS") remote AT command, unicast to each endpoint. This works perfectly well when there are up to two endpoints, but as soon as a third is added, one of the three always becomes non-responsive (times out with XBeeTimeoutException). It's not always the same physical unit that stops responding, but it is always the third one (for example, if I send Force I/O Sample to Device1, Device2, and Device3, Device3 will time out, and if I change the order to Device3, Device1, Device2, Device2 will time out.
If I set up more than three XBees, about 1 out of 3 will time out - but not every third one.
I've verified that the XBees themselves are fine. I've searched the Internet and Stack Overflow in particular to no avail. I've tried using a simple ZNetRemoteAtRequest. I've tried opening and closing the XBee coordinator serial connection once for all three devices, once per device, and once per program run. I've tried varying the distance between the coordinator and endpoints (never more than five feet apart). I've tried different coordinator configuration parameters (from the Digi documentation). I've tried changing out the XBee for the coordinator.
This is the code I'm using to send the Force I/O Sample request to each endpoint and read the response:
xbee = new XBee(); // Coordinator
xbee.open("/dev/ttyUSB0, 115200)); // Happens before any of the endpoints are contacted
... // Loop through known endpoint addresses
XBeeRequest request = new ZBForceSampleRequest(new XBeeAddress64(endpointAddress));
ZNetRemoteAtResponse response = null;
response = (ZNetRemoteAtResponse) xbee.sendSynchronous(request, remoteXBeeTimeout);
if (response.isOk()) {
// Process response payload
}
... // End loop and finally close coordinator connection
What might help polling I/O samples from more than two endpoints?
EDIT: I found that Andrew Rapp's XBee-API library fakes multithreaded behavior, which causes the synchronization issues described in this question. I wrote a replacement library that is actually multithreaded and correctly maps responses from multiple XBee endpoints: https://github.com/steveperkins/xbee-api-for-java-1-4. When I wrote it Java 1.4 was necessary for use on the BeagleBone, Plug, and Zotac single-board PCs but it's an easy conversion to 1.7+.
Are you using hardware flow control on your serial port? Is it possible that you're sending requests out when the local XBee has deasserted CTS (e.g., asking you to stop sending)? I assume you're running at 115200 bps, so the XBee serial port can keep up with the network data rate.
Can you turn on debugging information, or connect some port monitoring hardware/software to display the data going over the serial port to the local XBee?