We have been facing an issue where message rate of a xmitq is very slow comparing with what should be a normal performance.
We have many other Qmgrs with bigger MQ flows that don't have the same issue.
Our HUB qmgr connects to business line in the same company HUB qmgr, and even the destination queues on their side being empty the flow is really slow.
At OS and Network level they say nothing can be done. At MQ we have changed the Buffersizes so it matches the OS level and uses the system tcp windows.
Now at MQ level we have the channel SDR setup with BATCHSZ to 100, but seems the receiver is configured with 30.
We noticed that because we see messages flow in batches fof 30 messages. Also not sure if that is related but we see the XMITQ havs always 30 uncommited messages.
Our questiong for advice.
Would increase the BATCHSZ parameter on SDR/RCVR help the perfomance?
Is there any other parameter at MQ level that could help it?
DIS CHS(NAME) ALL
AMQ8417: Display Channel Status details.
CHANNEL(QMGRA.QMGRB.T7) CHLTYPE(SDR)
BATCHES(234) BATCHSZ(30)
BUFSRCVD(235) BUFSSENT(6391)
BYTSRCVD(6996) BYTSSENT(14396692)
CHSTADA(2020-04-16) CHSTATI(14.38.17)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(159.50.69.38(48702)) CURLUWID(398F3E5EEA43381C)
CURMSGS(30) CURRENT
CURSEQNO(43488865) EXITTIME(0,0)
HBINT(300) INDOUBT(YES)
JOBNAME(000051FC00000001) LOCLADDR(10.185.8.122(54908))
LONGRTS(999999999) LSTLUWID(398F3E5EE943381C)
LSTMSGDA(2020-04-16) LSTMSGTI(14.49.46)
LSTSEQNO(43488835) MCASTAT(RUNNING)
MONCHL(HIGH) MSGS(6386)
NETTIME(2789746,3087573) NPMSPEED(NORMAL)
RQMNAME(QMGRB) SHORTRTS(10)
SSLCERTI(*******************)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(*******************)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
XBATCHSZ(23,7) XMITQ(QMGRB.X7)
XQTIME(215757414,214033427) RVERSION(08000008)
RPRODUCT(MQMM)
qm.ini:
Log:
LogPrimaryFiles=10
LogSecondaryFiles=10
LogFilePages=16384
LogType=LINEAR
LogBufferPages=4096
LogPath=/apps/wmq/QMGR/log/QMGR/
LogWriteIntegrity=SingleWrite
Service:
Name=AuthorizationService
EntryPoints=13
TCP:
SvrSndBuffSize=0
SvrRcvBuffSize=0
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/opt/mqm75/lib64/amqzfu
ComponentDataSize=0
Channels:
MaxChannels=500
UPDATED: 15:41 GMT
Just to update the information, both sides are now with BATCHSZ 100 and seems slightly.
AMQ8417: Display Channel Status details.
CHANNEL(QMGRA.QMGRB.T7) CHLTYPE(SDR)
BATCHES(403) BATCHSZ(100)
BUFSRCVD(405) BUFSSENT(23525)
BYTSRCVD(11756) BYTSSENT(53751066)
CHSTADA(2020-04-17) CHSTATI(15.13.51)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(159.50.69.38(48702)) CURLUWID(6D66985E94343410)
CURMSGS(0) CURRENT
CURSEQNO(44115897) EXITTIME(0,0)
HBINT(300) INDOUBT(NO)
JOBNAME(0000172A00000001) LOCLADDR(10.185.8.122(2223))
LONGRTS(999999999) LSTLUWID(6D66985E93343410)
LSTMSGDA(2020-04-17) LSTMSGTI(15.30.06)
LSTSEQNO(44115897) MCASTAT(RUNNING)
MONCHL(HIGH) MSGS(23505)
NETTIME(101563,480206) NPMSPEED(NORMAL)
RQMNAME(QMGRB) SHORTRTS(10)
SSLCERTI(*************************************)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(****************************)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(MQGET)
XBATCHSZ(1,1) XMITQ(QMGRB.X7)
XQTIME(191225,794134) RVERSION(08000008)
RPRODUCT(MQMM)
AMQ8450: Display queue status details.
QUEUE(QMGRB.X7) TYPE(QUEUE)
CURDEPTH(0) IPPROCS(1)
LGETDATE(2020-04-17) LGETTIME(15.30.06)
LPUTDATE(2020-04-17) LPUTTIME(15.30.06)
MEDIALOG(S2488154.LOG) MONQ(LOW)
MSGAGE(0) OPPROCS(9)
QTIME(794134, 191225) UNCOM(NO)
I'll put a few observations in this answer, but based on any further feedback I may add more.
You are running a very old version of the software on the sender side, MQ 7.5 went out of support almost two years ago (April 30 2018). IBM for a cost will provide extended support for an additional three years, so maybe you fall in that group. The 7.5.0.2 maintenance release itself came out in July 11th 2013, so it is almost seven years old at this point. I would strongly suggest you move to a newer version.
Note that MQ v8.0 goes out of support April 30 2020, and IBM just announced a few days ago that MQ v9.0 goes out of support September 30 2021. When you do migrate you should go with either 9.1 which has no announced end of support (they give five years minimum so it could be 2023) or go with the next version of MQ that should be out some time this year.
You mention setting the following:
TCP:
SvrSndBuffSize=0
SvrRcvBuffSize=0
The above setting apply to the SVRCONN end of a client connection. You can see this in the MQ v7.5 Knowledge Center page WebSphere MQ>Configuring>Changing configuration information>Changing queue manager configuration information>TCP, LU62, NETBIOS, and SPX:
SvrSndBuffSize=32768|number
The size in bytes of the TCP/IP send buffer used by the server end of a client-connection
server-connection channel.
SvrRcvBuffSize=32768|number
The size in bytes of the TCP/IP receive buffer used by the server end of a client-connection
server-connection channel.
At IBM MQ v7.5.0.2 APAR IV58073 introduced the concept of setting various buffer settings to a value to 0 which means that it will allow the operating system defaults to be used. Unfortunately like many things in the Knowledge Center it does not look like IBM documented this correctly for 7.5.
You can however review the IBM MQ v8.0 Knowledge Center to get the full picture regarding these settings at the page Configuring>Changing configuration information>Changing queue manager configuration information>TCP, LU62, and NETBIOS, specifically you would want to set these two settings to have any impact on your Sender Channel:
SndBuffSize=number| 0
The size in bytes of the TCP/IP send buffer used by the sending end of
channels. This stanza value can be overridden by a stanza more
specific to the channel type, for example RcvSndBuffSize. If the
value is set as zero, the operating system defaults are used. If no
value is set, then the IBM MQ default, 32768, is used.
RcvSndBuffSize=number| 0
The size in bytes of the TCP/IP send buffer used by the sender end of
a receiver channel. If the value is set as zero, the operating system
defaults are used. If no value is set, then the IBM MQ default, 32768,
is used.
Starting at IBM MQ v8.0 any newly created queue manager will have all of the following in the qm.ini:
TCP:
SndBuffSize=0
RcvBuffSize=0
RcvSndBuffSize=0
RcvRcvBuffSize=0
ClntSndBuffSize=0
ClntRcvBuffSize=0
SvrSndBuffSize=0
SvrRcvBuffSize=0
However, any queue manager that is upgraded will not by default get those settings, meaning if those are not present they will not be added, if they are present they will remain the same. If the setting is not present then as it says above "the IBM MQ default, 32768, is used."
I had extensive discussions with IBM support on this topic and came to the conclusion that they did not see any reason to not set it to 0, they only saw benefit in doing so, but with an abundance of caution they do not change it to 0 for you.
I would recommend you add all of those to your qm.ini, but at minimum add the two I highlighted above.
These are good setting to implement but may not solve your problem if nothing changed recently on either end. If however something did change, for example a network difference, or MQ was upgraded to 8.0.0.8 on the remote side, then this setting just might solve your problem.
In the channel status output two values are interesting:
NETTIME(2789746,3087573)
XQTIME(215757414,214033427)
NETTIME means that based on recent activity it took 2.7 seconds to receive a response from the RCVR channel, over a longer period of time it took 3.1 seconds to receive a response from the RCVR channel. Can you compare this to a TCP ping from the sender channel server to the receive channel server, 2.7 seconds for a response over the network seems excessive. In the presentation Keeping MQ Channels Up and Running given at Capitalware's MQ Technical Conference v2.0.1.4, Paul Clarke who used to work for IBM states "NETTIME only measures network time, and does not include
the MQCMIT for example".
XQTIME means that based on recent activity and over a longer period of time it took ~215 seconds for a message on the XMITQ to be picked up by the SDR channel to be sent.
See below for how IBM documents these:
NETTIME
Amount of time, displayed in microseconds, to send a request to the remote end of the channel and receive a response. This time only measures the network time for such an operation. Two values are displayed:
A value based on recent activity over a short period.
A value based on activity over a longer period.
XQTIME
The time, in microseconds, that messages remained on the transmission queue before being retrieved. The time is measured from when the message is put onto the transmission queue until it is retrieved to be sent on the channel and, therefore, includes any interval caused by a delay in the putting application.
Two values are displayed:
A value based on recent activity over a short period.
A value based on activity over a longer period.
Information on the BATCHSZ channel parameter can be found in the IBM MQ v8.0 Knowledge Center page Reference>Configuration reference>Channel attributes>Channel attributes in alphabetical order>Batch size (BATCHSZ). I have quoted it and highlighted a few areas in bold.
This attribute is the maximum number of messages to be sent before a sync point is taken.
The batch size does not affect the way the channel transfers messages; messages are always transferred individually, but are committed or backed out as a batch.
To improve performance, you can set a batch size to define the maximum number of messages to be transferred between two sync points. The batch size to be used is negotiated when a channel starts, and the lower of the two channel definitions is taken. On some implementations, the batch size is calculated from the lowest of the two channel definitions and the two queue manager MAXUMSGS values. The actual size of a batch can be less; for example, a batch completes when there are no messages left on the transmission queue or the batch interval expires.
A large value for the batch size increases throughput, but recovery times are increased because there are more messages to back out and send again. The default BATCHSZ is 50, and you are advised to try that value first. You might choose a lower value for BATCHSZ if your communications are unreliable, making the need to recover more likely.
This attribute is valid for channel types of:
Sender
Server
Receiver
Requester
Cluster sender
Cluster receiver
Follow up questions:
Are the messages that are PUT to this XMITQ persistent?
Answer: Yes, in our PROD env all messages are pesistent.
Have you had a recent increase in volume going to this XMITQ?
Answer: No, we use a monitoring tools, we extracted a report that show very similar message rate during the period. The same rate over the last 2 weeks.
Do the putting applications set MQPMO_SYNCPOINT and then commit after 1 or more messages are PUT to the queue?
Answer: I will check with the application team.
A couple of things..
You have XBATCHSZ(1,1) so your recent batch size is 1 message per batch.
Total messages 23505 batches 403, so an average of 58 messages per batch. If your recent batch size is 1, then you must have had some larger (100?) batch sizes
XQTIME 191225 is number of microseconds messages were on the xmit queue before being sent. This is 0.1 second!
Nettime 101563 microseconds. This is a long time ( 0.1 seconds) 10,000 would be a good value. Compare this with a "TCP PING"
BUFSSENT 23525 is similar to number of messages - so message size is typically under 32K. Bytessent. messages gives 2286 so small messages.
Things to check
The queue at the remote end. Has it filled up? This would cause the sender queue to get more messages
The nettime seems very long. Compare this with TCP Ping. Nettime can include slow IO at the remote end - or a queue full at the remote end
XQTIME is high. This could be caused by sending applications not committing, or slow disk IO
I wrote "Why is my xmit queue filling up" in this blog
*Search for the title
have a read.
Capture these metrics over a day and see if they are typical
regards
Colin Paice
Related
We are currently having an issue with an MQ Cluster were a CLUSSDR channel is going into retry as the receiving MQ object is showing as damaged.
Configuration is many QMGR's (STAT00-11) sending messages to the Cluster of 4 QMGR's, 2 FullRepos (HUB01-02 and 2PartialRepos HUB03-04)
Problem is that on the STAT02 QMGR the CLUSSDR channel to HUB01 is in a retry state
with the MQ log error;
AMQ9506: Message receipt confirmation failed.
and on HUB01 the MQ log errors;
AMQ7472: Object %CHLBATCH.706, type scratchpad damaged. (many)
AMQ9999: Channel 'TO_HUB01' to host 'server02 (n.n.n.n)' ended abnormally.
AMQ9588: Program cannot update queue manager object. (single instance)
AMQ9587: Program cannot open queue manager object (many)
I have now stopped the CLUSSDR on STAT02 to HUB01 and there is no longer any log entries, however as the QMGR's have linear logging the log files are not being released on the HUB01 QMGR
this has introduced a new error
AMQ7084: Object syncfile, type syncfile damaged.
which is filling up the disk.
I have so far tried to recover the damaged object, the command used was on the HUB01 QMGR
rcrmqobj -m HUB01 -t channel TO_STAT02
and this returned the result, AMQ7085: Object TO_STAT02, type channel not found., although the following results contradict this;
DIS CLUSQMGR(STAT*) CHANNEL
outputs a list of all the STAT* QMGR's which includes the TO_STAT02 channel
and the channel status
DIS CHS(TO_STAT*) STATUS
shows all the channels in a RUNNING state, including the supposed non-existent TO_STAT02
Anyone had similar issues please, note that this is the second occurrence we have had in the last month to different clusters and last time we had to take the drastic action of rebuilding the QMGR once the disk space was exhausted and the QMGR crashed
rcrmqobj -m HUB01 -t syncfile
is the correct way to rebuild a corrupt syncfile and if using linear logging this will also repair any damaged scratchpad objects. Damaged scratchpad objects should only ever occur through operational or filesystem error, for example if files were deleted or partially restored from backup and so having a large number is something that you should try and identify the root cause.
rcrmqobj -t channel will be able to recover damage to channel object definitions, but it is the synchronization data and its index (syncfile) that is damaged/missing. TO_STAT02 sounds like it is a cluster sender that MQ clustering maintains from information shared within the cluster - you can check on whether a cluster channel has a local channel definition using DEFTYPE on DISPLAY CLUSQMGR.
I have an EA set in place that loops history trades and builds one large string with trade information. I then send this string every second from MT4 to the python backend using a plain PUSH/PULL pattern.
For whatever reason, the data isn't received on the pull side when the string transferred becomes too long. The backend PULL-socket slices each string and further processes it.
Any chance that the PULL-side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Talking about file sizes we are well below 5kb per second.
This is the PULL-socket, which manipulates the data after receiving it:
while True:
# check 24/7 for available data in the pull socket
try:
msg = zmq_socket.recv_string()
data = msg.split("|")
print(data)
# if data is available and msg is account info, handle as follows
if data[0] == "account_info":
[...]
except zmq.error.Again:
print("\nResource timeout.. please try again.")
sleep(0.000001)
I am a bit curious now since the pull socket seems to not even be able to process a string containing 40 trades with their according information on a single MT4 client - Python connection. I actually planned to set it up to handle more than 5.000 MT4 clients - python backend connections at once.
Q : Any chance that the pull side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Zero chance.
Sending 640 B each second is definitely no showstopper ( 5kb per second - is nowhere near a performance ceiling... )
The posted problem formulation is otherwise undecidable.
Step 1) POSACK/NACK prove whether a PUSH side accepts the payload for sending error-free.
Step 2) prove the PULL side is not to be blamed - [PUSH.send(640*chr(64+i)) for i in range( 10 )] via a python-2-python tcp://-transport-class solo-channel crossing host-to-host hop, over at least your local physical network ( no VMCI/emulated vLAN, no other localhost colocation )
Step 3) if either steps above got POSACK-ed, your next chances are the ZeroMQ configuration space and/or the MT4-based PUSH-side incompatibility, most probably "hidden" inside a (not mentioned) third party ZeroMQ wrapper used / first-party issues with string handling / processing ( which you must have already read about, as it has been so many times observed and mentioned in the past posts about this trouble with well "hidden" MQL4 internal eco-system changes ).
Anyway, stay tuned. ZeroMQ is a sure bet and a truly horsepower for professional and low-latency designs in distributed-system's domain.
We are testing the capacity of a Mail relay based on RHEL 7.6.
We are observing issues when sending an important number of msgs (e.g.: ~1000 msgs in 60 seconds).
While we have sent all the msgs and the recipient has received all the msgs, logs are missing in the /var/log/maillog_rfc5424.
We have the following message in the /var/log/messages:
rsyslogd: imjournal: XYZ messages lost due to rate-limiting
We adapted the /etc/rsyslog.conf with the following settings but without effect:
$SystemLogRateLimitInterval 0 # turn off rate limit
$SystemLogRateLimitBurst 0 # turn rate limit off
Any ideas ?
The error is from imjournal, but your configuration settings are for imuxsock.
According to the rsyslog configuration page you need to set
$imjournalRatelimitInterval 0
$imjournalRatelimitBurst 0
Note that for very high message rates you might like to change to imuxsock, as it says:
this module may be notably slower than when using imuxsock. The journal provides imuxsock with a copy of all “classical” syslog messages, however, it does not provide structured data. Only if that structured data is needed, imjournal must be used. Otherwise, imjournal may simply be replaced by imuxsock, and we highly suggest doing so.
I am using veins4.6 with sumo 0.30 and omnet++5.1.1 in ubuntu 14.04. I have created a custom network with a cross(one intersection with 4 roads) and ran the simulation with 200 vehicles. I did not observe this behaviour for 4vehicles. I have seen it with 50 vehicles too. I need to get the count of total lost packets for my masters project. So I was looking at statistics and found that RXTXLostPackets is not zero. As far as I understood from documentation it should be zero if allowTxDuringRx=false. Default is false(PhyLayer80211p.ned). As I did not change any code yet, I was confused if that is expected behaviour.
What I have done so far.
from Mac1609_4::handleLowerControl, statsTXRXLostPackets is updated when Decider80211p responds with RECWHILESEND.
In Decider80211p::processSignalEnd, if value of whileSending is true RECWHILESEND is sent to mac layer as control message.
In Decider80211p::processSignalEnd, if(frame->getWasTransmitting() || phy11p->getRadioState() == Radio::TX) , this frame was considered as received while sending and sets the value for whileSending as true.
The wasTransmitting varilable is set to true in Decider80211p::switchToTx and Decider80211p::processNewSignal functions.
currentFrame->setWasTransmitting(true);
currentFrame->setBitError(true);
in Decider80211p::processNewSignal:
if (phy11p->getRadioState() == Radio::TX ) {
frame->setBitError(true); --> tried disabling both these values and the RXTXLostPackets was zero.
frame->setWasTransmitting(true);
DBG_D11P << "AirFrame: " << frame->getId() << " (" << recvPower << ") received, while already sending. Setting BitErrors to true" << std::endl;
}
There is one thread with similar issue with the fix of adding this line in processSignalEnd function. But looks like veins4.6 does not use curSyncFrame anymore.
Veins - Unexpected behavior with lost packets in certain vehicles
if (!frame->getWasTransmitting()){
curSyncFrame = 0;
}
I could not clearly understand the issue. The code and configuration files I have used are here. https://github.com/Rajeswar59/veins_learning.
Can anyone please take a look and help me with this. Thanks in advance.
edit: I went through the logs. This is what I could understand as of now.
https://drive.google.com/open?id=0BzjDW8PQhkSmSEUtZ2lpcld4ZXc --> some portion of logs are here.
---> order of sending
#13332 0.247987176594 node[30] --> node[48] id=22266
#13375 0.247987796864 node[18] --> node[20] id=22447
#13384 0.247987864534 node[20] --> node[30] id=22573
From logs I have concentrated on node 18. Two nodes that transmitted before 30 are 32 and 4. These 2 messages are received successfully by all 3 nodes. When a message arrives decider tries to set channel state as busy in processnewsignal and set idle after processing packet. This calls mac1609_4.cc channelBusy and channelIdle functions respectively. So the channelIdle variable is set accordingly. Also if channel is to be set busy it will stop contention and calculate currentBackoff if any packet is waiting to be transmitted. If channel is being set idle at the end of reception, startContent is called. Based on this only the lastIdle variable is set which is used to calculate nextMacEvent. So when the last successful message was received all the nodes which have a packet to send decide nextMacEvent and it is sent as self message in Mac1609_4.cc. on receiving the nextMacEvent self message we will start transmitting without checking if any other node has started transmission. We can not identify that probably because we are setting channel busy when we receive messages after some propagation delay. So between last successful transmission and nextMacEvent other nodes also take decision to transmit without checking current channel state. That's why the node has some receive events while sending. As far as my understanding goes before transmission we should sense current state of channel and retry backoff accordingly. We do not check this at the nextMacEvent. It looks like a collision behaviour but should we not check the current state of channel when backoff counter reaches zero and retry. Please correct me if I am wrong anywhere.
Thanks for your patience.
Any help or advice??
My Learnings(probably last update):
After Some digging, these are my learnings if it helps some one. The basic CSMA mechanism says before attempting for transmission, the node has to sense the channel, initiate transmission if the channel is sensed idle for AIFS time, or go in to back off if channel is busy. In veins the channel busy status is stored in idleChannel variable whose status is checked in Mac1609_4:channelBusySelf() function before initiating transmission (nextMacEvent in Mac1609_4::handleSelfMsg). The idleChannel is updated in Mac1609_4::channelBusy and Mac1609_4::channelIdle functions when a message reception starts and when message reception ends respectively. So when a previously transmitting node sends a packet, all the recieving nodes will receive the packet with varying delay i.e., starts receiving at different times and update their channelIdle variable. After that they calculate best time to transmit and starts transmission. It does check if channel is idle or not but as the channelIdle status is updated at next reception and because of transmission delay it takes some time between transmission start at sender and reception start at receiver side, both the transmitting nodes cant see other transmission. As far as I understand this is called a collision when more than two nodes start transmission at the same time. So the BitError statistic is set and statsTXRXLostPackets is also set. So while calculating totalLostPackets we can take only one of these two values.
What's the topic of connection threshold events? How do I listen to connection count threshold events over the message bus, and how do I figure out what is the current connection count?
Connection threshold events can be published over the message bus to the following topics:
#LOG/WARNING/VPN/<router-name>/VPN_VPN_CONNECTIONS_HIGH/<vpn-name> when the connection count exceeds the high threshold.
#LOG/INFO/VPN/<router-name>/VPN_VPN_CONNECTIONS_HIGH_CLEAR/<vpn-name> when the connection count goes below the clear threshold.
If desired, you can apply wildcards to the topics. For example, #LOG/*/VPN/<router-name>/VPN_VPN_CONNECTIONS*/<vpn-name>.
Note that you will need to fill in <router-name> and <vpn-name> with appropriate values.
In order to have the connection count threshold events published over the message bus, you will need to do the following:
a. Configure the VPN to "Publish Message VPN Event Messages".
b. Your application needs to subscribe to the topic for connection threshold events.
In order to figure out the current connection count, you will need to send a SEMP over message bus query.
a. Enable SEMP over Message Bus Show Commands on the VPN.
b. Send a SEMP over Message Bus query. There's an SempGetOverMB sample in the API with detailed instructions to do this. You can also refer to the documentation for details.
<rpc semp-version="soltr/7_2">
<show>
<message-vpn>
<vpn-name>default</vpn-name>
</message-vpn>
</show>
</rpc>
c. Parse the XML based response.
<rpc-reply semp-version="soltr/7_2">
<rpc>
<show>
<message-vpn>
<vpn>
<name>default</name>
<connections-service-smf>3</connections-service-smf>
<connections-service-web>0</connections-service-web>
<connections-service-rest-incoming>0</connections-service-rest-incoming>
<connections-service-mqtt>0</connections-service-mqtt>
<connections-service-rest-outgoing>0</connections-service-rest-outgoing>
<max-connections>10</max-connections>
<max-connections-service-smf>9000</max-connections-service-smf>
<max-connections-service-web>9000</max-connections-service-web>
<max-connections-service-rest-incoming>9000</max-connections-service-rest-incoming>
<max-connections-service-mqtt>9000</max-connections-service-mqtt>
<max-connections-service-rest-outgoing>6000</max-connections-service-rest-outgoing>
... Removed non-relevant portions for clarity ...
</vpn>
</message-vpn>
</show>
</rpc>
<execute-result code="ok"/>
</rpc-reply>
Note that there is a system limit of 10 SEMP poll requests per second, and some topics should not be polled. Refer to the documentation for details.