I have an application which uses MDB, activation specifications and Queue Connection Factories to get/put Messages from WMQ. The application expects a max load of 80 tps. Both Websphere Application Server and WMQ are clustered and each application server connects to seperate host and channel. The application onMessage method is implemented in way so that both session and connection gets closed after message is consumed and response is sent.
As per our configuration, we have WAS version is 8.5, IBM MQ queue manager version 7, max server sessions for act spec set to 40 for each node. max connection count in Connection Factory to 40 for each node and max session in session pool of connection factory to 10.
Now on peak load we expect to make max 80 MQ Channel Instance but as per the investigation we can see it goes above 200 which is causing an issue as max instance limit is reached.
Is this happening because max session in session pool of connection factory is set to 10?
Is it possible that even though we are closing the session and connection in onMessage, still one connection can have more than one session. If that is the case, is it wise to set this property to 1?
Also can there be some property set at WMQ which could cause this increase in MQ Channel Instances.
You don't mention specific versions of WAS or MQ, and there could be known problems at a specific version that would change the behavior, but in general it should work as described below.
IBM has a nice Technote "TCP/IP Connection usage between WebSphere Application Server V7 and V8, and WebSphere MQ V7 (and later) explained" which goes into detail on this subject.
You do not mention what you have the SVRCONN channel's SHARECNV set to, as per below this will impact the number of channel instances observed, I'll assume the default of 10 for the calculations.
Note that block quotes below are from the Technote
we have set max server sessions for act spec to 40 for each node
The link above states:
Maximum number of conversations = Maximum server sessions + 1
Maximum number of conversations = 40 + 1 = 41
The link also states:
Maximum number of TCP/IP channel instances = Maximum number of conversations / SHARECNV for the channel being used
Maximum number of TCP/IP channel instances = 41 / 10 = 5 (rounded up to nearest connection)
max connection count in Connection Factory to 40 for each node
max session in session pool of connection factory to 10.
Maximum number of conversations = Connection Pool Maximum Connections + (Connection Pool Maximum Connections * Session Pool Maximum Connections)
Maximum number of conversations = 40 + (40 * 10) = 440
Maximum number of TCP/IP channel instances = Maximum number of conversations / SHARECNV for the channel being used
Maximum number of TCP/IP channel instances = 440 / 10 = 44
If your MQ SVRCONN channel's SHARECNV was set to 10, then you should have no more than 49 channel instances for each channel based on each node connecting to a separate channel.
If you are hitting 200 channel instances I would suspect your SHARECNV is less than 10. If it was 1 the the maximum number of channel instances WAS would try to create would go up to 481 which would be limited by the MAXINST of the the channel to 200.
After an application has finished with a JMS Connection and closed it off, it is moved from the Active Pool to the Free Pool, where it is available for reuse. The Connection Pool property Unused timeout defines how long a JMS Connection will stay in the Free Pool before it is disconnected. This property has the default value of 1800 seconds, which is 30 minutes.
Every JMS Connection that is created from a WebSphere MQ messaging provider Connection Factory has an associated JMS Session Pool, which work in the same way as Connection Pools. The maximum number of JMS Sessions that can be created from a single JMS Connection is determined by the Connection Factory Session Pool property Maximum connections. The default value of this property is 10.
A conversation is started when a JMS Session is first created, and will remain active until the JMS Session is closed because it has remained in the Free Pool for longer than the value of the Session Pool's Unused timeout property.
When your app closes the session and connection in onMessage, the connection is moved to the free pool for reuse and the session is moved to the free pool for reuse, the MQ Channel instance will not be closed until the respective timeout is hit.
If you want to keep your maximum channel count less than 200 then you could tune your Session Pool Maximum Connections) to 1 which combined with your Activation Specifications and a SHARECNV(1) would max out at 121 channel instances.
You can also increase the SHARECNV value of the channel which will result in dividing the channel instances by that number.
It is possible that your connections or sessions are not getting closed properly and you have a "leak".
Related
We have been facing an issue where message rate of a xmitq is very slow comparing with what should be a normal performance.
We have many other Qmgrs with bigger MQ flows that don't have the same issue.
Our HUB qmgr connects to business line in the same company HUB qmgr, and even the destination queues on their side being empty the flow is really slow.
At OS and Network level they say nothing can be done. At MQ we have changed the Buffersizes so it matches the OS level and uses the system tcp windows.
Now at MQ level we have the channel SDR setup with BATCHSZ to 100, but seems the receiver is configured with 30.
We noticed that because we see messages flow in batches fof 30 messages. Also not sure if that is related but we see the XMITQ havs always 30 uncommited messages.
Our questiong for advice.
Would increase the BATCHSZ parameter on SDR/RCVR help the perfomance?
Is there any other parameter at MQ level that could help it?
DIS CHS(NAME) ALL
AMQ8417: Display Channel Status details.
CHANNEL(QMGRA.QMGRB.T7) CHLTYPE(SDR)
BATCHES(234) BATCHSZ(30)
BUFSRCVD(235) BUFSSENT(6391)
BYTSRCVD(6996) BYTSSENT(14396692)
CHSTADA(2020-04-16) CHSTATI(14.38.17)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(159.50.69.38(48702)) CURLUWID(398F3E5EEA43381C)
CURMSGS(30) CURRENT
CURSEQNO(43488865) EXITTIME(0,0)
HBINT(300) INDOUBT(YES)
JOBNAME(000051FC00000001) LOCLADDR(10.185.8.122(54908))
LONGRTS(999999999) LSTLUWID(398F3E5EE943381C)
LSTMSGDA(2020-04-16) LSTMSGTI(14.49.46)
LSTSEQNO(43488835) MCASTAT(RUNNING)
MONCHL(HIGH) MSGS(6386)
NETTIME(2789746,3087573) NPMSPEED(NORMAL)
RQMNAME(QMGRB) SHORTRTS(10)
SSLCERTI(*******************)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(*******************)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
XBATCHSZ(23,7) XMITQ(QMGRB.X7)
XQTIME(215757414,214033427) RVERSION(08000008)
RPRODUCT(MQMM)
qm.ini:
Log:
LogPrimaryFiles=10
LogSecondaryFiles=10
LogFilePages=16384
LogType=LINEAR
LogBufferPages=4096
LogPath=/apps/wmq/QMGR/log/QMGR/
LogWriteIntegrity=SingleWrite
Service:
Name=AuthorizationService
EntryPoints=13
TCP:
SvrSndBuffSize=0
SvrRcvBuffSize=0
ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/opt/mqm75/lib64/amqzfu
ComponentDataSize=0
Channels:
MaxChannels=500
UPDATED: 15:41 GMT
Just to update the information, both sides are now with BATCHSZ 100 and seems slightly.
AMQ8417: Display Channel Status details.
CHANNEL(QMGRA.QMGRB.T7) CHLTYPE(SDR)
BATCHES(403) BATCHSZ(100)
BUFSRCVD(405) BUFSSENT(23525)
BYTSRCVD(11756) BYTSSENT(53751066)
CHSTADA(2020-04-17) CHSTATI(15.13.51)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(159.50.69.38(48702)) CURLUWID(6D66985E94343410)
CURMSGS(0) CURRENT
CURSEQNO(44115897) EXITTIME(0,0)
HBINT(300) INDOUBT(NO)
JOBNAME(0000172A00000001) LOCLADDR(10.185.8.122(2223))
LONGRTS(999999999) LSTLUWID(6D66985E93343410)
LSTMSGDA(2020-04-17) LSTMSGTI(15.30.06)
LSTSEQNO(44115897) MCASTAT(RUNNING)
MONCHL(HIGH) MSGS(23505)
NETTIME(101563,480206) NPMSPEED(NORMAL)
RQMNAME(QMGRB) SHORTRTS(10)
SSLCERTI(*************************************)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(****************************)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(MQGET)
XBATCHSZ(1,1) XMITQ(QMGRB.X7)
XQTIME(191225,794134) RVERSION(08000008)
RPRODUCT(MQMM)
AMQ8450: Display queue status details.
QUEUE(QMGRB.X7) TYPE(QUEUE)
CURDEPTH(0) IPPROCS(1)
LGETDATE(2020-04-17) LGETTIME(15.30.06)
LPUTDATE(2020-04-17) LPUTTIME(15.30.06)
MEDIALOG(S2488154.LOG) MONQ(LOW)
MSGAGE(0) OPPROCS(9)
QTIME(794134, 191225) UNCOM(NO)
I'll put a few observations in this answer, but based on any further feedback I may add more.
You are running a very old version of the software on the sender side, MQ 7.5 went out of support almost two years ago (April 30 2018). IBM for a cost will provide extended support for an additional three years, so maybe you fall in that group. The 7.5.0.2 maintenance release itself came out in July 11th 2013, so it is almost seven years old at this point. I would strongly suggest you move to a newer version.
Note that MQ v8.0 goes out of support April 30 2020, and IBM just announced a few days ago that MQ v9.0 goes out of support September 30 2021. When you do migrate you should go with either 9.1 which has no announced end of support (they give five years minimum so it could be 2023) or go with the next version of MQ that should be out some time this year.
You mention setting the following:
TCP:
SvrSndBuffSize=0
SvrRcvBuffSize=0
The above setting apply to the SVRCONN end of a client connection. You can see this in the MQ v7.5 Knowledge Center page WebSphere MQ>Configuring>Changing configuration information>Changing queue manager configuration information>TCP, LU62, NETBIOS, and SPX:
SvrSndBuffSize=32768|number
The size in bytes of the TCP/IP send buffer used by the server end of a client-connection
server-connection channel.
SvrRcvBuffSize=32768|number
The size in bytes of the TCP/IP receive buffer used by the server end of a client-connection
server-connection channel.
At IBM MQ v7.5.0.2 APAR IV58073 introduced the concept of setting various buffer settings to a value to 0 which means that it will allow the operating system defaults to be used. Unfortunately like many things in the Knowledge Center it does not look like IBM documented this correctly for 7.5.
You can however review the IBM MQ v8.0 Knowledge Center to get the full picture regarding these settings at the page Configuring>Changing configuration information>Changing queue manager configuration information>TCP, LU62, and NETBIOS, specifically you would want to set these two settings to have any impact on your Sender Channel:
SndBuffSize=number| 0
The size in bytes of the TCP/IP send buffer used by the sending end of
channels. This stanza value can be overridden by a stanza more
specific to the channel type, for example RcvSndBuffSize. If the
value is set as zero, the operating system defaults are used. If no
value is set, then the IBM MQ default, 32768, is used.
RcvSndBuffSize=number| 0
The size in bytes of the TCP/IP send buffer used by the sender end of
a receiver channel. If the value is set as zero, the operating system
defaults are used. If no value is set, then the IBM MQ default, 32768,
is used.
Starting at IBM MQ v8.0 any newly created queue manager will have all of the following in the qm.ini:
TCP:
SndBuffSize=0
RcvBuffSize=0
RcvSndBuffSize=0
RcvRcvBuffSize=0
ClntSndBuffSize=0
ClntRcvBuffSize=0
SvrSndBuffSize=0
SvrRcvBuffSize=0
However, any queue manager that is upgraded will not by default get those settings, meaning if those are not present they will not be added, if they are present they will remain the same. If the setting is not present then as it says above "the IBM MQ default, 32768, is used."
I had extensive discussions with IBM support on this topic and came to the conclusion that they did not see any reason to not set it to 0, they only saw benefit in doing so, but with an abundance of caution they do not change it to 0 for you.
I would recommend you add all of those to your qm.ini, but at minimum add the two I highlighted above.
These are good setting to implement but may not solve your problem if nothing changed recently on either end. If however something did change, for example a network difference, or MQ was upgraded to 8.0.0.8 on the remote side, then this setting just might solve your problem.
In the channel status output two values are interesting:
NETTIME(2789746,3087573)
XQTIME(215757414,214033427)
NETTIME means that based on recent activity it took 2.7 seconds to receive a response from the RCVR channel, over a longer period of time it took 3.1 seconds to receive a response from the RCVR channel. Can you compare this to a TCP ping from the sender channel server to the receive channel server, 2.7 seconds for a response over the network seems excessive. In the presentation Keeping MQ Channels Up and Running given at Capitalware's MQ Technical Conference v2.0.1.4, Paul Clarke who used to work for IBM states "NETTIME only measures network time, and does not include
the MQCMIT for example".
XQTIME means that based on recent activity and over a longer period of time it took ~215 seconds for a message on the XMITQ to be picked up by the SDR channel to be sent.
See below for how IBM documents these:
NETTIME
Amount of time, displayed in microseconds, to send a request to the remote end of the channel and receive a response. This time only measures the network time for such an operation. Two values are displayed:
A value based on recent activity over a short period.
A value based on activity over a longer period.
XQTIME
The time, in microseconds, that messages remained on the transmission queue before being retrieved. The time is measured from when the message is put onto the transmission queue until it is retrieved to be sent on the channel and, therefore, includes any interval caused by a delay in the putting application.
Two values are displayed:
A value based on recent activity over a short period.
A value based on activity over a longer period.
Information on the BATCHSZ channel parameter can be found in the IBM MQ v8.0 Knowledge Center page Reference>Configuration reference>Channel attributes>Channel attributes in alphabetical order>Batch size (BATCHSZ). I have quoted it and highlighted a few areas in bold.
This attribute is the maximum number of messages to be sent before a sync point is taken.
The batch size does not affect the way the channel transfers messages; messages are always transferred individually, but are committed or backed out as a batch.
To improve performance, you can set a batch size to define the maximum number of messages to be transferred between two sync points. The batch size to be used is negotiated when a channel starts, and the lower of the two channel definitions is taken. On some implementations, the batch size is calculated from the lowest of the two channel definitions and the two queue manager MAXUMSGS values. The actual size of a batch can be less; for example, a batch completes when there are no messages left on the transmission queue or the batch interval expires.
A large value for the batch size increases throughput, but recovery times are increased because there are more messages to back out and send again. The default BATCHSZ is 50, and you are advised to try that value first. You might choose a lower value for BATCHSZ if your communications are unreliable, making the need to recover more likely.
This attribute is valid for channel types of:
Sender
Server
Receiver
Requester
Cluster sender
Cluster receiver
Follow up questions:
Are the messages that are PUT to this XMITQ persistent?
Answer: Yes, in our PROD env all messages are pesistent.
Have you had a recent increase in volume going to this XMITQ?
Answer: No, we use a monitoring tools, we extracted a report that show very similar message rate during the period. The same rate over the last 2 weeks.
Do the putting applications set MQPMO_SYNCPOINT and then commit after 1 or more messages are PUT to the queue?
Answer: I will check with the application team.
A couple of things..
You have XBATCHSZ(1,1) so your recent batch size is 1 message per batch.
Total messages 23505 batches 403, so an average of 58 messages per batch. If your recent batch size is 1, then you must have had some larger (100?) batch sizes
XQTIME 191225 is number of microseconds messages were on the xmit queue before being sent. This is 0.1 second!
Nettime 101563 microseconds. This is a long time ( 0.1 seconds) 10,000 would be a good value. Compare this with a "TCP PING"
BUFSSENT 23525 is similar to number of messages - so message size is typically under 32K. Bytessent. messages gives 2286 so small messages.
Things to check
The queue at the remote end. Has it filled up? This would cause the sender queue to get more messages
The nettime seems very long. Compare this with TCP Ping. Nettime can include slow IO at the remote end - or a queue full at the remote end
XQTIME is high. This could be caused by sending applications not committing, or slow disk IO
I wrote "Why is my xmit queue filling up" in this blog
*Search for the title
have a read.
Capture these metrics over a day and see if they are typical
regards
Colin Paice
We are migrating from IBM MQ to Amazon MQ, at least we would like to do so. The problem is Amazon MQ is having bad performance when using JMS producer to put a large message on a queue compared to IBM MQ.
All messages are persistent and the system is High Available regarding IBM MQ, and Amazon MQ is multi AZ.
If we put this size of XML files to IBM MQ (2 cpu and 8GB RAM HA instance) we have this performance:
256 KB = 15ms
4,6 MB = 125ms
9,3 MB = 141ms
18,7 MB = 218ms
37,4 MB = 628ms
74,8 MB = 1463ms
If we put the same files on Amazon MQ (mq.m5.2xlarge = 8 CPU and 32 GB RAM) or ActiveMQ we have this performance:
256 KB = 967ms
4,6 MB = 1024ms
9,3 MB = 1828ms
18,7 MB = 3550ms
37,4 MB = 8900ms
74,8 MB = 14405ms
What we also see is that IBM MQ has equal response times for sending a message to a queue and getting a message from a queue, while Amazon MQ is real fast in getting a message (e.g. just takes 1 ms), but very slow on sending.
On Amazon MQ we use the OpenWire protocol. We use this config in Terraform style:
resource "aws_mq_broker" "default" {
broker_name = "bernardamazonmqtest"
deployment_mode = "ACTIVE_STANDBY_MULTI_AZ"
engine_type = "ActiveMQ
engine_version = "5.15.10"
host_instance_type = "mq.m5.2xlarge"
auto_minor_version_upgrade = "false"
apply_immediately = "false"
publicly_accessible = "false"
security_groups = [aws_security_group.pittensbSG-allow-mq-external.id]
subnet_ids = [aws_subnet.pittensbSN-public-1.id, aws_subnet.pittensbSN-public-3.id]
logs {
general = "true"
audit = "true"
}
We use Java 8 with JMS ActiveMQ library via POM (Maven):
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-client</artifactId>
<version>5.15.8</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-pool</artifactId>
<version>5.15.8</version>
</dependency>
In JMS we have this Java code:
private ActiveMQConnectionFactory mqConnectionFactory;
private PooledConnectionFactory mqPooledConnectionFactory;
private Connection connection;
private Session session;
private MessageProducer producer;
private TextMessage textMessage;
private Queue queue;
this.mqConnectionFactory = new ActiveMQConnectionFactory();
this.mqPooledConnectionFactory = new PooledConnectionFactory();
this.mqPooledConnectionFactory.setConnectionFactory(this.mqConnectionFactory);
this.mqConnectionFactory.setBrokerURL("ssl://tag-1.mq.eu-west-1.amazonaws.com:61617");
this.mqPooledConnectionFactory.setMaxConnections(10);
this.connection = mqPooledConnectionFactory.createConnection());
this.connection.start();
this.session = this.connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
this.session.createQueue("ExampleQueue");
this.producer = this.session.createProducer(this.queue);
long startTimeSchrijf = 0;
startTimeWrite= System.currentTimeMillis();
producer.send("XMLFile.xml"); // here we send the files
logger.debug("EXPORTTIJD_PUT - Put to queue takes: " + (System.currentTimeMillis() - startTimeWrite));
// close session, producer and connection after 10 cycles
We also have run the performance test as a Single Instance AmazonMQ. But same results.
We have also run the performance test with a mq.m5.4xlarge (16 cpu, 96 GB RAM) engine but still no improvement of the bad performance.
Performance test configuration:
We first push the messages(XML files) according above one by one to a queue. We do that 5 times. After 5 times we read those messages(XML files) from the queue. We call this 1 cycle.
We run 10 cycles one after another, so in total we have pushed 300 files to the queue and we have getted 300 files from the queue.
We run 3 tests in parallel: One from AWS Region Londen, one from AWS Region Frankfurt in a different VPC and 1 from Frankfurt in the same VPC as the Amazon MQ broker and in the same subnet. Alle clients run on an EC2 instance: m4.xlarge.
If we run a test with only one VPC for example only the local VPC which is in the same subnet as the AmazonMQ broker the performance improves and we have these results:
256 KB = 72ms
4,6 MB = 381ms
9,3 MB = 980ms
18,7 MB = 2117ms
37,4 MB = 3985ms
74,8 MB = 7781ms
The client and server are in the same subnet, so we have nothing to do with firewalling etc.
Maybe somebody can tell me what is wrong, and why we have such a terrible performance with Amazon MQ or ActiveMQ?
extra info:
Response times are measured in the JMS Java app with Java starttime just before the producer.send('XML') and just endtime just after the producer.send('XML'). Difference is the recorded time. Times are average times over 300 calls.
IBM MQ server is located in our datacenter, and client app is running at a server in the same datacenter.
extra info test:
The jms app starts create connectionFactory queues sessions. Then it uploads the files to MQ 1 by 1. This is a cycle, then it run this cycle 10 times in a for lus without opening or closing sessions queues or connectionfactorys. Then all 60 messages are read from queue and written to files on the local drive. Then it closes the connection factory and session and producer/consumer. This is one batch.
Then we run 5 batches. So between the batches connectionFactory, queue, session are recreated.
In response to Sam:
When I also execute the test with the same size of files like you did Sam I approach the same response times, I set the persistence mode also to false value between () :
500 KB = 30ms (6ms)
1 MB = 50ms (13ms)
2 MB = 100ms (24ms)
I removed the connection pooling and I set
concurrentStoreAndDispatchQueues="false"
The system I have used broker: mq.m5.2xlarge and client: m4.xlarge.
But if I test with bigger files, this are the response times:
256 KB = 72ms
4,6 MB = 381ms
9,3 MB = 980ms
18,7 MB = 2117ms
37,4 MB = 3985ms
74,8 MB = 7781ms
I am having a very simple requirement. I have a system what puts messages on a queue and the messages are get from the queue by another system, sometimes at the same time sometimes not, sometimes there are 20 or 30 messages on the system before they get unloaded. Thats why I need a queue and messages must be persistent and it must be a Java JMS implementation.
I think Amazon MQ might be a solution for small files but for big files it is not. I think we have to use IBM MQ for this case which has better performance. But one important thing: I tested IBM MQ only on premis in our LAN. We tried to test IBM MQ on Amazon but we didn't succeed yet.
I tried to reproduce the scenario you were testing. When I ran a JMS client in the same VPC as the AmazonMQ broker for mq.m5.4xlarge broker with an Active and Standby instance, I see the following roundtrip latencies - measuring the moment from which a producer sends a message to the moment when consumer receives the message.
2MB - 50ms
1MB - 31ms
500KB - 15ms
My code just created a connection and a session. I did not use a PooledConnectionFactory (stating this as a matter of fact, not saying/suspecting that's the cause). Also it is better to strip down the code to bare minimum in order to establish a baseline and remove noise when doing performance testing. That way, when you introduce additional code, you can easily see if the new code introduced a performance issue. I used the default broker configuration.
In ActiveMQ, there is a concept of Fast Producer and Fast Consumer, this means, if consumer can process the messages at the same rate as the Producer, the broker transfers the message from producer to consumer via memory and then it writes the message to disk. This is the default behavior and is controlled by a broker configuration setting named concurrentStoreAndDispatch which is true (default)
If consumer is unable to keep up with producer, and thus becomes a "slow" consumer and with the concurrentStoreAndDispatch flag set to true, you take a performance hit.
ActiveMQ provides advisory topics which you can subscribe to detect slow consumers. If in fact, you detected that the consumer is slower than the producer, it is better to set concurrentStoreAndDispatch flag to false to get better performance.
I don't get any response.
I think its because there is no solution for this performance problem. Amazon MQ is a cloud service and mabye thats the reason why performance is this bad.
IBM MQ is a different architecture, and it is on premise.
I have to investigate the performance of ActiveMQ some more before I can tell what exactly the reason is for this problem.
I was facing this issue for my springboot application that connects to a DB and MQ, and uses Atomikos Transaction manager.
com.atomikos.jms.AtomikosJMSException|Connection pool exhausted - try increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the AtomikosConnectionFactoryBean.
com.atomikos.datasource.pool.PoolExhaustedException: ConnectionPool: pool is empty - increase either maxPoolSize or borrowConnectionTimeout
at com.atomikos.datasource.pool.ConnectionPool.waitForAtLeastOneAvailableConnection(ConnectionPool.java:326)
at com.atomikos.datasource.pool.ConnectionPool.findOrWaitForAnAvailableConnection(ConnectionPool.java:144)
at com.atomikos.datasource.pool.ConnectionPool.borrowConnection(ConnectionPool.java:132)
at com.atomikos.datasource.pool.ConnectionPoolWithSynchronizedValidation.borrowConnection(ConnectionPoolWithSynchronizedValidation.java:23)
at com.atomikos.jms.AtomikosConnectionFactoryBean.createConnection(AtomikosConnectionFactoryBean.java:601)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:196)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$100(AbstractPollingMessageListenerContainer.java:77)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createConnection(AbstractPollingMessageListenerContainer.java:490)
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:281)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:245)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
at java.lang.Thread.run(Thread.java:748)
I tried printing the maxPoolSize and found that it is 1. This page came across in between (https://www.atomikos.com/Documentation/ConfiguringJms) and I found the line where they increased the MaxPoolSize to 5. I just tried setting it to 2 and it worked.
AtomikosConnectionFactoryBean xaConnectionFactory = new AtomikosConnectionFactoryBean();
xaConnectionFactory.setXaConnectionFactory(ibmMQXAConnectionFactory);
xaConnectionFactory.setMaxPoolSize(2);
Can someone help me to understand what should be the ideal poolsize. what it is for etc?
In order to process messages Atomikos uses DB and JMS connections (in your case).
These connections are taken from the pools of available connections. To get the idea why connection pools are needed, please follow this link as a starting point - Connection_pool
To put it simple - in order to process one message at a time Atomikos needs one DB and one JMS connection/session. So if you plan to process 10 messages in parallel, each connection pool size must be at least 10 (10 for DB and 10 for JMS connection pools respectively).
Does HikariCP supports command timeout in Spring Boot application similar to C#
I am using Hikari Connection Pool in my Spring boot application. I have enabled connectionTimeout using the following configuration
spring.datasource.hikari.connectionTimeout: 30000
If I increase the number of concurrent users I'm getting the following exception in logs
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection;
nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
I am perfectly find with the above exception. I can increase the number of connections. But my concern is, there are few endpoints which are taking more than 2 minutest to respond. Those endpoints got DB Connection from pool but took more time to process. Is there a setting in which I can mentioned some timeout so that if DB takes more than some time (Operation time -Say 40 seconds) then it should send SQL exception. Similar to command timeout in C#
Application.properties
# A list of all Hikari parameters with a good explanation is available on https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby
# This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. Default: same as maximumPoolSize
spring.datasource.hikari.minimumIdle: 10
# This property controls the maximum size that the pool is allowed to reach, including both idle and in-use connections. Basically this value will determine the maximum number of actual connections to the database backend.
# Default: 10
spring.datasource.hikari.maximumPoolSize: 20
#This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown.
#Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
spring.datasource.hikari.connectionTimeout: 30000
# This property controls the maximum amount of time that a connection is allowed to sit idle in the pool. This setting only applies when minimumIdle is defined to be less than maximumPoolSize
# Default: 600000 (10 minutes)
spring.datasource.hikari.idleTimeout: 600000
# This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed.
# Default: 1800000 (30 minutes)
spring.datasource.hikari.maxLifetime: 1800000
# This property sets a SQL statement that will be executed after every new connection creation before adding it to the pool. Default: none
spring.datasource.hikari.connectionInitSql: SELECT 1 FROM DUAL
As explained in this comment, seems that connectionTimeout value is used for acquire connection timeout.
You can try setting that to a higher value than 30 secs and see if that helps.
Druid connection pool have this setting that allows you to set up the timeout:
spring.datasource.druid.query-timeout=10000
We're running an enterprise scale SAP application with front-end springboot clients connecting via Jco adapter 3.0 on Oracle VM using the connection pool (size 100). We're experiencing unsystematic long-running requests > 10s that are not visible in the SAP application server log, i.e. the bottleneck does not appear to be on SAP side.
Looking at the trace files (level 4) for an example request we can see that the time seems lost when the adapter thread tries to get the client from the pool (other threads continue execution, removed the irrelevant threads for clarity):
[20:05:50:259]: [JCoAPI] JCoContext.isStateful(P-foo-CPIC0) in session ID Client-53-1 returns false
[20:05:50:259]: [JCoAPI] JCoContext.begin(P-foo-CPIC0) in session ID Client-53-1
[20:05:50:259]: [JCoAPI] Started context for session Client-53-1
[20:05:50:259]: [JCoAPI] JCoContext.begin() for destination PFOO_200 (P-foo-CPIC0) on context with id Client-53-1; current state counter is 1
[20:05:50:259]: [JCoAPI] destination PFOO_200 destinationID=P-foo-CPIC0 executes Z_foo sessionID=Client-53-1, threadID=0x35
[20:05:50:259]: [JCoAPI] Context.getConnection on destination PFOO_200 (state: destination = STATEFUL, default = STATELESS)
[20:05:50:259]: [JCoAPI] PoolingFactory.getClient() on pool P-foo-CPIC0
--> time lost here
[20:06:20:840]: [JCoAPI] PoolingFactory.getClient() returns handle [3/84977415]
[20:06:20:840]: [JCoAPI] Context.getConnection on destination PFOO_200 nothing found in the context - got client from ConnectionManager [3/84977415]
[20:06:20:840]: [JCoAPI] JCoClient before execute(Z_foo) on handle [3/84977415]
[20:06:20:840]: [JCoRFC] Executing function Z_foo on handle [3/84977415]
[20:06:20:866]: [JCoAPI] JCoClient after execute(Z_foo) on handle [3/84977415] returns after 26 ms
[20:06:20:866]: [JCoAPI] Context.releaseConnection on destination PFOO_200 [3/84977415]
[20:06:20:867]: [JCoAPI] JCoContext.end(P-foo-CPIC0) in session ID Client-53-1
[20:06:20:867]: [JCoAPI] PoolingFactory.releaseClient() handle [3/84977415] into pool P-foo-CPIC0 [pool size: 3, peak limit: 100, waiting threads: 0, currently used: 1]
[20:06:20:879]: [JCoAPI] Finished context for session Client-53-1
[20:06:20:879]: [JCoAPI] JCoContext.end() for destination PFOO_200 (P-foo-CPIC0) on context with id Client-53-1; current state counter is 0
For a typical request the step is handled in milliseconds.
Are there any known limitations or configurations regarding pool handling for the Jco adapter, either on adapter or on SAP side?
Update we've on Jco adapter 3.0.16 and will double-check 3.0.17 now. DNS seems unlikely since we're monitoring dig/nslookup and they're running without delays.
Which JCo patch level do you use?
Did you try to update to the latest JCo patch level 3.0.17 first?
In your time gap the RFC connection will be opened and the RFC logon will be done, if the pool is empty at that time. Did you have a closer look with a higher trace level, or did you have a look into the RFC trace?
This can be anything from not having a free dialog work process at ABAP side, to SAP system database issues (required for the RFC logon authentication checks), slow response times from the SAP message server (if using load balanced logons), SNC handshake issues (if using SNC) or general network issues with the DNS (try using the IP address instead of a hostname).
Another point worth checking: you say your connection pool has size 100. Is it possible, that your program has more than 100 threads? Then it may happen from time to time, that all connections are currently busy in other threads and the current thread has to wait until a function call in another thread completes and a connection is returned to the pool.
(How long a thread waits on an empty pool can be customized via the "pool wait time" parameter.)