Does HikariCP supports command timeout in Spring Boot application similar to C# - spring

Does HikariCP supports command timeout in Spring Boot application similar to C#
I am using Hikari Connection Pool in my Spring boot application. I have enabled connectionTimeout using the following configuration
spring.datasource.hikari.connectionTimeout: 30000
If I increase the number of concurrent users I'm getting the following exception in logs
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection;
nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
I am perfectly find with the above exception. I can increase the number of connections. But my concern is, there are few endpoints which are taking more than 2 minutest to respond. Those endpoints got DB Connection from pool but took more time to process. Is there a setting in which I can mentioned some timeout so that if DB takes more than some time (Operation time -Say 40 seconds) then it should send SQL exception. Similar to command timeout in C#
Application.properties
# A list of all Hikari parameters with a good explanation is available on https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby
# This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. Default: same as maximumPoolSize
spring.datasource.hikari.minimumIdle: 10
# This property controls the maximum size that the pool is allowed to reach, including both idle and in-use connections. Basically this value will determine the maximum number of actual connections to the database backend.
# Default: 10
spring.datasource.hikari.maximumPoolSize: 20
#This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown.
#Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
spring.datasource.hikari.connectionTimeout: 30000
# This property controls the maximum amount of time that a connection is allowed to sit idle in the pool. This setting only applies when minimumIdle is defined to be less than maximumPoolSize
# Default: 600000 (10 minutes)
spring.datasource.hikari.idleTimeout: 600000
# This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed.
# Default: 1800000 (30 minutes)
spring.datasource.hikari.maxLifetime: 1800000
# This property sets a SQL statement that will be executed after every new connection creation before adding it to the pool. Default: none
spring.datasource.hikari.connectionInitSql: SELECT 1 FROM DUAL

As explained in this comment, seems that connectionTimeout value is used for acquire connection timeout.
You can try setting that to a higher value than 30 secs and see if that helps.

Druid connection pool have this setting that allows you to set up the timeout:
spring.datasource.druid.query-timeout=10000

Related

ConnectionPool: pool is empty - increase either maxPoolSize or borrowConnectionTimeout

I was facing this issue for my springboot application that connects to a DB and MQ, and uses Atomikos Transaction manager.
com.atomikos.jms.AtomikosJMSException|Connection pool exhausted - try increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the AtomikosConnectionFactoryBean.
com.atomikos.datasource.pool.PoolExhaustedException: ConnectionPool: pool is empty - increase either maxPoolSize or borrowConnectionTimeout
at com.atomikos.datasource.pool.ConnectionPool.waitForAtLeastOneAvailableConnection(ConnectionPool.java:326)
at com.atomikos.datasource.pool.ConnectionPool.findOrWaitForAnAvailableConnection(ConnectionPool.java:144)
at com.atomikos.datasource.pool.ConnectionPool.borrowConnection(ConnectionPool.java:132)
at com.atomikos.datasource.pool.ConnectionPoolWithSynchronizedValidation.borrowConnection(ConnectionPoolWithSynchronizedValidation.java:23)
at com.atomikos.jms.AtomikosConnectionFactoryBean.createConnection(AtomikosConnectionFactoryBean.java:601)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:196)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$100(AbstractPollingMessageListenerContainer.java:77)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createConnection(AbstractPollingMessageListenerContainer.java:490)
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:281)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:245)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
at java.lang.Thread.run(Thread.java:748)
I tried printing the maxPoolSize and found that it is 1. This page came across in between (https://www.atomikos.com/Documentation/ConfiguringJms) and I found the line where they increased the MaxPoolSize to 5. I just tried setting it to 2 and it worked.
AtomikosConnectionFactoryBean xaConnectionFactory = new AtomikosConnectionFactoryBean();
xaConnectionFactory.setXaConnectionFactory(ibmMQXAConnectionFactory);
xaConnectionFactory.setMaxPoolSize(2);
Can someone help me to understand what should be the ideal poolsize. what it is for etc?
In order to process messages Atomikos uses DB and JMS connections (in your case).
These connections are taken from the pools of available connections. To get the idea why connection pools are needed, please follow this link as a starting point - Connection_pool
To put it simple - in order to process one message at a time Atomikos needs one DB and one JMS connection/session. So if you plan to process 10 messages in parallel, each connection pool size must be at least 10 (10 for DB and 10 for JMS connection pools respectively).

HikariPool-1 – Connection is not available, request timed out after 30000ms

I am using the standard HikariCP implementation in my SpringBoot 2.0.1 application. But after a while I get the same error over and over
HikariPool-1 – Connection is not available, request timed out after 30000ms
I first checked the code if there are any not closed connection or missing Transactional annotations but I did not find anything. I also tried then to increase the pool and decrease the time out in my application.yml but it seems that this does not have any effect.
The weird thing is that HikariCP seems to create only 4 pooled connection and I am not able to override these properties.
HikariPool-1 - Timeout failure stats (total=4, active=4, idle=0, waiting=100)
This is my application.yml file
spring:
jpa:
hibernate:
ddl-auto: update
datasource:
hikari:
maximum-pool-size: 64
connection-timeout: 5000
Your application is not able to get the connection within the 30s which is the default connection timeout for HikariCP. There can be reasons for that such as:
The Pool size is small, and the requests take too long to execute (more than 30 s).
Possible explanation is that for e.g your pool size is 1 and your one request/query takes approximately 500ms to execute. suppose you have opened for example approximately 200 concurrent query requests, then there might be a chance that some requests will be timed out. (one request has waited for the 30s but a connection was not available for it).
Disclaimer: Numbers are taken randomly to explain and is not tested.
To solve this you might want to increase the connectionTimeout or increase the pool size. ( Also check for the memory requirements).
In application.properties set the values
spring.datasource.hikari.minimumIdle=10
spring.datasource.hikari.maximumPoolSize=10
spring.datasource.hikari.idleTimeout=120000
spring.datasource.hikari.maxLifetime=500000
Official Hikari CP documentation link
There can be another reason for this is that your connection is not getting closed properly use try-with-resources to close your connection
e.g
try(Connection c = dataSource.getConnection()) {
//ToDo: Add your logic statement here
log.debug("[d] Tenant '{}' added.", tenantId);
}

Hikari connection is not getting timed out

I Have the following Hikari configuration in my spring boot app. Queries are taking more than the connection-timeout time set. However, timeout never happened. I am keeping as low as possible to simulate the connection timeout.
HikariConfig dataSourceConfig = new HikariConfig();
dataSourceConfig.setDriverClassName(config.driver);
dataSourceConfig.setJdbcUrl(config.url);
dataSourceConfig.setUsername(config.user);
dataSourceConfig.setPassword(config.password);
dataSourceConfig.setConnectionTestQuery(config.validationQuery);
dataSourceConfig.setMaximumPoolSize(config.poolMax);
dataSourceConfig.setConnectionTimeout(300);
dataSourceConfig.setIdleTimeout(10000);
dataSourceConfig.setMaxLifetime(60000);
JdbcTemplate jdbcTemplate = new JdbcTemplate(new HikariDataSource(dataSourceConfig));
Here is some log that shows the query ran for more than 300ms.
Time elapsed to run query......2913
Using Hikari 3.2 and mariadb
Thanks.
From: https://github.com/brettwooldridge/HikariCP
connectionTimeout
This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown. Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
So this property is more about how long your application will wait for a connection, not how long a query is allowed to execute.
I think what you want is "max_statement_time": https://mariadb.com/kb/en/library/server-system-variables/#max_statement_time
Maximum time in seconds that a query can execute before being aborted. This includes all queries, not just SELECT statements, but excludes statements in stored procedures. If set to 0, no limit is applied.

HikariPool-1 - Connection marked as broken because of SQLSTATE(08S01), ErrorCode(-99999)

HikariPool-1 - Connection marked as broken because of SQLSTATE(08S01), ErrorCode(-99999)
java.sql.SQLNonTransientConnectionException: Communication link failure. (Read timed out)
I kept getting this problem, when creating a table with data, which was about 6million records. On ACS, an sql developer, it took around 1½ minute.
I found out that default connection validation (spring.datasource.hikari.validationTimeout) was set to 5000 (5 seconds).
This lead to hikari, not being able to validate the connection, because it was busy for long time.
Solution was setting this property to a high enough number (i set it to 5minutes - 300000)

WebSphere MQ Connection Tuning

I have an application which uses MDB, activation specifications and Queue Connection Factories to get/put Messages from WMQ. The application expects a max load of 80 tps. Both Websphere Application Server and WMQ are clustered and each application server connects to seperate host and channel. The application onMessage method is implemented in way so that both session and connection gets closed after message is consumed and response is sent.
As per our configuration, we have WAS version is 8.5, IBM MQ queue manager version 7, max server sessions for act spec set to 40 for each node. max connection count in Connection Factory to 40 for each node and max session in session pool of connection factory to 10.
Now on peak load we expect to make max 80 MQ Channel Instance but as per the investigation we can see it goes above 200 which is causing an issue as max instance limit is reached.
Is this happening because max session in session pool of connection factory is set to 10?
Is it possible that even though we are closing the session and connection in onMessage, still one connection can have more than one session. If that is the case, is it wise to set this property to 1?
Also can there be some property set at WMQ which could cause this increase in MQ Channel Instances.
You don't mention specific versions of WAS or MQ, and there could be known problems at a specific version that would change the behavior, but in general it should work as described below.
IBM has a nice Technote "TCP/IP Connection usage between WebSphere Application Server V7 and V8, and WebSphere MQ V7 (and later) explained" which goes into detail on this subject.
You do not mention what you have the SVRCONN channel's SHARECNV set to, as per below this will impact the number of channel instances observed, I'll assume the default of 10 for the calculations.
Note that block quotes below are from the Technote
we have set max server sessions for act spec to 40 for each node
The link above states:
Maximum number of conversations = Maximum server sessions + 1
Maximum number of conversations = 40 + 1 = 41
The link also states:
Maximum number of TCP/IP channel instances = Maximum number of conversations / SHARECNV for the channel being used
Maximum number of TCP/IP channel instances = 41 / 10 = 5 (rounded up to nearest connection)
max connection count in Connection Factory to 40 for each node
max session in session pool of connection factory to 10.
Maximum number of conversations = Connection Pool Maximum Connections + (Connection Pool Maximum Connections * Session Pool Maximum Connections)
Maximum number of conversations = 40 + (40 * 10) = 440
Maximum number of TCP/IP channel instances = Maximum number of conversations / SHARECNV for the channel being used
Maximum number of TCP/IP channel instances = 440 / 10 = 44
If your MQ SVRCONN channel's SHARECNV was set to 10, then you should have no more than 49 channel instances for each channel based on each node connecting to a separate channel.
If you are hitting 200 channel instances I would suspect your SHARECNV is less than 10. If it was 1 the the maximum number of channel instances WAS would try to create would go up to 481 which would be limited by the MAXINST of the the channel to 200.
After an application has finished with a JMS Connection and closed it off, it is moved from the Active Pool to the Free Pool, where it is available for reuse. The Connection Pool property Unused timeout defines how long a JMS Connection will stay in the Free Pool before it is disconnected. This property has the default value of 1800 seconds, which is 30 minutes.
Every JMS Connection that is created from a WebSphere MQ messaging provider Connection Factory has an associated JMS Session Pool, which work in the same way as Connection Pools. The maximum number of JMS Sessions that can be created from a single JMS Connection is determined by the Connection Factory Session Pool property Maximum connections. The default value of this property is 10.
A conversation is started when a JMS Session is first created, and will remain active until the JMS Session is closed because it has remained in the Free Pool for longer than the value of the Session Pool's Unused timeout property.
When your app closes the session and connection in onMessage, the connection is moved to the free pool for reuse and the session is moved to the free pool for reuse, the MQ Channel instance will not be closed until the respective timeout is hit.
If you want to keep your maximum channel count less than 200 then you could tune your Session Pool Maximum Connections) to 1 which combined with your Activation Specifications and a SHARECNV(1) would max out at 121 channel instances.
You can also increase the SHARECNV value of the channel which will result in dividing the channel instances by that number.
It is possible that your connections or sessions are not getting closed properly and you have a "leak".

Resources