HikariPool-1 - Connection marked as broken because of SQLSTATE(08S01), ErrorCode(-99999)
java.sql.SQLNonTransientConnectionException: Communication link failure. (Read timed out)
I kept getting this problem, when creating a table with data, which was about 6million records. On ACS, an sql developer, it took around 1½ minute.
I found out that default connection validation (spring.datasource.hikari.validationTimeout) was set to 5000 (5 seconds).
This lead to hikari, not being able to validate the connection, because it was busy for long time.
Solution was setting this property to a high enough number (i set it to 5minutes - 300000)
Related
I am using the standard HikariCP implementation in my SpringBoot 2.0.1 application. But after a while I get the same error over and over
HikariPool-1 – Connection is not available, request timed out after 30000ms
I first checked the code if there are any not closed connection or missing Transactional annotations but I did not find anything. I also tried then to increase the pool and decrease the time out in my application.yml but it seems that this does not have any effect.
The weird thing is that HikariCP seems to create only 4 pooled connection and I am not able to override these properties.
HikariPool-1 - Timeout failure stats (total=4, active=4, idle=0, waiting=100)
This is my application.yml file
spring:
jpa:
hibernate:
ddl-auto: update
datasource:
hikari:
maximum-pool-size: 64
connection-timeout: 5000
Your application is not able to get the connection within the 30s which is the default connection timeout for HikariCP. There can be reasons for that such as:
The Pool size is small, and the requests take too long to execute (more than 30 s).
Possible explanation is that for e.g your pool size is 1 and your one request/query takes approximately 500ms to execute. suppose you have opened for example approximately 200 concurrent query requests, then there might be a chance that some requests will be timed out. (one request has waited for the 30s but a connection was not available for it).
Disclaimer: Numbers are taken randomly to explain and is not tested.
To solve this you might want to increase the connectionTimeout or increase the pool size. ( Also check for the memory requirements).
In application.properties set the values
spring.datasource.hikari.minimumIdle=10
spring.datasource.hikari.maximumPoolSize=10
spring.datasource.hikari.idleTimeout=120000
spring.datasource.hikari.maxLifetime=500000
Official Hikari CP documentation link
There can be another reason for this is that your connection is not getting closed properly use try-with-resources to close your connection
e.g
try(Connection c = dataSource.getConnection()) {
//ToDo: Add your logic statement here
log.debug("[d] Tenant '{}' added.", tenantId);
}
Does HikariCP supports command timeout in Spring Boot application similar to C#
I am using Hikari Connection Pool in my Spring boot application. I have enabled connectionTimeout using the following configuration
spring.datasource.hikari.connectionTimeout: 30000
If I increase the number of concurrent users I'm getting the following exception in logs
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection;
nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
I am perfectly find with the above exception. I can increase the number of connections. But my concern is, there are few endpoints which are taking more than 2 minutest to respond. Those endpoints got DB Connection from pool but took more time to process. Is there a setting in which I can mentioned some timeout so that if DB takes more than some time (Operation time -Say 40 seconds) then it should send SQL exception. Similar to command timeout in C#
Application.properties
# A list of all Hikari parameters with a good explanation is available on https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby
# This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. Default: same as maximumPoolSize
spring.datasource.hikari.minimumIdle: 10
# This property controls the maximum size that the pool is allowed to reach, including both idle and in-use connections. Basically this value will determine the maximum number of actual connections to the database backend.
# Default: 10
spring.datasource.hikari.maximumPoolSize: 20
#This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown.
#Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
spring.datasource.hikari.connectionTimeout: 30000
# This property controls the maximum amount of time that a connection is allowed to sit idle in the pool. This setting only applies when minimumIdle is defined to be less than maximumPoolSize
# Default: 600000 (10 minutes)
spring.datasource.hikari.idleTimeout: 600000
# This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed.
# Default: 1800000 (30 minutes)
spring.datasource.hikari.maxLifetime: 1800000
# This property sets a SQL statement that will be executed after every new connection creation before adding it to the pool. Default: none
spring.datasource.hikari.connectionInitSql: SELECT 1 FROM DUAL
As explained in this comment, seems that connectionTimeout value is used for acquire connection timeout.
You can try setting that to a higher value than 30 secs and see if that helps.
Druid connection pool have this setting that allows you to set up the timeout:
spring.datasource.druid.query-timeout=10000
I Have the following Hikari configuration in my spring boot app. Queries are taking more than the connection-timeout time set. However, timeout never happened. I am keeping as low as possible to simulate the connection timeout.
HikariConfig dataSourceConfig = new HikariConfig();
dataSourceConfig.setDriverClassName(config.driver);
dataSourceConfig.setJdbcUrl(config.url);
dataSourceConfig.setUsername(config.user);
dataSourceConfig.setPassword(config.password);
dataSourceConfig.setConnectionTestQuery(config.validationQuery);
dataSourceConfig.setMaximumPoolSize(config.poolMax);
dataSourceConfig.setConnectionTimeout(300);
dataSourceConfig.setIdleTimeout(10000);
dataSourceConfig.setMaxLifetime(60000);
JdbcTemplate jdbcTemplate = new JdbcTemplate(new HikariDataSource(dataSourceConfig));
Here is some log that shows the query ran for more than 300ms.
Time elapsed to run query......2913
Using Hikari 3.2 and mariadb
Thanks.
From: https://github.com/brettwooldridge/HikariCP
connectionTimeout
This property controls the maximum number of milliseconds that a client (that's you) will wait for a connection from the pool. If this time is exceeded without a connection becoming available, a SQLException will be thrown. Lowest acceptable connection timeout is 250 ms. Default: 30000 (30 seconds)
So this property is more about how long your application will wait for a connection, not how long a query is allowed to execute.
I think what you want is "max_statement_time": https://mariadb.com/kb/en/library/server-system-variables/#max_statement_time
Maximum time in seconds that a query can execute before being aborted. This includes all queries, not just SELECT statements, but excludes statements in stored procedures. If set to 0, no limit is applied.
A JBoss 5.2 application server log was filled with thousands of the following exception:
Caused by: javax.resource.ResourceException: Unable to get managed connection for jdbc_TestDB
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:441)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:96)
... 9 more
Caused by: javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] )
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:311)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:689)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
... 13 more
I've stripped off the first part of the exception, which is basically our internal JDBC wrapper code which tries to get a DB connection from the pool.
Looking at the Oracle DB side I ran the query:
select resource_name, current_utilization, max_utilization, limit_value
from v$resource_limit
where resource_name in ('sessions', 'processes');
This produced the output:
RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION LIMIT_VALUE
processes 1387 1500 1500
sessions 1434 1586 2272
Given the fact that that PROCESSES limit of 1500 was reached, would this cause the JBoss exceptions we experienced? I've also been investigating the possibility of connection leaks, but haven't found any evidence of that so far.
What is the recommended course of action here? Is simply increasing the limit a valid solution?
Usually when max_utilization gets the processes value listener will refuse new connections to database. you can see the errors relates to it in alert log. to solve this in database side you should increase the processes parameter.
hmm strange. is it possible, that exception wrapping in JBOSS hides the original error? You should get some sql exception whose text starts with ORA-. Maybe your JDBC wrapper does not handle errors properly.
The recommended actions is to:
check configured size of connection pool against processes sessions Oracle startup paramters.
check Oracles view v$session, especially columns STATUS, LAST_CALL_ET, SQL_ID, PREV_SQL_ID.
translate sql_id(prev_sql_id) into sql_text via v$sql.
if you application has a connection leak, sql_id and pred_sql_id might point you onto a place in your source code, where a connection was used last (i.e. where it was leaked).
In our application, we are facing an issue where in for certain hibernate queries, the queries are taking longer (sometimes not completing) than usual and when profiled using a profiler, we are observing that the connection objects related to these queries are open but not closed.
Because of this behavior, eventually the application runs out of connections and goes into high CPU and heap utilization.
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <3a685292> (a oracle.jdbc.pool.OracleImplicitConnectionCache)
at oracle.jdbc.pool.OracleImplicitConnectionCache.processConnectionWaitTimeout(OracleImplicitConnectionCache.java:2955)
at oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(OracleImplicitConnectionCache.java:374)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:374)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:178)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:156)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy75.prepareStatement(Unknown Source)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1854)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1831)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1811)
at org.hibernate.loader.Loader.doQuery(Loader.java:899)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:311)
at org.hibernate.loader.Loader.loadEntity(Loader.java:2111)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:82)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:72)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3917)
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:460)
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:429)
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:206)
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:262)
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:150)
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1091)
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:174)
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2473)
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:991)
at org.hibernate.event.internal.DefaultMergeEventListener.entityIsDetached(DefaultMergeEventListener.java:271)
at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:151)
at org.hibernate.event.internal.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:76)
at org.hibernate.internal.SessionImpl.fireMerge(SessionImpl.java:913)
at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:897)
at org.hibernate.internal.SessionImpl.merge(SessionImpl.java:901)
In such a scenario could you please suggest what kind of timeout property is preferable for the connection cache.
As per JDBC documentation, we came across the following properties, please help:
InactivityTimeout
TimeToLiveTimeout
AbandonedConnectionTimeout
Ref: http://docs.oracle.com/cd/B14117_01/java.101/b10979/conncache.htm#CDEBCBJC
Please use Oracle Universal Connection Pool for Java (UCP) a replacement for the Implicit Connection Cache (ICC) which has been de-supported in Oracle Database 12c. The documentation is found on OTN.