Why are my connections not returning to the pool with Hikari? - spring

Question

We have this database layer with JdbcTemplate:
#Override
#Transactional
protected SomeObj insert(SomeObj obj) {
this.getNamedParameterJdbcTemplate().update(INSERT_QUERY, MyJdbcUtils.prepareQueryParams(obj));
synchronized (obj.getSomeList()) { decidable.getReasons().forEach(someItem -> { if (someItem != null) getJdbcTemplate().update(INSERT_ITEM_QUERY, obj.getId(), someItem ); }); }
return obj;
}
#Override
#Transactional
public SomeObj update(SomeObj obj) {
this.getNamedParameterJdbcTemplate().update(UPDATE_QUERY, MyJdbcUtils.prepareQueryParams(obj));
synchronized (obj.getSomeList()) { decidable.getReasons().forEach(someItem -> { if (someItem != null) getJdbcTemplate().update(UPDATE_ITEM_QUERY, obj.getId(), someItem ); }); }
return obj;
}
#Override
public SomeObj upsert(SomeObj obj) {
try {
return this.insert(obj);
} catch (DuplicateKeyException exception) {
final SomeObj existingObj = this.find(obj.getSomeId()).get(0);
// Populate some fields from existing object
}
return this.update(transaction);
}
MyJdbcUtils is as follows:
public static Map<String, ?> prepareQueryParams(SomeObj obj) {
return new HashMap<String, Object>() {{
put("someColumn", obj.getSomeColumn());
...
}};
}
My database related application.properties are as follows:
spring.datasource.jdbc-url=...
spring.datasource.username=...
spring.datasource.password=...
spring.datasource.initialization-mode=never
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.leakDetectionThreshold=5000
spring.datasource.maximumPoolSize=100
spring.datasource.minimumIdle=20
spring.datasource.idleTimeout=30000
After a while, Hikari pool logs looks like this:
HikariPool-1 - Timeout failure stats (total=100, active=100, idle=0, waiting=2)
with a lot of:
Connection leak detection triggered for org.postgresql.jdbc.PgConnection#17b19afa on thread http-nio-8080-exec-100, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128) ~[HikariCP-3.4.1.jar!/:?]
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83) ~[hibernate-core-5.4.6.Final.jar!/:5.4.6.Final]
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:184) ~[spring-orm-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:402) ~[spring-orm-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at com.company.project.core.domain.transaction.JdbcTransactionDaoImpl$$EnhancerBySpringCGLIB$$87fab16e.update(<generated>) ~[classes!/:?]
at com.company.project.core.integration.spring.TransactionIntegrationConfiguration.asyncPersistenceService(TransactionIntegrationConfiguration.java:208) ~[classes!/:?]
at sun.reflect.GeneratedMethodAccessor326.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171) ~[spring-messaging-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120) ~[spring-messaging-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:508) ~[spring-integration-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:478) ~[spring-integration-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.integration.gateway.GatewayProxyFactoryBean$Invoker.get(GatewayProxyFactoryBean.java:1012) ~[spring-integration-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_212]
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:341) ~[spring-context-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:451) ~[spring-integration-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at com.sun.proxy.$Proxy181.processTransaction(Unknown Source) ~[?:?]
at com.company.project.core.services.transaction.TransactionServiceImpl.async(TransactionServiceImpl.java:101) ~[classes!/:?]
at com.company.project.core.services.transaction.TransactionServiceImpl$$FastClassBySpringCGLIB$$73b9cf27.invoke(<generated>) ~[classes!/:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at com.company.project.core.services.transaction.TransactionServiceImpl$$EnhancerBySpringCGLIB$$eae305f2.async(<generated>) ~[classes!/:?]
at com.company.project.core.controllers.TransactionController.async(TransactionController.java:52) ~[classes!/:?]
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.27.jar!/:9.0.27]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
until:
HikariPool-1 - Connection is not available, request timed out after 30000ms
DBA confirms that the longest query is ~1second and I'm sure that the queries should run around that long. What am I doing wrong? Why aren't my connections returning to the pool and are still marked as active?
Update:
As per comments, I switched the logic to:
private final TransactionTemplate transactionTemplate;
#Autowired
public MyDao (PlatformTransactionManager transactionManager, DataSource dataSource) {
setDataSource(dataSource); // Class extends NamedParameterJdbcDaoSupport
this.transactionTemplate = new TransactionTemplate(transactionManager);
}
#Override
protected SomeObj insert(SomeObj obj) {
return this.transactionTemplate.execute(status -> {
this.getNamedParameterJdbcTemplate().update(INSERT_QUERY, MyJdbcUtils.prepareQueryParams(obj));
for (String someItem : obj.getSomeList()) if (someItem != null) getJdbcTemplate().update(INSERT_ITEM_QUERY, obj.getId(), someItem );
return obj;
});
}
#Override
public SomeObj update(SomeObj obj) {
return this.transactionTemplate.execute(status -> {
this.getNamedParameterJdbcTemplate().update(UPDATE_QUERY, MyJdbcUtils.prepareQueryParams(obj));
for (String someItem : obj.getSomeList()) if (someItem != null) getJdbcTemplate().update(UPDATE_ITEM_QUERY, obj.getId(), someItem );
return obj;
});
}
However the issue did not resolve.