Spring custom SQL error code translator not catching connection exception - spring

Question

Using Spring JdbcTemplate with HikariCP and JTDS to connect to MS SQL Server database in legacy code that I'm maintaining. I'm trying to handle the case where the application has started and later for whatever reason the database becomes unavailable and a connection exception is thrown.
I already have a custom SQL error code translator (below) that captures many other exceptions thrown by jdbcTemplate. But the connection refused exception is not being caught even though it is being encapsulated as a java.sql.SQLException (see the stack trace).
Why is the translator not capturing this exception?
public class CustomSqlErrorCodeTranslator extends
SQLErrorCodeSQLExceptionTranslator {
private final Log log = LogFactory.getLog(getClass());
#Override
protected DataAccessException customTranslate
(String task, String sql, SQLException sqlException) {
/*
* We want to capture all codes.
*/
StringBuilder logMsg = new StringBuilder("*********************SQL Exception:");
logMsg.append(", errorCode: " + sqlException.getErrorCode())
.append(", message: " + sqlException.getMessage())
;
log.error(logMsg.toString());
return null;
}
}
public class Dao {
//HikariDataSource dataSource;
DataSource dataSource;
protected JdbcTemplate jdbcTemplate;
private final Log log = LogFactory.getLog(getClass());
private boolean connected = false;
private CustomSqlErrorCodeTranslator exceptionHandler;
public void setDataSource(DataSource ds) {
//this.jdbcTemplate = new JdbcTemplate(ds);
try {
this.jdbcTemplate = new JdbcTemplate(ds);
exceptionHandler = new CustomSqlErrorCodeTranslator();
this.jdbcTemplate.setExceptionTranslator(exceptionHandler);
connected = true;
} catch (Exception e) {
if (e.getCause() != null && e.getCause() instanceof SQLException) {
log.error("******Cannot connect to database.***************************");
log.debug(e.getMessage());
}
}
}
}
16:59:33,542 [main] WARN PoolBase - springHikariCP - Connection net.sourceforge.jtds.jdbc.JtdsConnection#42206bce failed alive test with exception I/O Error: Connection reset by peer: socket write error
16:59:33,542 [Hikari connection closer (pool springHikariCP)] DEBUG PoolBase - springHikariCP - Closing connection net.sourceforge.jtds.jdbc.JtdsConnection#42206bce: (connection evicted or dead)
16:59:34,530 [main] DEBUG HikariPool - Timeout failure pool springHikariCP stats (total=0, active=0, idle=0, waiting=0)
16:59:34,544 [Hikari connection adder (pool springHikariCP)] DEBUG HikariPool - springHikariCP - Cannot acquire connection from data source
java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:436)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:288)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:251)
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:331)
... 13 more
16:59:35,746 [Hikari connection adder (pool springHikariCP)] DEBUG HikariPool - springHikariCP - Cannot acquire connection from data source
java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:436)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:288)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:251)
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:331)
... 13 more
16:59:37,010 [Hikari connection adder (pool springHikariCP)] DEBUG HikariPool - springHikariCP - Cannot acquire connection from data source
Sorry that it took me this long to realize I needed to add to the stack trace.
Here is the stack trace when I stop the database. Now it is clear that JDBC Template is involved. And if JdbcTemplate is involved it seems to me the custom exception handler should pick it up. Has anyone else had any experience capturing the database connection exception using the custom error code translator?
5:06:05,076 [ActiveMQ Session Task-1] DEBUG HikariPool - Timeout failure pool HikariPool-0 stats (total=1, active=1, idle=0, waiting=2)
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 10000ms.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:866)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:927)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:937)
at com.pevco.persist.resources.AvailabilityDaoImpl.create(AvailabilityDaoImpl.java:51)
at com.pevco.nexgen.health.handlers.SystemModelListener.onMessage(SystemModelListener.java:135)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:746)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:684)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:622)
at org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:330)
at org.springframework.jms.listener.SimpleMessageListenerContainer$2.onMessage(SimpleMessageListenerContainer.java:306)
at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1321)
at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 10000ms.
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 20 more
Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:436)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:288)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:251)
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:331)
... 12 more
15:06:06,217 [ActiveMQ Session Task-1] DEBUG HikariPool - Timeout failure pool HikariPool-0 stats (total=1, active=1, idle=0, waiting=2)
15:06:06,217 [ActiveMQ Session Task-1] WARN SimpleMessageListenerContainer - Execution of JMS message listener failed, and no ErrorHandler has been set.
(I'll award bounty before time expires, but would like to get the accurate answer.)

Answer 1

As "reos" pointed out, there is nothing related to JdbcTemplate in the stack trace. That means you tried to open the connection outside the JdbcTemplate.
Even if you used the JdbcTemplate, the exception would not get translated. Look at the execute method in the source code. They don't use the custom exception translator if the data source fails to obtain the connection.

Answer 2

The problem is because the connection is being asked by Hikari component. Probably Hikari lost the connection and it is trying to reconnect.
In the log there is not any part related to the JdbcTemplate. That is because the problem is cause by a reconnection try and not because a new transaction.
If the transaction is not passing by the template the error translator is never called. The event that throws the exception is no fired by the business logic. It is fired by the reconnect or create connection functionality of Hikari