Unable to get a connection from pool of a PoolingDataSource - jdbc

Question

This production issue is now taking over my head. I have configured to use Bitronix as transaction manager with the jBPM5 tool. It works fine from a long time but sudden started breaking with below exception.
Exception stacktrace
Caused by: java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 1 connection(s) (1 still available)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:262)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 78 more
Caused by: bitronix.tm.internal.BitronixRuntimeException: cannot get valid connection from an XAPool of resource jdbc/jbpm-ds with 1 connection(s) (1 still available) after trying for 30s
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:160)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:258)
... 80 more
Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:899)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3657)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1495)
at bitronix.tm.resource.jdbc.JdbcPooledConnection.testConnection(JdbcPooledConnection.java:215)
at bitronix.tm.resource.jdbc.JdbcPooledConnection.getConnectionHandle(JdbcPooledConnection.java:299)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:130)
... 82 more
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at oracle.net.ns.Packet.receive(Packet.java:308)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:324)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:268)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:190)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:107)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:350)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:886)
... 90 more
Strange thing is that application has sufficient number of XA connections are available in XAPool. It takes the connection, tried to connect, unable to connect and just discard the connection and take new connection to process.
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 9 connection(s) (9 still available)
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 8 connection(s) (8 still available)
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 7 connection(s) (7 still available)
Could anybody please shed some light to recover from this issue?
Update1:
After some try I am able to reproduce the same issue in local. Once database connection is resetted or network connectivity lost between my application & database, pool connections there in PoolingDataSource (provided by Bitronix) pool becomes invalid. My query is Why it is unable auto-recover? Is there any why through which we can auto-recover or establish the new connections?
I am using btm version 2.1.4.
Thanks,
Mayank

Answer 1

I was having the same issue but it was due to user permissions. My user does not have enough permissions so I was getting this issue. After that, I ran these 3 below-mentioned commands to create a new user and it works for me. I hope, it helps someone.
CREATE USER 'newuser'#'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON * . * TO 'newuser'#'localhost';
FLUSH PRIVILEGES;