I have a spring application where its home page fire multiple ajax calls which in turn fetch data from the DB and return back. The DB has been configured with connection pooling with minPoolSize as 50 and maxPoolSize as 100.
now when I open the home page, around 7 connections are established with the DB, which is expected as around 7 ajax calls are made and I assume all create their own connection. Now when I refresh the page, I see 7 more new connection are established (I see total 14 physical connections from db2 monitoring), which seems to be unexpected, as I assume jdbcTemplate do close the connection after the query from the first access and refresh in this case should reuse the connections ?
Now question here is does resultsets are also getting closed by jdbcTemplate along with connection close ? or Do i need to explicitly close the resultSet so that connection can be closed automatically. Opened resultSet may be a reason of connection not getting close ? Attaching the code for connection pooling configuration
<dataSource jdbcDriverRef="db2-driver" jndiName="jdbc/dashDB-Development" transactional="true" type="javax.sql.DataSource">
<properties.db2.jcc databaseName="BLUDB" id="db2-dashDB-Development-props" password="********" portNumber="*****" serverName="*********" sslConnection="false" user="*****"/>
<connectionManager id="db2-DashDB-Development-conMgr" maxPoolSize="100" minPoolSize="50" numConnectionsPerThreadLocal="2"/>
My initial theory was that the reuse of the connection will happen only when minPoolSize is reached and till that time it will always create new physical connection. HOwever I see this behavior even after reaching that limit. I refreshed my page 10 time and I see 70 physical connections. Now my only doubt is that connection are somehow are not getting close and spring is seeing those connection busy ? This may be because resultsets are not closed or some other reason ? Is it a way to say jdbctemplate not to wait for closing resultset beyond a time limit ?
Thanks
Manoj
If you look at the org.springframework.jdbc.core.JdbcTemplate.query method source code you see calls to -
JdbcUtils.closeResultSet(rs);
In the finally blocks - so yes JDBCTemplate does call rs.close
The template also closes or returns the connection to the pool
Related
In EAR application running on Websphere application server 8.5.5 we have to execute CallableStatement (call stored procedure in Oracle DB) which runs more than five minutes or more depending on input data. The operation is automatically rolled back because of transaction timeout (code WTRN0006W) which is set to 120 seconds by default in Websphere AS. We can't change this value due to customers requirements.
We can split input data to smaller chunks and execute CallableStatement several times to achieve shorter run time (30 seconds or so). Processing whole data chunks still takes more than 120 seconds (as expected). But the transaction timeout still occurs although for every statement execution with small chunk (in loop) we are getting connection from datasource configured in WAS, set autocommit to false, after statement execution doing commit and closing connection. Then again with next chunk in next loop cycle.
The whole process of statement executions is done in Stateless EJB which is called from Singleton EJB scheduled to run twice a day. We are not using JTA neither JPA, just JDBC.
Is it possible to avoid transaction timeout if we execute statement several times?
How we obtain datasource during application start:
javax.naming.Context ctx = new InitialContext();
javax.sql.Datasource ds = (javax.sql.Datasource) ctx.lookup("jndi/datasource1");
How we obtain Connection:
java.sql.Connection conn = m24sb.getConnection();
conn.setAutoCommit(false)
How we execute statement:
try (CallableStatement sta = conn.prepareCall("{ call SOME_STORED_PROC }"))) {
// ... setting statement params
sta.execute();
// ... resolving returned values
}
and then commit and closing connection.
Thanks in advance for answers!
Try marking your stateless session bean method as transaction NOT_SUPPORTED or NEVER, which will cause it to run outside of a global transaction. (Note that you would need to do this anyway for your connection.commit() call to be valid -- it likely just isn't getting that far due to the timeout).
#javax.ejb.TransactionAttribute(javax.ejb.TransactionAttributeType.NEVER)
public void yourStatelessEJBMethod() {
... code that invokes stored procedure and commits transaction
}
Hikari: 2.4.7
PostgreSQL JDBC driver: 9.4-1201-jdbc41
I'm trying to understand what must be done to a java.sql.Connection object for it to be available again in the
connection pool?
I've just introduced connection pooling to a multi threaded application that was
previously standing up / tearing down connections with each SQL statement.
What I have noticed, after introducing Hikari, is that as soon as I hit maximumPoolSize every attempt
thereafter to HikariDataSource.getConnection will fail due to connectionTimeout. So it seems like I'm not "releasing" this connection somehow.
The typical use of the Connection object is:
# omits Exception handling, parameter substitution, result evaluation.
PreparedStatement preparedStatement = hikariDataSource.getConnection().prepareStatement(sql);
preparedStatement.executeQuery();
preparedStatement.close();
Is there anything else that is expected to be done on this connection to get it eligible for reuse in the connection pool?
Autocommit is on. Connection.close(), unless doing something special when provided by Hikari, seemed like the exact thing I wanted to avoid.
I don't know Hikari specifically, but for every connection you take out of a connection pool, you have to return that connection when you are done with it.
Typically this is done using Connection.close() - the pool hands out a wrapper function where close() doesn't physically close the connection, only returns it.
So your code should look like this:
Connection con = hikariDataSource.getConnection();
PreparedStatement preparedStatement = con.prepareStatement(sql);
preparedStatement.executeQuery();
preparedStatement.close();
con.close(); // this returns the connection to the pool
Of course the two close() methods should be called in a finally block.
I'm getting random " The connection is closed: The connection is closed" errors when using Oracle UCP, v 12.1.0.2.0. It looks like connection is marked as closed in
oracle.ucp.jdbc.proxy.JDBCConnectionProxyFactory#invoke :
if(Clock.isBefore(this.creationTS, this.m_jdbcPooledConnection.getAvailableStartTime()) || Clock.isBefore(this.creationTS, this.m_jdbcPooledConnection.getBorrowedStartTime())) {
this.m_closed = Boolean.valueOf(true);
}
The Clock.isBefore(this.creationTS, this.m_jdbcPooledConnection.getAvailableStartTime()) returns true.
Could somebody please explain what this check is for?
The getAvailableStartTime is set when connection is retured to the pool, the creationTS - is set when JDBCConnectionProxyFactory is being created and it's being created when giving connection away.
The isBefore looks like this:
public static boolean isBefore(long time1, long time2) {
return time1 < time2 - 1000L;
}
So, is the condition for the cases when connection was returned less than a second ago?
ps: tried validation query "select 1 from dual" - no effect
If Clock.isBefore(this.creationTS, this.m_jdbcPooledConnection.getAvailableStartTime()) returns true then it means that UCP has recollected the connection and made it available again. This typically happens if you turn on connection harvesting in UCP. UCP detects when a connection is borrowed but not used for too long (poorly designed application) and to avoid connection leaks it will grab the connection back and make it available in the pool. If the original thread then wakes up and attempts to use the connection it gets a connection is closed error.
Does BoneCP (or any other pool) close connection's statements when connection is returned to pool? As I understand, it does not call actual connection's close method, so there is no automatic statement closing. So, does it close statements in any other way or do I need to close them manually?
The JDBC spec is very unclear on what should happen under normal connection close so, irrespective of the pool you use, you should always make sure to close off the statements manually. Consider what would happen to your application if you opt to switch to a different pool in the future that does not do what you expect it to do for you.
As regards BoneCP, the answer is no, it will not close off your statements for you though it can be configured to close off your connections if you forget. This is for performance reasons since some JDBC drivers will close off any still active statements internally if you close off the connection.
However, BoneCP will close off any cached statements if you have statements caching enabled.
EDIT: As of v0.8.0, support has been added to close off unclosed statements (+ print out stack trace of location where statement was opened if you want).
BoneCP (0.8.0 -RC3), there are 2 possible results,
close off with some configuration for non-cached statement only
non-close off no matter how you configure it for cached statement even you invoke the statement.close() explicitly.
There is a StatementCache class to cache the preparedStatement & callableStatement. The default is disabled. You need call BoneCPConfig.setStatementsCacheSize() with the >0 parameter to enable it. After enable the cache,
1 BoneCP.Statement.Close() will bypass the underlying statement close if it is cached.
public void close() throws SQLException {
this.connectionHandle.untrackStatement(this);
this.logicallyClosed.set(true);
if (this.logStatementsEnabled){
this.logParams.clear();
this.batchSQL = new StringBuilder();
}
if (this.cache == null || !this.inCache){ // no cache = throw it away right now
this.internalStatement.close();
}
}
2 BoneCP.Connection.close()
Will just simply clear the cache through the function "clearStatementCaches()"
The good news is MYSQL JDBC driver, Connector/J, will close all the opened statements when you close the connection through the function "closeAllOpenStatements()"
We have our JBoss and Oracle on separate servers. The connections seem to be dropped and is causing issues with JBoss. How can I have the JBoss reconnect to Oracle if the connection is bad while we figure out why the connections are being dropped in the first place?
Whilst you can use the old "select 1 from dual" trick, the downside with this is that it issues an extra query each and every time you borrow a connection from the pool. For high volumes, this is wasteful.
JBoss provides a special connection validator which should be used for Oracle:
<valid-connection-checker-class-name>
org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker
</valid-connection-checker-class-name>
This makes use of the proprietary ping() method on the Oracle JDBC Connection class, and uses the driver's underlying networking code to determine if the connection is still alive.
However, it's still wasteful to run this each and every time a connection is borrowed, so you may want to use the facility where a background thread checks the connections in the pool, and silently discards the dead ones. This is much more efficient, but means that if the connections do go dead, any attempt to use them before the background thread runs its check will fail.
See the wiki docs for how to configure the background checking (look for background-validation-millis).
There is usually a configuration option on the pool to enable a validation query to be executed on borrow. If the validation query executes successfully, the pool will return that connection. If the query does not execute successfully, the pool will create a new connection.
The JBoss Wiki documents the various attributes of the pool.
<check-valid-connection-sql>select 1 from dual</check-valid-connection-sql>
Seems like it should do the trick.
Not enough rep for a comment, so it's in a form of an answer. The 'Select 1 from dual' and skaffman's org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker method are equivalent , although the connection check does provide a level of abstraction. We had to decompile the oracle jdbc drivers for a troubleshooting exercise and Oracle's internal implementation of the ping is to perform a 'Select 'x' from dual'. Natch.
JBoss provides 2 ways to Validate connection:
- Ping based AND
- Query based
You can use as per requirement. This is scheduled by separate thread as per duration defined in datasource configuration file.
<background-validation>true</background-validation> <background-validation-minutes>1</background-validation-minutes>
Some time if you are not having right oracle driver at Jboss, you may get classcast or related error and for that connection may start dropout from connection pool. You can try creating your own ConnectionValidator class by implementing org.jboss.resource.adapter.jdbc.ValidConnectionChecker interface. This interface provides only single method 'isValidConnection()' and expecting 'NULL' in return for valid connection.
Ex:
public class OracleValidConnectionChecker implements ValidConnectionChecker, Serializable {
private Method ping;
// The timeout (apparently the timeout is ignored?)
private static Object[] params = new Object[] { new Integer(5000) };
public SQLException isValidConnection(Connection c) {
try {
Integer status = (Integer) ping.invoke(c, params);
if (status.intValue() < 0) {
return new SQLException("pingDatabase failed status=" + status);
}
}
catch (Exception e) {
log.warn("Unexpected error in pingDatabase", e);
}
// OK
return null;
}
}
A little update to #skaffman's answer. In JBoss 7 you have to use "class-name" attribute when setting valid connection checker and also package is different:
<valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker" />
We've recently had some floating request handling failures caused by orphaned oracle DBMS_LOCK session locks that retained indefinitely in client-side connection pool.
So here is a solution that forces session expiry in 30 minutes but doesn't affect application's operation:
<check-valid-connection-sql>select case when 30/60/24 > sysdate-LOGON_TIME then 1 else 1/0 end
from V$SESSION where AUDSID = userenv('SESSIONID')</check-valid-connection-sql>
This may involve some slow down in process of obtaining connections from pool. Make sure to test this under load.