Every once in a while in my app I get this error:
ERROR 2015-04-09 08:30:13,724 [http-bio-8080-exec-2] mojo.jdbc.MojoAlertDataAccess: Invalid or Stale Connection found in the Connection Cache
java.sql.SQLException: Invalid or Stale Connection found in the Connection Cache
at oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(OracleImplicitConnectionCache.java:421)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:395)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:179)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:157)
at mojo.jdbc.MojoAlertDataAccess.getAllAlertTypes(MojoAlertDataAccess.java:807)
So currently I have a datasource in Spring defined as:
<bean id="globalDSRead" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
<property name="connectionCachingEnabled" value="true" />
<property name="URL" value="${mojo.jdbc.read.url}"/>
<property name="user" value="${mojo.jdbc.read.username}"/>
<property name="password" value="${mojo.jdbc.read.password}"/>
<property name="connectionCacheProperties">
<value>
PropertyCheckInterval:10
MinLimit:1
MaxLimit:200
InitialLimit:1
ConnectionWaitTimeout:30
InactivityTimeout:30
ValidateConnection:true
</value>
</property>
</bean>
As far as I can tell this happens when the app has been sitting idle for a while, but it's not all together easy to reproduce.
Also - Another thing that occurs in this function is the retrieval of the connection can take a long time to come back, again this usually happens when the app's been sitting idle for a long time.
Anyone have any ideas whatcould be misconfigured?
Thanks
Yes, I have seen that before. I recommend to use Apache DBCP (https://commons.apache.org/proper/commons-dbcp/) for the connection pool management.
Related
Can a HikariCP Datasource be started with a Lazy configuration?
For that, i'm using Spring LazyConnectionDataSourceProxy.
<bean id="hikariConfig" class="com.zaxxer.hikari.HikariConfig" lazy-init="true">
<property name="poolName" value="TargetHikariCP" />
<property name="dataSourceClassName" value="oracle.jdbc.pool.OracleDataSource" />
<property name="connectionInitSql" value="SELECT 1 FROM DUAL"/>
<property name="leakDetectionThreshold" value="300000"/>
<property name="minimumIdle" value="1"/>
<property name="maximumPoolSize" value="10"/>
<property name="autoCommit" value="false"/>
<property name="dataSourceProperties"> <props> ... </props> </property>
</bean>
<bean id="dataSourceLazy" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close" lazy-init="true">
<constructor-arg ref="hikariConfig" />
</bean>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy">
<property name="targetDataSource" ref="dataSourceLazy" />
</bean>
<bean id="txManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager" lazy-init="true">
<property name="dataSource" ref="dataSource" />
</bean>
Nevertheless, its not working, as the Datasource is started on project startup.
The same configuration, when using a org.springframework.jdbc.datasource.DriverManagerDataSource, works correctly.
In the version > 3 we can set setInitializationFailTimeout(-1);
According to docs:
Any value greater than zero will be treated as a timeout for pool initialization.The calling thread will be blocked from continuing until a successful connection
to the database, or until the timeout is reached. If the timeout is reached, then
a PoolInitializationException will be thrown.
A value of zero will not prevent the pool from starting in the
case that a connection cannot be obtained. However, upon start the pool will
attempt to obtain a connection and validate that the connectionTestQuery
and connectionInitSql are valid. If those validations fail, an exception
will be thrown. If a connection cannot be obtained, the validation is skipped
and the the pool will start and continue to try to obtain connections in the
background. This can mean that callers to DataSource#getConnection() may
encounter exceptions.
A value less than zero will bypass any connection attempt and validation during
startup, and therefore the pool will start immediately. The pool will continue to
try to obtain connections in the background. This can mean that callers to
DataSource#getConnection() may encounter exceptions.
HikariCP has a property, initializationFailFast, that controls whether the pool will "fail fast" if the pool cannot be seeded with initial connections successfully:
This property controls whether the pool will "fail fast" if the pool cannot be seeded with initial connections successfully. If you want your application to start even when the database is down/unavailable, set this property to false. Default: true
This property was documented in their site, but per version 2.6.2 its not, but it seems its still supported.
In my use case, the use of this property should be enough to solve my problem.
In spring I have a datasource defined in this way:
<bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
<property name="URL" value="${jdbc.url}" />
<property name="user" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
<property name="connectionCachingEnabled" value="true"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
This datasource is used by my REST service and everything works fine...anyway first REST call is very slow (about ~5 secs), after that EVERY other call is fast.
I think this is an initialization related problem, in the sense that initialization is made when first DB call is received.
Is there a way to tell spring to initialize this datasource on server startup?
I think this is an initialization related problem, in the sense that
initialization is made when first DB call is received.
With your current config I think that's what's happening.
Is there a way to tell spring to initialize this datasource on server
startup?
It's the behavior of the connection pool, not Spring. Spring is creating the bean when your app starts (you aren't using lazy-init="true" on the bean). However, the connection pool isn't creating connections to the database when Spring instantiates it. From the Oracle docs:
The initial pool size property specifies the number of available
connections that are created when the connection pool is initially
created or re-initialized. This property is typically used to reduce
the ramp-up time incurred by priming the pool to its optimal size.
A value of 0 indicates that no connections are pre-created. The
default value is 0.
Try setting a non-zero value for initialPoolSize.
Edit: Try setting ConnectionCacheProperties instead:
<property name="connectionCacheProperties">
<props merge="default">
<prop key="InitialLimit">5</prop>
</props>
</property>
When uploaded our application on prod server we faced strange behaviour - sometimes it stopped to get data from data base as though the logic and everything was right and local version was working perfect. After remote debugging we found that GenericObjectPool was blocking thread. And after some time spent on searching we found situation that fits our problem.
The pool just got exhausted when filled with 8 connections (default value) and by default it's behaviour is set to block thread when exhausted.
Here is my datamodel-context.xml config of BasicDataSource
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="url_to_db"/>
<property name="username" value="username"/>
<property name="password" value="12211"/>
<property name="defaultAutoCommit" value="false"/>
<property name="poolPreparedStatements" value="true"/>
<property name="initialSize" value="10"/>
<property name="maxIdle" value="5"/>
<property name="testOnBorrow" value="true"/>
<property name="logAbandoned" value="true"/>
</bean>
Found here two solutions - increase max active number or change pool's behaviour.
The first one seems to be simple - set property maxActive in BasicDataSource to some number (please correct me if I am mistaking). This way is unwanted because we can not know the exact number of possible simultanious connections.
It was decided to try the second way - to change behaviour to WHEN_EXHAUSTED_GROW.
So is there any way to configure spring default pool to change it's whenExhaustedAction behaviour? Or should I define my own connection pool for BasicDataSource? If so, please can you provide examples?
I appreciate any comments or advices. Thanks.
I'm facing a weird production problem. Environment is the following:
JBOSS 4.0.2
SQL Server 2005
Driver JTDS 1.2.5
From time to time the following szenario occurs.
A SQL command fails to Excute with
java.sql.SQLException: I/O Error: Read timed out
(I can live with that, if it just happens twice a day or so)
But from that moment on the connection seems to be wasted without the pool recognizing it, as I continously receive
java.sql.SQLException: Invalid state, the Connection object is closed.
from that moment on. The only thing that helps is restarting JBOSS. This occurs despite of the fact that I have
<check-valid-connection-sql>select getdate()</check-valid-connection-sql>
set up in my Datasource definition.
I was wondering if I can use a custom ValidConnectionChecker, that either rebuilds the connection itself, or explicitly throws a Exception to fix this. Maybe anyone has other suggestions.
Here is my complete DS definition.
<local-tx-datasource>
<jndi-name>MyDS</jndi-name>
<connection-url>jdbc:jtds:sqlserver://192.168.35.235:1433/MyDb;user=user1;password=pwd;appName=MyApp;loginTimeout=15;socketTimeout=120</connection-url>
<driver-class>net.sourceforge.jtds.jdbc.Driver</driver-class>
<user-name>user1</user-name>
<password>pwd</password>
<min-pool-size>10</min-pool-size>
<max-pool-size>25</max-pool-size>
<blocking-timeout-millis>60000</blocking-timeout-millis>
<idle-timeout-minutes>1</idle-timeout-minutes>
<check-valid-connection-sql>select getdate()</check-valid-connection-sql>
</local-tx-datasource>
Any help appriciated.
Regards
Try changing your driver class line to
net.sourceforge.jtds.jdbcx.JtdsDataSource.
net.sourceforge.jtds.jdbc.Driver doesn't implement the javax.sql.ConnectionPoolDataSource interface.
source:
http://jtds.sourceforge.net/faq.html#features
Probably too late the solution, but I am stuck with the jtds driver here. Hope this saves half an hour of your productive time.
The fix is to specify a validationQuery to the Apache dbcp2 Connection Pool implementation.
For jtds/sql server
I specified the spring configuration as follows:
<bean id="sqlServerDS" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close" >
<property name="driverClassName" value="${jdbc.driverClassName}" />
<property name="url" value="${jdbc.url}" />
<property name="username" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="defaultReadOnly" value="true" />
<property name="validationQuery" value="select 1" />
</bean>
In case you are not using Spring, call setValidationQuery method on BasicDataSource in your java code.
BasicDataSource bds = new BasicDataSource();
bds.setValidationQuery("select 1");
Connection.isValid() isn't implemented in JTDS.
I found even catching the exception and forcing a complete restart of the connection didn't work.
I have connection pooling implemented in spring using Oracle Data Source. Currently we are facing an issue where connections are becoming invalid after a period of time. (May be Oracle is dropping those idle connections after a while). Here are my questions:
Can Oracle database be configured to drop idle connections automatically after a specific period of time. Since we expect those connections to lie idle for a while; if there is any such configuration; it may be happening.
In our connection pooling properties in spring we didn't have "validateConnection" property. I understand that it validates the connection before handing it over to web application? But does that mean that if a connection passes validateConnection test then it'll always connect to database correctly. I ask this, as I read following problem here:
http://forum.springsource.org/showthread.php?t=69759
If suppose validateConnection doesn't do the whole 9 yards of ensuring that connection is valid, is there any other option like "testBeforBorrow" in DBCP , which runs a test query to ensure that connection is active before handing it over to webapp?
I'll be grateful if you could provide answers to one ore more queries listed above.
Cheers
You don't say what application server you are using, or how you are configuring the datasource, so I can't give you specific advice.
Connection validation often sounds like a good idea, but you have to be careful with it. For example, we once used it in our JBoss app servers to validate connections in the pool before handing them to the application. This Oracle-proprietary mechanism used the ping() method on the Oracle JDBC driver, which checks that the connection is still alive. It worked fine, but it turns out that ping() executes "select 'x' from dual' on the server, which is a surprisingly expensive query when it's run dozens of times per second.
So the moral is, if you have a high-traffic server, be very careful with connection validation, it can actually bring your database server to its knees.
As for DBCP, that has the ability to validate connections as their borrowed from the pool, as well as returned to the pool, and you can tell it what SQL to send to the database to perform this validation. However, if you're not using DBCP for your connection pooling, then that's not much use to you. C3PO does something similar.
If you're using an app server's data source mechanism, then you have to find out if you can configure that to validate connections, and that's specific to your server.
One last thing: Spring isn't actually involved here. Spring just uses the DataSource that you give it, it's up to the DataSource implementation to perform connection validation.
Configuration of data source "was" as follows:
<bean id="datasource2"
class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName">
<value>org.apache.commons.dbcp.BasicDataSource</value>
</property>
<property name="url">
<value>ORACLE URL</value>
</property>
<property name="username">
<value>user id</value>
</property>
<property name="password">
<value>user password</value>
</property>
<property name="initialSize" value="5"/>
<property name="maxActive" value="20"/>
</bean>
have changed it to:
<bean id="connectionPool1" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
<property name="connectionCachingEnabled" value="true" />
<property name="URL">
<value>ORACLE URL</value>
</property>
<property name="user">
<value>user id</value>
</property>
<property name="password">
<value>user password</value>
</property>
<property name="connectionCacheProperties">
<value>
MinLimit:1
MaxLimit:5
InitialLimit:1
ConnectionWaitTimeout:120
InactivityTimeout:180
ValidateConnection:true
</value>
</property>
</bean>