Intermittent PersistenceException/GenericJDBCException in opening Connection (Hibernate, Oracle) - spring

Question

In my Stand One Java Application, we are using Hibernate JPA based Pooled DataSource to connect with Oracle Database. We are intermittently getting the following exception. But on retry it often succeeds to create connection pool. Also, other service(s) which are also accessing same DB server, can create connection pool successfully.
How to address this problem?
Can there be any connection leakage and how to diagnose it?
Exception in thread "main" com.xyz.abc.inf.console.ConsoleException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
at com.xyz.abc.inf.console.service.GatewayService.postStart(GatewayService.java:104)
at com.xyz.abc.inf.console.service.AbstractLaunchableService.start(AbstractLaunchableService.java:205)
at com.xyz.abc.inf.console.ConsoleLauncher.main(ConsoleLauncher.java:85)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:273)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.orm.jpa.SharedEntityManagerCreator$DeferredQueryInvocationHandler.invoke(SharedEntityManagerCreator.java:311)
at $Proxy102.getResultList(Unknown Source)
at com.nrift.finch.inf.domain.repository.RepositoryBase.query(RepositoryBase.java:139)
at com.xyz.abc.inf.domain.dao.jpa.impl.ServiceRouteParticipantDaoImpl.findByServiceId(ServiceRouteParticipantDaoImpl.java:55)
at com.xyz.abc.inf.service.impl.RouteResolverServiceImpl.getRouteByService(RouteResolverServiceImpl.java:21)
at com.xyz.abc.inf.console.service.GatewayService.startCamelContextIfNotStarted(GatewayService.java:143)
at com.xyz.abc.inf.console.service.GatewayService.postStart(GatewayService.java:102)
... 2 more
Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:304)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.extractPhysicalConnection(ConnectionProxyHandler.java:82)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.continueInvocation(ConnectionProxyHandler.java:138)
at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at $Proxy103.prepareStatement(Unknown Source)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:147)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:166)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:145)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1854)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1831)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1811)
at org.hibernate.loader.Loader.doQuery(Loader.java:899)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
at org.hibernate.loader.Loader.doList(Loader.java:2516)
at org.hibernate.loader.Loader.doList(Loader.java:2502)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2332)
at org.hibernate.loader.Loader.list(Loader.java:2327)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:490)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1247)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:264)
... 13 more
Caused by: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
Database Server : Oracle 11g
Client configuration :
[root#eighamq1 ~]# uname -a
Linux eighamq1.nrifintech.com 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root#eighamq1 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
[root#eighamq1 ~]# java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
[root#eighamq1 ~]# mvn -v
Apache Maven 3.0.4 (r1232337; 2012-01-17 14:14:56+0530)
Maven home: /usr/local/maven-3.0.4
Java version: 1.7.0, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.18-194.el5", arch: "amd64", family: "unix"
JDBC.properties
jdbc.database.driverClass = oracle.jdbc.OracleDriver
jdbc.database.url = jdbc:oracle:thin:#172.16.29.92:1521:d11gr21
jdbc.database.initialLimit = 1
jdbc.database.maxLimit = 5
jdbc.database.minLimit = 1
jpa.hibernate.dialectClass = org.hibernate.dialect.Oracle10gDialect
jdbc.database.initialPoolSize = 10
jdbc.database.minPoolSize = 5
jdbc.database.maxPoolSize = 20
jdbc.database.maxStatements = 0
jdbc.database.acquireIncrement = 5
jdbc.database.acquireRetryAttempts = 5
jdbc.database.checkoutTimeout = 1000
Artifacts used :
ojdbc16-11.2.0.1.0.jar
c3p0-0.9.1.1.jar
hibernate-entitymanager : 4.1.12.Final
hibernate-commons-annotations : 4.0.1.Final
hibernate-c3p0 : 4.1.12.Final
hibernate-commons-annotations : 4.0.1.Final
hibernate-jpa-2.0-api : 1.0.1.Final

Answer 1

The oracle thin driver use the Java Security api internally to connect to oracle. Java Security in linux use the /dev/random as Entropy Gathering device and this device in linux sometimes are very slow returning values. It is probable that this is the reason than the connection couldn't be opened fast.
You can change the devices putting this in the command line:
-Djava.security.egd=file:///dev/urandom
With this property you indicate that you can use the /dev/urandom to get random values and this device is very fast because it uses a pseudo random algorithm.
It is good to put this property when you use the Oracle Thin Driver to connect database to improve the performance to connect to database.