Connection pool gets exhausted, Not able to receive free connection - spring

Question

After serving around some request on heavy load the server starts showing these error continuously...... strong text
This is a stacktrace:
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 62 more
org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; Cannot get a connection, pool error Timeout waiting for idle object; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:424)
at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:410)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:912)
at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:904)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:153)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:87)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:113)
at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:97)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:461)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:149)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:108)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2533)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:452)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at org.springframework.orm.hibernate3.HibernateTemplate$30.doInHibernate(HibernateTemplate.java:921)
at org.springframework.orm.hibernate3.HibernateTemplate$30.doInHibernate(HibernateTemplate.java:1)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
... 44 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 63 more
These are the spring and hibernate dependency i am using
spring 3.1.0
commons-dbcp-1.4.jar
hibernate-3.6.1.Final
mysql-connector-java-5.1.2.jar
I am using these configuration for handling the request... in applicationContext.xml
<bean id="springDS"
class="org.springframework.jdbc.datasource.DriverManagerDataSource"
lazy-init="true">
<property name="driverClassName">
<value>com.mysql.jdbc.Driver</value>
</property>
<property name="url">
<value>jdbc:mysql://xxxxx/xxxxxx
</value>
</property>
<property name="username">
<value>xxxxxx</value>
</property>
<property name="password">
<value>xxxxx</value>
</property>
</bean>
<bean id="dbcpDS" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close" lazy-init="true">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://xxxx/xxxx"/>
<property name="username" value="xxxxxx" />
<property name="password" value="xxxxx" />
<property name="initialSize" value="10" />
<property name="maxActive" value="200" />
<property name="maxIdle" value="50" />
<property name="minIdle" value="10" />
<property name="maxWait" value="10000" />
</bean>
<!-- Hibernate SessionFactory -->
<bean id="sessionFactory"
class="org.springframework.orm.hibernate3.LocalSessionFactoryBean"
lazy-init="true">
<property name="dataSource">
<ref local="dbcpDS" />
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect
</prop>
<prop key="hibernate.hbm2ddl.auto">validate</prop>
<prop key="hibernate.show_sql">false</prop>
<prop key="hibernate.connection.isolation">2</prop>
<prop key="hibernate.connection.release_mode">after_transaction</prop>
<!-- prop key="hibernate.c3p0.min_size">5</prop> <prop key="hibernate.c3p0.max_size">100</prop>
<prop key="hibernate.c3p0.timeout">1800</prop> <prop key="hibernate.c3p0.max_statements">100</prop -->
</props>
</property>
</bean>
Anyone having any idea please give a response.

Answer 1

Here is typical option you have to resolve connection leak / pool exhaustion:
Increase maximum pool capacity
Make you queries run faster so connection get returned to the pool sooner
Check for code that doesn't close connection when exception is thrown
I would start with analysing queries running on mysql. If you have MySQL workbench you can use the Server Administration module to see what queries currently running, how long they run for etc.
If your program indeed issuing long running queriesbyou need to fix this. Other possibility is your code throwing exceptions (most likely unchecked) but not closing the connection afterwards
I'm sure there's also a tool that can analyze what's your most long-running SQL, server throughput etc
Since you mention Spring 3.1.x, also check the list of JIRA ticket fixed between 3.0.5 to 3.1.0 and see if any might affect your program