Can a HikariCP Datasource be started with a Lazy configuration?
For that, i'm using Spring LazyConnectionDataSourceProxy.
<bean id="hikariConfig" class="com.zaxxer.hikari.HikariConfig" lazy-init="true">
<property name="poolName" value="TargetHikariCP" />
<property name="dataSourceClassName" value="oracle.jdbc.pool.OracleDataSource" />
<property name="connectionInitSql" value="SELECT 1 FROM DUAL"/>
<property name="leakDetectionThreshold" value="300000"/>
<property name="minimumIdle" value="1"/>
<property name="maximumPoolSize" value="10"/>
<property name="autoCommit" value="false"/>
<property name="dataSourceProperties"> <props> ... </props> </property>
</bean>
<bean id="dataSourceLazy" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close" lazy-init="true">
<constructor-arg ref="hikariConfig" />
</bean>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy">
<property name="targetDataSource" ref="dataSourceLazy" />
</bean>
<bean id="txManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager" lazy-init="true">
<property name="dataSource" ref="dataSource" />
</bean>
Nevertheless, its not working, as the Datasource is started on project startup.
The same configuration, when using a org.springframework.jdbc.datasource.DriverManagerDataSource, works correctly.
In the version > 3 we can set setInitializationFailTimeout(-1);
According to docs:
Any value greater than zero will be treated as a timeout for pool initialization.The calling thread will be blocked from continuing until a successful connection
to the database, or until the timeout is reached. If the timeout is reached, then
a PoolInitializationException will be thrown.
A value of zero will not prevent the pool from starting in the
case that a connection cannot be obtained. However, upon start the pool will
attempt to obtain a connection and validate that the connectionTestQuery
and connectionInitSql are valid. If those validations fail, an exception
will be thrown. If a connection cannot be obtained, the validation is skipped
and the the pool will start and continue to try to obtain connections in the
background. This can mean that callers to DataSource#getConnection() may
encounter exceptions.
A value less than zero will bypass any connection attempt and validation during
startup, and therefore the pool will start immediately. The pool will continue to
try to obtain connections in the background. This can mean that callers to
DataSource#getConnection() may encounter exceptions.
HikariCP has a property, initializationFailFast, that controls whether the pool will "fail fast" if the pool cannot be seeded with initial connections successfully:
This property controls whether the pool will "fail fast" if the pool cannot be seeded with initial connections successfully. If you want your application to start even when the database is down/unavailable, set this property to false. Default: true
This property was documented in their site, but per version 2.6.2 its not, but it seems its still supported.
In my use case, the use of this property should be enough to solve my problem.
Related
I am developping a Spring MVC web application that use the dbcp database connection pool.
<bean id="datasourceAR_XXX" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close" scope="singleton">
<property name="driverClassName"><value>oracle.jdbc.driver.OracleDriver</value></property>
<property name="url"><value>jdbc:oracle:thin:#XXX.XXX.com:1500:SERVICE</value></property>
<property name="maxActive"><value>100</value></property>
<property name="maxIdle"><value>10</value></property>
<property name="username"><value>XXX</value></property>
<property name="password"><value>XXX</value></property>
</bean>
I recently moved the scope of those beans to singleton because the amount of connection per session started to be a bit too much.
The problem is :
Our database is shutting down every sunday and the spring application seems to act strangely by keeping the socket open and does not refresh the connection as I thought it would do.
Is there a way to refresh the beans scoped as singleton in a way that will refresh the connection everyday and not be obliged to relaunch the application every monday?
What you want to do is to configure validation for your connections. When a connection is borrowed from the pool you want to make sure that that connection is valid. For this you can specify the validationQuery property on your datasource.
<bean id="datasourceAR_XXX" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close" scope="singleton">
<property name="driverClassName"><value>oracle.jdbc.driver.OracleDriver</value></property>
<property name="url"><value>jdbc:oracle:thin:#XXX.XXX.com:1500:SERVICE</value></property>
<property name="maxActive"><value>100</value></property>
<property name="maxIdle"><value>10</value></property>
<property name="username"><value>XXX</value></property>
<property name="password"><value>XXX</value></property>
<property name="validationQuery" value="select 1 from dual" />
</bean>
See DBCP - validationQuery for different Databases for a list of possible validation queries for different databases.
There are some issues with Commons DBCP and it is pretty old (although there is a DBCP 2.x now). I would suggest moving to a different datasource like HikariCP this datasource is also a JDBC 4.x based datasource which allows for easier connection validation (it is part of the JDBC 4 spec).
<bean id="datasourceAR_XXX" class="com.zaxxer.hikari.HikariDataSource">
<property name="datasourceClassName" value="oracle.jdbc.pool.OracleDataSource"/>
<property name="maximumPoolSize" value="20" />
<property name="username" value="XXX" />
<property name="password" value="XXX" />
<property name="datasourceProperties">
<props>
<prop key="serverName">XXX.XXX.com</prop>
<prop key="port">1500</prop>
<prop key="databaseName">SERVICE</prop>
</props>
</property>
</bean>
If your oracle driver is new enough you don't need a validation query anymore as validation is provided by the driver instead of needing to be done with a query. Next to that you probably have better results with this pool.
Also you might have a bit of a large pool size, nice article/presentation about pool sizing can be found here.
In spring I have a datasource defined in this way:
<bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
<property name="URL" value="${jdbc.url}" />
<property name="user" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
<property name="connectionCachingEnabled" value="true"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
This datasource is used by my REST service and everything works fine...anyway first REST call is very slow (about ~5 secs), after that EVERY other call is fast.
I think this is an initialization related problem, in the sense that initialization is made when first DB call is received.
Is there a way to tell spring to initialize this datasource on server startup?
I think this is an initialization related problem, in the sense that
initialization is made when first DB call is received.
With your current config I think that's what's happening.
Is there a way to tell spring to initialize this datasource on server
startup?
It's the behavior of the connection pool, not Spring. Spring is creating the bean when your app starts (you aren't using lazy-init="true" on the bean). However, the connection pool isn't creating connections to the database when Spring instantiates it. From the Oracle docs:
The initial pool size property specifies the number of available
connections that are created when the connection pool is initially
created or re-initialized. This property is typically used to reduce
the ramp-up time incurred by priming the pool to its optimal size.
A value of 0 indicates that no connections are pre-created. The
default value is 0.
Try setting a non-zero value for initialPoolSize.
Edit: Try setting ConnectionCacheProperties instead:
<property name="connectionCacheProperties">
<props merge="default">
<prop key="InitialLimit">5</prop>
</props>
</property>
We have an OracleXADataSource that is being wrapped by Apache Aries in Fuse Fabric (like in this article). If I keep sending a lot of request to the server, it starts throwing the following error:
Caused by: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
When I check the sessions using the following query, after every request in Oracle, it keeps showing an increased number under current utilization.
select resource_name, current_utilization, max_utilization, limit_value
from v$resource_limit
where resource_name in ('sessions', 'processes', 'transactions');
CURRENT_UTILIZATION MAX_UTILIZATION LIMIT_VALUE
processes 545 768 800
sessions 553 774 1222
transactions 0 0 UNLIMITED
Most of the recommendations for this issue says to increase the processes and session limits in Oracle, but this would solve the problem temporarily, until we reach a certain load I'm affraid.
I found/tried the followings so far:
Perodically when the load increases (or certain amount of time spent) the session and processes get decreased with a bigger amount (100-200). (I guess Geronimo periodically releases the sessions). At the same time when a number of sessions are released, the active transactions column shows the same amount:
CURRENT_UTILIZATION MAX_UTILIZATION LIMIT_VALUE
processes 355 768 800
sessions 363 774 1222
transactions 122 122 UNLIMITED
If I shut down Fuse, the processes values goes back to initial size immediately (so the issue is on client side)
If I turn off the distributed transaction support, then everything is fine and processes doesn't increase at all
I tried adding pooling to the OracleXADataSource, but nothing has changed (it's deprecated, but I assume it still works. We don't have the UCP jar unfortunately, so I couldn't test it with that)
<property name="connectionCachingEnabled" value="true"/>
<property name="connectionCacheProperties">
<props merge="default">
<prop key="InitialLimit">1</prop>
<prop key="MinLimit">1</prop>
<prop key="MaxLimit">1</prop>
</props>
</property>
I couldn't resolve this issue using Aries unfortunately. I consider it a bug. However I managed to make it properly work using Atomikos, which I strongly recommend. Much more straightforward than using Aries' built in auto-proxy behavior: you declare everything so you know what actually happens.
<bean id="transactionManager" class="com.atomikos.icatch.jta.UserTransactionManager" init-method="init" destroy-method="close">
<property name="forceShutdown" value="false" />
</bean>
<bean id="userTransaction" class="com.atomikos.icatch.jta.UserTransactionImp">
<property name="transactionTimeout" value="300" />
</bean>
<bean id="jtaTransactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="transactionManager" />
<property name="userTransaction" ref="userTransaction" />
</bean>
<bean id="dataSource" class="com.atomikos.jdbc.AtomikosDataSourceBean">
<property name="uniqueResourceName" value="oracledb" />
<property name="xaDataSource">
<bean class="oracle.jdbc.xa.client.OracleXADataSource">
<property name="URL" value="jdbc:oracle:thin:#${db.host}:${db.port}:${db.sid}"/>
<property name="user" value="${db.schema}" />
<property name="password" value="${db.password}" />
</bean>
</property>
</bean>
I have the following AMQ consumer configuration file which tries to consume 'persistent' messages from the queue. Also, the messages are 'transacted' (as I need to rollback if a message can't be processed in an expected way).
I see a problem with this configuration:
Whenever the consumer calls session.commit() after consuming the message, I see the commit call taking ~8 seconds to come out. I believe this is not expected.
Can someone point me if I have any issues with the below config?
<bean id="amqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://localhost:61616"/>
</bean>
<bean id="simpleMessageListener" class="listener.StompSpringListener" scope="prototype" />
<bean id="destination" class="org.apache.activemq.command.ActiveMQQueue">
<property name="physicalName" value="JOBS.notifications"/>
</bean>
<bean id="poolMessageListener" class="org.springframework.aop.target.CommonsPoolTargetSource">
<property name="targetBeanName" value="simpleMessageListener"/>
<property name="maxSize" value="200"/>
</bean>
<bean id="messageListenerBean" class="org.springframework.aop.framework.ProxyFactoryBean">
<property name="targetSource" ref="poolMessageListener"/>
</bean>
<jms:listener-container
container-type="default"
connection-factory="amqConnectionFactory"
acknowledge="transacted"
concurrency="200-200"
cache="consumer"
prefetch="10">
<jms:listener destination="JOBS.notifications" ref="messageListenerBean" method="onMessage"/>
</jms:listener-container>
Take a look at the the ActiveMQ Spring support page. What I can see immediately is that you should be using a PoolingConnectionFactory instead of an ActiveMQConnectionFactory. This will ensure that only a single TCP connection is set up to the broker from your code, and will be shared between polling threads.
I am using Spring MVC 3 and Hibernate 3.6, I use xml configured transaction management,
my code works greate but my JDBC are not being released although it says it does.
I checked it with JProfiler and it says the connection is open.
this is my spring-config code
<bean id="ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://localhost:3306/parse_web?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8"/>
<property name="username" value="root" />
<property name="password" value="miles106" />
<property name="initialSize" value="5"/>
<property name="maxActive" value="50000"/>
</bean>
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="ds" />
<property name="mappingResources">
<list>
<value>com/mubasher/parsewebpage/entities/Changes.hbm.xml</value>
<value>com/mubasher/parsewebpage/entities/Owners.hbm.xml</value>
<value>com/mubasher/parsewebpage/entities/Ownerships.hbm.xml</value>
<value>com/mubasher/parsewebpage/entities/TargetCompanies.hbm.xml</value>
<value>com/mubasher/parsewebpage/entities/TempData.hbm.xml</value>
<value>com/mubasher/parsewebpage/entities/Exceptions.hbm.xml</value>
</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="dialect">org.hibernate.dialect.MySQL5Dialect</prop>
<prop key="hibernate.connection.useUnicode">true</prop>
<prop key="hibernate.connection.characterEncoding">UTF-8</prop>
<prop key="hibernate.connection.charSet">UTF-8</prop>
<prop key="hibernate.connection.release_mode">after_statement</prop>
</props>
</property>
</bean>
<bean id="transactionManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
and this is my debug code
DEBUG [myExec-2] (JDBCTransaction.java:223) - re-enabling autocommit
DEBUG [myExec-2] (JDBCTransaction.java:143) - committed JDBC Connection
DEBUG [myExec-2] (ConnectionManager.java:427) - aggressively releasing JDBC connection
DEBUG [myExec-2] (ConnectionManager.java:464) - releasing JDBC connection [ (open PreparedStatements: 0, globally: 0) (open ResultSets: 0, globally: 0)]
DEBUG [myExec-2] (HibernateTransactionManager.java:734) - Closing Hibernate Session [org.hibernate.impl.SessionImpl#52ab7af2] after transaction
DEBUG [myExec-2] (SessionFactoryUtils.java:789) - Closing Hibernate Session
but in JProfiler i can see that the connection is still open as you see
this is realy causing me problems, my application is doing massive database work, so I need the connection to close as soon as the work is done, should I use maxIdle ?
Connections are not closed, they are reused. This is the whole purpose of commons-dbcp, which stands for Database Connection Pool.
Establishing a new connection is usually an expensive operation. So what DBCP is doing is that instead of closing connection, it leaves it open and returns it to the connection pool for another use.
If you want your database connections to get closed and re-opened with each request, then you need to use a different data source (e.g. org.springframework.jdbc.datasource.SimpleDriverDataSource).
UPDATE 1: Also note, that in your example you are setting maximum number of parallel connections (maxActive) to be 50000. That is some extreme number (default is 8!!!), which IMO can cause a lot of problems.
UPDATE 2: Using maxIdle is a good idea if you don't wan't to get rid of the pool. But that will not save you from "having non-closed connections". If you are thinking about setting maxIdle=0, then drop the pool completely.
UPDAET 3: I just need to stress out this one again - If you need 50000 parallel connections, then there is really something wrong with your code.