How do we properly configure Sybase datasource in a Spring Batch?
Currently I am using a JdbcPagingItemReader for reading DB and creating the data source using Tomcat DBCP. However the following error occurs on the batch run.
Caused by: java.lang.IllegalArgumentException: DatabaseType not found for product name: [Sybase IQ]
at org.springframework.batch.support.DatabaseType.fromProductName(DatabaseType.java:77)
at org.springframework.batch.support.DatabaseType.fromMetaData(DatabaseType.java:108)
at org.springframework.batch.core.repository.support.JobRepositoryFactoryBean.afterPropertiesSet(JobRepositoryFactoryBean.java:162)
at org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer.createJobRepository(DefaultBatchConfigurer.java:82)
at org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer.initialize(DefaultBatchConfigurer.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:344)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:295)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
... 13 more
For reference, I am currently JConn4-7.0 jar for connectivity. There are no issues connecting to a Oracle Database though [with pool properties changed for Oracle].
Any insight would be helpful.
Please feel free to edit the question for clarity/readability.
The error you're getting is probably from not explicitly configuring the database type on your JobRepositoryFactoryBean. Without it set, we attempt to determine what type it is from the JDBC metadata. To explicitly specify Sybase as your job repository's database type, configure it via the databaseType attribute on the JobRepositoryFactoryBean like below:
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="transactionManager" />
<property name="databaseType" value="SYBASE" />
</bean>
You can read more about this feature in the Spring Batch documentation here: http://docs.spring.io/spring-batch/reference/html/configureJob.html#nonStandardDatabaseTypesInRepository
Related
I am getting the below exception sometimes when posting message to a queue. I am using the JMS template and CachingConnectionFactory with spring boot. In the documentation of CachingConnectionFactory, I can see that session need to be closed. But as I am using the JMS template, not understanding how to close the session. Is that causing the issue. The error is not happening always and the failure is random in behavior. Please help with your suggestion
Caused by: javax.jms.JMSException: Connection has been terminated by the server
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:676)
at com.tibco.tibjms.TibjmsxSessionImp._confirmTransacted(TibjmsxSessionImp.java:3576)
at com.tibco.tibjms.TibjmsxSessionImp._confirm(TibjmsxSessionImp.java:3982)
at com.tibco.tibjms.TibjmsxSessionImp._commit(TibjmsxSessionImp.java:3211)
at com.tibco.tibjms.TibjmsxSessionImp.commit(TibjmsxSessionImp.java:5374)
at com.fedex.mi.decorator.jms.FedexJmsSession.commit(FedexJmsSession.java:139)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:383)
at com.sun.proxy.$Proxy130.commit(Unknown Source)
at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:218)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:612)
at org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:586)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504)
... 68 more
I don't think the exception related to not closing connection. Most likely TibcoEMS server was not available for some reason or connection terminated by some network issue.
You need to define reconnectOnException settings in you CachingConnectionFactory definition to reconnect automatically if you don't have it already.
something like:
<bean id="tibcoConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="tibcoServerJndiObjectFactoryBean"/>
<property name="reconnectOnException" value="true"/>
<property name="sessionCacheSize" value="100"/>
<property name="cacheConsumers" value="false"/>
</bean>
You may also need to define QueueConnectionFactory fault-tolerant failover settings on TibcoEMS side
https://docs.tibco.com/pub/ems/8.5.1/doc/html/GUID-270D7C10-7686-44B0-BC11-D7BFF1F33A7E.html
I am using c3p0.ComboPooledDataSource to connect to my oracle database and here is the code snippet and i have deployed this application in Jboss Fuse ESB
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
destroy-method="close">
<property name="driverClass" value="oracle.jdbc.driver.OracleDriver" />
<property name="jdbcUrl" value="jdbc:oracle:thin:#localhost:1523:xe" />
<property name="user" value="test" />
<property name="password" value="test" />
</bean
when i try to insert values into the database am getting
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#339d05df -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
java.sql.SQLException: No suitable driver
at java.sql.DriverManager.getDriver(DriverManager.java:278)[:1.7.0_55]
when I use org.springframework.jdbc.datasource.SimpleDriverDataSource instead of ComboPooledDataSource it is working fine
I've previously had issues setting up connection pools in Fuse/OSGi because the connection pool bundle (in this case the bundle containing com.mchange.v2.c3p0.ComboPooledDataSource) doesn't include the driver it's using in its Import-Package. Could this be the problem?
If so, there are a number of options to resolve the problem. For example, you could, e.g.:
repackage the pool to include the Import-Package
make the JDBC driver a fragment bundle of the pool bundle
You'll find that searching for 'OSGI JDBC pool' normally fires up lots of additional articles and hints around this problem.
I have a web application which using PostgreSQL as database.
Assume the scenario like i am trying to Save() data using Hibernate. Now just before save the data if database server will down then what happened ?
Hibernate will re-try save again or do i have to write the re-try code.
Any help will be appreciated.
No, Hibernate will not re-try to save your data. The db failure will cause your transaction to fail, and all the change will be rollbacked.
Even worse, if you don't configure correctly your connection pool, you will not be able to re-establish the connection when the db will be running again. Take a look to this question for more info. I copy and paste c3p0 configuration.
<c3p0-config>
<default-config>
<!-- Configuring Connection Testing -->
<!-- property name="automaticTestTable">TEST_EMS_HIBERNATE_CONN</property -->
<property name="checkoutTimeout">0</property>
<property name="testConnectionOnCheckout">true</property>
<property name="testConnectionOnCheckin">false</property>
<property name="preferredTestQuery">SELECT 1 from dual</property>
<!-- Configuring Recovery From Database Outages -->
<property name="acquireRetryAttempts">0</property>
<property name="acquireRetryDelay">1000</property>
<property name="breakAfterAcquireFailure">false</property>
<!-- Configuring to Debug and Workaround Broken Client Apps -->
<property name="unreturnedConnectionTimeout">1800</property>
<property name="debugUnreturnedConnectionStackTraces">true</property>
</default-config>
I'm using Apache Derby with the Spring JdbcTemplate inside a web app running on Tomcat.
Spring is managing the data source. I've noticed that if I update the .war file and Tomcat undeploys/redeploys the app, I get this error:
java.sql.SQLException: Another instance of Derby may have already booted the database /tmp/manager_db/manager.
Restarting Tomcat fixes the problem, but as a purist, I'd like to clean things up properly when the webapp is undeployed.
The Embedded driver doesn't seem to have a 'close' method to put in the bean declaration under 'destroy-method'. I know the shutdown is normally achieved using a 'shutdown' connection URL, "jdbc:derby:;shutdown=true".
Any suggestions?
Here's the declaration in the Spring config file for my data source (the db won't be under /tmp/, just there for now).
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.apache.derby.jdbc.EmbeddedDriver"/>
<property name="url" value="jdbc:derby:/tmp/manager_db/manager;create=true"/>
<property name="username" value=""/>
<property name="password" value=""/>
</bean>
I think a better answer is to use the Tomcat JNDI data source pool with Spring. Spring's JDBC template will return the connection to pool when it's done.
I originally set up spring with xapool, but it turns out that's a dead project and seems to have lots of problems.
I switched to c3p0, but now I learn that the #Transactional annotations don't actually create transactions when used with c3p0. If I do the following it will insert the row into Foo even through an exception was thrown inside the method:
#Service
public class FooTst
{
#PersistenceContext(unitName="accessControlDb") private EntityManager em;
#Transactional
public void insertFoo() {
em.createNativeQuery("INSERT INTO Foo (id) VALUES (:id)")
.setParameter("id", System.currentTimeMillis() % Integer.MAX_VALUE )
.executeUpdate();
throw new RuntimeException("Foo");
}
}
This is strange because if I comment out the #Transactional annotation it will actually fail and complain about having a transaction set to rollback only:
java.lang.IllegalStateException: Cannot get Transaction for setRollbackOnly
at org.objectweb.jotm.Current.setRollbackOnly(Current.java:568)
at org.hibernate.ejb.AbstractEntityManagerImpl.markAsRollback(AbstractEntityManagerImpl.java:421)
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:576)
at org.hibernate.ejb.QueryImpl.executeUpdate(QueryImpl.java:48)
at com.ipass.rbac.svc.FooTst.insertFoo(FooTst.java:21)
at com.ipass.rbac.svc.SingleTst.testHasPriv(SingleTst.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.test.context.junit4.SpringTestMethod.invoke(SpringTestMethod.java:160)
at org.springframework.test.context.junit4.SpringMethodRoadie.runTestMethod(SpringMethodRoadie.java:233)
at org.springframework.test.context.junit4.SpringMethodRoadie$RunBeforesThenTestThenAfters.run(SpringMethodRoadie.java:333)
at org.springframework.test.context.junit4.SpringMethodRoadie.runWithRepetitions(SpringMethodRoadie.java:217)
at org.springframework.test.context.junit4.SpringMethodRoadie.runTest(SpringMethodRoadie.java:197)
at org.springframework.test.context.junit4.SpringMethodRoadie.run(SpringMethodRoadie.java:143)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.invokeTestMethod(SpringJUnit4ClassRunner.java:160)
at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:97)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:45)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
So, clearly it notices the #Transactional annotation. But, it doesn't actually set autocommit to off at the start of the method.
Here is how I have transactional stuff setup up in the applicationContext.xml. Is this correct? If not, what is this supposed to be?
<bean id="jotm" class="org.springframework.transaction.jta.JotmFactoryBean"/>
<bean id="txManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="jotm"/>
<property name="userTransaction" ref="jotm"/>
<property name="allowCustomIsolationLevels" value="true"/>
</bean>
<tx:annotation-driven transaction-manager="txManager" proxy-target-class="false"/>
After a bunch of searching I found a connection pool called Bitronix, but their spring setup page describes stuff about JMS which doesn't even make any sense. What does JMS have to do with setting up a connection pool?
So I'm stuck. What am I actually supposed to do? I don't understand why the connection pool needs to support transactions. All connections support turning autocommit on and off, so I have no idea what the problem is here.
It took a lot of searching and experimentation, but I finally got things working. Here are my results:
enhydra xapool is a terrible connection pool. I won't enumerate the problems it caused because it doesn't matter. The latest version of that pool hasn't been updated since Dec 2006. It's a dead project.
I put c3p0 into my application context and got it working fairly easily. But, for some reason it just doesn't seem to support rollback even inside a single method. If I mark a method as #Transactional then do an insert into a table and then throw a RuntimeException (one that's definitely not declared in the throws list of the method because there is no throws list on the method) it will still keep the insert into that table. It will not roll back.
I was going to try Apache DBCP, but my searching turned up lots of complaints about it, so I didn't bother.
I tried Bitronix and had plenty of trouble getting it to work properly under Tomcat, but once I figured out the magic configuration it works beautifully. What follows is all the things you need to do to set it up properly.
I dabbled briefly with the Atomkos connection pool. It looks like it should be good, but I got Bitronix working first, so I didn't try using it much.
The configuration below works in standalone unit tests and under Tomcat. That was the major problem I had. Most of the examples I found about how to set up Spring with Bitronix assume that I'm using JBoss or some other full container.
The first bit of configuration is the part that sets up the Bitronix transaction manager.
<!-- Bitronix transaction manager -->
<bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices">
<property name="disableJmx" value="true" />
</bean>
<bean id="btmManager" factory-method="getTransactionManager" class="bitronix.tm.TransactionManagerServices" depends-on="btmConfig" destroy-method="shutdown"/>
<bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="btmManager" />
<property name="userTransaction" ref="btmManager" />
<property name="allowCustomIsolationLevels" value="true" />
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
The major difference between that code and the examples I found is the "disableJmx" property. It throws exceptions at runtime if you don't use JMX but leave it enabled.
The next bit of configuration is the connection pool data source. Note that the connection pool classname is not the normal oracle class "oracle.jdbc.driver.OracleDriver". It's an XA data source. I don't know what the equivalent class would be in other databases.
<bean id="dataSource" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init" destroy-method="close">
<property name="uniqueName" value="dataSource-BTM" />
<property name="minPoolSize" value="1" />
<property name="maxPoolSize" value="4" />
<property name="testQuery" value="SELECT 1 FROM dual" />
<property name="driverProperties"><props>
<prop key="URL">${jdbc.url}</prop>
<prop key="user">${jdbc.username}</prop>
<prop key="password">${jdbc.password}</prop>
</props></property>
<property name="className" value="oracle.jdbc.xa.client.OracleXADataSource" />
<property name="allowLocalTransactions" value="true" />
</bean>
Note also that the uniqueName needs to be different than any other data sources you have configured.
The testQuery of course needs to be specific to the database that you are using. The driver properties are specific to the database class that I'm using. OracleXADataSource for some silly reason has different setter names for OracleDriver for the same value.
The allowLocalTransactions had to be set to true for me. I found recommendations NOT to set it to true online. But, that seems to be impossible. It just won't work if it's set to false. I am not sufficiently knowledgeable about these things to know why that is.
Lastly we need to configure the entity manager factory.
<util:map id="jpa_property_map">
<entry key="hibernate.transaction.manager_lookup_class" value="org.hibernate.transaction.BTMTransactionManagerLookup"/>
<entry key="hibernate.current_session_context_class" value="jta"/>
</util:map>
<bean id="dataSource-emf" name="accessControlDb" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="persistenceXmlLocation" value="classpath*:META-INF/foo-persistence.xml" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="true"/>
<property name="databasePlatform" value="org.hibernate.dialect.Oracle10gDialect"/>
</bean>
</property>
<property name="jpaPropertyMap" ref="jpa_property_map"/>
<property name="jpaDialect"><bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect"/></property>
</bean>
Note the dataSource property refers to the id of the dataSource I declared. The persistenceXmlLocation refers to a persistence xml file that exists in the classpath somewhere. The classpath*: indicates it may be in any jar. Without the * it won't find it if it's in a jar for some reason.
I found util:map to be a handy way to put the jpaPropertyMap values in one place so that I don't need to repeat them when I use multiple entity manager factories in one application context.
Note that the util:map above won't work unless you include the proper settings in the outer beans element. Here is the header of the xml file that I use:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:tx="http://www.springframework.org/schema/tx" xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-2.5.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-2.5.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
Lastly, in order for Bitronix (or apparently any cpool which supports two phase commit) to work with Oracle you need to run the following grants as user SYS. (See http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/rtrb_dsaccess2.html and http://docs.codehaus.org/display/BTM/FAQ and http://docs.codehaus.org/display/BTM/JdbcXaSupportEvaluation#JdbcXaSupportEvaluation-Oracle)
grant select on pending_trans$ to <someUsername>;
grant select on dba_2pc_pending to <someUsername>;
grant select on dba_pending_transactions to <someUsername>;
grant execute on dbms_system to <someUsername>;
Those grants need to be run for any user that a connection pool is set up for regardless of whether you actually do any modifying of anything. It apparently looks for those tables when a connection is established.
A few other misc issues:
You can't query tables which are remote synonyms in Oracle while inside a Spring #Transactional block while using Bitronix (you'll get an ORA-24777). Use materialized views or a separate EntityManager which directly points at the other DB instead.
For some reason the btmConfig in the applicationContext.xml has problems setting config values. Instead create a bitronix-default-config.properties file. The config values you can use are found at http://docs.codehaus.org/display/BTM/Configuration13 . Some other config info for that file is at http://docs.codehaus.org/display/BTM/JdbcConfiguration13 but I haven't used it.
Bitronix uses some local files to store transactional stuff. I don't know why, but I do know that if you have multiple webapps with local connection pools you will have problems because they will both try to access the same files. To fix this specify different values for bitronix.tm.journal.disk.logPart1Filename and bitronix.tm.journal.disk.logPart2Filename in the bitronix-default-config.properties for each app.
Bitronix javadocs are at http://www.bitronix.be/uploads/api/index.html .
That's pretty much it. It's very fiddly to get it to work, but it's working now and I'm happy. I hope that all this helps others who are going through the same troubles I did to get this all to work.
When I do connection pooling I tend to use the one provided by the app server I'm deploying on. It's just a JNDI name to Spring at that point.
Since I don't want to worry about an app server when I'm testing, I use a DriverManagerDataSource and its associated transaction manager when I'm unit testing. I'm not as concerned about pooling or performance when testing. I do want the tests to run efficiently, but pooling isn't a deal breaker in that case.