I am new to spring batch , the csv file for example
Employee Name Id age year
Department DeptName deptId desc
Department DeptName deptId desc
Employee Name Id age year
Department DeptName deptId desc
Department DeptName deptId desc
This is my format
If that employer with name exists mysql will throw DataIntegrityViolationException
So i added
<batch:skippable-exception-classes>
<batch:include class="org.springframework.dao.DataIntegrityViolationException" />
</batch:skippable-exception-classes>
So this will skip the whole unit the corresponding Employer and Department records
what I have to do even if the Employee exists and throws this Exception
Then to continue with Department.
I would recommend to use item classifier
public class ItemTypeClassifier {
#Classifier
public String classify(Item item) {
return item.getType();// returns "Employee", "Department"
}
}
Then a router implementation with two different item writer, one for employee and one for department.
<bean id="classifier" class="org.springframework.batch.classify.BackToBackPatternClassifier">
<property name="routerDelegate">
<bean class="ItemTypeClassifier" />
</property>
<property name="matcherMap">
<map>
<entry key="Employee" value-ref="emplyeeItemWriter" />
<entry key="Department" value-ref="departmentItemWriter" />
</map>
</property>
</bean>
And then
<bean id="ItemWriter" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter">
<property name="classifier" ref="classifier" />
</bean>
Now that you have two different writer, you can run two steps inside your job by defining two separate skippable exceptions configuration for them.
Related
I am using a int-jdbc:inbound-channel-adapter.
I am facing issue with my update statement.I need pass single value ,since I have hard coded that value in RowMapper ,that's why I am receiving 5 values in place of 1.(max-rows-per-poll=5)
My Requirement :- I need to pass a system variable to the node_id column and 5 ids select from select query to in condition
<int-jdbc:inbound-channel-adapter id="itemsInboundJdbcChannelAdapter"
auto-startup="true" data-source="dataSource" channel="ItemsInboundJdbcChannel"
query="SELECT QXXXX_ID,BXXXX_ID,TXXXX_ID,RXXXX_ID,EXXXXID,NODE_ID FROM XXXX_QXXXX WHERE XXXX_STATUS = :identier ORDER BY QXXXX_ID asc FOR UPDATE SKIP LOCKED"
update="UPDATE XXXX_QXXXX SET XXXX_STATUS ='IT_PROCESSED',NODE_ID=(:NODE_ID),UPDATE_BY='BISWO',UPDATED_ON=SYSDATE WHERE QXXXX_ID IN (:QXXXX_ID)"
row-mapper="xxDataRowMapper" max-rows-per-poll="${item.transfer.jdbc.max.rows}"
select-sql-parameter-source="myItemsSelectSqlParameterSource"
update-sql-parameter-source-factory="myUpdateParameterSource">
<int:poller id="jdbcPoller" fixed-delay="${item.transfer.poller.jdbc.fixed.delay}" task-executor="notificationExecutor" error-channel="chainToFailedOut">
<int:transactional transaction-manager="transactionManager" />
</int:poller>
</int-jdbc:inbound-channel-adapter>
<int:service-activator input-channel="ItemsInboundJdbcChannel"
ref="getMyService" >
<int:poller fixed-delay="${item.transfer.poller.jdbc.fixed.delay}" />
</int:service-activator>
<bean id="myUpdateParameterSource"
class="org.springframework.integration.jdbc.ExpressionEvaluatingSqlParameterSourceFactory">
<property name="parameterExpressions">
<map>
<entry key="NODE_ID" value=" #this['nodeId']"/>
<entry key="QXXXX_ID" value="#this['qxxxxId']" />
</map>
</property>
</bean>
xxxDataRowMapper.java
#Override
public XXXDataModel mapRow(ResultSet rs, int rowNum) throws SQLException {
XXDataModel object=null;
if(rs!=null){
long queueId=rs.getLong("QXXXX_ID");
long batchId=rs.getLong("BXXXX_ID");
long tradeRunId=rs.getLong("TXXXX_ID");
long riskRunId=rs.getLong("RXXXX_RUN_ID");
long eventId=rs.getLong("EXXXXID");
object=new XXDataModel(queueId,batchId,tradeRunId,riskRunId,eventId,this.nodeId);
}
return object;
}
I think you simply can go the same properties placeholder way and do something like this:
update="UPDATE XXXX_QXXXX SET XXXX_STATUS ='IT_PROCESSED',NODE_ID='${node_id.from-system.properties}', UPDATE_BY='BISWO',UPDATED_ON=SYSDATE WHERE QXXXX_ID IN (:QXXXX_ID)"
Spring will understand your PP pattern, resolves it against an Environment and the final value will be present in the target SQL statement to use.
I have to read from a file (FlatFile) and update a column if that ID present in the file matches the id in the column.The file is being read properly but only the last id value is getting updated here . Please find the snippet
Job-Config.xml
<bean id="abcitemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="file:datafile/outputs/ibdData.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="ID,NAM,TYPE" />
<property name="delimiter" value="|"/>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.pershing.intraware.springbatch.mapper.abcFieldsetMapper" />
</property>
</bean>
</property>
</bean>
<bean id="abcitemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter" scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"><value>UPDATE TEST_abc SET BIZ_ARNG_CD = CASE WHEN ID IN (SELECT ID FROM TEST_abc WHERE ID= ? and MONTH=(to_char(sysdate, 'MM')) AND YR =(to_char(sysdate, 'YY'))) THEN 'Y' ELSE 'N' END</value></property>
<!-- It will take care matching between object property and sql name parameter -->
<property name="itemPreparedStatementSetter" ref="testPrepStatementSetter" />
</bean>
</beans>
Setter.java
public class IDItemPreparedStatementSetter implements ItemPreparedStatementSetter<Test> {
#Override
public void setValues(Test item, PreparedStatement ps) throws SQLException {
// TODO Auto-generated method stub
ps.setString(1, item.getID());
}
}
Your query is updating each row of database every time it is fired. You need to restrict that. Currently; it must be setting the BIZ_ARNG_CD to 'Y' for records with ID equal to the ID of the last record passed to the writer.
You can fix this in 2 ways -
Default the database column to 'N' and don't set it to 'N' in the update statement
Add where clause in update script ( BIZ_ARNG_CD != 'Y')
In spring batch, I have a requirement where i have list of customer objects from database and from this list i need to create multiple text files.
clas Customer{
long customerId;
string name;
Address add;
Phone phn;
}
class Address{
string address;
long pincode;
string street;
}
class Phone{
string phoneNumber;
string desc;
}
I want to write the data of each customer to different text files such as
customer.txt -> customerId, name
address.txt -> address, pincode, street
phone.txt -> phoneNumber, desc
i tried using CompositeItemWriter which can delegate object to different writer but i could not get how to retrive specific object or properties in the writer.
I couldnt get any sample or explaination to implement CompositeWriter. can someone please help me with the writer implementation? Is there any other better way of doing it?
Thanks in advance.
I was able to write data of each customer to different text file using CompositeItemWriter. Below is how i did for customer phone extract. I hope it helps someone.
<bean id="compositeCustomerWriter"
class="org.springframework.batch.item.support.CompositeItemWriter">
<property name="delegates">
<list>
<ref bean="customerWriter" />
<ref bean="customerPhoneWriter" />
<ref bean="customerAddressWriter" />
</list>
</property>
</bean>
<bean id="customerPhoneWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:customer_phone_extract.txt" />
<property name="lineAggregator" ref="customerPhonelineAggregator" />
</bean>
<bean id="customerPhonelineAggregator"
class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="|" />
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names"
value="phn.phoneNumber,phn.desc" />
</bean>
</property>
</bean>
I can't get spring to initialize my database, although from what I can see I'm doing this right. I keep getting a org/springframework/dao/DataAccessResourceFailureException.
Here is my springapp-servlet.xml:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
</bean>
<jdbc:initialize-database data-source="dataSource" ignore-failures="DROPS">
<jdbc:script location="classpath:create_products.sql" />
<jdbc:script location="classpath:load_data.sql" />
</jdbc:initialize-database>
The jdbc.properties file is in Java Resources/src and is this:
jdbc.driverClassName=org.hsqldb.jdbcDriver
jdbc.url=jdbc:hsqldb:hsql://localhost
jdbc.username=sa
jdbc.password=
The create_products.sql is also in Java Resources/src and is this:
CREATE TABLE products (
id INTEGER NOT NULL PRIMARY KEY,
description varchar(255),
price decimal(15,2)
);
The load_data.sql is in the same location and is this:
INSERT INTO products (id, description, price) values(1, 'Lamp', 5.78);
INSERT INTO products (id, description, price) values(2, 'Table', 75.29);
INSERT INTO products (id, description, price) values(3, 'Chair', 22.81);
Any suggestions?
Thanks.
Dave probably you are also getting this error in your springapp-servlet.xml cvc-complex-type.2.4.b: The content of element 'jdbc:initialize-database' is not complete.
One of '{"http://www.springframework.org/schema/jdbc":script}' is expected. ?
The org.springframework.jdbc.datasource.init package provides support for initializing an existing DataSource. Just check if your spring-jdbc version.
Ref: [http://static.springsource.org/spring/docs/3.0.0.RC3/reference/html/ch12s09.html][1]
Just open the spring-jdbc jar and see under: org/springframework/jdbc/datasource if you can see any folder by name init .If init folder doesnt exist, then the version of Spring Framework you are using does not support this feature.
I'm creating a web application with Spring 3.1.0.RELEASE and JPA 2 with Hibernate Provider.
I'm doing the test with junit 4.10 , dbunit 2.4.8, unitils 3.3, hsqldb 2.2.8.
I try to test the service layer, a create operation. In my DAO i have this method:
#Override
#Transactional
public void createQuestion(Question question) {
logger.debug("createQuestion");
entityManager.persist(question);
logger.info("New question created [id] {}", question.getId());
}
My QuestionServiceTest test class :
#SpringApplicationContext("test-applicationContext.xml")
public class QuestionServiceTest extends UnitilsJUnit4 {
#SpringBeanByName
private QuestionService questionService;
#SpringBeanByName
private ThemeService themeService;
#Test
#DataSet("QuestionServiceTest.testCreateQuestion.xml")
#ExpectedDataSet("QuestionServiceTest.testCreateQuestion-result.xml")
public void testCreateQuestion() {
final Question newQuestion = new Question();
newQuestion.setCountryCode("FR");
newQuestion.setEmail("test#mytest.com");
newQuestion.setFirstName("FirstTest");
newQuestion.setLastName("LastTest");
newQuestion.setOriginalLang(LanguageEnum.FR);
newQuestion.setOriginalQuestion("This is the original question");
final Calendar calendar = Calendar.getInstance();
calendar.set(2012, 5, 12);
newQuestion.setCreationDate(calendar.getTime());
final Theme theme = themeService.findThemeById(new Integer(1));
newQuestion.setTheme(theme);
questionService.createQuestion(newQuestion);
}
}
I use the property hibernate.hbm2ddl.auto = create-drop for generate the schema, the question table is:
create table question (
id integer generated by default as identity (start with 1),
country_code varchar(10) not null,
creation_date timestamp not null,
email varchar(255) not null,
firstname varchar(100) not null,
lastname varchar(100) not null,
original_lang varchar(255) not null,
original_question clob not null,
theme_id integer not null,
primary key (id)
)
theme_id is a foreign key to table theme.
When i launch the test with ExpectedDataSet, the insert works but the test never finish.
The test block on :
DEBUG: org.dbunit.database.AbstractResultSetTable - Query: select
"ID", "COUNTRY_CODE", "CREATION_DATE", "EMAIL", "FIRSTNAME",
"LASTNAME", "ORIGINAL_LANG", "ORIGINAL_QUESTION", "THEME_ID" from
"PUBLIC"."QUESTION" order by "ID"
This is the last line on debug.
My unitils.properties is :
# Defaults and other keys with explanations can be found there: http://unitils.org/unitils-default.properties
database.driverClassName=org.hsqldb.jdbcDriver
database.url=jdbc:hsqldb:mem:testOpen
database.userName=sa
database.password=
database.dialect=hsqldb
# This schema is the initial schema when a new session is started in HSQLDB, don't change it or test won't works !
database.schemaNames=PUBLIC
dbUnit.datasetresolver.prefixWithPackageName=false
dbUnit.datasetresolver.pathPrefix=dataSets
My persistence.xml :
<persistence-unit name="OpenTestPU" transaction-type="RESOURCE_LOCAL">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect" />
<property name="hibernate.show_sql" value="true" />
<property name="hibernate.format_sql" value="true" />
<property name="hibernate.connection.driver_class" value="org.hsqldb.jdbcDriver" />
<property name="hibernate.connection.username" value="sa" />
<property name="hibernate.connection.password" value="" />
<property name="hibernate.connection.url" value="jdbc:hsqldb:mem:testOpen" />
<property name="hibernate.hbm2ddl.auto" value="create-drop" />
</properties>
</persistence-unit>
What should I do? I already try previous release of dbunit or unitils but it doesn't change anything. Expected Dataset is really cool feature.
Thanks.
I had the same issue and failed to solve it. Then I gave a try to
spring-test-dbunit
and this lib run smoothly.