Spring Batch: Footer callback of multiple file writer not giving exact count - spring

If item writers are writing 2 records to file A and 1 record in file B then Trailer count of both the files(A & B) are 3.
I have a reader, processor and ClassifierCompositeItemWriter. In classifier i have two item writers those are giving valid outputs but the footer callback is not proper. in both the file trailer count is same though the record counts are different.
<batch:job id="abc-job" parent="xyzJob">
<batch:step id="inputfile">
<batch:tasklet>
<batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="1000" >
<batch:streams>
<batch:stream ref="AFileWriter"/>
<batch:stream ref="BFileWriter"/>
</batch:streams>
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<beans:bean id="itemWriter" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter">
<beans:property name="classifier" ref="classifier" />
</beans:bean>
<beans:bean id="classifier" class="org.springframework.batch.classify.BackToBackPatternClassifier">
<beans:property name="routerDelegate">
<beans:bean class="com.abc.classifier.MyClassifier" />
</beans:property>
<beans:property name="matcherMap">
<beans:map>
<beans:entry key="A" value-ref="AFileWriter" />
<beans:entry key="B" value-ref="BFileWriter" />
</beans:map>
</beans:property>
</beans:bean>
<beans:bean id="1FileWriter" parent="parentItemWriter1">
<beans:property name="name" value="AFileWriter"/>
<beans:property name="resource" ref="AFile"/>
</beans:bean>
<beans:bean id="2FileWriter" parent="parentItemWriter2">
<beans:property name="name" value="BFileWriter"/>
<beans:property name="resource" ref="BFile"/>
</beans:bean>
Footer callback-
public class ItemCountFooterCallback implements FlatFileFooterCallback
{
private AtomicInteger count;
public ItemCountFooterCallback(final AtomicInteger count)
{
this.count = count;
}
public void writeFooter(final Writer writer) throws IOException
{
writer.append("Trailer " + this.count.toString());
}
}
I expect the output of A and B file's trailer record to be exact number rows of that particular file.

footerCallback is registered at the step level, hence it will use the write.count of the step, which is the total count of written items (3 in your case).
What you can do is to have a write count for each writer (writer1.count and writer2.count for example) and set a footer callback on each writer (not at the step level). Each footer callback should write the item count of the writer it is attached to.

Related

how to indicate the end of csv file using spring batch?

I have a csv file that contains 4 lines.
Step is executed normal, But it throws an error when it goes beyond the 4th line and I have only 4 lines.
in fieledSetMapper class I display the lines of my file
public class BatchFieldSetMapper implements FieldSetMapper<Batch>{
#Override
public Batch mapFieldSet(FieldSet fieldSet) throws BindException {
Batch result = new Batch();
result.setInstitution(fieldSet.readString(0));
System.out.println("Institution ==> " + result.getInstitution());
result.setType(fieldSet.readString(1));
System.out.println("Type ==> " + result.getType());
result.setNom(fieldSet.readString(2));
System.out.println("Nom ==> " + result.getNom());
result.setRubrique(fieldSet.readString(3));
System.out.println("Rubrique ==> " + result.getRubrique());
result.setMontantPaye(fieldSet.readDouble(4));
System.out.println("MT P ==> " + result.getMontantPaye());
result.setMontantRetenu(fieldSet.readDouble(5));
System.out.println("MT R ==> " + result.getMontantRetenu());
return result;
}
}
And this error appears
org.springframework.batch.item.file.FlatFileParseException: Parsing error at line: 5 in resource=[URL [file:C:/Temp/kk/xx.csv]], input=[;;;;;]
But I don't know how to indicate the end of the file normally it should do it automatically? no ?
PS : I upload the file using primefaces as UploadedFile And I convert it using this method to put it in a temporary file and for the batch to retrieve it and apply subsequent processing
public void uploadFile(FileUploadEvent e) throws IOException{
UploadedFile uploadedCsv=e.getFile();
String filePath="C:/Temp/kk/xx.csv";
byte[] bytes=null;
if(uploadedCsv != null){
bytes=uploadedCsv.getContents();
BufferedOutputStream stream = new BufferedOutputStream(new FileOutputStream(new File(filePath)));
String filename = FilenameUtils.getName(uploadedCsv.getFileName());
stream.write(bytes);
stream.close();
}
ApplicationContext context = new ClassPathXmlApplicationContext("spring-batch-context.xml");
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
Job job = (Job) context.getBean("batchJob");
try {
JobExecution execution = jobLauncher.run(job, new JobParameters());
} catch (JobExecutionException e1) {
System.out.println("Job Batch failed");
e1.printStackTrace();
}
}
And here is my spring-batch-context
<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- à voir comment récuperer le nom du fichier et le mettre au value -->
<bean id="multiResourceItemReader" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/Temp/kk/xx.csv" />
<property name="delegate" ref="flatFileItemReader" />
</bean>
<!-- ItemReader reads a complete line one by one from input file -->
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<!-- Mapper which maps each individual items in a record to properties in POJO -->
<bean class="ma.controle.gestion.springbatch.BatchFieldSetMapper" />
</property>
<property name="lineTokenizer">
<!-- A tokenizer class to be used when items in input record are separated by specific characters -->
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- ItemWriter which writes data to database -->
<bean id="databaseItemWriter" class="org.springframework.batch.item.database.HibernateItemWriter">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<!-- Optional ItemProcessor to perform business logic/filtering on the input records -->
<bean id="itemProcessor" class="ma.controle.gestion.springbatch.BatchItemProcessor" />
<!-- Optional JobExecutionListener to perform business logic before and after the job -->
<bean id="jobListener" class="ma.controle.gestion.springbatch.BatchJobItemListener" />
<!-- Actual Job -->
<batch:job id="batchJob">
<batch:step id="step1">
<batch:tasklet transaction-manager="txManager">
<batch:chunk reader="multiResourceItemReader" writer="databaseItemWriter"
processor="itemProcessor" commit-interval="10" />
</batch:tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="jobListener" />
</batch:listeners>
</batch:job>
your 5th line is having empty values seems your BatchFieldSetMapper mapper is failing so your reader is throwing an exception.Can you check your mapper for null values.

Is it possible to using expression on Spring SecurityMetadataSource?

i want to manage url authorization by Database. So, i'm implement Security MetadataSource. It was perfect except cann't using expression.
below is my code and xml settings.
xml
<beans:bean id="filterSecurityInterceptor" class="org.springframework.security.web.access.intercept.FilterSecurityInterceptor">
<beans:property name="authenticationManager" ref="authenticationManager" />
<beans:property name="accessDecisionManager" ref="accessDecisionManager" />
<beans:property name="securityMetadataSource" ref="securityMetadataSource" />
</beans:bean>
<beans:bean id="accessDecisionManager" class="org.springframework.security.access.vote.AffirmativeBased">
<beans:constructor-arg>
<beans:list>
<beans:bean class="org.springframework.security.access.vote.RoleVoter">
<beans:property name="rolePrefix" value="" />
</beans:bean>
</beans:list>
</beans:constructor-arg>
<beans:property name="allowIfAllAbstainDecisions" value="false" />
</beans:bean>
<beans:bean id="securityMetadataSource" class="my.package.CustomSecurityMetadataSource">
</beans:bean>
java
public class CustomSecurityMetadataSource implements FilterInvocationSecurityMetadataSource {
#Override
public Collection<ConfigAttribute> getAttributes(Object object) throws IllegalArgumentException {
FilterInvocation fi = (FilterInvocation) object;
String url = fi.getRequestUrl();
HttpServletRequest request = fi.getHttpRequest();
// TODO get url authorization from db and caching
String[] roles = new String[] { "ROLE_ANONYMOUS", "ROLE_USER"};
return SecurityConfig.createList(roles);
}
#Override
public Collection<ConfigAttribute> getAllConfigAttributes() {
return null;
}
#Override
public boolean supports(Class<?> clazz) {
return FilterInvocation.class.isAssignableFrom(clazz);
}
}
i want to using expression like hasAnyRole("ROLE_ADMIN", "ROLE_USER").
how can i use expression?

Spring JdbcCursorItemReader in spring batch application

My use case is as follows:
There is an Employee table and columns are as follows:
employee_id;
empoyee_dob;
employee_lastName;
employee_firstName;
employee_zipCode
Now there is an use-case to build a list of Employees present in Dept 'A' and zipcode 11223 and also employees present in Dept B and zipcode 33445.
I have configured a spring job as follows:
<batch:job id="EmployeeDetailsJob" job-repository="EmpDaoRepository">
<batch:step id="loadEmployeeDetails" >
<batch:tasklet transaction-manager="EmployeeDAOTranManager">
<batch:chunk reader="EmpDaoJdbcCursorItemReader" writer="EmpDaoWriter" commit-interval="200" skip-limit="100">
<batch:skippable-exception-classes>
</batch:skippable-exception-classes>
</batch:chunk>
<batch:listeners>
<batch:listener ref="EmpDaoStepListener" />
</batch:listeners>
<batch:transaction-attributes isolation="DEFAULT" propagation="REQUIRED" timeout="300" />
</batch:tasklet>
</batch:step>
</batch:job>
The configuration of reader is as follows:
<bean id="EmpDaoJdbcCursorItemReader" class="EmpDaoJdbcCursorItemReader">
<property name="dataSource" ref="EmpDataSource" />
<property name="sql">
<value><![CDATA[select * from Employee where employee_id=? and employee_zipCode=? ]]>
</value>
</property>
<property name="fetchSize" value="100"></property>
<property name="rowMapper" ref="EmployeeMapper" />
</bean>
There is class EmployeeQueryCriteria which has two fields employee_id and employee_zipCode.
In on of the steps i will create an ArrayList of EmployeeQueryCriteria objects for which the data has to be fetched.
So my question is:
1.Is there a way i can pass this List to the EmpDaoJdbcCursorItemReader and it will iterate through the object and set the parameter values from the EmployeeQueryCriteria object
2.Can i loop through the step to read data for every item in the ArrayList created containing EmployeeQueryCriteria and fetch the data.
The class EmpDaoJdbcCursorIte‌​mReader:
public class EmpDaoJdbcCursorIte‌​mReader extends JdbcCursorItemReader{
#BeforeStep
public void beforeStep(StepExecution stepExecution)
{
StringBuffer sqlQuerySB= new StringBuffer(super.getSql());
sqlQuerySB.append((").append(/*I am adding a comma seperated list of employee ids*/).append(")");
super.setSql(sqlQuerySB.toString());
}
}
My Spring configurations are as follows:
Spring-batch-core 2.2.2
Spring-beans 3.2.3
Spring-context 3.2.3
Can someone please provide suggestions on how to solve this problem.
you can iterate through the steps by following code model
<decision id="testLoop" decider="iterationDecider">
<next on="CONTINUABLE" to="pqrStep" />
<end on="FINISHED" />
</decision>
<step id="pqrStep" next="xyzStep">
<tasklet ref="someTasklet" />
</step>
<step id="xyzStep" next="testLoop">
<tasklet ref="someOtherTasklet" />
</step>
and Configuration is
<bean id="iterationDecider" class="com.xyz.StepFlowController" />
Following class will handle the flow based on the condition
public class StepFlowController implements JobExecutionDecider{
#Override
public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
FlowExecutionStatus status = null;
try {
if (conditionIsTrue) {
status = new FlowExecutionStatus("CONTINUABLE");
}else {
status = new FlowExecutionStatus("FINISHED");
}
} catch (Exception e) {
e.printStackTrace();
}
return status;
}

Spring batch-Circular reference error while using Java Beans but the application runs properly by using xml beans

Bean "delegateItemWriter" and "itemWriter" have cyclic calls to each other.
When I comment xml bean(which runs fine) and run by using java beans it gives me error as "Requested bean is currently in creation".
Can anyone help why this is happening with java beans only and not for xml beans?
Any help would be appreciated.
Batch Job defination:
<job id="fileToFileWithHeaderFooterJob" job-repository="jobRepository">
<step id="step">
<tasklet>
<chunk reader="itemReader" processor="itemProcessor" writer="itemWriter"
commit-interval="2">
<streams>
<stream ref="delegateItemWriter" />
</streams>
</chunk>
<listeners>
<listener ref="itemWriter" />
</listeners>
</tasklet>
</step>
</job>
If I comment below xml bean(delegateItemWriter) my application breaks and gives me error as Error creating bean with name 'delegateItemWriter': Requested bean is currently in creation: Is there an unresolvable circular reference?
<beans:bean id="delegateItemWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter">
<beans:property name="resource" ref="outputFileResource" />
<beans:property name="shouldDeleteIfExists" value="true" />
<beans:property name="lineAggregator">
<beans:bean
class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<beans:property name="delimiter" value="," />
<beans:property name="fieldExtractor">
<beans:bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<beans:property name="names"
value="newId,newName,newDate,newParty,newPrice" />
</beans:bean>
</beans:property>
</beans:bean>
</beans:property>
<beans:property name="headerCallback" ref="headerCallback" />
<beans:property name="footerCallback" ref="itemWriter" />
</beans:bean>
Java Config:
#Bean
public FlatFileItemReader<Object> itemReader() {
FlatFileItemReader<Object> itemReader = new FlatFileItemReader<>();
DefaultLineMapper<Object> lineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setNames(new String[] {"dealId", "price", "name", "party", "Date"});
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(new TradeDataFieldSetMapper());
itemReader.setResource(new ClassPathResource("inputData.txt"));
itemReader.setLineMapper(lineMapper);
return itemReader;
}
#Bean
public SimpleItemProcessor itemProcessor() {
return new SimpleItemProcessor();
}
#Bean
public FileToFileFooterCallback itemWriter(final ItemWriter<TradeDataOutput> delegetItemWriter) {
FileToFileFooterCallback footerCallback = new FileToFileFooterCallback();
footerCallback.setDelegate(delegetItemWriter);
return footerCallback;
}
#Bean
public FlatFileItemWriter<TradeDataOutput> delegateItemWriter(final FileToFileFooterCallback itemWriter) {
FlatFileItemWriter<TradeDataOutput> writer = new FlatFileItemWriter<>();
LineAggregator<TradeDataOutput> delimitedLineAggregator = new DelimitedLineAggregator<>();
FieldExtractor<TradeDataOutput> beanWrapperFieldExtractor = new BeanWrapperFieldExtractor<>();
String[] names = {"newID", "newName", "newDate", "newParty", "newPrice"};
((BeanWrapperFieldExtractor<TradeDataOutput>) beanWrapperFieldExtractor).setNames(names);
((DelimitedLineAggregator<TradeDataOutput>) delimitedLineAggregator).setDelimiter(",");
((DelimitedLineAggregator<TradeDataOutput>) delimitedLineAggregator).setFieldExtractor(beanWrapperFieldExtractor);
writer.setLineAggregator(delimitedLineAggregator);
writer.setResource(new FileSystemResource(
"batch/filetoFileWithHeaderFooterOutputFile.data"));
writer.setShouldDeleteIfExists(true);
writer.setHeaderCallback(new FileToFileHeaderCallback());
writer.setFooterCallback((FlatFileHeaderCallback) itemWriter);
return writer;
}

Spring ContentNegotiatingViewResolver - How to use bean name for jsp view not full url with path parameters

My servlet-context file has
<beans:bean
class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver">
<beans:property name="useNotAcceptableStatusCode"
value="false" />
<beans:property name="contentNegotiationManager">
<beans:bean
class="org.springframework.web.accept.ContentNegotiationManager">
<beans:constructor-arg>
<beans:bean
class="org.springframework.web.accept.PathExtensionContentNegotiationStrategy">
<beans:constructor-arg>
<beans:map>
<beans:entry key="html" value="text/html" />
<beans:entry key="json" value="application/json" />
</beans:map>
</beans:constructor-arg>
</beans:bean>
</beans:constructor-arg>
</beans:bean>
</beans:property>
<beans:property name="viewResolvers">
<beans:list>
<beans:bean
class="org.springframework.web.servlet.view.BeanNameViewResolver"/>
<beans:bean id="jspView"
class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<beans:property name="prefix" value="/WEB-INF/jsp/" />
<beans:property name="suffix" value=".jsp" />
</beans:bean>
</beans:list>
</beans:property>
<beans:property name="defaultViews">
<beans:list>
<beans:bean
class="org.springframework.web.servlet.view.json.MappingJackson2JsonView" />
</beans:list>
</beans:property>
</beans:bean>
My Controller File has
#Controller("resources")
public class Resources {
#RequestMapping(value = "/resources/{name}", method = RequestMethod.GET)
public Map getResource(#PathVariable String name) {
return new HashMap();
}
}
But whenever i try to access /server/resources/myfilename.html
Server throws 404 saying /server/WEB-INF/jsp/resources/myfilename.jsp is not found.
But it should load /server/WEB-INF/jsp/resources.jsp as im using BeanNameViewResolver. Please help.
What you get:
Controller return a null view name, so DefaultRequestToViewNameTranslator generates one from URI = path from servlet path without slashes and filename extension: resources/myfilename
BeanNameViewResolver try to get a View from context named resources/myfilename, seem that fails and chain to next ViewResolver
InternalResourceViewResolver return a JstlView pointing to jsp prefix + viewName + suffix = /WEB-INF/jsp/resources/myfilename.jsp
So BeanNameViewResolver seems that don't resolve the view and the return of InternalResourceViewResolver is the expected one.
What you want (I think)
You want to remove the filename from the default view name, not only the extension.
Implements a RequestToViewNameTranslator and declare it in the DispatcherServlet context with name viewNameTranslator.
For example:
public class StripFileNameViewNameTranslator extends DefaultRequestToViewNameTranslator {
#Override
protected String transformPath(String lookupPath) {
String path = super.transformPath(lookupPath);
return StringUtils.substringBeforeLast(path, "/");
}
}

Resources