I am invoking a spring batch job through quartz scheduler, which should run every 1 minute.
When the job runs the first time, the ItemReader is opened successfully and the job runs. However when the job attempts to run a second time, it's using the same instance it did the first time which is already initialized and receiving "java.lang.IllegalStateException: Stream is already initialized. Close before re-opening." I have set scope as step for both itemreader and itemwriter.
Please let me know if I am doing anything wrong in configuration?
<?xml version="1.0" encoding="UTF-8"?>
<import resource="context.xml"/>
<import resource="database.xml"/>
<bean id="MyPartitioner" class="com.MyPartitioner" />
<bean id="itemProcessor" class="com.MyProcessor" scope="step" />
<bean id="itemReader" class="com.MyItemReader" scope="step">
<property name="dataSource" ref="dataSource"/>
<property name="sql" value="query...."/>
<property name="rowMapper">
<bean class="com.MyRowMapper" scope="step"/>
</property>
</bean>
<job id="MyJOB" xmlns="http://www.springframework.org/schema/batch">
<step id="masterStep">
<partition step="slave" partitioner="MyPartitioner">
<handler grid-size="10" task-executor="taskExecutor"/>
</partition>
</step>
</job>
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
<tasklet>
<chunk reader="itemReader" writer="mysqlItemWriter" processor="itemProcessor" commit-interval="100"/>
</tasklet>
</step>
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="20"/>
<property name="maxPoolSize" value="20"/>
<property name="allowCoreThreadTimeOut" value="true"/>
</bean>
<bean id="mysqlItemWriter" class="com.MyItemWriter" scope="step">
<property name="dataSource" ref="dataSource"/>
<property name="sql">
<value>
<![CDATA[
query.....
]]>
</value>
</property>
<property name="itemPreparedStatementSetter">
<bean class="com.MyPreparedStatementSetter" scope="step"/>
</property>
</bean>
Quartz job invoker-
Scheduler scheduler = new StdSchedulerFactory("quartz.properties").getScheduler();
JobKey jobKey = new JobKey("QUARTZJOB", "QUARTZJOB");
JobDetail jobDetail = JobBuilder.newJob("com.MySpringJobInvoker").withIdentity(jobKey).build();
jobDetail.getJobDataMap().put("jobName", "SpringBatchJob");
SimpleTrigger smplTrg = newTrigger().withIdentity("QUARTZJOB", "QUARTZJOB").startAt(new Date(startTime))
.withSchedule(simpleSchedule().withIntervalInSeconds(frequency).withRepeatCount(repeatCnt))
.forJob(jobDetail).withPriority(5).build();
scheduler.scheduleJob(jobDetail, smplTrg);
Quartz job -
public class MySpringJobInvoker implements Job
{
#Override
public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException
{
JobDataMap data = jobExecutionContext.getJobDetail().getJobDataMap();
ApplicationContext applicationContext =ApplicationContextUtil.getInstance();
JobLauncher jobLauncher = (JobLauncher) applicationContext.getBean("jobLauncher");
org.springframework.batch.core.Job job = (org.springframework.batch.core.Job) applicationContext.getBean(data.getString("jobName"));
JobParameters param = new JobParametersBuilder().addString("myparam","myparam").addString(Long.toString(System.currentTimeMillis(),Long.toString(System.currentTimeMillis())).toJobParameters();
JobExecution execution = jobLauncher.run(job, param);
}
}
Singletonclass -
public class ApplicationContextUtil
{
private static ApplicationContext applicationContext;
public static synchronized ApplicationContext getInstance()
{
if(applicationContext == null)
{
applicationContext = new ClassPathXmlApplicationContext("myjob.xml");
}
return applicationContext;
}
}
what are the parameters that you are passing to the Spring Batch job from Quartz? Could you post the exception stack trace?
If your trying to execute the second instance of the batch with the same parameters - it won't work. Spring Batch identifies a unique instance of the job based on parameters passed - so every new instance of the job requires different parameters to be passed.
Related
I have a csv file that contains 4 lines.
Step is executed normal, But it throws an error when it goes beyond the 4th line and I have only 4 lines.
in fieledSetMapper class I display the lines of my file
public class BatchFieldSetMapper implements FieldSetMapper<Batch>{
#Override
public Batch mapFieldSet(FieldSet fieldSet) throws BindException {
Batch result = new Batch();
result.setInstitution(fieldSet.readString(0));
System.out.println("Institution ==> " + result.getInstitution());
result.setType(fieldSet.readString(1));
System.out.println("Type ==> " + result.getType());
result.setNom(fieldSet.readString(2));
System.out.println("Nom ==> " + result.getNom());
result.setRubrique(fieldSet.readString(3));
System.out.println("Rubrique ==> " + result.getRubrique());
result.setMontantPaye(fieldSet.readDouble(4));
System.out.println("MT P ==> " + result.getMontantPaye());
result.setMontantRetenu(fieldSet.readDouble(5));
System.out.println("MT R ==> " + result.getMontantRetenu());
return result;
}
}
And this error appears
org.springframework.batch.item.file.FlatFileParseException: Parsing error at line: 5 in resource=[URL [file:C:/Temp/kk/xx.csv]], input=[;;;;;]
But I don't know how to indicate the end of the file normally it should do it automatically? no ?
PS : I upload the file using primefaces as UploadedFile And I convert it using this method to put it in a temporary file and for the batch to retrieve it and apply subsequent processing
public void uploadFile(FileUploadEvent e) throws IOException{
UploadedFile uploadedCsv=e.getFile();
String filePath="C:/Temp/kk/xx.csv";
byte[] bytes=null;
if(uploadedCsv != null){
bytes=uploadedCsv.getContents();
BufferedOutputStream stream = new BufferedOutputStream(new FileOutputStream(new File(filePath)));
String filename = FilenameUtils.getName(uploadedCsv.getFileName());
stream.write(bytes);
stream.close();
}
ApplicationContext context = new ClassPathXmlApplicationContext("spring-batch-context.xml");
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
Job job = (Job) context.getBean("batchJob");
try {
JobExecution execution = jobLauncher.run(job, new JobParameters());
} catch (JobExecutionException e1) {
System.out.println("Job Batch failed");
e1.printStackTrace();
}
}
And here is my spring-batch-context
<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- à voir comment récuperer le nom du fichier et le mettre au value -->
<bean id="multiResourceItemReader" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="file:C:/Temp/kk/xx.csv" />
<property name="delegate" ref="flatFileItemReader" />
</bean>
<!-- ItemReader reads a complete line one by one from input file -->
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<!-- Mapper which maps each individual items in a record to properties in POJO -->
<bean class="ma.controle.gestion.springbatch.BatchFieldSetMapper" />
</property>
<property name="lineTokenizer">
<!-- A tokenizer class to be used when items in input record are separated by specific characters -->
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- ItemWriter which writes data to database -->
<bean id="databaseItemWriter" class="org.springframework.batch.item.database.HibernateItemWriter">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<!-- Optional ItemProcessor to perform business logic/filtering on the input records -->
<bean id="itemProcessor" class="ma.controle.gestion.springbatch.BatchItemProcessor" />
<!-- Optional JobExecutionListener to perform business logic before and after the job -->
<bean id="jobListener" class="ma.controle.gestion.springbatch.BatchJobItemListener" />
<!-- Actual Job -->
<batch:job id="batchJob">
<batch:step id="step1">
<batch:tasklet transaction-manager="txManager">
<batch:chunk reader="multiResourceItemReader" writer="databaseItemWriter"
processor="itemProcessor" commit-interval="10" />
</batch:tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="jobListener" />
</batch:listeners>
</batch:job>
your 5th line is having empty values seems your BatchFieldSetMapper mapper is failing so your reader is throwing an exception.Can you check your mapper for null values.
I am trying to convert the spring batch configuration from xml based to annotation based.
Below is my xml based configuration.
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- Step will need a transaction manager -->
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="dbMapper" class="org.test.DBValueMapper">
</bean>
<bean id="dbMapperFlatfile" class="org.test.FlatFileRowMapper">
</bean>
<bean id="paramSetter" class="org.test.DBParamSetter">
</bean>
<bean id="dbReader" class="org.test.DBValueReader"
scope="step">
<property name="paramSetter" ref="paramSetter"/>
<property name="verifyCursorPosition" value="false" />
<property name="dataSource" ref="dataSource" />
<property name="sql" value="#{jobParameters['SQL_QUERY']}" />
<property name="rowMapper" ref="dbMapper" />
<property name="fetchSize" value="5000" />
</bean>
<bean id="dbWriterIO" class="org.test.TemplateWritterIO"
scope="step">
<property name="velocityEngine" ref="velocityEngine" />
<!-- <property name="rptConfig" value="#{jobParameters['RPT_CONFIGVAL']}" /> -->
<property name="headerCallback" ref="dbWriterIO" />
<property name="footerCallback" ref="dbWriterIO" />
</bean>
<batch:job id="fileGenJobNio">
<batch:step id="fileGenJobStempNio">
<batch:tasklet>
<batch:chunk reader="dbReader" writer="dbWriterNIO"
commit-interval="5000">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
Below is the equivalent Java based configuration:
#EnableBatchProcessing
#Import({ServiceConfiguration.class})
public class SRBatchGenerator extends DefaultBatchConfigurer{
#Autowired
private JobBuilderFactory jobBuilders;
#Autowired
private StepBuilderFactory stepBuilders;
#Autowired
private VelocityEngine velocityEngine;
#Autowired
private DBValueMapper mapper;
#Autowired
private DbHelper dbhelper;
#Autowired
private DataSource datasource;
#Bean
public Step step(){
return stepBuilders.get("step")
.chunk(5000)
.reader(reader())
//.processor(processor())
.writer(writer())
//.listener(logProcessListener())
.faultTolerant()
//.skipLimit(10)
//.skip(UnknownGenderException.class)
//.listener(logSkipListener())
.build();
}
#Bean
public Job fileGeneratorJob(){
return jobBuilders.get("fileGeneratorJob")
//.listener(protocolListener())
.start(step())
.build();
}
#Bean
public DBValueMapper mapper(){
return new DBValueMapper();
}
#Bean
#StepScope
public DBValueReader3 reader(){
String query="Select Test1,Test2,test3,test4 from RPt_TEST";
DBValueReader3 dbread = new DBValueReader3();
dbread.setSql(query);
dbread.setRowMapper(mapper);
dbread.setDataSource(datasource);
return dbread;
}
#Bean
#StepScope
public TemplateWritterIO writer(){
TemplateWritterIO writer=new TemplateWritterIO();
writer.setVelocityEngine(velocityEngine);
return writer;
}
#Override
protected JobRepository createJobRepository() throws Exception {
MapJobRepositoryFactoryBean factory =
new MapJobRepositoryFactoryBean();
factory.afterPropertiesSet();
return (JobRepository) factory.getObject();
}
}
When I execute my Job using xml based it took 27sec to write 1 Million record into Flat file.
But to write same 1 million record, Java based job took about 2 hours to write.
I am not sure what I am missing here. Can anyone help me or guide me why it is slow in Java based configuration.
I am trying to configure a spring batch step without an item writer using below configuraion. However i get error saying that writer
element has neither a 'writer' attribute nor a element.
I went through the link spring batch : Tasklet without ItemWriter. But could not resolve issue.
Could any one tell me the specific changes to be made in the code snippet I mentioned
<batch:job id="helloWorldJob">
<batch:step id="step1">
<batch:tasklet>
<batch:chunk reader="cvsFileItemReader"
commit-interval="10">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="classpath:cvs/input/report.csv" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="id,sales,qty,staffName,date" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.mkyong.ReportFieldSetMapper" />
<!-- if no data type conversion, use BeanWrapperFieldSetMapper to map by name
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="report" />
</bean>
-->
</property>
</bean>
</property>
</bean>
For chunk-based step reader and writer are mandatory.
If you don't want a writer use a No-operation ItemWriter that does nothing.
EDIT:
A no-op implementation is an empty implementation of interface tha does...nothing!
Just let your class implements desiderable inteface(s) with empty methods.
No-op ItemWriter:
public class NoOpItemWriter implements ItemWriter {
void write(java.util.List<? extends T> items) throws java.lang.Exception {
// no-op
}
}
I hope you got answer but I want to explain it for other readers, When we use chunk then usually we declare reader, processor and writer. In chunk reader and writer are mandatory and processor is optional. In your case if you don't need writer then u need to make a class which implements ItemWriter. Override write method and keep it blank. Now create a bean of writer class and pass it as reference of writer.
<batch:step id="recordProcessingStep" >
<batch:tasklet>
<batch:chunk reader="fileReader" processor="recordProcessor"
writer="rocordWriter" commit-interval="1" />
</batch:tasklet>
</batch:step>
Your writer class will look like .
public class RecordWriter<T> implements ItemWriter<T> {
#Override
public void write(List<? extends T> items) throws Exception {
// TODO Auto-generated method stub
}
}
In maven repo you can find the framework "spring-batch-samples".
In this framework you will find this Writer :
org.springframework.batch.sample.support.DummyItemWriter
I have written a simple spring batch tasklet which calls a dao method which in turn does some deletes. But I am not sure what I should be doing to call the job.
public class RemoveSpringBatchHistoryTasklet implements Tasklet {
#Autowired
private SpringBatchDao springBatchDao;
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
contribution.incrementWriteCount(springBatchDao.purge());
return RepeatStatus.FINISHED;
}
}
So far to execute my spring batch jobs I am using quartz triggers with a setup like so. Each job has it's own xml file which has a read and a writer.
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="jobDetails">
<list>
<ref bean="dailyTranCountJobDetail" />
</list>
</property>
<property name="triggers">
<list>
<ref bean="dailyTranCountCronTrigger" />
</list>
</property>
</bean>
<bean id="dailyTranCountCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="dailyTranCountJobDetail" />
<property name="cronExpression" value="#{batchProps['cron.dailyTranCounts']}" />
</bean>
<bean id="dailyTranCountJobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="com.myer.reporting.batch.JobLauncherDetails" />
<property name="group" value="quartz-batch" />
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="job-daily-tran-counts" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
</map>
</property>
</bean>
And then here is an example of the job file itself with a reader and a writer.
<job id="job-daily-tran-counts" xmlns="http://www.springframework.org/schema/batch">
<step id="job-daily-tran-counts-step1">
<tasklet transaction-manager="custDbTransactionManager">
<chunk
reader="dailyTranCountJdbcCursorItemReader"
writer="dailyTranCountItemWriter"
commit-interval="1000" />
</tasklet>
</step>
</job>
<bean id="dailyTranCountJdbcCursorItemReader"
class="com.myer.reporting.dao.itemreader.DailyTranCountJdbcCursorItemReader"
scope="step"
parent="abstractEposJdbcDao">
<property name="rowMapper">
<bean class="com.myer.reporting.dao.mapper.DailyTranCountMapper" />
</property>
</bean>
<bean id="dailyTranCountItemWriter"
class="com.myer.reporting.dao.itemwriter.DailyTranCountItemWriter"
parent="abstractCustDbJdbcDao"/>
Obviously for this new job there is no reader or writer. So what it he best/correct way for me to execute my new tasklet?
thanks
I prefer java configuration instead of xml. You can configure your tasklet with the following code:
#Configuration
#EnableBatchProcessing
public class BatchCleanUpJobsConfiguration {
#Bean
public Job batchCleanUpJob(final JobBuilderFactory jobBuilderFactory,
final StepBuilderFactory stepBuilderFactory,
final RemoveSpringBatchHistoryTasklet removeSpringBatchHistoryTasklet) {
return jobBuilderFactory.get("batchCleanUpJob")
.start(stepBuilderFactory.get("batchCleanUpStep")
.tasklet(removeSpringBatchHistoryTasklet)
.build())
.build();
}
#Bean
public RemoveSpringBatchHistoryTasklet batchCleanUpTasklet(final JdbcTemplate jdbcTemplate) {
final var tasklet = new RemoveSpringBatchHistoryTasklet();
tasklet.setJdbcTemplate(jdbcTemplate);
return tasklet;
}
}
To schedule your new job use the following code:
#Component
#RequiredArgsConstructor
public class BatchCleanUpJobsScheduler {
private final Job batchCleanUpJob;
private final JobLauncher launcher;
#Scheduled(cron = "0 0 0 * * MON-FRI")
public void launchBatchCleanupJob()
throws JobParametersInvalidException, JobExecutionAlreadyRunningException,
JobRestartException, JobInstanceAlreadyCompleteException {
launcher.run(
batchCleanUpJob,
new JobParametersBuilder()
.addLong("launchTime", System.currentTimeMillis())
.toJobParameters());
}
}
Is there any way to make the currentResource processed by MultiResourceItemReader to make it available in the beforeStep method.Kindly provide me a working code sample. I tried injected multiresourcereader reference in to stepexecutionlistener , but the spring cglib only accepts an interface type to be injected ,i dont know whether to use ItemReader or ItemStream interface.
The MultiResourceItemReader now has a method getCurrentResource() that returns the current Resource.
Retrieving of currentResource from MultiResourceItemReader is not possible at the moment. If you need this API enhancement, create one in Spring Batch JIRA.
Even if there is a getter for currentResource, it's value is not valid in beforeStep(). It is valid between open() and close().
it is possible, if you use a Partition Step and the Binding Input Data to Steps concept
simple code example, with concurrency limit 1 to imitate "serial" processing:
<bean name="businessStep:master" class="org.springframework.batch.core.partition.support.PartitionStep">
<property name="jobRepository" ref="jobRepository"/>
<property name="stepExecutionSplitter">
<bean class="org.springframework.batch.core.partition.support.SimpleStepExecutionSplitter">
<constructor-arg ref="jobRepository"/>
<constructor-arg ref="concreteBusinessStep"/>
<constructor-arg>
<bean class="org.spring...MultiResourcePartitioner" scope="step">
<property name="resources" value="#{jobParameters['input.file.pattern']}"/>
</bean>
</constructor-arg>
</bean>
</property>
<property name="partitionHandler">
<bean class="org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler">
<property name="taskExecutor">
<bean class="org.springframework.core.task.SimpleAsyncTaskExecutor">
<property name="concurrencyLimit" value="1" />
</bean>
</property>
<property name="step" ref="concreteBusinessStep"/>
</bean>
</property>
</bean>
<bean id="whateverClass" class="..." scope="step">
<property name="resource" value="#{stepExecutionContext['fileName']}" />
</bean>
example step configuration:
<job id="renameFilesPartitionJob">
<step id="businessStep"
parent="businessStep:master" />
</job>
<step id="concreteBusinessStep">
<tasklet>
<chunk reader="itemReader"
writer="itemWriter"
commit-interval="5" />
</tasklet>
</step>
potential drawbacks:
distinct steps for each file instead of one step
more complicated configuration
Using the getCurrentResource() method from MultiResourceItemReader, update the stepExecution. Example code as below
private StepExecution stepExecution;
#Override
public Resource getCurrentResource() {
this.stepExecution.getJobExecution().getExecutionContext().put("FILE_NAME", super.getCurrentResource().getFilename());
return super.getCurrentResource();
}
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
If you make the item you are reading implement ResourceAware, the current resource is set as it is read
public class MyItem implements ResourceAware {
private Resource resource;
//others
public void setResource(Resource resource) {
this.resource = resource;
}
public Resource getResource() {
return resource;
}
}
and in your reader, processor or writer
myItem.getResource()
will return the resource it was loaded from