Is there any way to make the currentResource processed by MultiResourceItemReader to make it available in the beforeStep method.Kindly provide me a working code sample. I tried injected multiresourcereader reference in to stepexecutionlistener , but the spring cglib only accepts an interface type to be injected ,i dont know whether to use ItemReader or ItemStream interface.
The MultiResourceItemReader now has a method getCurrentResource() that returns the current Resource.
Retrieving of currentResource from MultiResourceItemReader is not possible at the moment. If you need this API enhancement, create one in Spring Batch JIRA.
Even if there is a getter for currentResource, it's value is not valid in beforeStep(). It is valid between open() and close().
it is possible, if you use a Partition Step and the Binding Input Data to Steps concept
simple code example, with concurrency limit 1 to imitate "serial" processing:
<bean name="businessStep:master" class="org.springframework.batch.core.partition.support.PartitionStep">
<property name="jobRepository" ref="jobRepository"/>
<property name="stepExecutionSplitter">
<bean class="org.springframework.batch.core.partition.support.SimpleStepExecutionSplitter">
<constructor-arg ref="jobRepository"/>
<constructor-arg ref="concreteBusinessStep"/>
<constructor-arg>
<bean class="org.spring...MultiResourcePartitioner" scope="step">
<property name="resources" value="#{jobParameters['input.file.pattern']}"/>
</bean>
</constructor-arg>
</bean>
</property>
<property name="partitionHandler">
<bean class="org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler">
<property name="taskExecutor">
<bean class="org.springframework.core.task.SimpleAsyncTaskExecutor">
<property name="concurrencyLimit" value="1" />
</bean>
</property>
<property name="step" ref="concreteBusinessStep"/>
</bean>
</property>
</bean>
<bean id="whateverClass" class="..." scope="step">
<property name="resource" value="#{stepExecutionContext['fileName']}" />
</bean>
example step configuration:
<job id="renameFilesPartitionJob">
<step id="businessStep"
parent="businessStep:master" />
</job>
<step id="concreteBusinessStep">
<tasklet>
<chunk reader="itemReader"
writer="itemWriter"
commit-interval="5" />
</tasklet>
</step>
potential drawbacks:
distinct steps for each file instead of one step
more complicated configuration
Using the getCurrentResource() method from MultiResourceItemReader, update the stepExecution. Example code as below
private StepExecution stepExecution;
#Override
public Resource getCurrentResource() {
this.stepExecution.getJobExecution().getExecutionContext().put("FILE_NAME", super.getCurrentResource().getFilename());
return super.getCurrentResource();
}
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
If you make the item you are reading implement ResourceAware, the current resource is set as it is read
public class MyItem implements ResourceAware {
private Resource resource;
//others
public void setResource(Resource resource) {
this.resource = resource;
}
public Resource getResource() {
return resource;
}
}
and in your reader, processor or writer
myItem.getResource()
will return the resource it was loaded from
Related
I am trying to configure a spring batch step without an item writer using below configuraion. However i get error saying that writer
element has neither a 'writer' attribute nor a element.
I went through the link spring batch : Tasklet without ItemWriter. But could not resolve issue.
Could any one tell me the specific changes to be made in the code snippet I mentioned
<batch:job id="helloWorldJob">
<batch:step id="step1">
<batch:tasklet>
<batch:chunk reader="cvsFileItemReader"
commit-interval="10">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="classpath:cvs/input/report.csv" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="id,sales,qty,staffName,date" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.mkyong.ReportFieldSetMapper" />
<!-- if no data type conversion, use BeanWrapperFieldSetMapper to map by name
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="report" />
</bean>
-->
</property>
</bean>
</property>
</bean>
For chunk-based step reader and writer are mandatory.
If you don't want a writer use a No-operation ItemWriter that does nothing.
EDIT:
A no-op implementation is an empty implementation of interface tha does...nothing!
Just let your class implements desiderable inteface(s) with empty methods.
No-op ItemWriter:
public class NoOpItemWriter implements ItemWriter {
void write(java.util.List<? extends T> items) throws java.lang.Exception {
// no-op
}
}
I hope you got answer but I want to explain it for other readers, When we use chunk then usually we declare reader, processor and writer. In chunk reader and writer are mandatory and processor is optional. In your case if you don't need writer then u need to make a class which implements ItemWriter. Override write method and keep it blank. Now create a bean of writer class and pass it as reference of writer.
<batch:step id="recordProcessingStep" >
<batch:tasklet>
<batch:chunk reader="fileReader" processor="recordProcessor"
writer="rocordWriter" commit-interval="1" />
</batch:tasklet>
</batch:step>
Your writer class will look like .
public class RecordWriter<T> implements ItemWriter<T> {
#Override
public void write(List<? extends T> items) throws Exception {
// TODO Auto-generated method stub
}
}
In maven repo you can find the framework "spring-batch-samples".
In this framework you will find this Writer :
org.springframework.batch.sample.support.DummyItemWriter
I have written a simple spring batch tasklet which calls a dao method which in turn does some deletes. But I am not sure what I should be doing to call the job.
public class RemoveSpringBatchHistoryTasklet implements Tasklet {
#Autowired
private SpringBatchDao springBatchDao;
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext)
throws Exception {
contribution.incrementWriteCount(springBatchDao.purge());
return RepeatStatus.FINISHED;
}
}
So far to execute my spring batch jobs I am using quartz triggers with a setup like so. Each job has it's own xml file which has a read and a writer.
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="jobDetails">
<list>
<ref bean="dailyTranCountJobDetail" />
</list>
</property>
<property name="triggers">
<list>
<ref bean="dailyTranCountCronTrigger" />
</list>
</property>
</bean>
<bean id="dailyTranCountCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="dailyTranCountJobDetail" />
<property name="cronExpression" value="#{batchProps['cron.dailyTranCounts']}" />
</bean>
<bean id="dailyTranCountJobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="com.myer.reporting.batch.JobLauncherDetails" />
<property name="group" value="quartz-batch" />
<property name="jobDataAsMap">
<map>
<entry key="jobName" value="job-daily-tran-counts" />
<entry key="jobLocator" value-ref="jobRegistry" />
<entry key="jobLauncher" value-ref="jobLauncher" />
</map>
</property>
</bean>
And then here is an example of the job file itself with a reader and a writer.
<job id="job-daily-tran-counts" xmlns="http://www.springframework.org/schema/batch">
<step id="job-daily-tran-counts-step1">
<tasklet transaction-manager="custDbTransactionManager">
<chunk
reader="dailyTranCountJdbcCursorItemReader"
writer="dailyTranCountItemWriter"
commit-interval="1000" />
</tasklet>
</step>
</job>
<bean id="dailyTranCountJdbcCursorItemReader"
class="com.myer.reporting.dao.itemreader.DailyTranCountJdbcCursorItemReader"
scope="step"
parent="abstractEposJdbcDao">
<property name="rowMapper">
<bean class="com.myer.reporting.dao.mapper.DailyTranCountMapper" />
</property>
</bean>
<bean id="dailyTranCountItemWriter"
class="com.myer.reporting.dao.itemwriter.DailyTranCountItemWriter"
parent="abstractCustDbJdbcDao"/>
Obviously for this new job there is no reader or writer. So what it he best/correct way for me to execute my new tasklet?
thanks
I prefer java configuration instead of xml. You can configure your tasklet with the following code:
#Configuration
#EnableBatchProcessing
public class BatchCleanUpJobsConfiguration {
#Bean
public Job batchCleanUpJob(final JobBuilderFactory jobBuilderFactory,
final StepBuilderFactory stepBuilderFactory,
final RemoveSpringBatchHistoryTasklet removeSpringBatchHistoryTasklet) {
return jobBuilderFactory.get("batchCleanUpJob")
.start(stepBuilderFactory.get("batchCleanUpStep")
.tasklet(removeSpringBatchHistoryTasklet)
.build())
.build();
}
#Bean
public RemoveSpringBatchHistoryTasklet batchCleanUpTasklet(final JdbcTemplate jdbcTemplate) {
final var tasklet = new RemoveSpringBatchHistoryTasklet();
tasklet.setJdbcTemplate(jdbcTemplate);
return tasklet;
}
}
To schedule your new job use the following code:
#Component
#RequiredArgsConstructor
public class BatchCleanUpJobsScheduler {
private final Job batchCleanUpJob;
private final JobLauncher launcher;
#Scheduled(cron = "0 0 0 * * MON-FRI")
public void launchBatchCleanupJob()
throws JobParametersInvalidException, JobExecutionAlreadyRunningException,
JobRestartException, JobInstanceAlreadyCompleteException {
launcher.run(
batchCleanUpJob,
new JobParametersBuilder()
.addLong("launchTime", System.currentTimeMillis())
.toJobParameters());
}
}
In my springbatch+quartz setup, I am reading a CSV File using FlatFileItemReader. I want to set the cursor for the reader to start the next jobinstance with the given parameters for reader. Is it possible?
<bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<!-- Read a csv file -->
<property name="resource" value="classpath:cvs/input/report.csv" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="id,impressions" />
</bean>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="report" />
</bean>
</property>
</bean>
</property>
</bean>
The idea is to continue reading the file where last failure occured in the next execution. I am putting an integer 'writecursor' for each line written in my customWriter.
public void write(List<? extends Report> items) throws Exception {
System.out.println("writer..." + items.size() + " > ");
for(Report item : items){
System.out.println("writing item id: " + item.getId());
System.out.println(item);
}
//getting stepExecution by implementing StepExecutionListener
this.stepExecution.getExecutionContext().putInt("writecursor", ++writecursor);
}
Now, in the customItemReadListener, I want to get the update writecursor value and then skip the lines from the top to start reading from writecursor
public class CustomItemReaderListener implements ItemReadListener<Report>, StepExecutionListener {
ApplicationContext context = ApplicationContextUtils.getApplicationContext();
private StepExecution stepExecution;
#Override
public void beforeRead() {
//Skip lines somehow
}
Another thing I saw as a possible solution is to set linestoskip dynamically in itemreader. There is a thread here http://thisisurl.com/dynamic-value-for-linestoskip-in-itemreader but not answered yet. And here,
http://forum.spring.io/forum/spring-projects/batch/100950-accessing-job-execution-context-from-itemwriter
Use FlatFileItemReader.linesToSkip property setted injecting job Parameter value.
<bean id="myReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="linesToSkip" value="file:#{jobParameters['cursor']}" />
</bean>
A more easy way for implementing the lines to skip is by the following:
create a reader flat file reader in the xml, autowire the reader to the beforeStep of step execution listener as shown below
public class CustomStepListener implements StepExecutionListener {
#Autowired
#Qualifier("cvsFileItemReader")
FlatFileItemReader cvsFileItmRdr;
#Override
public void beforeStep(StepExecution stepExecution) {
System.out.println("StepExecutionListener -------- beforeStep");
cvsFileItmRdr.setLinesToSkip(4);
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
System.out.println("StepExecutionListener - afterStep");
return null;
}
}
it is working fine for me.....
I have a simple job as below:
<batch:step id="step">
<batch:tasklet>
<batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit- interval="5000" />
</batch:tasklet>
</batch:step>
itemReader is as below:
<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
<property name="linesToSkip" value="1"></property>
<property name="skippedLinesCallback" ref="skippedLinesCallback" ></property>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="lineTokenizer">
<property name="delimiter" value="," />
</bean>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
</property>
</bean>
</property>
<property name="resource" value="#{stepExecutionContext['inputKeyName']}" />
</bean>
<bean id"lineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<bean id="skippedLinesCallback" class="com.test.IteMReaderHeader" >
<property name="lineTokenizer" ref="lineTokenizer">
</bean>
I am setting the "names" of the input fields in "com.test.IteMReaderHeader" class by injecting "lineTokenizer" in it.
I need to validate the header lines which is the 1st line in the input csv file with a fixed header value and if the header line invalidates then in that case I need to fail the step and skip the entire file so that the next file can be used for reading.
Please suggest a suitable way of achieving it.
I would really appreciate your time and valuable inputs.
Thanks !!
Looking code of FlatFileItemReader file stop condition is managed;
with private field boolean noInput
with private function readLine() used in protected doRead()
IMHO the best solution is to throw a runtime exception from your skippedLineCallback and manage error as reader exhaustion condition.
Foe example writing your delegate in this way
class SkippableItemReader<T> implements ItemStreamReader<T> {
private ItemStreamReader<T> flatFileItemReader;
private boolean headerError = false;
void open(ExecutionContext executionContext) throws ItemStreamException {
try {
flatFileItemReader.open(executionContext);
} catch(MyCustomExceptionHeaderErrorException e) {
headerError = true;
}
}
public T read() {
if(headerError)
return null;
return flatFileItemReader.read();
}
// Other functions delegation
}
(you have to register delegate as stream manually,of course)
or extending FlatFileItemReader as
class SkippableItemReader<T> extends FlatFileItemReader<T> {
private boolean headerError = false;
protected void doOpen() throws Exception {
try {
super.doOpen();
} catch(MyCustomExceptionHeaderErrorException e) {
headerError = true;
}
}
protected T doRead() throws Exception {
if(headerError)
return null;
return super.doRead();
}
}
The code has been written directly without test so there can be errors, but I hope you can understand my point.
Hope can solve your problem
We disable the quartz scheduler locally by commenting out the scheduler factory bean in the jobs.xml file.
Is there a setting for doing something similar in the quartz.properties file?
If you use Spring Framework you can make subclass from org.springframework.scheduling.quartz.SchedulerFactoryBean and override afterPropertiesSet() method.
public class MySchedulerFactoryBean extends org.springframework.scheduling.quartz.SchedulerFactoryBean {
#Autowired
private #Value("${enable.quartz.tasks}") boolean enableQuartzTasks;
#Override
public void afterPropertiesSet() throws Exception {
if (enableQuartzTasks) {
super.afterPropertiesSet();
}
}
}
Then change declaration of factory in xml file and set "enable.quartz.tasks" property in properties file. That's all.
Of course, instead using #Autowired you can write and use setter method and add
<property name="enableQuartzTasks" value="${enable.quartz.tasks}"/>
to MySchedulerFactoryBean declaration in xml.
No. But the properties file doesn't start the scheduler.
The scheduler doesn't start until/unless some code invokes scheduler.start().
It seems that there is a property autoStartup in org.springframework.scheduling.quartz.SchedulerFactoryBean. So you can configure it in XML config like this:
<bean id="quartzFactory" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="autoStartup" value="${cron.enabled}"/>
<property name="triggers">
<list>
<ref bean="someTriggerName"/>
</list>
</property>
</bean>
Thanks to https://chrisrng.svbtle.com/configure-spring-to-turn-quartz-scheduler-onoff
You can disable Quartz Scheduler if you use Spring Framework 3.1 for creating and starting it.
On my Spring configuration file I use the new profiles feature of Spring 3.1 in this way:
<beans profile="production,test">
<bean name="bookingIndexerJob" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<property name="jobClass" value="com.xxx.indexer.scheduler.job.BookingIndexerJob" />
<property name="jobDataAsMap">
<map>
<entry key="timeout" value="10" />
</map>
</property>
</bean>
<bean id="indexerSchedulerTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerFactoryBean">
<property name="jobDetail" ref="bookingIndexerJob" />
<property name="startDelay" value="1000" />
<property name="repeatInterval" value="5000" />
</bean>
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<list>
<ref bean="indexerSchedulerTrigger" />
</list>
</property>
<property name="dataSource" ref="ds_quartz-scheduler"></property>
<property name="configLocation" value="classpath:quartz.properties" />
<property name="applicationContextSchedulerContextKey" value="applicationContext" />
</bean>
</beans>
Only when I want to start the Scheduler (for example on the production environment), I set the spring.profiles.active system property, with the list of active profiles:
-Dspring.profiles.active="production"
More info here:
http://blog.springsource.com/2011/02/11/spring-framework-3-1-m1-released/
http://java.dzone.com/articles/spring-profiles-or-not
I personally like the answer from Demis Gallisto. If you can work with profiles, this would be my recommendation.
Nowadays people most likely prefer to work with Annotations, so as an addition to his answer.
#Configuration
#Profile({ "test", "prod" })
public class SchedulerConfig {
#Bean
// ... some beans to setup your scheduler
}
This will trigger the scheduler only when the profile test OR prod is active. So if you set an different profile, e.g. -Dspring.profiles.active=dev nothing will happen.
If for some reasons you cannot use the profile approach, e.g. overlap of profiles ...
The solution from miso.belica seems also to work.
Define a property. e.g. in application.properties: dailyRecalculationJob.cron.enabled=false and use it in your SchedulerConfig.
#Configuration
public class SchedulerConfig {
#Value("${dailyRecalculationJob.cron.enabled}")
private boolean dailyRecalculationJobCronEnabled;
#Bean
public SchedulerFactoryBean schedulerFactoryBean(JobFactory jobFactory, Trigger trigger) throws
SchedulerFactoryBean factory = new SchedulerFactoryBean();
factory.setAutoStartup(dailyRecalculationJobCronEnabled);
// ...
return factory;
}
// ... the rest of your beans to setup your scheduler
}
I had similar issue: disable scheduler in test scope.
Here is part of my applicationContext.xml
<task:annotation-driven scheduler="myScheduler" />
<task:scheduler id="myScheduler" pool-size="10" />
And I've disabled scheduler using 'primary' attribute and Mockito. Here is my applicationContext-test.xml
<bean id="myScheduler" class="org.mockito.Mockito" factory-method="mock" primary="true">
<constructor-arg value="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler"/>
</bean>
Hope this help!
The simplest way I've found in a spring boot context for tests is to simply:
#MockBean
Scheduler scheduler;
This scala code works：
#Bean
def schedulerFactoryBean(): SchedulerFactoryBean = {
new SchedulerFactoryBean {
override def afterPropertiesSet(): Unit = {}
}
}