spring batch ItemWriter creating output twice - spring

Question

I have written a spring batch job which reads from a db and then write to a csv.
The job is pretty simple and it works HOWEVER the ItemWriter looks like it is being executed twice. i.e. I am ending up with two CSV's (exactly the same), instead of one.
Can someone help me work out why please? It does so with a different jobExecution.id.
Attaching batch job and console output of run below...
#Configuration
#EnableBatchProcessing
public class ExportMasterListCsvJobConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
public Job exportMasterListCsvJob(
Step readStgDbAndExportMasterListStep,
Step writeToCsvStep) {
return jobBuilderFactory.get("exportMasterListCsvJob")
.incrementer(new RunIdIncrementer())
.flow(readStgDbAndExportMasterListStep)
.end()
.build();
}
#Bean
public Step readStgDbAndExportMasterListStep(
#Value("${exportMasterListCsv.generateMasterListRows.chunkSize}") int chunkSize,
#Qualifier("queryOnlineStagingDbReader") ItemReader<MasterList> queryOnlineStagingDbReader,
#Qualifier("masterListFileWriter") ItemWriter<MasterList> masterListFileWriter) {
return stepBuilderFactory.get("readStgDbAndExportMasterListStep")
.<MasterList,MasterList>chunk(chunkSize)
.reader(queryOnlineStagingDbReader)
.writer(masterListFileWriter)
.build();
}
#Bean
public ItemReader<MasterList> queryOnlineStagingDbReader(
DataSource onlineStagingDb,
#Value("${exportMasterListCsv.generateMasterListRows.masterListSql}") String masterListSql) {
JdbcCursorItemReader<MasterList> reader = new JdbcCursorItemReader<>();
reader.setDataSource(onlineStagingDb);
reader.setSql(masterListSql);
reader.setRowMapper(new RowMapper<MasterList>() {
#Override
public MasterList mapRow(ResultSet resultSet, int i) throws SQLException {
MasterList masterList = new MasterList();
masterList.setL2(resultSet.getString("l2"));
masterList.setL2Name(resultSet.getString("l2_name"));
return masterList;
}
});
return reader;
}
#Bean
#StepScope
public ItemWriter<MasterList> masterListFileWriter(
FileSystemResource masterListFile,
#Value("#{stepExecutionContext}")Map<String, Object> executionContext) {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(masterListFile);
DelimitedLineAggregator<MasterList> lineAggregator = new DelimitedLineAggregator<>();
lineAggregator.setDelimiter(",");
BeanWrapperFieldExtractor<MasterList> extractor = new BeanWrapperFieldExtractor<MasterList>();
extractor.setNames(new String[] { "l2", "l2Name"});
lineAggregator.setFieldExtractor(extractor);
writer.setLineAggregator(lineAggregator);
writer.setForceSync(true);
writer.open(new ExecutionContext(executionContext));
return writer;
}
#Bean
#JobScope
public FileSystemResource masterListFile(
#Value("${temp.dir}") String tempDir,
#Value("#{jobExecution.id}") Long jobId
) throws IOException {
return new FileSystemResource(new File(tempDir, "masterList" + jobId + ".csv"));
}
}
And Console Output...
13:03:45.815 [main] DEBUG org.springframework.test.context.junit4.SpringJUnit4ClassRunner - SpringJUnit4ClassRunner constructor called with [class com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.827 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating CacheAwareContextLoaderDelegate from class [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
13:03:45.836 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating BootstrapContext using constructor [public org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
13:03:45.853 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating TestContextBootstrapper for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest] from class [org.springframework.test.context.support.DefaultTestContextBootstrapper]
13:03:45.884 [main] DEBUG org.springframework.test.context.support.DefaultTestContextBootstrapper - Found explicit ContextLoader class [org.springframework.boot.test.SpringApplicationContextLoader] for context configuration attributes [ContextConfigurationAttributes#3cbbc1e0 declaringClass = 'com.myer.pricing.onlinestore.export.ExportMasterListCsvTest', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', locations = '{}', inheritLocations = true, initializers = '{}', inheritInitializers = true, name = [null], contextLoaderClass = 'org.springframework.boot.test.SpringApplicationContextLoader']
13:03:45.899 [main] DEBUG org.springframework.test.context.support.AbstractContextLoader - Did not detect default resource location for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]: class path resource [com/myer/pricing/onlinestore/export/ExportMasterListCsvTest-context.xml] does not exist
13:03:45.900 [main] DEBUG org.springframework.test.context.support.AbstractContextLoader - Did not detect default resource location for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]: class path resource [com/myer/pricing/onlinestore/export/ExportMasterListCsvTestContext.groovy] does not exist
13:03:45.900 [main] INFO org.springframework.test.context.support.AbstractContextLoader - Could not detect default resource locations for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]: no resource found for suffixes {-context.xml, Context.groovy}.
13:03:45.927 [main] INFO org.springframework.test.context.support.DefaultTestContextBootstrapper - Using TestExecutionListeners: [org.springframework.boot.test.IntegrationTestPropertiesListener#724af044, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener#4678c730, org.springframework.test.context.support.DependencyInjectionTestExecutionListener#6767c1fc, org.springframework.test.context.support.DirtiesContextTestExecutionListener#29ee9faa, org.springframework.test.context.transaction.TransactionalTestExecutionListener#c038203, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener#cc285f4]
13:03:45.931 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.931 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.946 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.946 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.948 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.948 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.950 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.950 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.953 [main] DEBUG org.springframework.test.context.support.AbstractDirtiesContextTestExecutionListener - Before test class: context [DefaultTestContext#6a4f787b testClass = ExportMasterListCsvTest, testInstance = [null], testMethod = [null], testException = [null], mergedContextConfiguration = [MergedContextConfiguration#685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]], class annotated with #DirtiesContext [false] with mode [null].
13:03:45.954 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved #ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.954 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.959 [main] DEBUG org.springframework.test.util.ReflectionTestUtils - Getting field 'mergedContextConfiguration' from target object [[DefaultTestContext#6a4f787b testClass = ExportMasterListCsvTest, testInstance = com.myer.pricing.onlinestore.export.ExportMasterListCsvTest#53b32d7, testMethod = [null], testException = [null], mergedContextConfiguration = [MergedContextConfiguration#685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]]] or target class [class org.springframework.test.context.support.DefaultTestContext]
13:03:45.960 [main] DEBUG org.springframework.test.util.ReflectionTestUtils - Setting field 'propertySourceProperties' of type [null] on target object [[MergedContextConfiguration#685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]] or target class [class org.springframework.test.context.MergedContextConfiguration] to value [[Ljava.lang.String;#4667ae56]
13:03:45.961 [main] DEBUG org.springframework.test.context.support.DependencyInjectionTestExecutionListener - Performing dependency injection for test context [[DefaultTestContext#6a4f787b testClass = ExportMasterListCsvTest, testInstance = com.myer.pricing.onlinestore.export.ExportMasterListCsvTest#53b32d7, testMethod = [null], testException = [null], mergedContextConfiguration = [MergedContextConfiguration#685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{org.springframework.boot.test.IntegrationTest=true}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]]].
13:03:46.072 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [systemProperties] PropertySource with lowest search precedence
13:03:46.073 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [systemEnvironment] PropertySource with lowest search precedence
13:03:46.074 [main] DEBUG org.springframework.core.env.StandardEnvironment - Initialized StandardEnvironment with PropertySources [systemProperties,systemEnvironment]
13:03:46.074 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [test] PropertySource with highest search precedence
13:03:46.075 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [integrationTest] PropertySource with search precedence immediately lower than [systemEnvironment]
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.3.6.RELEASE)
2016-07-22 13:03:46.541 INFO 10084 --- [ main] c.m.p.o.export.ExportMasterListCsvTest : Starting ExportMasterListCsvTest on IGSM473537 with PID 10084 (C:\design_centre_dev\myer-pricing-engine\java-onlinestore-feed-exporter\target\test-classes started by rriviere in C:\design_centre_dev\myer-pricing-engine\java-onlinestore-feed-exporter)
2016-07-22 13:03:46.542 INFO 10084 --- [ main] c.m.p.o.export.ExportMasterListCsvTest : The following profiles are active: batchtest
2016-07-22 13:03:46.875 INFO 10084 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext#4e08711f: startup date [Fri Jul 22 13:03:46 AEST 2016]; root of context hierarchy
2016-07-22 13:03:48.151 INFO 10084 --- [ main] o.s.b.f.config.PropertiesFactoryBean : Loading properties file from URL [jar:file:/C:/.m2/repository/org/springframework/integration/spring-integration-core/4.2.8.RELEASE/spring-integration-core-4.2.8.RELEASE.jar!/META-INF/spring.integration.default.properties]
2016-07-22 13:03:48.157 INFO 10084 --- [ main] o.s.i.config.IntegrationRegistrar : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2016-07-22 13:03:48.716 WARN 10084 --- [ main] o.s.c.a.ConfigurationClassEnhancer : #Bean method ScopeConfiguration.stepScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as #Autowired, #Resource and #PostConstruct within the method's declaring #Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see #Bean javadoc for complete details.
2016-07-22 13:03:48.728 WARN 10084 --- [ main] o.s.c.a.ConfigurationClassEnhancer : #Bean method ScopeConfiguration.jobScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as #Autowired, #Resource and #PostConstruct within the method's declaring #Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see #Bean javadoc for complete details.
2016-07-22 13:03:48.742 INFO 10084 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2016-07-22 13:03:48.751 INFO 10084 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2016-07-22 13:03:48.900 INFO 10084 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$402d705e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-07-22 13:03:49.060 INFO 10084 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:hsqldb:mem:testdb', username='sa'
2016-07-22 13:03:49.459 INFO 10084 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from class path resource [db/sql/staging_db.sql]
2016-07-22 13:03:49.465 INFO 10084 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from class path resource [db/sql/staging_db.sql] in 6 ms.
2016-07-22 13:03:49.710 WARN 10084 --- [ main] o.s.b.c.l.AbstractListenerFactoryBean : org.springframework.batch.item.ItemWriter is an interface. The implementing class will not be queried for annotation based listener configurations. If using #StepScope on a #Bean method, be sure to return the implementing class so listner annotations can be used.
2016-07-22 13:03:49.797 INFO 10084 --- [ main] o.s.aop.framework.CglibAopProxy : Unable to proxy method [public final java.lang.String org.springframework.core.io.FileSystemResource.getPath()] because it is final: All calls to this method via a proxy will NOT be routed to the target instance.
2016-07-22 13:03:50.236 INFO 10084 --- [ main] o.s.b.f.config.PropertiesFactoryBean : Loading properties file from URL [jar:file:/C:/.m2/repository/org/springframework/integration/spring-integration-core/4.2.8.RELEASE/spring-integration-core-4.2.8.RELEASE.jar!/META-INF/spring.integration.default.properties]
2016-07-22 13:03:50.730 INFO 10084 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'taskScheduler'
2016-07-22 13:03:51.258 INFO 10084 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2016-07-22 13:03:51.281 INFO 10084 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 23 ms.
2016-07-22 13:03:51.813 INFO 10084 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
2016-07-22 13:03:51.814 INFO 10084 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2016-07-22 13:03:51.814 INFO 10084 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'application:batchtest.errorChannel' has 1 subscriber(s).
2016-07-22 13:03:51.814 INFO 10084 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started _org.springframework.integration.errorLogger
2016-07-22 13:03:51.832 INFO 10084 --- [ main] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: []
2016-07-22 13:03:51.844 INFO 10084 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: HSQL
2016-07-22 13:03:51.973 INFO 10084 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2016-07-22 13:03:52.044 INFO 10084 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=exportMasterListCsvJob]] launched with the following parameters: [{run.id=1}]
2016-07-22 13:03:52.073 INFO 10084 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [readStgDbAndExportMasterListStep]
2016-07-22 13:03:52.167 INFO 10084 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=exportMasterListCsvJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
2016-07-22 13:03:52.169 INFO 10084 --- [ main] c.m.p.o.export.ExportMasterListCsvTest : Started ExportMasterListCsvTest in 6.091 seconds (JVM running for 6.895)
2016-07-22 13:03:52.196 INFO 10084 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=exportMasterListCsvJob]] launched with the following parameters: [{random=93853}]
2016-07-22 13:03:52.234 INFO 10084 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [readStgDbAndExportMasterListStep]
2016-07-22 13:03:52.290 INFO 10084 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=exportMasterListCsvJob]] completed with the following parameters: [{random=93853}] and the following status: [COMPLETED]
2016-07-22 13:03:52.305 INFO 10084 --- [ Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#4e08711f: startup date [Fri Jul 22 13:03:46 AEST 2016]; root of context hierarchy
2016-07-22 13:03:52.307 INFO 10084 --- [ Thread-1] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 0
2016-07-22 13:03:52.308 INFO 10084 --- [ Thread-1] o.s.i.endpoint.EventDrivenConsumer : Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2016-07-22 13:03:52.309 INFO 10084 --- [ Thread-1] o.s.i.channel.PublishSubscribeChannel : Channel 'application:batchtest.errorChannel' has 0 subscriber(s).
2016-07-22 13:03:52.309 INFO 10084 --- [ Thread-1] o.s.i.endpoint.EventDrivenConsumer : stopped _org.springframework.integration.errorLogger
2016-07-22 13:03:52.314 INFO 10084 --- [ Thread-1] o.s.s.c.ThreadPoolTaskScheduler : Shutting down ExecutorService 'taskScheduler'
2016-07-22 13:03:52.317 INFO 10084 --- [ Thread-1] o.s.j.d.e.EmbeddedDatabaseFactory : Shutting down embedded database: url='jdbc:hsqldb:mem:testdb'
And this produces two CSV's each with a different jobExecution.id which I have appended to the file name. masterList0.csv AND masterList1.csv
thanks in advance

Answer 1

I had the following setting in my yml file...
spring:
batch:
job.enabled: true
This was causing my job to run twice. After changing it to false it fixed my problem.