In my first Spring Batch chunk job (I'm a beginner),
I wrote db utility class for using in the chunk step,
since I need to overwrite default connection attributes set by application.properties.
However, it causes rollback problem (never rollback) on errors.
Any advices to improve these logics?
DBUtility class
// Construct
public DBUtility() {
... some business logic to get connection attributes dynamically.
}
#Bean
#BatchDataSource
#Primary
public DriverManagerDataSource getConnection() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverNameGottenAbove);
dataSource.setUrl(urlGottenAbove);
dataSource.setUsername(userNameGottenAbove);
dataSource.setPassword(passwordGottenAbove);
return dataSource;
}
#Bean
public DataSourceTransactionManager transactionManager(){
DataSourceTransactionManager dtm = new DataSourceTransationManager(getConnection());
dtm.setRollbackOnCommitFailure(true);
return dtm;
}
#Bean
public JdbcTemplate jdbcTemplate(){
return new JdbcTemplate(getConnection());
}
BatchConfiguration class(chunk)
#SpringBootApplication
#EnableBatchProcessing
#Transactional
public class BatchConfiguration extends DefaultBatchConfigurer {
...
private DBUtility dbUtil;
private DriverManagerDataSource ds;
private PlatformTransactionManager ptm;
// Construct
public BatchConfiguration(){
dbUtil = new DBUtility();
ds = dbUtil.getConnection();
ptm = dbUtil.transactionManager();
}
// Override
#Override
public JobRepository createJobRepository() throws Exception {
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(ds);
factory.setTransactionManager(ptm);
JobRepository repo = factory.getObject();
return repo;
}
#Bean
public Job BatchJob() {
return jobBuilderFactory.get("BatchJob")
.incrementer(new RunIdIncrementer())
.listener(jobListener())
.flow(step1())
.end().build();
}
#Bean
public step step1() {
DefaultTransactionAttribute att = new DefaultTransactionAttribute();
att.setPropagationBehavior(Propagation.REQUIRED.value());
return stepBuilderFactory.get("step1")
.transactionManager(ptm)
.listener(stepListener())
.<Entity, Entity>chunk(COMMIT_INTERVAL)
.reader(reader())
.processor(processore())
.writer(writer())
.readerIsTransactionalQueue()
.transactionAttribute(att)
.build();
}
...
}
Thank you for checking this post.
I have requirement to develop a spring boot batch using quartz as scheduler.
I have created spring batch application and also configured quartz to trigger job and persist job/trigger data in quartz related RDBMS tables.
Spring boot application starts successfully but when it triggers the job as per cron config. Its throws below error.
2020-04-27 20:42:09.967 INFO 1460 --- [ main] c.b.g.batch.IemsBatchApplication : Started IemsBatchApplication in 12.112 seconds (JVM running for 13.22)
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = ? and JOB_KEY = ?]; nested exception is java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
I am not able to find the root cause that why application is trying to persist data in spring meta data table rather than quartz tables.
Quartz related tables are already created in my db.
Please help me.
Please find below the code for the same.
#Configuration
public class BatchSchedulerConfig {
#Autowired
#Qualifier("batchDataSource")
private DataSource batchDataSource;
#Autowired
JobLauncher jobLauncher;
#Autowired
JobLocator jobLocator;
#Bean
public JobDetailFactoryBean jobDetailFactoryBean() {
JobDetailFactoryBean jobDetailFactoryBean = new JobDetailFactoryBean();
jobDetailFactoryBean.setJobClass(BatchJobLauncherConfig.class);
Map<String, Object> map = new HashMap<String, Object>();
map.put("jobName", IEMSBatchConstant.MonthlyElectricityMeterReadingJob);
jobDetailFactoryBean.setJobDataAsMap(map);
jobDetailFactoryBean.setGroup("etl_group");
jobDetailFactoryBean.setName("etl_job");
return jobDetailFactoryBean;
}
#Bean
public CronTriggerFactoryBean cronTriggerFactoryBean(JobDetailFactoryBean jobDetailFactoryBean) {
CronTriggerFactoryBean cronTriggerFactoryBean = new CronTriggerFactoryBean();
cronTriggerFactoryBean.setJobDetail(jobDetailFactoryBean.getObject());
cronTriggerFactoryBean.setStartDelay(3000);
cronTriggerFactoryBean.setName("cron_trigger");
cronTriggerFactoryBean.setGroup("cron_group");
cronTriggerFactoryBean.setCronExpression("0 0/2 * 1/1 * ? *");
return cronTriggerFactoryBean;
}
#Bean
public SchedulerFactoryBean schedulerFactoryBean(CronTriggerFactoryBean cronTriggerFactoryBean) throws IOException {
SchedulerFactoryBean scheduler = new SchedulerFactoryBean();
Map<String, Object> map = new HashMap<String, Object>();
map.put("joblocator", jobLocator);
map.put("JobLauncher", jobLauncher);
scheduler.setSchedulerContextAsMap(map);
scheduler.setTriggers(cronTriggerFactoryBean.getObject());
scheduler.setDataSource(batchDataSource);
scheduler.setQuartzProperties(quartzProperties());
// scheduler.setSchedulerName(schedulerName);
return scheduler;
}
public Properties quartzProperties() throws IOException {
PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
propertiesFactoryBean.setLocation(new ClassPathResource("/quartz.properties"));
propertiesFactoryBean.afterPropertiesSet();
return propertiesFactoryBean.getObject();
}
}
#Configuration
#EnableConfigurationProperties
public class BatchBaseConfig {
private static final Logger log = LoggerFactory.getLogger(BatchBaseConfig.class);
#Bean
public JobLauncher jobLauncher(JobRepository jobRepository) throws Exception {
SimpleJobLauncher simpleJobLauncher = new SimpleJobLauncher();
simpleJobLauncher.setJobRepository(jobRepository);
return simpleJobLauncher;
}
#Bean
public JobRepository jobRepository(DataSource batchDataSource) throws Exception {
DataSourceTransactionManager batchTransactionManager = new DataSourceTransactionManager();
JobRepositoryFactoryBean jobRepositoryFactoryBean = new JobRepositoryFactoryBean();
batchTransactionManager.setDataSource(batchDataSource);
jobRepositoryFactoryBean.setTransactionManager(batchTransactionManager);
jobRepositoryFactoryBean.setDatabaseType(DatabaseType.ORACLE.getProductName());
//jobRepositoryFactoryBean.setIsolationLevelForCreate("ISOLATION_DEFAULT");
jobRepositoryFactoryBean.setDataSource(batchDataSource);
jobRepositoryFactoryBean.setTablePrefix("QRTZ_");
jobRepositoryFactoryBean.afterPropertiesSet();
return jobRepositoryFactoryBean.getObject();
}
#Bean(name = "batchDataSource")
#Primary
#ConfigurationProperties(prefix = "quartz")
public DataSource batchDataSource() {
DataSource batchDbSrc = DataSourceBuilder.create().build();
return batchDbSrc;
}
#Bean(name = "appDataSource")
#ConfigurationProperties(prefix = "app")
public DataSource appDataSource() {
DataSource appDbSrc = DataSourceBuilder.create().build();
return appDbSrc;
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
return jobRegistryBeanPostProcessor;
}
}
#Getter
#Setter
public class BatchJobLauncherConfig extends QuartzJobBean {
private static final Logger log = LoggerFactory.getLogger(BatchJobLauncherConfig.class);
private JobLauncher jobLauncher;
private JobLocator joblocator;
private String jobName;
#Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
try {
JobExecution jobExecution = jobLauncher.run(joblocator.getJob(jobName), new
JobParameters());
log.info("{}_{} was completed successfully", jobExecution.getJobConfigurationName(),
jobExecution.getId());
} catch (JobExecutionAlreadyRunningException | JobRestartException |
JobInstanceAlreadyCompleteException
| JobParametersInvalidException | NoSuchJobException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}
#Configuration
public class JobConfig {
private static final Logger log = LoggerFactory.getLogger(JobConfig.class);
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private ReadingFromDB readingFromDB;
#Autowired
private ReadingFileWriter readingFileWriter;
#Qualifier(value = "meterReadingJob")
#Bean
public Job meterReadingJob() throws Exception {
log.info("Enter into meterReadingJob method ");
return this.jobBuilderFactory.get("meterReadingJob")
.start(createReadingFile()).build();
}
#Bean
public Step createReadingFile() throws Exception {
return this.stepBuilderFactory.get("createReadingFile").chunk(1)
.reader(readingFromDB).writer(readingFileWriter).build();
}
}
Application.properties
spring.batch.job.enabled=false
spring.main.allow-bean-definition-overriding=true
#Spring's default configuration to generate datasource
app.jdbcUrl=jdbc:oracle:thin:#//<Removed>
app.username=<Removed>
app.password=<Removed>
app.driverClassName=oracle.jdbc.OracleDriver
spring.app.jpa.database-platform=org.hibernate.dialect.Oracle10gDialect
#Quartz dataSource
quartz.jdbcUrl=jdbc:oracle:thin:#//<Removed>
quartz.username=<Removed>
quartz.password=<Removed>
quartz.driverClassName=oracle.jdbc.OracleDriver
Quartz.properties
#scheduler name will be "MyScheduler"
org.quartz.scheduler.instanceName=MyNewScheduler
org.quartz.scheduler.instanceId=AUTO
#maximum of 3 jobs can be run simultaneously
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=50
#Quartz persistent jobStore config
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.tablePrefix=QRTZ_
#org.quartz.jobStore.dataSource=batchDataSource
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.isClustered=false
Both datasources are getting created and entries are also going for triggers and jobs in datasource names batchDataSource.
Issue comes when job gets triggered and batch tries to insert job details in DB.
Thanks in advance.
I created a Spring Boot with Spring Batch Application and Scheduling. When i create only one job, things are working fine . But when i try to create another job using the modular approach. The jobs and it's step are running many times and they are getting duplicated.
Getting the below error.
2017-08-24 16:05:00.581 INFO 16172 --- [cTaskExecutor-2] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=importDeptJob]] completed with the following parameters: [{JobID1=1503579900035}] and the following status: [FAILED]
2017-08-24 16:05:00.581 ERROR 16172 --- [cTaskExecutor-2] o.s.batch.core.step.tasklet.TaskletStep : JobRepository failure forcing rollback
org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=1 with wrong version (3), where current version is 1
Can anyone Please guide me how to resolve these issues and run the jobs in a parallel way independent of each other ?
Below are the configuration Classes : ModularJobConfiguration.java , DeptBatchConfiguration.java ,CityBatchConfiguration.java and BatchScheduler.java
#Configuration
#EnableBatchProcessing(modular=true)
public class ModularJobConfiguration {
#Bean
public ApplicationContextFactory firstJob() {
return new GenericApplicationContextFactory(DeptBatchConfiguration.class);
}
#Bean
public ApplicationContextFactory secondJob() {
return new GenericApplicationContextFactory(CityBatchConfiguration.class);
}
}
#Configuration
#EnableBatchProcessing
#Import({BatchScheduler.class})
public class DeptBatchConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(DeptBatchConfiguration.class);
#Autowired
private SimpleJobLauncher jobLauncher;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public JobExecutionListener listener;
public ItemReader<DepartmentModelReader> deptReaderSO;
#Autowired
#Qualifier("dataSourceReader")
private DataSource dataSourceReader;
#Autowired
#Qualifier("dataSourceWriter")
private DataSource dataSourceWriter;
#Scheduled(cron = "0 0/1 * * * ?")
public void performFirstJob() throws Exception {
long startTime = System.currentTimeMillis();
LOGGER.info("Job1 Started at :" + new Date());
JobParameters param = new JobParametersBuilder().addString("JobID1",String.valueOf(System.currentTimeMillis())).toJobParameters();
JobExecution execution = (JobExecution) jobLauncher.run(importDeptJob(jobBuilderFactory,stepdept(deptReaderSO,customWriter()),listener), param);
long endTime = System.currentTimeMillis();
LOGGER.info("Job1 finished at " + (endTime - startTime) / 1000 + " seconds with status :" + execution.getExitStatus());
}
#Bean
public ItemReader<DepartmentModelReader> deptReaderSO() {
//LOGGER.info("Inside deptReaderSO Method");
JdbcCursorItemReader<DepartmentModelReader> deptReaderSO = new JdbcCursorItemReader<>();
//deptReaderSO.setSql("select id, firstName, lastname, random_num from reader");
deptReaderSO.setSql("SELECT DEPT_CODE,DEPT_NAME,FULL_DEPT_NAME,CITY_CODE,CITY_NAME,CITY_TYPE_NAME,CREATED_USER_ID,CREATED_G_DATE,MODIFIED_USER_ID,MODIFIED_G_DATE,RECORD_ACTIVITY,DEPT_CLASS,DEPT_PARENT,DEPT_PARENT_NAME FROM TBL_SAMPLE_SAFTY_DEPTS");
deptReaderSO.setDataSource(dataSourceReader);
deptReaderSO.setRowMapper(
(ResultSet resultSet, int rowNum) -> {
if (!(resultSet.isAfterLast()) && !(resultSet.isBeforeFirst())) {
DepartmentModelReader recordSO = new DepartmentModelReader();
recordSO.setDeptCode(resultSet.getString("DEPT_CODE"));
recordSO.setDeptName(resultSet.getString("DEPT_NAME"));
recordSO.setFullDeptName(resultSet.getString("FULL_DEPT_NAME"));
recordSO.setCityCode(resultSet.getInt("CITY_CODE"));
recordSO.setCityName(resultSet.getString("CITY_NAME"));
recordSO.setCityTypeName(resultSet.getString("CITY_TYPE_NAME"));
recordSO.setCreatedUserId(resultSet.getInt("CREATED_USER_ID"));
recordSO.setCreatedGDate(resultSet.getDate("CREATED_G_DATE"));
recordSO.setModifiedUserId(resultSet.getString("MODIFIED_USER_ID"));
recordSO.setModifiedGDate(resultSet.getDate("MODIFIED_G_DATE"));
recordSO.setRecordActivity(resultSet.getInt("RECORD_ACTIVITY"));
recordSO.setDeptClass(resultSet.getInt("DEPT_CLASS"));
recordSO.setDeptParent(resultSet.getString("DEPT_PARENT"));
recordSO.setDeptParentName(resultSet.getString("DEPT_PARENT_NAME"));
// LOGGER.info("RowMapper record : {}", recordSO.getDeptCode() +" | "+recordSO.getDeptName());
return recordSO;
} else {
LOGGER.info("Returning null from rowMapper");
return null;
}
});
return deptReaderSO;
}
#Bean
public ItemProcessor<DepartmentModelReader, DepartmentModelWriter> processor() {
//LOGGER.info("Inside Processor Method");
return new RecordProcessor();
}
#Bean
public ItemWriter<DepartmentModelWriter> customWriter(){
//LOGGER.info("Inside customWriter Method");
return new CustomItemWriter();
}
#Bean
public Job importDeptJob(JobBuilderFactory jobs, Step stepdept,JobExecutionListener listener){
return jobs.get("importDeptJob")
.incrementer(new RunIdIncrementer())
.listener(listener())
.flow(stepdept).end().build();
}
#Bean
public Step stepdept(ItemReader<DepartmentModelReader> deptReaderSO,
ItemWriter<DepartmentModelWriter> writerSO) {
LOGGER.info("Inside stepdept Method");
return stepBuilderFactory.get("stepdept").<DepartmentModelReader, DepartmentModelWriter>chunk(5)
.reader(deptReaderSO).processor(processor()).writer(customWriter()).transactionManager(platformTransactionManager(dataSourceWriter)).build();
}
#Bean
public JobExecutionListener listener() {
return new JobCompletionNotificationListener();
}
#Bean
public JdbcTemplate jdbcTemplate(DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
#Bean
public BatchWriteService batchWriteService() {
return new BatchWriteService();
}
#Bean
public PlatformTransactionManager platformTransactionManager(#Qualifier("dataSourceWriter") DataSource dataSourceWriter) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setDataSource(dataSourceWriter);
return transactionManager;
}
}
#Configuration
#EnableBatchProcessing
#Import({BatchScheduler.class})
public class CityBatchConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(CityBatchConfiguration.class);
#Autowired
private SimpleJobLauncher jobLauncher;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public JobExecutionListener listener;
public ItemReader<CitiesModelReader> citiesReaderSO;
#Autowired
#Qualifier("dataSourceReader")
private DataSource dataSourceReader;
#Autowired
#Qualifier("dataSourceWriter")
private DataSource dataSourceWriter;
#Scheduled(cron = "0 0/1 * * * ?")
public void performSecondJob() throws Exception {
long startTime = System.currentTimeMillis();
LOGGER.info("\n Job2 Started at :" + new Date());
JobParameters param = new JobParametersBuilder().addString("JobID2",String.valueOf(System.currentTimeMillis())).toJobParameters();
JobExecution execution = (JobExecution) jobLauncher.run(importCitiesJob(jobBuilderFactory,stepcity(citiesReaderSO,customCitiesWriter()),listener), param);
long endTime = System.currentTimeMillis();
LOGGER.info("Job2 finished at " + (endTime - startTime) / 1000 + " seconds with status :" + execution.getExitStatus());
}
#Bean
public ItemReader<CitiesModelReader> citiesReaderSO() {
//LOGGER.info("Inside readerSO Method");
JdbcCursorItemReader<CitiesModelReader> readerSO = new JdbcCursorItemReader<>();
readerSO.setSql("SELECT CITY_CODE,CITY_NAME,PARENT_CITY,CITY_TYPE,CITY_TYPE_NAME,CREATED_G_DATE,CREATED_USER_ID,MODIFIED_G_DATE,MODIFIED_USER_ID,RECORD_ACTIVITY FROM TBL_SAMPLE_SAFTY_CITIES");
readerSO.setDataSource(dataSourceReader);
readerSO.setRowMapper(
(ResultSet resultSet, int rowNum) -> {
if (!(resultSet.isAfterLast()) && !(resultSet.isBeforeFirst())) {
CitiesModelReader recordSO = new CitiesModelReader();
recordSO.setCityCode(resultSet.getLong("CITY_CODE"));
recordSO.setCityName(resultSet.getString("CITY_NAME"));
recordSO.setParentCity(resultSet.getInt("PARENT_CITY"));
recordSO.setCityType(resultSet.getString("CITY_TYPE"));
recordSO.setCityTypeName(resultSet.getString("CITY_TYPE_NAME"));
recordSO.setCreatedGDate(resultSet.getDate("CREATED_G_DATE"));
recordSO.setCreatedUserId(resultSet.getString("CREATED_USER_ID"));
recordSO.setModifiedGDate(resultSet.getDate("MODIFIED_G_DATE"));
recordSO.setModifiedUserId(resultSet.getString("MODIFIED_USER_ID"));
recordSO.setRecordActivity(resultSet.getInt("RECORD_ACTIVITY"));
//LOGGER.info("RowMapper record : {}", recordSO.toString());
return recordSO;
} else {
LOGGER.info("Returning null from rowMapper");
return null;
}
});
return readerSO;
}
#Bean
public ItemProcessor<CitiesModelReader,CitiesModelWriter> citiesProcessor() {
//LOGGER.info("Inside Processor Method");
return new RecordCitiesProcessor();
}
#Bean
public ItemWriter<CitiesModelWriter> customCitiesWriter(){
LOGGER.info("Inside customCitiesWriter Method");
return new CustomCitiesWriter();
}
#Bean
public Job importCitiesJob(JobBuilderFactory jobs, Step stepcity,JobExecutionListener listener) {
LOGGER.info("Inside importCitiesJob Method");
return jobs.get("importCitiesJob")
.incrementer(new RunIdIncrementer())
.listener(listener())
.flow(stepcity).end().build();
}
#Bean
public Step stepcity(ItemReader<CitiesModelReader> readerSO,
ItemWriter<CitiesModelWriter> writerSO) {
LOGGER.info("Inside stepCity Method");
return stepBuilderFactory.get("stepcity").<CitiesModelReader, CitiesModelWriter>chunk(5)
.reader(readerSO).processor(citiesProcessor()).writer(customCitiesWriter()).transactionManager(platformTransactionManager(dataSourceWriter)).build();
}
#Bean
public JobExecutionListener listener() {
return new JobCompletionNotificationListener();
}
#Bean
public JdbcTemplate jdbcTemplate(DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
#Bean
public BatchWriteService batchWriteService() {
return new BatchWriteService();
}
#Bean
public PlatformTransactionManager platformTransactionManager(#Qualifier("dataSourceWriter") DataSource dataSourceWriter) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setDataSource(dataSourceWriter);
return transactionManager;
}
}
#Configuration
#EnableScheduling
public class BatchScheduler {
private static final Logger LOGGER = LoggerFactory.getLogger(BatchScheduler.class);
#Bean
public ResourcelessTransactionManager resourcelessTransactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
public MapJobRepositoryFactoryBean mapJobRepositoryFactory(
ResourcelessTransactionManager txManager) throws Exception {
LOGGER.info("Inside mapJobRepositoryFactory method");
MapJobRepositoryFactoryBean factory = new
MapJobRepositoryFactoryBean(txManager);
factory.afterPropertiesSet();
return factory;
}
#Bean
public JobRepository jobRepository(
MapJobRepositoryFactoryBean factory) throws Exception {
LOGGER.info("Inside jobRepository method");
return factory.getObject();
}
#Bean
public SimpleJobLauncher jobLauncher(JobRepository jobRepository) {
LOGGER.info("Inside jobLauncher method");
SimpleJobLauncher launcher = new SimpleJobLauncher();
launcher.setJobRepository(jobRepository);
final SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
launcher.setTaskExecutor(simpleAsyncTaskExecutor);
return launcher;
}
}
The map-based SimpleJobRepository created from MapJobRepositoryFactoryBean is not thread-safe.
From the Javadocs:
A FactoryBean that automates the creation of a SimpleJobRepository using non-persistent in-memory DAO implementations. This repository is only really intended for use in testing and rapid prototyping. In such settings you might find that ResourcelessTransactionManager is useful (as long as your business logic does not use a relational database). Not suited for use in multi-threaded jobs with splits, although it should be safe to use in a multi-threaded step.
You can create a JDBC-based SimpleJobRepository from JobRepositoryFactoryBean, which may utilize an in-memory H2 database if you don't require that Batch metadata be persisted.
Since you are using Spring Boot, to use an H2-backed JobRepository simply remove your JobRepository bean and add the following dependency to your pom.xml file:
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
Spring Boot will automatically configure a DataSource as though you had configured the following in your application.properties file and automatically use that DataSource in the creation of a JobRepository.
spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=
Alternatively, to use some other JDBC-backed JobRepository add the JDBC dependencies for your RDBMS of choice to your project, and configure a DataSource for it (either in code as a DataSource bean, or in application.properties using the spring.datasource prefix as shown above). Spring Boot will automatically use this DataSource during the creation of the JobRepository bean.
I have a JavaConfig configurated Spring Batch job. The main job configuration file is CrawlerJobConfiguration. Before now, I have all the configuration (infrastructure, autowired beans, etc) in this class and it works fine. So I decided to separate the job configuration from autowired beans and infracstruture beans configuration and create another 2 configuration classes Beans and MysqlInfrastructureConfiguration.
But now I am having problems to run my job. I'm receiving a NullPointerException when the application try to use autowired fields indicating that the autowired is not working.
I put a breakpoint in the methods that create autowired beans to make sure that they are being called and really are, so I cannot realize what can be the problem.
java.lang.NullPointerException: null
at br.com.alexpfx.supermarket.batch.tasklet.StartCrawlerTasklet.execute(StartCrawlerTasklet.java:27) ~[classes/:na]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406) ~[spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
Main job configuration class:
#Configuration
#EnableBatchProcessing
public class CrawlerJobConfiguration {
#Autowired
private InfrastructureConfiguration infrastructureConfiguration;
#Autowired
private StepBuilderFactory steps;
#Autowired
Environment environment;
#Bean
public Job job(JobBuilderFactory jobs) {
Job theJob = jobs.get("job").start(crawlerStep()).next(processProductStep()).build();
((AbstractJob) theJob).setRestartable(true);
return theJob;
}
#Bean
public Step crawlerStep() {
TaskletStep crawlerStep = steps.get("crawlerStep").tasklet(crawlerTasklet()).build();
crawlerStep.setAllowStartIfComplete(true);
return crawlerStep;
}
#Bean
public Step processProductStep() {
TaskletStep processProductStep = steps.get("processProductStep")
.<TransferObject, Product>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
processProductStep.setAllowStartIfComplete(true);
return processProductStep;
}
private Tasklet crawlerTasklet() {
return new StartCrawlerTasklet();
}
private ItemProcessor<TransferObject, Product> processor() {
return new ProductProcessor();
}
private ItemReader<TransferObject> reader() {
return new ProductItemReader();
}
private ItemWriter<Product> writer() {
return new HibernateProductsItemWriter();
}
}
Beans configuration class:
#Configuration
#EnableBatchProcessing
public class Beans {
#Bean
public Crawler crawler() {
return new RibeiraoCrawler(new UserAgentFactory());
}
#Bean
public ProductBo productBo() {
return new ProductBoImpl();
}
#Bean
public ProductDao productDao() {
return new ProductDaoImpl();
}
#Bean
public CrawlerListener listener() {
CrawlerListener listener = new RibeiraoListener();
return listener;
}
#Bean
public ProductList getProductList() {
return new ProductList();
}
}
MysqlInfrastructureConfiguration:
#Configuration
#EnableBatchProcessing
#PropertySource("classpath:database.properties")
#EnableJpaRepositories(basePackages = {"br.com.alexpfx.supermarket.domain"})
public class MysqlInfrastructureConfiguration implements InfrastructureConfiguration {
#Value("${jdbc.url}")
String url;
#Value("${jdbc.driverClassName}")
String driverClassName;
#Value("${jdbc.username}")
String username;
#Value("${jdbc.password}")
String password;
#Bean
#Override
public DataSource getDataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverClassName);
dataSource.setUrl(url);
dataSource.setUsername(username);
dataSource.setPassword(password);
return dataSource;
}
#Bean
#Override
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory());
transactionManager.setDataSource(getDataSource());
return transactionManager;
}
#Bean
#Override
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(getDataSource());
em.setPackagesToScan(new String[]{"br.com.alexpfx.supermarket.domain"});
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalJpaProperties());
em.afterPropertiesSet();
return em.getObject();
}
private Properties additionalJpaProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.hbm2ddl.auto", "create");
properties.setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5Dialect");
properties.setProperty("hibernate.show_sql", "true");
properties.setProperty("current_session_context_class", "thread");
return properties;
}
}
tasklet:
public class StartCrawlerTasklet implements Tasklet {
#Autowired
private Crawler crawler;
#Autowired
private CrawlerListener listener;
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
crawler.setListener(listener);
crawler.setStopCondition(new TimeLimitStopCondition(1, TimeUnit.MINUTES));
crawler.crawl();
return RepeatStatus.FINISHED;
}
}
StartCrawlerTasklet use the autowired annotation, so it should be a bean as well. So change your code :
private Tasklet crawlerTasklet() {
return new StartCrawlerTasklet();
}
to a bean definition:
#Bean
public Tasklet crawlerTasklet() {
return new StartCrawlerTasklet();
}
I have a spring mvc application with a batch process powered by spring batch . If i remove the batch configurations, all transactions commit. If a batch job is run, the batch job completes successfully but not data is commited to the database
my configurations are as follows
#Configuration
#EnableWebMvc
#EnableAsync
#EnableScheduling
#EnableBatchProcessing(modular = false)
#EnableTransactionManagement
#EnableRabbit
#EnableJpaRepositories(basePackages = "zw.co.econet.workstation.repositories")
#ComponentScan(basePackages = {"zw.co.workstation"})
#PropertySource(value = {"classpath:application.properties"})
public class WebConfiguration extends WebMvcConfigurerAdapter {
#Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler("/resources/**").addResourceLocations("/resources/");
}
#Bean
public InternalResourceViewResolver jspViewResolver() {
InternalResourceViewResolver bean = new InternalResourceViewResolver();
bean.setPrefix("/WEB-INF/pages/");
bean.setSuffix(".jsp");
return bean;
}
}
Spring batch configuration :
#Configuration
public class BatchConfiguration {
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Autowired
DataSource dataSource;
#Qualifier("creditQueueItemWriter")
#Autowired
private ItemWriter queueItemWriter;
#Qualifier("creditQueueProcessor")
#Autowired
private CreditQueueProcessor creditQueueProcessor;
#Qualifier("creditQueueReader")
#Autowired
private CreditQueueReader creditQueueReader;
#Qualifier("transactionManager")
#Autowired
private PlatformTransactionManager transactionManager;
#Bean
public AsyncTaskExecutor taskExecutor() {
return new SimpleAsyncTaskExecutor();
}
#Bean
#Autowired
protected Step creditSubscriberStep() throws Exception {
return steps.get("creditSubscriberStep")
.allowStartIfComplete(true)
.startLimit(3)
.chunk(10)
.reader(creditQueueReader)
.processor(creditQueueProcessor)
.writer(queueItemWriter)
.faultTolerant()
.build();
}
#Bean
public Job creditSubscribersJob() throws Exception {
JobBuilder builder = jobs.get("creditSubscriberJob");
return builder
.start(creditSubscriberStep())
.build();
}
#Bean
public JobLauncher jobLauncher() throws Exception {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository());
jobLauncher.setTaskExecutor(taskExecutor());
jobLauncher.afterPropertiesSet();
return jobLauncher;
}
#Bean
public JobRepository jobRepository() {
try {
JobRepositoryFactoryBean factoryBean = new JobRepositoryFactoryBean();
factoryBean.setDataSource(dataSource);
factoryBean.setTransactionManager(transactionManager);
factoryBean.setIsolationLevelForCreate("DEFAULT");
return factoryBean.getObject();
} catch (Exception e) {
return null;
}
}
#Bean
public DataSourceInitializer databasePopulator() {
ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
populator.addScript(new ClassPathResource("org/springframework/batch/core/schema-mysql.sql"));
populator.setContinueOnError(true);
populator.setIgnoreFailedDrops(true);
DataSourceInitializer initializer = new DataSourceInitializer();
initializer.setDatabasePopulator(populator);
initializer.setDataSource(dataSource);
return initializer;
}
}
Credit writer :
#Service
public class CreditQueueItemWriter implements ItemWriter {
private Logger logger = LoggerFactory.getLogger(getClass());
#Qualifier("creditQueueService")
#Autowired
private CreditQueueService creditQueueService;
#Override
public void write(List<? extends CreditQueue> list) throws Exception {
logger.info("Processing credit list with size {}", list.size());
for (CreditQueue creditQueue : list) {
logger.info("Updating >>>> {} ", creditQueue);
creditQueue.setProcessingState("PROCESSED");
creditQueueService.save(creditQueue);
}
logger.info("chunk processed");
}
}