I am trying to copy entites from one DB to another using SpringBatch. My idea is to switch datasources during runtime using AbstractRoutingDataSource depending on thread context passed in ItemReaderListener methods. When debugging JobExecution after job execution completes, it contains correct info about number of rows written and about number of commits. However data are not persisted into database.
I have defined 2 datasources and routing datasource:
#Bean
#Primary
public DataSource dataSourceRouting() {
DataSourceRouting dataSourceRouting = new DataSourceRouting(targetDataSource, sourceDataSource);
Map<Object, Object> dataSourceMap = new HashMap<>();
dataSourceMap.put(DataSourceContextHolder.DataSourceEnum.SOURCE, sourceDataSource);
dataSourceMap.put(DataSourceContextHolder.DataSourceEnum.TARGET, targetDataSource);
dataSourceRouting.setTargetDataSources(dataSourceMap);
dataSourceRouting.setDefaultTargetDataSource(targetDataSource);
return dataSourceRouting;
}
#Bean("transactionRoutingManager")
public PlatformTransactionManager platformTransactionManager(DataSource dataSourceRouting) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setDataSource(dataSourceRouting);
return transactionManager;
}
next I have defined reader and writer:
#Bean
#StepScope
public RepositoryItemReader<DataType> dataTypeReader() {
RepositoryItemReader<DataType> itemReader = new RepositoryItemReader<>();
itemReader.setRepository(dataTypeRepository);
itemReader.setPageSize(10);
itemReader.setMethodName("findAll");
itemReader.setSort(new HashMap<>());
return itemReader;
}
#Bean
#StepScope
public RepositoryItemWriter<DataType> dataTypeWriter() {
RepositoryItemWriter<DataType> itemWriter = new RepositoryItemWriter<>();
itemWriter.setRepository(dataTypeRepository);
return itemWriter;
}
and ItemReaderListener to switch DS using thread context:
#Override
#BeforeRead
public void beforeRead() {
DataSourceContextHolder.setBranchContext(DataSourceContextHolder.DataSourceEnum.SOURCE);
}
#Override
#AfterRead
public void afterRead(Object o) {
DataSourceContextHolder.setBranchContext(DataSourceContextHolder.DataSourceEnum.TARGET);
}
finally I have defined step as follows:
Step step1 = stepBuilderFactory.get("DATA_TYPE")
.<DataType, DataType>chunk(10)
.reader(dataTypeReader())
.writer(dataTypeWriter())
.transactionManager(transactionRoutingManager)
.throttleLimit(1)
.startLimit(1)
.listener(new WISItemReadListener())
.build();
Related
I have a spring batch job using partions and reader is JdbcCursorItemReader, so in this reader I need an authorisation to read correctely crypted data, so when I declare my reader a call the method just bellow .
the problem is that somme partions read null value for the field which need to be decrypted , the only reason is that the authorisation is not set ( I check in database and data are not null), so why it's work for some partions and not for all?
private void authorize() {
//Authorize
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update(setClientInfo, authorization);
}
and this is how i declare my reader
#Bean
#StepScope
public JdbcCursorItemReader<MyEntity> reader(#Value("#{stepExecutionContext['modulo']}") Integer modulo)
throws IOException {
ClassPathResource resource = new ClassPathResource(SQL_FILE);
BufferedReader reader = new BufferedReader(new InputStreamReader(resource.getInputStream()));
String query = FileCopyUtils.copyToString(reader);
query = query.replace(MODULO_LABEL, String.valueOf(modulo));
query = query.replace(GRID_SIZE_LABEL, String.valueOf(gridSize));
authorize();
JdbcCursorItemReader<MyEntity> cursorItemReader = new JdbcCursorItemReader<>();
cursorItemReader.setSql(query);
final int partitionSize = maxNumberCards / gridSize;
cursorItemReader.setMaxItemCount(partitionSize);
cursorItemReader.setDataSource(dataSource);
cursorItemReader.setRowMapper(myRowMapper);
return cursorItemReader;
}
and my job configuration
#Configuration
#EnableBatchProcessing
#RefreshScope
public class MyFunctionJobConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
JdbcCursorItemReader<MyEntity> reader;
#Value("${max-number-card-to-process}")
private Integer MAX_NUMBER_CARD;
#Value("${chunck-size:10}")
private int chunckSize;
#Value("${grid-size:1}")
private int gridSize;
private final static String JOB_DISABLED = "job is disabled, check the configuration file !";
#Value("${job.enabled}")
private boolean batchIsEnabled;
private static final Logger LOGGER = LoggerFactory.getLogger("FUNCTIONAL_LOGGER");
#Bean
#StepScope
#RefreshScope
public MyEntityWriter writer() {
return new MyEntityWriter();
}
#Bean
#StepScope
#RefreshScope
public MyFunctionProcessor processor() throws IOException {
return new MyFunctionProcessor();
}
#Bean
public MyPrationner partitioner() {
return new MyPrationner();
}
#Bean
public Step masterStep() throws SQLException, IOException, ClassNotFoundException {
return stepBuilderFactory.get("masterStep")
.partitioner("MyFunctionStep", partitioner())
.step(MyFunctionStep())
.gridSize(gridSize)
.taskExecutor(MyFunctionTaskExecutor())
.build();
}
#Bean
public TaskExecutor myFunctionTaskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setThreadNamePrefix("MyFunctionTaskExecutor_");
int corePoolSize = gridSize + 2;
int maxPoolSize = corePoolSize * 2;
taskExecutor.setMaxPoolSize(maxPoolSize);
taskExecutor.setAllowCoreThreadTimeOut(true);
taskExecutor.setCorePoolSize(corePoolSize);
taskExecutor.setQueueCapacity(Integer.MAX_VALUE);
return taskExecutor;
}
#Bean
public Step myFunctionStep() throws IOException, ClassNotFoundException, SQLException {
return stepBuilderFactory.get("MyFunctionStep")
.<MyEntity, MyEntity>chunk(chunckSize)
.reader(reader)
.faultTolerant()
.skipLimit(MAX_NUMBER_CARD)
.skip(InvalidCardNumberException.class)
.skip(TokenManagementException.class)
.processor(processor())
.listener(new MyEntityProcessListener())
.writer(writer())
.listener(new MyEntityWriteListener())
.build();
}
#Bean
public Job myFunctionJob(#Qualifier("MyFunctionStep") Step myFunctionStep)
throws SQLException, IOException, ClassNotFoundException {
if (!batchIsEnabled) {
LOGGER.error(JOB_DISABLED);
System.exit(0);
}
return jobBuilderFactory.get("MyFunctionJob")
.listener(new MyFunctionJobListener())
.incrementer(new RunIdIncrementer())
.flow(masterStep())
.end()
.build();
}
}
I try to run a spring batch job with partions to read data from oracle database( in the sql there is a decryption function ) this need to set an authorisation for every session of connexion
the problem when th batch run some partion not decrypt data and return null an the only reason fo that , is that the authorisation is not set
The JdbcCursorItemReader does not use a JdbcTemplate. It directly creates connections to the database from the data source object passed to it. So you should not be expecting to call authorize which operates on a separate JdbcTemplate instance to impact the behaviour of the JdbcCursorItemReader. You said it works for some partitions, and that's really surprising.
If you want to take control on how the connection to the database is configured and override the default settings (for example by adding some authorization attributes), you need to extend JdbcCursorItemReader and override the protected void openCursor(Connection con) method, something like:
class MyCustomJdbcCursorItemReader extends JdbcCursorItemReader {
#Override
protected void openCursor(Connection con) {
super.openCursor(con);
// con.setClientInfo(); // set client info as needed here
}
}
In my first Spring Batch chunk job (I'm a beginner),
I wrote db utility class for using in the chunk step,
since I need to overwrite default connection attributes set by application.properties.
However, it causes rollback problem (never rollback) on errors.
Any advices to improve these logics?
DBUtility class
// Construct
public DBUtility() {
... some business logic to get connection attributes dynamically.
}
#Bean
#BatchDataSource
#Primary
public DriverManagerDataSource getConnection() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverNameGottenAbove);
dataSource.setUrl(urlGottenAbove);
dataSource.setUsername(userNameGottenAbove);
dataSource.setPassword(passwordGottenAbove);
return dataSource;
}
#Bean
public DataSourceTransactionManager transactionManager(){
DataSourceTransactionManager dtm = new DataSourceTransationManager(getConnection());
dtm.setRollbackOnCommitFailure(true);
return dtm;
}
#Bean
public JdbcTemplate jdbcTemplate(){
return new JdbcTemplate(getConnection());
}
BatchConfiguration class(chunk)
#SpringBootApplication
#EnableBatchProcessing
#Transactional
public class BatchConfiguration extends DefaultBatchConfigurer {
...
private DBUtility dbUtil;
private DriverManagerDataSource ds;
private PlatformTransactionManager ptm;
// Construct
public BatchConfiguration(){
dbUtil = new DBUtility();
ds = dbUtil.getConnection();
ptm = dbUtil.transactionManager();
}
// Override
#Override
public JobRepository createJobRepository() throws Exception {
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(ds);
factory.setTransactionManager(ptm);
JobRepository repo = factory.getObject();
return repo;
}
#Bean
public Job BatchJob() {
return jobBuilderFactory.get("BatchJob")
.incrementer(new RunIdIncrementer())
.listener(jobListener())
.flow(step1())
.end().build();
}
#Bean
public step step1() {
DefaultTransactionAttribute att = new DefaultTransactionAttribute();
att.setPropagationBehavior(Propagation.REQUIRED.value());
return stepBuilderFactory.get("step1")
.transactionManager(ptm)
.listener(stepListener())
.<Entity, Entity>chunk(COMMIT_INTERVAL)
.reader(reader())
.processor(processore())
.writer(writer())
.readerIsTransactionalQueue()
.transactionAttribute(att)
.build();
}
...
}
Thank you for checking this post.
I had spring batch application like below (table name and query are edited for some general names)
when i execute this program, it was able to read 7500 events , i.e 3 times of chunk size and not able to read remaining records in oracle database. I had a table contain 50 million records and able to copy to another noSql database.
#EnableBatchProcessing
#SpringBootApplication
#EnableAutoConfiguration
public class MultiThreadPagingApp extends DefaultBatchConfigurer{
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
#Bean
public DataSource dataSource() {
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("oracle.jdbc.OracleDriver");
dataSource.setUrl("jdbc:oracle:thin:#***********");
dataSource.setUsername("user");
dataSource.setPassword("password");
return dataSource;
}
#Override
public void setDataSource(DataSource dataSource) {}
#Bean
#StepScope
ItemReader<UserModel> dbReader() throws Exception {
JdbcPagingItemReader<UserModel> reader = new JdbcPagingItemReader<UserModel>();
final SqlPagingQueryProviderFactoryBean sqlPagingQueryProviderFactoryBean = new SqlPagingQueryProviderFactoryBean();
sqlPagingQueryProviderFactoryBean.setDataSource(dataSource);
sqlPagingQueryProviderFactoryBean.setSelectClause("select * ");
sqlPagingQueryProviderFactoryBean.setFromClause("from user");
sqlPagingQueryProviderFactoryBean.setWhereClause("where id>0");
sqlPagingQueryProviderFactoryBean.setSortKey("name");
reader.setQueryProvider(sqlPagingQueryProviderFactoryBean.getObject());
reader.setDataSource(dataSource);
reader.setPageSize(2500);
reader.setRowMapper(new BeanPropertyRowMapper<>(UserModel.class));
reader.afterPropertiesSet();
reader.setSaveState(true);
System.out.println("Reading users anonymized in chunks of {}"+ 2500);
return reader;
}
#Bean
public Dbwriter writer() {
return new Dbwriter(); // I had another class for this
}
#Bean
public Step step1() throws Exception {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setCorePoolSize(4);
taskExecutor.setMaxPoolSize(10);
taskExecutor.afterPropertiesSet();
return this.stepBuilderFactory.get("step1")
.<UserModel, UserModel>chunk(2500)
.reader(dbReader())
.writer(writer())
.taskExecutor(taskExecutor)
.build();
}
#Bean
public Job multithreadedJob() throws Exception {
return this.jobBuilderFactory.get("multithreadedJob")
.start(step1())
.build();
}
#Bean
public PlatformTransactionManager getTransactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
public JobRepository getJobRepo() throws Exception {
return new MapJobRepositoryFactoryBean(getTransactionManager()).getObject();
}
public static void main(String[] args) {
SpringApplication.run(MultiThreadPagingApp.class, args);
}
}
Can you help me how can i efficiently read all the records using spring batch, or help me any other approach to handle this. I had tried one approch mentioned here : http://techdive.in/java/jdbc-handling-huge-resultset
its taken 120 mins to read and save all records with single thread application. Since spring batch is best fit for this, I assume we can handle this scenario in quick time.
You are setting the saveState flag to true (BTW, it should be set before calling afterPropertiesSet) on a JdbcPagingItemReader and using this reader in a multithreaded step. However, it is documented to set this flag to false in a multi-threaded context.
Multi-threading with database readers is usually not the best option, I would recommend to use partitioning in your case.
I had the same problem, i fix it by changing my sortKey. I realize that the previous one wasn't different for every data records. So i replace it with ID whitch were different for every records of the Database
I have been unable to get this to work even after following Victor Jabor blog very comprehensive example. I have followed his configuration as he described and used all the latest dependencies. I, as Victor am trying to read from one db and write to another. I have this working without partitioning but need partitioning to improve performance as I need to be able to read 5 to 10 million rows within 5mins.
The following seems to work:
1) ColumnRangePartitioner
2) TaskExecutorPartitionHandler builds the correct number of step tasks based on the gridsize and spawns the correct number of threads
3) setPreparedStatementSetter from the stepExecution set by the ColumnRangePartitioner.
But when I run the application I get errors from JdbcCursorItemReader which are not consistent and which I dont understand. As a last resort I will have to debug the JdbcCursorItemReader. I am hoping to get some help before this and hopefully it will be a configuration issue.
ERROR:
Caused by: java.sql.SQLException: Exhausted Resultset
at oracle.jdbc.driver.OracleResultSetImpl.getInt(OracleResultSetImpl.java:901) ~[ojdbc6-11.2.0.2.0.jar:11.2.0.2.0]
at org.springframework.jdbc.support.JdbcUtils.getResultSetValue(JdbcUtils.java:160) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.BeanPropertyRowMapper.getColumnValue(BeanPropertyRowMapper.java:370) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.jdbc.core.BeanPropertyRowMapper.mapRow(BeanPropertyRowMapper.java:291) ~[spring-jdbc-4.3.4.RELEASE.jar:4.3.4.RELEASE]
at org.springframework.batch.item.database.JdbcCursorItemReader.readCursor(JdbcCursorItemReader.java:139) ~[spring-batch-infrastructure-3.0.7.RELEASE.jar:3.0.7.RELEASE]
Configuration classes:
#Configuration #EnableBatchProcessing public class BatchConfiguration {
#Bean
public ItemProcessor<Archive, Archive> processor(#Value("${etl.region}") String region) {
return new ArchiveProcessor(region);
}
#Bean
public ItemWriter<Archive> writer(#Qualifier(value = "postgres") DataSource dataSource) {
JdbcBatchItemWriter<Archive> writer = new JdbcBatchItemWriter<>();
writer.setSql("insert into tdw_src.archive (id) " +
"values (:id)");
writer.setDataSource(dataSource);
writer.setItemSqlParameterSourceProvider(new org.springframework.batch.item.database.
BeanPropertyItemSqlParameterSourceProvider<>());
return writer;
}
#Bean
public Partitioner archivePartitioner(#Qualifier(value = "gmDataSource") DataSource dataSource,
#Value("ROWNUM") String column,
#Value("archive") String table,
#Value("${gm.datasource.username}") String schema) {
return new ColumnRangePartitioner(dataSource, column, schema + "." + table);
}
#Bean
public Job archiveJob(JobBuilderFactory jobs, Step partitionerStep, JobExecutionListener listener) {
return jobs.get("archiveJob")
.preventRestart()
.incrementer(new RunIdIncrementer())
.listener(listener)
.start(partitionerStep)
.build();
}
#Bean
public Step partitionerStep(StepBuilderFactory stepBuilderFactory,
Partitioner archivePartitioner,
Step step1,
#Value("${spring.batch.gridsize}") int gridSize) {
return stepBuilderFactory.get("partitionerStep")
.partitioner(step1)
.partitioner("step1", archivePartitioner)
.gridSize(gridSize)
.taskExecutor(taskExecutor())
.build();
}
#Bean(name = "step1")
public Step step1(StepBuilderFactory stepBuilderFactory, ItemReader<Archive> customReader,
ItemWriter<Archive> writer, ItemProcessor<Archive, Archive> processor) {
return stepBuilderFactory.get("step1")
.listener(customReader)
.<Archive, Archive>chunk(5)
.reader(customReader)
.processor(processor)
.writer(writer)
.build();
}
#Bean
public TaskExecutor taskExecutor(){
return new SimpleAsyncTaskExecutor();
}
#Bean
public SimpleJobLauncher getJobLauncher(JobRepository jobRepository) {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
return jobLauncher;
}
Custom Reader:-
public class CustomReader extends JdbcCursorItemReader<Archive> implements StepExecutionListener {
private StepExecution stepExecution;
#Autowired
public CustomReader(#Qualifier(value = "gmDataSource") DataSource geomangerDataSource,
#Value("${gm.datasource.username}") String schema) throws Exception {
super();
this.setSql("SELECT TMP.* FROM (SELECT ROWNUM AS ID_PAGINATION, id FROM " + schema + ".archive) TMP " +
"WHERE TMP.ID_PAGINATION >= ? AND TMP.ID_PAGINATION <= ?");
this.setDataSource(geomangerDataSource);
BeanPropertyRowMapper<Archive> rowMapper = new BeanPropertyRowMapper<>(Archive.class);
this.setRowMapper(rowMapper);
this.setFetchSize(5);
this.setSaveState(false);
this.setVerifyCursorPosition(false);
// not sure if this is needed? this.afterPropertiesSet();
}
#Override
public synchronized void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
this.setPreparedStatementSetter(getPreparedStatementSetter());
}
private PreparedStatementSetter getPreparedStatementSetter() {
ListPreparedStatementSetter listPreparedStatementSetter = new ListPreparedStatementSetter();
List<Integer> list = new ArrayList<>();
list.add(stepExecution.getExecutionContext().getInt("minValue"));
list.add(stepExecution.getExecutionContext().getInt("maxValue"));
listPreparedStatementSetter.setParameters(list);
LOGGER.debug("getPreparedStatementSetter list: " + list);
return listPreparedStatementSetter;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
}
I've got this all working.
First I needed to order my select statement in my CustomReader so the rownum remains the same for all threads and lastly I had to scope the beans by using #StepScope for each bean used in the step.
In reality I wont be using rownum since this needs to be ordered which reduce loose performance and therefore I will use a pk column to get the best performance.
Context
I'm trying to develop a batch service with Spring Boot, using JPA Repository. Using two different datasources, I want the batch-related tables created in a in-memory database, so that it does not pollute my business database.
Following multiple topics on the web, I came up with this configuration of my two datasources :
#Configuration
public class DataSourceConfiguration {
#Bean(name = "mainDataSource")
#Primary
#ConfigurationProperties(prefix="spring.datasource")
public DataSource mainDataSource(){
return DataSourceBuilder.create().build();
}
#Bean(name = "batchDataSource")
public DataSource batchDataSource( #Value("${batch.datasource.url}") String url ){
return DataSourceBuilder.create().url( url ).build();
}
}
The first one, mainDataSource, uses the default Spring database configuration. The batchDataSource defines an embedded HSQL database, in which I want the batch and step tables to be created.
# DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource.url=jdbc:mariadb://localhost:3306/batch_poc
spring.datasource.username=root
spring.datasource.password=
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.max-age=10000
spring.datasource.initialize=false
# JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
spring.jpa.generate-ddl=false
spring.jpa.show-sql=true
spring.jpa.database=MYSQL
# SPRING BATCH (BatchDatabaseInitializer)
spring.batch.initializer.enabled=false
# ----------------------------------------
# PROJECT SPECIFIC PROPERTIES
# ----------------------------------------
# BATCH DATASOURCE
batch.datasource.url=jdbc:hsqldb:file:C:/tmp/hsqldb/batchdb
Here is my batch config :
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
private static final Logger LOG = Logger.getLogger( BatchConfiguration.class );
#Bean
public BatchConfigurer configurer(){
return new CustomBatchConfigurer();
}
#Bean
public Job importElementsJob( JobBuilderFactory jobs, Step step1 ){
return jobs.get("importElementsJob")
.incrementer( new RunIdIncrementer() )
.flow( step1 )
.end()
.build();
}
#Bean
public Step step1( StepBuilderFactory stepBuilderFactory, ItemReader<InputElement> reader,
ItemWriter<List<Entity>> writer, ItemProcessor<InputElement, List<Entity>> processor ){
return stepBuilderFactory.get("step1")
.<InputElement, List<Entity>> chunk(100)
.reader( reader )
.processor( processor )
.writer( writer )
.build();
}
#Bean
public ItemReader<InputElement> reader() throws IOException {
return new CustomItemReader();
}
#Bean
public ItemProcessor<InputElement, List<Entity>> processor(){
return new CutsomItemProcessor();
}
#Bean
public ItemWriter<List<Entity>> writer(){
return new CustomItemWriter();
}
}
The BatchConfigurer, using the in-memory database :
public class CustomBatchConfigurer extends DefaultBatchConfigurer {
#Override
#Autowired
public void setDataSource( #Qualifier("batchDataSource") DataSource dataSource) {
super.setDataSource(dataSource);
}
}
And, finally, my writer :
public class CustomItemWriter implements ItemWriter<List<Entity>> {
private static final Logger LOG = Logger.getLogger( EntityWriter.class );
#Autowired
private EntityRepository entityRepository;
#Override
public void write(List<? extends List<Entity>> items)
throws Exception {
if( items != null && !items.isEmpty() ){
for( List<Entity> entities : items ){
for( Entity entity : entities ){
Entity fromDb = entityRepository.findById( entity.getId() );
// Insert
if( fromDb == null ){
entityRepository.save( entity );
}
// Update
else {
// TODO : entityManager.merge()
}
}
}
}
}
}
The EntityRepository interface extends JpaRepository.
Problem
When I separate the datasources this way, nothing happens when I call the save method of the repository. I see the select queries from the call of findById() in the logs. But nothing for the save. And my output database is empty at the end.
When I come back to a unique datasource configuration (removing the configurer bean and letting Spring Boot manage the datasource alone), the insert queries work fine.
Maybe the main datasource configuration is not good enough for JPA to perform the inserts correctly. But what is missing ?
I finally solved the problem implementing my own BatchConfigurer, on the base of the Spring class BasicBatchConfigurer, and forcing the use of Map based jobRepository and jobExplorer. No more custom datasource configuration, only one datasource which I let Spring Boot manage : it's easier that way.
My custom BatchConfigurer :
public class CustomBatchConfigurer implements BatchConfigurer {
private static final Logger LOG = Logger.getLogger( CustomBatchConfigurer.class );
private final EntityManagerFactory entityManagerFactory;
private PlatformTransactionManager transactionManager;
private JobRepository jobRepository;
private JobLauncher jobLauncher;
private JobExplorer jobExplorer;
/**
* Create a new {#link CustomBatchConfigurer} instance.
* #param entityManagerFactory the entity manager factory
*/
public CustomBatchConfigurer( EntityManagerFactory entityManagerFactory ) {
this.entityManagerFactory = entityManagerFactory;
}
#Override
public JobRepository getJobRepository() {
return this.jobRepository;
}
#Override
public PlatformTransactionManager getTransactionManager() {
return this.transactionManager;
}
#Override
public JobLauncher getJobLauncher() {
return this.jobLauncher;
}
#Override
public JobExplorer getJobExplorer() throws Exception {
return this.jobExplorer;
}
#PostConstruct
public void initialize() {
try {
// transactionManager:
LOG.info("Forcing the use of a JPA transactionManager");
if( this.entityManagerFactory == null ){
throw new Exception("Unable to initialize batch configurer : entityManagerFactory must not be null");
}
this.transactionManager = new JpaTransactionManager( this.entityManagerFactory );
// jobRepository:
LOG.info("Forcing the use of a Map based JobRepository");
MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean( this.transactionManager );
jobRepositoryFactory.afterPropertiesSet();
this.jobRepository = jobRepositoryFactory.getObject();
// jobLauncher:
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(getJobRepository());
jobLauncher.afterPropertiesSet();
this.jobLauncher = jobLauncher;
// jobExplorer:
MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
jobExplorerFactory.afterPropertiesSet();
this.jobExplorer = jobExplorerFactory.getObject();
}
catch (Exception ex) {
throw new IllegalStateException("Unable to initialize Spring Batch", ex);
}
}
}
My configuration class looks like this now :
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Bean
public BatchConfigurer configurer( EntityManagerFactory entityManagerFactory ){
return new CustomBatchConfigurer( entityManagerFactory );
}
[...]
}
And my properties files :
# DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource.url=jdbc:mariadb://localhost:3306/inotr_poc
spring.datasource.username=root
spring.datasource.password=root
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.max-age=10000
spring.datasource.initialize=true
# JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
spring.jpa.generate-ddl=false
spring.jpa.show-sql=true
spring.jpa.database=MYSQL
# SPRING BATCH (BatchDatabaseInitializer)
spring.batch.initializer.enabled=false
Thanks for the above posts! I have been struggling for the past couple of days to get my Spring Boot with Batch application working with an in-memory Map based job repository and a resourceless transaction manager. ( I cannot let Spring Batch to use my application datasource for the Batch meta data tables as I don't have the DDL access to create the BATCH_ tables there )
Finally arrived at the below configuration after looking at the above posts and it worked perfectly!!
public class CustomBatchConfigurer implements BatchConfigurer {
private static final Logger LOG = LoggerFactory.getLogger(CustomBatchConfigurer.class);
// private final EntityManagerFactory entityManagerFactory;
private PlatformTransactionManager transactionManager;
private JobRepository jobRepository;
private JobLauncher jobLauncher;
private JobExplorer jobExplorer;
/**
* Create a new {#link CustomBatchConfigurer} instance.
* #param entityManagerFactory the entity manager factory
public CustomBatchConfigurer( EntityManagerFactory entityManagerFactory ) {
this.entityManagerFactory = entityManagerFactory;
}
*/
#Override
public JobRepository getJobRepository() {
return this.jobRepository;
}
#Override
public PlatformTransactionManager getTransactionManager() {
return this.transactionManager;
}
#Override
public JobLauncher getJobLauncher() {
return this.jobLauncher;
}
#Override
public JobExplorer getJobExplorer() throws Exception {
return this.jobExplorer;
}
#PostConstruct
public void initialize() {
try {
// transactionManager:
LOG.info("Forcing the use of a Resourceless transactionManager");
this.transactionManager = new ResourcelessTransactionManager();
// jobRepository:
LOG.info("Forcing the use of a Map based JobRepository");
MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean( this.transactionManager );
jobRepositoryFactory.afterPropertiesSet();
this.jobRepository = jobRepositoryFactory.getObject();
// jobLauncher:
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(getJobRepository());
jobLauncher.afterPropertiesSet();
this.jobLauncher = jobLauncher;
// jobExplorer:
MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
jobExplorerFactory.afterPropertiesSet();
this.jobExplorer = jobExplorerFactory.getObject();
}
catch (Exception ex) {
throw new IllegalStateException("Unable to initialize Spring Batch", ex);
}
}
}
And below is the bean i added in my job configuration class
#Bean
public BatchConfigurer configurer(){
return new CustomBatchConfigurer();
}
Eria's answer worked! However, I had modified it to use:
org.springframework.batch.support.transaction.ResourcelessTransactionManager
From CustomBatchConfigurer:
#PostConstruct
public void initialize() {
try {
// transactionManager:
LOGGER.info("Forcing the use of ResourcelessTransactionManager for batch db");
this.transactionManager = new ResourcelessTransactionManager();
//the rest of the code follows...
}