Is there a way to integrate spring batch with spring data? I see RepositoryItemReader and RepositoryItemWriter in spring documentation.
You are totally right. Spring batch can be easily be integrated with spring data. Here example of item reader:
#Bean(name = "lotteryInfoReader")
#StepScope
public RepositoryItemReader<LotteryInfo> reader() {
RepositoryItemReader<LotteryInfo> reader = new RepositoryItemReader<>();
reader.setRepository(lotteryInfoRepository);
reader.setMethodName("findAll");
reader.setSort(Collections.singletonMap("name", Sort.Direction.ASC));
return reader;
}
Here is another example with using hibernate without spring data :
#Bean(name = "drawsWriter")
#StepScope
public ItemWriter<? super List<Draw>> writer() {
return items -> items.stream()
.flatMap(Collection::stream)
.forEach(entityManager::merge);
}
well i'm doing that in a different way i'm injecting the service into the job configuration and invoke the method that's available on the JpaRepository, like this example
#Bean
#StepScope
public ItemWriter<Customer> customerItemWriter2() {
return items -> {
for (Customer item : items) {
customerService.save(item);
}
};
}
Related
I am trying to create a Spring bean based on a MyBatis databaseId like so:
#Bean
fun getSomeBean(#Autowired databaseId: String): SomeBean {
if(databaseId.equals("mysql")){
return SomeMySqlBean()
} else {
return SomeNonMysqlBean()
}
}
How do I get the databaseId to be injected with MyBatis?
I had a look at some of the API documentation and came up with this - works nicely and now I can construct a couple of the vendor-specific objects I need:
#Bean
fun databaseId(sqlSessionFactory: SqlSessionFactory): String {
return sqlSessionFactory.configuration.databaseId
}
I am trying to configure the spring data source programmatically. The idea is to override the default data source if tenant information is specified or else use the default data source for backward compatibility. I tried coming up with something like :
#Component
#AllArgsConstructor
public class TenancyDataSource {
private TenantConfigs tenantConfigs;
private DataSource dataSource;
#Bean
public DataSource dataSource() {
String tenant = System.getenv(AppConstants.TENANT);
if (!ObjectUtils.isEmpty(tenant)) {
TenantConfigs config =
tenantConfigs
.getDataSources()
.stream()
.filter(TenantConfig -> tenantConfig.getTenantName().equals(tenant))
.findFirst()
.orElse(null);
if (!ObjectUtils.isEmpty(config)) {
return DataSourceBuilder.create()
.url(config.getUrl())
.password(config.getPassword())
.username(config.getUsername())
.build();
}
}
return dataSource;
}
}
But this does not work due to circular dependency.
What would be the best way to implement it ?
SpringBoot treats environment variables as properties by default. So using #ConditionalOnProperty with the "tenant" property worked
#Bean
#ConditionalOnProperty(name = AppConstants.TENANT)
public DataSource dataSource() {
return DataSourceBuilder.create()
.url(config.getUrl())
.password(config.getPassword())
.username(config.getUsername())
.build();
}
I want to migrate my app to WebFlux, but the tricky part that I have bean which connects to 6 data sources by such mechanism
public class MultiRoutingDataSource extends AbstractRoutingDataSource {
#Override
protected Object determineCurrentLookupKey() {
return //code which sets context for chosen db;
}
}
Then I'm creating 6 data sources which is then managed by multiRoutingDataSource
#Bean(name = "multiRoutingDataSource")
public DataSource multiRoutingDataSource() {
Map<Object, Object> targetDataSources = new HashMap<>();
targetDataSources.put(ident, MyDataSourceBean());
MultiRoutingDataSource multiRoutingDataSource = new MultiRoutingDataSource();
multiRoutingDataSource.setTargetDataSources(targetDataSources);
return multiRoutingDataSource;
}
and this data sources could be changed in runtime. This multiRouting then set into entity manager.
Is there something similar with WebFlux?
I found
public class MultiRoutingDataSource extends AbstractRoutingConnectionFactory {
#Override
protected Mono<Object> determineCurrentLookupKey() {
return null;
}
But how to create beans with connections and switch them in runtime like I'm doing in Spring MVC?
If you want multi R2dbc connectionfactories at the same application, check my example multi-r2dbc-connectionfactories.
For multi-tenancy support, check multi-tenancy-r2dbc.
I am all new to Spring Boot and I have read som documentation about how you create a Spring boot application.
I have created an application in Spring Boot, this application should run a Spring Batch job (I am rewriting an old job in Spring Batch to a stand alone Spring Boot applikation). I have created the structure of the job with steps and so on. All the job does right now is moving files and that works. I work with embedded databases during development, h2, and Spring Boot has generated the whole database for Spring Batch. Really nice :)
So now my problem, in one of the steps I have to fetch and store data in another database. And I don't know (understand) how I should create this database and access the database in the job.
So in my application.properties
spring.h2.console.enabled=true
spring.h2.console.path=/h2
spring.datasource.url=jdbc:h2:mem:springdb
spring.datasource.username=sa
spring.datasource.password=
spring.datasource.driver-class-name=org.h2.Driver
persons.h2.console.enabled=true
persons.h2.console.path=/h2
persons.datasource.url=jdbc:h2:mem:persons
persons.datasource.username=sa
persons.datasource.password=
persons.datasource.driverClassName=org.h2.Driver
In test I will change database to a SQL-database on another server.
I have some entitys (example)
public class Person {
private Long id;
private String name;
private String familyname;
private Long birthDate;
public Person () {
}
...with getters and setters
In the configuration I have
#Primary
#Bean
#ConfigurationProperties(prefix = "persons.datasource")
public DataSource personsDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource batchDataSource() {
return DataSourceBuilder.create().build();
}
And the job
#Autowired
PersonItemWriter itemWriter;
#Autowired
PersonItemProcessor itemProcessor;
#Autowired
PersonItemReader workReader;
#Bean(name = "personsImportJob")
public Job personImportJob() {
Step downloadFiles = stepBuilderFactory.get("download-files")
.tasklet(downloadTasklet())
.build();
Step syncDbAndSavePersons = stepBuilderFactory.get("syncandsave-persons")
.<File, Person>chunk(50)
.reader(workReader)
.processor(itemProcessor)
.writer(itemWriter)
.build();
Step deleteFiles = stepBuilderFactory.get("delete-files")
.tasklet(deleteTasklet())
.build();
Job job = jobBuilderFactory.get("personsimport-job")
.incrementer(new RunIdIncrementer())
.flow(downloadFiles)
.next(syncDbAndSavePersons)
.next(deleteFiles)
.end()
.build();
return job;
}
My writer
#Component
public class PersonItemWriter implements ItemWriter<Person> {
private static final Logger LOGGER = LoggerFactory.getLogger(PersonItemWriter.class);
#Override
public void write(List<? extends Person> list) throws Exception {
LOGGER.info("Write Person to db");
}
}
Now this works, the step syncAndSavePersons does not do anything right now, but I want this step to access another database and update posts to the persons-database.
Can I do this without JPA? Because the existing job doesn't use JPA and if I have to use JPA there will be a lot of code changes and I want to avoid this. I just want to move the job with minimum of changes,
If I run my application with this the only database that is created is the database for spring batch. How can I make sure that the other database is also created? Or is that impossible when I am working with h2 embedded databases?
Before I added the second database I didn't have the datasource configuration at all in my code, but the database was created anyway. I think Spring Boot just created the batch datasource. So maybe I won't need that configuration?
UPDATE:
I solved it by removing the properties for the database in te properties file. The only thing I left is:
spring:
datasource:
initialization-mode: never
h2:
console:
enabled: true
I then created a class called EmbeddedDataSourceConfig:
#Profile("default")
#Configuration
public class EmbeddedDataSourceConfig {
#Bean
#Primary
public DataSource dataSource() {
EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
return builder
.setType(EmbeddedDatabaseType.H2)
.addScript("classpath:/org/springframework/batch/core/schema-h2.sql")
.build();
}
#Bean(name = "personsDataSource")
public DataSource personsDataSource() {
EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
return builder
.setName("persons")
.setType(EmbeddedDatabaseType.H2)
.addScript("classpath:persons_schema.sql")
.build();
}
#Bean(name = "personsTransactionManager")
public PlatformTransactionManager personsTransactionManager(
#Qualifier("personsDataSource") DataSource dataSource) {
return new DataSourceTransactionManager(dataSource);
}
#Bean(name = "personsJdbcTemplate")
public JdbcTemplate personsJdbcTemplate(#Qualifier("personsDataSource") DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
}
and in the job I changed to following
#Bean(name = "personsImportJob")
public Job personImportJob(#Qualifier("personsTransactionManager") PlatformTransactionManager personsTransactionManager) {
Step downloadFiles = stepBuilderFactory.get("download-files")
.tasklet(downloadTasklet())
.build();
Step syncDbAndSavePersons = stepBuilderFactory.get("syncandsave-persons")
.<File, Person>chunk(50)
.reader(workReader)
.processor(itemProcessor)
.writer(itemWriter)
.transactionManager(personsTransactionManager)
.build();
...
}
That's it. Now it generates two h2 in-memory databases.
I am using Spring Batch and JPA to process a Batch Job and perform updates. I am using the default Repository implementations.
And I am using a repository.save to save the modified object in the processor.
Also,I don't have any #Transactional Annotation specified in the processor or writer.
I don't see any exceptions throughout. The selects happen fine.
Is there any setting like "setAutoCommit(true)" that I should be using for the JPA to save the data in the DB.
Here is my step,reader and writer config:
Also, my config class is annotated with EnableBatchProcessing
#EnableBatchProcessing
public class UpgradeBatchConfiguration extends DefaultBatchConfigurer{
#Autowired
private PlatformTransactionManager transactionManager;
#Override
protected JobRepository createJobRepository() throws Exception {
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(getdataSource());
factory.setTransactionManager(transactionManager);
factory.setTablePrefix("CFTES_OWNER.BATCH_");
factory.afterPropertiesSet();
return factory.getObject();
}
#Bean(name = "updateFilenetJobStep")
public Step jobStep(StepBuilderFactory stepBuilderFactory,
#Qualifier("updateFileNetReader") RepositoryItemReader reader,
#Qualifier("updateFileNetWriter") ItemWriter writer,
#Qualifier("updateFileNetProcessor") ItemProcessor processor) {
return stepBuilderFactory.get("jobStep").allowStartIfComplete(true).chunk(1).reader(reader).processor(processor)
.writer(writer).transactionManager(transactionManager).build();
}
#Bean(name = "updateFileNetWriter")
public ItemWriter getItemWriter() {
return new BatchItemWriter();
}
#Bean(name = "updateFileNetReader")
public RepositoryItemReader<Page<TermsAndConditionsErrorEntity>> getItemReader(
TermsAndConditionsErrorRepository repository) {
RepositoryItemReader<Page<TermsAndConditionsErrorEntity>> reader = new RepositoryItemReader<Page<TermsAndConditionsErrorEntity>>();
reader.setRepository(repository);
reader.setMethodName("findAll");
HashMap<String, Direction> map = new HashMap<String, Direction>();
map.put("transactionId", Direction.ASC);
reader.setSort(map);
return reader;
}
}
And in the writer this is what I am using Repository.save
repository.save(entity);
I was able to solve this by injecting a JPATransactionManager throughout(the job repository, job, step etc) , instead of the Autowired PlatformTransactionManager.