How do I configure 2 jdbc connections using Spring4 java config classes ?
Should 2 transaction managers be configure for those two connections ?
Thx
Edit:
I'm want to use just JdbcTemplate no JPA, Spring Data.
Example configuration might looks something like the following. I also pushed a full sample to GitHub which can be found here
#Configuration
public class DataSourceConfiguration {
#Bean
public PlatformTransactionManager firstDataSourceTransactionManager() {
return new DataSourceTransactionManager(firstDataSource());
}
#Bean(destroyMethod = "shutdown")
#Primary
public DataSource firstDataSource() {
return new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2)
.generateUniqueName(true)
.build();
}
#Bean
public JdbcTemplate firstJdbcTemplate() {
return new JdbcTemplate(firstDataSource());
}
#Bean
public PlatformTransactionManager secondDataSourceTransactionManager() {
return new DataSourceTransactionManager(secondDataSource());
}
#Bean(destroyMethod = "shutdown")
public DataSource secondDataSource() {
return new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2)
.generateUniqueName(true)
.build();
}
#Bean
public JdbcTemplate secondJdbcTemplate() {
return new JdbcTemplate(secondDataSource());
}
}
Related
I am trying to migrate some data between a Postgres database and MongoDB using Spring Batch. I have a very simple ItemReader, ItemProcessor, and ItemWriter configured, and it everything works as intended. However, if I switch to a RepositoryItemReader, I'm getting the following error:
java.lang.IllegalStateException: Already value [org.springframework.jdbc.datasource.ConnectionHolder#684430c1] for key [HikariDataSource (HikariPool-1)] bound to thread
If I understand correctly, there is something wrong with the EntityManager or TransactionManager, but I cannot figure out what, and why it's working with a simple ItemReader that doesn't work with a repository, but it uses the same data source.
I would be very grateful for any help.
Here is my source db configuration:
package com.example.batch.primary;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "primaryEntityManagerFactory",
transactionManagerRef = "primaryTransactionManager",
basePackages = {"com.example.batch.primary"}
)
public class PrimaryDBConfig {
#Bean(name = "primaryDataSource")
#Primary
public DataSource primaryDatasource(){
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create()
.driverClassName("org.postgresql.Driver")
.url("jdbc:postgresql://localhost:5432/postgres")
.username("test")
.password("test");
return dataSourceBuilder.build();
}
#Bean(name = "primaryEntityManagerFactory")
#Primary
public LocalContainerEntityManagerFactoryBean primaryEntityManagerFactory(EntityManagerFactoryBuilder builder,
#Qualifier("primaryDataSource")
DataSource primaryDataSource){
return builder.dataSource(primaryDataSource)
.packages("com.example.batch.primary")
.build();
}
#Bean(name = "primaryTransactionManager")
public PlatformTransactionManager primaryTransactionManager(
#Qualifier("primaryEntityManagerFactory") EntityManagerFactory primaryEntityManagerFactory)
{
return new JpaTransactionManager(primaryEntityManagerFactory);
}
}
Here is the configuration of MongoDB:
package com.example.batch.secondary;
#EnableMongoRepositories(basePackages = "com.example.batch.secondary")
#Configuration
public class MongoDBConfig {
#Bean
public MongoClient mongo() {
ConnectionString connectionString = new ConnectionString("mongodb+srv://mongoadmin:blablabla.mongodb.net/?retryWrites=true&w=majority");
MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applyConnectionString(connectionString)
.build();
return MongoClients.create(mongoClientSettings);
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
return new MongoTemplate(mongo(), "test");
}
}
Here is the RepositoryItemReader:
package com.example.batch.stepcomponents;
#Component
public class RepositoryReader extends RepositoryItemReader<Partner> {
public RepositoryReader(#Autowired PartnerRepository partnerRepository){
setRepository(partnerRepository);
setPageSize(1);
setSort(Map.of("id", Sort.Direction.ASC));
setMethodName("findAll");
}
}
Batch Config:
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
RepositoryReader repositoryReader;
#Autowired
CustomWriter customWriter;
#Autowired
CustomProcessor customProcessor;
#Bean
public Job createJob() {
return jobBuilderFactory.get("MyJob")
.incrementer(new RunIdIncrementer())
.flow(createStep())
.end()
.build();
}
#Bean
public Step createStep() {
return stepBuilderFactory.get("MyStep")
.<Partner, Student> chunk(1)
.reader(repositoryReader)
.processor(customProcessor)
.writer(customWriter)
.build();
}
}
So I tried taking out the EntityManagerFactory and the TransactionManager, and now it works. I guess they are already initialized automatically when starting up the server..
Yes, by default, if you provide a DataSource bean, Spring Batch will use a DataSourceTransactionManager, not the JPA one as you expect. This is explained in the Javadoc of EnableBatchProcessing:
The transaction manager provided by this annotation will be of type:
* ResourcelessTransactionManager if no DataSource is provided within the context
* DataSourceTransactionManager if a DataSource is provided within the context
In order to use the JPA transaction manager, you need to configure a custom a BatchConfigurer and override getTransactionManager, something like:
#Bean
public BatchConfigurer batchConfigurer(DataSource dataSource, EntityManagerFactory entityManagerFactory) {
return new DefaultBatchConfigurer(dataSource) {
#Override
public PlatformTransactionManager getTransactionManager() {
return new JpaTransactionManager(entityManagerFactory);
}
};
}
Note this will not be required anymore starting from v5, see:
Revisit the configuration of infrastructure beans with #EnableBatchProcessing
Spring Batch 5.0.0-M6 and 4.3.7 are out!
You can also set the JPA transaction manager on your step:
#Bean
public Step createStep(JpaTransactionManager jpaTransactionManager) {
return stepBuilderFactory.get("MyStep")
.<Partner, Student> chunk(1)
.reader(repositoryReader)
.processor(customProcessor)
.writer(customWriter)
.transactionManager(jpaTransactionManager)
.build();
}
Adding 'spring-data-jpa' as a dependency will automatically configure aJpaTransactionManager if no other TransactionManager is defined
My application.properties
spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.username=${env.H2_USER}
spring.datasource.password=${env.H2_PASS}
spring.token-datasource.url=jdbc:mysql://${env.MYSQL_HOST}/${env.MYSQL_DB}
spring.token-datasource.username=${env.MYSQL_USER}
spring.token-datasource.password=${env.MYSQL_PASS}
My Configuration.java code snippet
#Configuration
public class DataSourceConfiguration {
#Bean
#Primary
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource regularDataSource() {
DataSource dataSource = DataSourceBuilder.create().build();
return dataSource;
}
#Bean
#ConfigurationProperties(prefix = "spring.token-datasource")
public DataSource tokenDataSource() {
DataSource dataSource = DataSourceBuilder.create().build();
return dataSource;
}
}
However, if I put a break point at each of return dataSource;, I see the DataSource object not populated with the correponding property values, e.g. jdbcUrl would be null.
Any help?
Thanks!
Followed https://docs.spring.io/spring-boot/docs/2.4.2/reference/html/howto.html#howto-two-datasources and crafted following code which works:
#Configuration
public class DataSourceConfiguration {
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSourceProperties regularDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource.configuration")
public HikariDataSource regularDataSource() {
HikariDataSource dataSource = regularDataSourceProperties().initializeDataSourceBuilder()
.type(HikariDataSource.class).build();
return dataSource;
}
#Bean
#ConfigurationProperties("spring.token-datasource")
public DataSourceProperties tokenDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#ConfigurationProperties("spring.token-datasource.configuration")
public DataSource tokenDataSource() {
DataSource dataSource = tokenDataSourceProperties().initializeDataSourceBuilder().build();
return dataSource;
}
}
It's interesting that the original code snippet does NOT work, while it's still the approach recommened by many Java tutorials and stackoverflow posts. For example,
https://www.baeldung.com/spring-data-jpa-multiple-databases
Spring Boot Configure and Use Two DataSources
In my first Spring Batch chunk job (I'm a beginner),
I wrote db utility class for using in the chunk step,
since I need to overwrite default connection attributes set by application.properties.
However, it causes rollback problem (never rollback) on errors.
Any advices to improve these logics?
DBUtility class
// Construct
public DBUtility() {
... some business logic to get connection attributes dynamically.
}
#Bean
#BatchDataSource
#Primary
public DriverManagerDataSource getConnection() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverNameGottenAbove);
dataSource.setUrl(urlGottenAbove);
dataSource.setUsername(userNameGottenAbove);
dataSource.setPassword(passwordGottenAbove);
return dataSource;
}
#Bean
public DataSourceTransactionManager transactionManager(){
DataSourceTransactionManager dtm = new DataSourceTransationManager(getConnection());
dtm.setRollbackOnCommitFailure(true);
return dtm;
}
#Bean
public JdbcTemplate jdbcTemplate(){
return new JdbcTemplate(getConnection());
}
BatchConfiguration class(chunk)
#SpringBootApplication
#EnableBatchProcessing
#Transactional
public class BatchConfiguration extends DefaultBatchConfigurer {
...
private DBUtility dbUtil;
private DriverManagerDataSource ds;
private PlatformTransactionManager ptm;
// Construct
public BatchConfiguration(){
dbUtil = new DBUtility();
ds = dbUtil.getConnection();
ptm = dbUtil.transactionManager();
}
// Override
#Override
public JobRepository createJobRepository() throws Exception {
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(ds);
factory.setTransactionManager(ptm);
JobRepository repo = factory.getObject();
return repo;
}
#Bean
public Job BatchJob() {
return jobBuilderFactory.get("BatchJob")
.incrementer(new RunIdIncrementer())
.listener(jobListener())
.flow(step1())
.end().build();
}
#Bean
public step step1() {
DefaultTransactionAttribute att = new DefaultTransactionAttribute();
att.setPropagationBehavior(Propagation.REQUIRED.value());
return stepBuilderFactory.get("step1")
.transactionManager(ptm)
.listener(stepListener())
.<Entity, Entity>chunk(COMMIT_INTERVAL)
.reader(reader())
.processor(processore())
.writer(writer())
.readerIsTransactionalQueue()
.transactionAttribute(att)
.build();
}
...
}
Thank you for checking this post.
What should I change in the below spring-boot H2 datasource config, to make sure XA is enabled?
#Bean
//#ConfigurationProperties(prefix = "spring.datasource")
public DataSource h2() {
String url = "jdbc:h2:mem:mydatabase;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE";
return DataSourceBuilder.create()
.url(url)
.driverClassName("org.h2.Driver")
.build();
}
#Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean entityManagerFactory(EntityManagerFactoryBuilder builder) {
return builder
.dataSource(h2())
.packages("demo.h2xa.domain")
.persistenceUnit("mydomainPersistanceUnit")
.build();
}
Ref: http://www.h2database.com/javadoc/org/h2/jdbcx/JdbcDataSource.html
Adding org.h2.jdbcx.JdbcDataSource.class as type helps.
public DataSource h2() {
String url = "jdbc:h2:mem:mydatabase;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE";
return DataSourceBuilder.create()
.url(url)
.driverClassName("org.h2.Driver")
.type(org.h2.jdbcx.JdbcDataSource.class) //xa datasource
.build();
}
I used spring boot + jdbctemplate, and in my business I have to access multi datasource, e.g.
application.properties
foo.datasource.url=jdbc:mysql://127.0.0.1/foo
foo.datasource.username=root
foo.datasource.password=12345678
bar.datasource.url=jdbc:mysql://127.0.0.1/bar
bar.datasource.username=root
bar.datasource.password=12345678
Java Config
#Bean(name = "fooDb")
#ConfigurationProperties(prefix = "foo.datasource")
public DataSource fooDataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "fooJdbcTemplate")
public JdbcTemplate fooJdbcTemplate(#Qualifier("fooDb") DataSource ds) {
return new JdbcTemplate(ds);
}
and there are some common configuration for all datasources
spring.datasource.test-while-idle=true
spring.datasource.time-between-eviction-runs-millis=30000
spring.datasource.validation-query=select 1
How could I populate these common properties to every jdbctemplate, e.g.
#Bean(name = "commonDb")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource commonDataSource() {
return DataSourceBuilder.create().build();
}
Now my way is
#Value("${spring.datasource.test-while-idle}")
private boolean testWhileIdle;
#Value("${spring.datasource.time-between-eviction-runs-millis}")
private int timeBetweenEvictionRunsMillis;
#Value("${spring.datasource.validation-query}")
private String validationQuery;
private DataSource commonProcess(DataSource build) {
org.apache.tomcat.jdbc.pool.DataSource ds = (org.apache.tomcat.jdbc.pool.DataSource) build;
ds.setTestWhileIdle(testWhileIdle);
ds.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis);
ds.setValidationQuery(validationQuery);
return build;
}
#Bean(name = "fooDb")
#ConfigurationProperties(prefix = "foo.datasource")
public DataSource fooDataSource() {
return commonProcess(DataSourceBuilder.create().build());
}