I have a spring boot application which connects to a Postgres SQL database which has 2 separate schemas.
Can I set up Liquidbase to be able to manage/apply changes in the tables that reside in separate shemas inside the same database?
You can create multiple SpringLiquibase beans, each with their own changelog files. For example...
#Configuration
public class DatabaseConfiguration {
#Bean
#ConfigurationProperties(prefix = "datasource.schema1")
public DataSource schema1DataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix = "datasource.schema2")
public DataSource schema2DataSource() {
return DataSourceBuilder.create().build();
}
#Bean
public SpringLiquibase schema1Liquibase() {
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setChangeLog("classpath:/META-INF/liquibase/changelog/schema1-changelog-master.xml");
liquibase.setDataSource(schema1DataSource());
return liquibase;
}
#Bean
public SpringLiquibase schema2Liquibase() {
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setChangeLog("classpath:/META-INF/liquibase/changelog/schema2-changelog-master.xml");
liquibase.setDataSource(schema2DataSource());
return liquibase;
}
}
Related
I am trying to migrate some data between a Postgres database and MongoDB using Spring Batch. I have a very simple ItemReader, ItemProcessor, and ItemWriter configured, and it everything works as intended. However, if I switch to a RepositoryItemReader, I'm getting the following error:
java.lang.IllegalStateException: Already value [org.springframework.jdbc.datasource.ConnectionHolder#684430c1] for key [HikariDataSource (HikariPool-1)] bound to thread
If I understand correctly, there is something wrong with the EntityManager or TransactionManager, but I cannot figure out what, and why it's working with a simple ItemReader that doesn't work with a repository, but it uses the same data source.
I would be very grateful for any help.
Here is my source db configuration:
package com.example.batch.primary;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "primaryEntityManagerFactory",
transactionManagerRef = "primaryTransactionManager",
basePackages = {"com.example.batch.primary"}
)
public class PrimaryDBConfig {
#Bean(name = "primaryDataSource")
#Primary
public DataSource primaryDatasource(){
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create()
.driverClassName("org.postgresql.Driver")
.url("jdbc:postgresql://localhost:5432/postgres")
.username("test")
.password("test");
return dataSourceBuilder.build();
}
#Bean(name = "primaryEntityManagerFactory")
#Primary
public LocalContainerEntityManagerFactoryBean primaryEntityManagerFactory(EntityManagerFactoryBuilder builder,
#Qualifier("primaryDataSource")
DataSource primaryDataSource){
return builder.dataSource(primaryDataSource)
.packages("com.example.batch.primary")
.build();
}
#Bean(name = "primaryTransactionManager")
public PlatformTransactionManager primaryTransactionManager(
#Qualifier("primaryEntityManagerFactory") EntityManagerFactory primaryEntityManagerFactory)
{
return new JpaTransactionManager(primaryEntityManagerFactory);
}
}
Here is the configuration of MongoDB:
package com.example.batch.secondary;
#EnableMongoRepositories(basePackages = "com.example.batch.secondary")
#Configuration
public class MongoDBConfig {
#Bean
public MongoClient mongo() {
ConnectionString connectionString = new ConnectionString("mongodb+srv://mongoadmin:blablabla.mongodb.net/?retryWrites=true&w=majority");
MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applyConnectionString(connectionString)
.build();
return MongoClients.create(mongoClientSettings);
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
return new MongoTemplate(mongo(), "test");
}
}
Here is the RepositoryItemReader:
package com.example.batch.stepcomponents;
#Component
public class RepositoryReader extends RepositoryItemReader<Partner> {
public RepositoryReader(#Autowired PartnerRepository partnerRepository){
setRepository(partnerRepository);
setPageSize(1);
setSort(Map.of("id", Sort.Direction.ASC));
setMethodName("findAll");
}
}
Batch Config:
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
RepositoryReader repositoryReader;
#Autowired
CustomWriter customWriter;
#Autowired
CustomProcessor customProcessor;
#Bean
public Job createJob() {
return jobBuilderFactory.get("MyJob")
.incrementer(new RunIdIncrementer())
.flow(createStep())
.end()
.build();
}
#Bean
public Step createStep() {
return stepBuilderFactory.get("MyStep")
.<Partner, Student> chunk(1)
.reader(repositoryReader)
.processor(customProcessor)
.writer(customWriter)
.build();
}
}
So I tried taking out the EntityManagerFactory and the TransactionManager, and now it works. I guess they are already initialized automatically when starting up the server..
Yes, by default, if you provide a DataSource bean, Spring Batch will use a DataSourceTransactionManager, not the JPA one as you expect. This is explained in the Javadoc of EnableBatchProcessing:
The transaction manager provided by this annotation will be of type:
* ResourcelessTransactionManager if no DataSource is provided within the context
* DataSourceTransactionManager if a DataSource is provided within the context
In order to use the JPA transaction manager, you need to configure a custom a BatchConfigurer and override getTransactionManager, something like:
#Bean
public BatchConfigurer batchConfigurer(DataSource dataSource, EntityManagerFactory entityManagerFactory) {
return new DefaultBatchConfigurer(dataSource) {
#Override
public PlatformTransactionManager getTransactionManager() {
return new JpaTransactionManager(entityManagerFactory);
}
};
}
Note this will not be required anymore starting from v5, see:
Revisit the configuration of infrastructure beans with #EnableBatchProcessing
Spring Batch 5.0.0-M6 and 4.3.7 are out!
You can also set the JPA transaction manager on your step:
#Bean
public Step createStep(JpaTransactionManager jpaTransactionManager) {
return stepBuilderFactory.get("MyStep")
.<Partner, Student> chunk(1)
.reader(repositoryReader)
.processor(customProcessor)
.writer(customWriter)
.transactionManager(jpaTransactionManager)
.build();
}
Adding 'spring-data-jpa' as a dependency will automatically configure aJpaTransactionManager if no other TransactionManager is defined
I am using Spring JPA Repository to connect to Oracle.
My Repository package is com.demo.infrastructure.repository;
Repository class is StoreRepo.java
#Repository
public interface StoreRepo extends JpaRepository<StoreAttribute, String> {
#Query("select storeAttributeName from StoreAttribute order by storeAttributeName asc")
List<String> fetchAllStoreAttributeNames();
List<StoreAttribute> findAllByOrderByStoreAttributeNameAsc();
}
Problem:
I am using JNDI config to configure data source. Currently it has only one JNDI entry. Now I want to use two user names for the same database, one with admin(read-write) access and the other with user(read-only) access. Both these users will access the same Repository and same entity.
I tried the solutions already available which uses two different repository packages for each data source. But I want the Repository "StoreRepo" to be the same.
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "entityManagerFactoryAdmin",
basePackages = { "com.demo.infrastructure.repository" }
)
public class DataSourceAdminConfig {
#Primary
#Bean(name = "dataSourceAdmin")
public DataSource dataSource() {
return new JndiDataSourceLookup().getDataSource("jdbc/myds_admin");
}
#Primary
#Bean(name = "entityManagerFactoryAdmin")
public LocalContainerEntityManagerFactoryBean
entityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSourceAdmin") DataSource dataSource
) {
return builder.dataSource(dataSource).
packages("com.demo.domain.model.entities").
persistenceUnit("read-write").
build();
}
#Primary
#Bean(name = "transactionManagerAdmin")
public PlatformTransactionManager transactionManager(
#Qualifier("entityManagerFactoryAdmin") EntityManagerFactory
entityManagerFactory
) {
return new JpaTransactionManager(entityManagerFactory);
}
}
I should have two classes like this with different package (Refer basePackages). But I dont want this solution instead want to use single repository package and the same repository class.
Solution that worked for me.
1) Created separate config classes one for admin user and app user each
2) Created seprate entity manager references one for admin user and app user each
3) Instantiated the same Repositoy class (without using #Repository annotation) through java code and using respective entity manager
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "entityManagerFactoryAdmin"
)
public class AdminUserConfig {
#Primary
#Bean(name = "dataSourceAdmin")
public DataSource dataSourceAdmin(#Value("${spring.datasource.admin-user.jndi-name}") String key) {
return new JndiDataSourceLookup().getDataSource(key);
}
#Primary
#Bean(name = "entityManagerFactoryAdmin")
public LocalContainerEntityManagerFactoryBean entityManagerFactoryAdmin(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSourceAdmin") DataSource dataSource
) {
return builder.dataSource(dataSource).
packages("com.demo.domain.model.entities").
persistenceUnit("read-write").
build();
}
#Bean(name = "entityManagerAdmin")
public EntityManager entityManagerAdmin(#Qualifier("entityManagerFactoryAdmin") EntityManagerFactory
entityManagerFactory) {
return SharedEntityManagerCreator.createSharedEntityManager(entityManagerFactory);
}
#Bean(name = "adminRepository")
public StoreRepo readWriteDimStoreRepository(#Qualifier("jpaRepositoryFactoryAdmin")
JpaRepositoryFactory repositoryFactory) {
return repositoryFactory.getRepository(StoreRepo.class);
}
}
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(
entityManagerFactoryRef = "entityManagerFactoryApp"
)
public class AppUserConfig {
#Primary
#Bean(name = "dataSourceApp")
public DataSource dataSourceApp(#Value("${spring.datasource.App-user.jndi-name}") String key) {
return new JndiDataSourceLookup().getDataSource(key);
}
#Primary
#Bean(name = "entityManagerFactoryApp")
public LocalContainerEntityManagerFactoryBean entityManagerFactoryApp(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSourceApp") DataSource dataSource
) {
return builder.dataSource(dataSource).
packages("com.demo.domain.model.entities").
persistenceUnit("read-only").
build();
}
#Bean(name = "entityManagerAdmin")
public EntityManager entityManagerApp(#Qualifier("entityManagerFactoryApp") EntityManagerFactory
entityManagerFactory) {
return SharedEntityManagerCreator.createSharedEntityManager(entityManagerFactory);
}
#Bean(name = "AppRepository")
public StoreRepo readOnlyStoreRepository(#Qualifier("jpaRepositoryFactoryApp")
JpaRepositoryFactory repositoryFactory) {
return repositoryFactory.getRepository(StoreRepo.class);
}
}
//#Repository
public interface StoreRepo extends JpaRepository<StoreAttribute, String> {
#Query("select storeAttributeName from StoreAttribute order by
storeAttributeName asc")
List<String> fetchAllStoreAttributeNames();
List<StoreAttribute> findAllByOrderByStoreAttributeNameAsc();
}
I was trying this code from github:https://github.com/kodinor/spring-data-many-dbs
It's an example of how to use multiple db's from a Spring-boot application. it worked fine but I added the Spring-boot-starter-web dependency and now I'm getting an error:
>Method requestMappingHandlerMapping in >org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration$En>ableWebMvcConfiguration required a single bean, but 2 were found:
> - productDSEmFactory: defined by method 'productDSEmFactory' in class >path resource [com/kodinor/configuration/ProductDBConfiguration.class]
> - userDSEmFactory: defined by method 'userDSEmFactory' in class path >resource [com/kodinor/configuration/UserDBConfiguration.class]
>
>
>Action:
>
>Consider marking one of the beans as #Primary, updating the consumer to >accept multiple beans, or using #Qualifier to identify the bean that should >be consumed<br>
I have two config files:
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackageClasses = UserRepository.class, entityManagerFactoryRef = "userDSEmFactory", transactionManagerRef = "userDSTransactionManager")
public class UserDBConfiguration {
#Primary
#Bean
#ConfigurationProperties("spring.datasource1")
public DataSourceProperties userDSProperties() {
return new DataSourceProperties();
}
#Primary
#Bean
public DataSource userDS(#Qualifier("userDSProperties") DataSourceProperties userDSProperties) {
return userDSProperties.initializeDataSourceBuilder().build();
}
#Bean
public LocalContainerEntityManagerFactoryBean userDSEmFactory(#Qualifier("userDS") DataSource userDS, EntityManagerFactoryBuilder builder) {
return builder.dataSource(userDS).packages(User.class).build();
}
#Primary
#Bean
public PlatformTransactionManager userDSTransactionManager(EntityManagerFactory userDSEmFactory) {
return new JpaTransactionManager(userDSEmFactory);
}
}
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackageClasses = ProductRepository.class, entityManagerFactoryRef = "productDSEmFactory", transactionManagerRef = "productDSTransactionManager")
public class ProductDBConfiguration {
#Bean
#ConfigurationProperties("spring.datasource2")
public DataSourceProperties productDSProperties() {
return new DataSourceProperties();
}
#Bean
public DataSource productDS(#Qualifier("productDSProperties") DataSourceProperties productDSProperties) {
return productDSProperties.initializeDataSourceBuilder().build();
}
#Bean
public LocalContainerEntityManagerFactoryBean productDSEmFactory(#Qualifier("productDS") DataSource productDS, EntityManagerFactoryBuilder builder) {
return builder.dataSource(productDS).packages(Product.class).build();
}
#Bean
public PlatformTransactionManager productDSTransactionManager(EntityManagerFactory productDSEmFactory) {
return new JpaTransactionManager(productDSEmFactory);
}
}
Application.properties:
spring.jpa.hibernate.ddl-auto=create
spring.jpa.generate-ddl=true
spring.datasource1.url=jdbc:mysql://localhost:3306/userdb?autoReconnect=true&useSSL=false
spring.datasource1.username=user
spring.datasource1.password=pass
spring.datasource2.url=jdbc:mysql://localhost:3306/productdb?autoReconnect=true&useSSL=false
spring.datasource2.username=user
spring.datasource2.password=pass
And a simple main app that adds some init data:
#SpringBootApplication
public class SpringDataManyDbsApplication {
public static void main(String[] args) {
SpringApplication.run(SpringDataManyDbsApplication.class, args);
}
#Autowired
private UserRepository userRepository;
#Autowired
private ProductRepository productRepository;
#PostConstruct
void init () {
User user = new User("john", "doe");
userRepository.save(user);
Product product = new Product("chair", 5);
productRepository.save(product);
}
}
I tried to add the #Primary annotation to userDSEmFactory. The error goes away but the product data isn't saved anymore. Any ideas how to save this problem? I don't have a lot of experience with Spring-boot and I've read dozens of articles but many seem to do things in a different way. Thanks so much for helping me out!
update
I've added the #Primary annotation like this:
#Primary
#Bean
public LocalContainerEntityManagerFactoryBean userDSEmFactory(#Qualifier("userDS") DataSource userDS, EntityManagerFactoryBuilder builder) {
return builder.dataSource(userDS).packages(User.class).build();
}
No there are no more errors, but only the user is being saved in the primary db and nothing is being saved to the product (second) database.
So if anybody has suggestions what could be the cause of that..
You are missing #Primary annotation on your userDSEmFactory
#Primary
#Bean
public LocalContainerEntityManagerFactoryBean userDSEmFactory(#Qualifier("userDS") DataSource userDS, EntityManagerFactoryBuilder builder) {
return builder.dataSource(userDS).packages(User.class).build();
}
And also it is goot to use once #EnableTransactionManagement at your main class.
How do I configure 2 jdbc connections using Spring4 java config classes ?
Should 2 transaction managers be configure for those two connections ?
Thx
Edit:
I'm want to use just JdbcTemplate no JPA, Spring Data.
Example configuration might looks something like the following. I also pushed a full sample to GitHub which can be found here
#Configuration
public class DataSourceConfiguration {
#Bean
public PlatformTransactionManager firstDataSourceTransactionManager() {
return new DataSourceTransactionManager(firstDataSource());
}
#Bean(destroyMethod = "shutdown")
#Primary
public DataSource firstDataSource() {
return new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2)
.generateUniqueName(true)
.build();
}
#Bean
public JdbcTemplate firstJdbcTemplate() {
return new JdbcTemplate(firstDataSource());
}
#Bean
public PlatformTransactionManager secondDataSourceTransactionManager() {
return new DataSourceTransactionManager(secondDataSource());
}
#Bean(destroyMethod = "shutdown")
public DataSource secondDataSource() {
return new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2)
.generateUniqueName(true)
.build();
}
#Bean
public JdbcTemplate secondJdbcTemplate() {
return new JdbcTemplate(secondDataSource());
}
}
I used spring boot + jdbctemplate, and in my business I have to access multi datasource, e.g.
application.properties
foo.datasource.url=jdbc:mysql://127.0.0.1/foo
foo.datasource.username=root
foo.datasource.password=12345678
bar.datasource.url=jdbc:mysql://127.0.0.1/bar
bar.datasource.username=root
bar.datasource.password=12345678
Java Config
#Bean(name = "fooDb")
#ConfigurationProperties(prefix = "foo.datasource")
public DataSource fooDataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "fooJdbcTemplate")
public JdbcTemplate fooJdbcTemplate(#Qualifier("fooDb") DataSource ds) {
return new JdbcTemplate(ds);
}
and there are some common configuration for all datasources
spring.datasource.test-while-idle=true
spring.datasource.time-between-eviction-runs-millis=30000
spring.datasource.validation-query=select 1
How could I populate these common properties to every jdbctemplate, e.g.
#Bean(name = "commonDb")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource commonDataSource() {
return DataSourceBuilder.create().build();
}
Now my way is
#Value("${spring.datasource.test-while-idle}")
private boolean testWhileIdle;
#Value("${spring.datasource.time-between-eviction-runs-millis}")
private int timeBetweenEvictionRunsMillis;
#Value("${spring.datasource.validation-query}")
private String validationQuery;
private DataSource commonProcess(DataSource build) {
org.apache.tomcat.jdbc.pool.DataSource ds = (org.apache.tomcat.jdbc.pool.DataSource) build;
ds.setTestWhileIdle(testWhileIdle);
ds.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis);
ds.setValidationQuery(validationQuery);
return build;
}
#Bean(name = "fooDb")
#ConfigurationProperties(prefix = "foo.datasource")
public DataSource fooDataSource() {
return commonProcess(DataSourceBuilder.create().build());
}