Spring batch – using two data source in one step - spring

Currently my spring-batch-app is configured to use only one data source (properties file). When running the app spring will pick the default configuration.
spring.datasource.url=jdbc:sqlserver ...
spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.jpa.show-sql=true
spring.jpa.generate-ddl=false
spring.jpa.hibernate.ddl-auto=none
spring.jpa.hibernate.naming.physical-
strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect
I have a task to create a job which required some data from another database. Basically the step will retrieve data from one data source and write the data to another data source.
For the new data source I have created a bean:
#Bean
public DataSource melDataSource() {
DriverManagerDataSource melDataSource = new DriverManagerDataSource();
melDataSource.setDriverClassName("prestosql....");
melDataSource.setUrl("....");
melDataSource.setUsername("....");
melDataSource.setPassword("....");
return melDataSource;
}
And this is how I am calling the dataSource:
#Autowired
private DataSource dataSource;
#Autowired
private DataSource melDataSource;
When running the program I get the following error:
Error creating bean with name
'org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration': Unsatisfied
dependency expressed through field 'dataSource'; nested exception is
org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name
'melDataSource': Requested bean is currently in creation: Is there an unresolvable circular
reference?
How can I add another data source ?
Thank you

You need to move your data source bean declaration in a separate class and import that class in your batch configuration class. Something like:
class DataSourceConfiguration {
#Bean
public DataSource melDataSource() {
DriverManagerDataSource melDataSource = new DriverManagerDataSource();
melDataSource.setDriverClassName("prestosql....");
melDataSource.setUrl("....");
melDataSource.setUsername("....");
melDataSource.setPassword("....");
return melDataSource;
}
}
#Configuration
#Import(DataSourceConfiguration.class)
class BatchConfiguration {
// use datasource bean here
}

Related

#DataJpaTest loads KafkaConfiguration and fails test

#DataJpaTest
class DataJpaVerificationTest {
#Autowired
private JdbcTemplate template;
#Test
public void testTemplate() {
assertThat(template).isNotNull();
}
}
When I run this test I get the following error:
java.lang.IllegalStateException: Failed to load ApplicationContext
Caused by:
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'kafkaConfig' defined in file
[***\config\KafkaConfig.class]:
Unsatisfied dependency expressed through constructor parameter 0;
nested exception is
org.springframework.beans.factory.NoSuchBeanDefinitionException: No
qualifying bean of type
'org.springframework.boot.autoconfigure.kafka.KafkaProperties'
available: expected at least 1 bean which qualifies as autowire
candidate. Dependency annotations: {}
where KafkaConfig class is a part of my application (app read data from Kafka and saves data to DB) and looks like:
#Configuration
#RequiredArgsConstructor
public class KafkaConfig {
private final KafkaProperties kafkaProperties;
#Bean
public Properties consumerProperties() {
Properties props = new Properties();
props.putAll(this.kafkaProperties.buildConsumerProperties());
return props;
}
}
Based on information that I googled about DataJpaTest annotation:
created application context will not contain the whole context needed
for our Spring Boot application, but instead only a “slice” of it
containing the components needed to initialize any JPA-related
components like our Spring Data repository.
So the question is: why Spring tries to load to the context Kafka specific bean for DataJpaTest?

Spring batch set data source when having 3 datasources without any #Primary

My system has got 3 data sources, all exposed as beans named datasourceA, datasourceB, datasourceC. I am trying to set the data source of spring batch to datasourceB but I am getting quite some issues.
My spring batch class
#Configuration
#EnableBatchProcessing
public class JobBatchConfiguration extends DefaultBatchConfigurer {
#Override
public void setDataSource(#Qualifier("dataSourceB") DataSource dataSource) {
super.setDataSource(dataSource);
}
#Bean
public BatchDataSourceInitializer batchDatabaseInitializer(#Qualifier("dataSourceB") DataSource dataSource, ResourceLoader resourceLoader){
BatchProperties batchProperties = new BatchProperties();
batchProperties.setInitializeSchema(DataSourceInitializationMode.ALWAYS);
BatchDataSourceInitializer batchDatabaseInitializer = new BatchDataSourceInitializer(dataSource, resourceLoader, batchProperties);
return batchDatabaseInitializer;
}
}
With this setup I am getting this error upon startup
Field dataSource in org.springframework.batch.core.configuration.annotation.AbstractBatchConfiguration required a single bean, but 3 were found:
Consider marking one of the beans as #Primary, updating the consumer to accept multiple beans, or using #Qualifier to identify the bean that should be consumed
I cannot set any of my datasources to #Primary since my spring batch writer reads and writes using all 3 datasources. I am using JPA repository and spring data.
Any solution? I thought overriding setDataSource should be enough
You should place #Qualifier both on your bean definition and in place when the bean is being autowired.
// data sources config
#Bean
#Qualifier("dataSourceB")
public DataSource dataSourceB() { ... }
// batch processing config
public void setDataSource(#Qualifier("dataSourceB") DataSource dataSource) { ... }
Also, there is no need to use #Qualifier because Spring uses a bean name as a fallback qualifier. Cite from https://docs.spring.io/spring-framework/docs/current/reference/html/core.html#beans-autowired-annotation-qualifiers:
For a fallback match, the bean name is considered a default qualifier value.
Thus, you can rename your variable to match the bean name you want to autowire.
// data sources config
#Bean
public DataSource dataSourceB() { ... }
// batch processing config
public void setDataSource(DataSource dataSourceB) { ... }

SpringBoot manage connection pool errors

I have a spring-boot application with 3 webservices that access to two different databases declared in application.properties file.
spring.datasource.url = jdbc:oracle
spring.datasource.username = aa
spring.datasource.password = aa
spring.seconddatasource.url = jdbc:oracle2
spring.seconddatasource.username = aa
spring.seconddatasource.password = aa
When I run the application, if a connection fails it ends the whole application even if one of the connections works.
I need to connect to all databases, and if a database isn't working, try to reconnect, but the application can not end.
I have tried these configurations but with no success :
testOnBorrow=true
validationQuery=SELECT 1
timeBetweenEvictionRunsMillis = 60000
Also I have a DataBaseConfig.java with
#Configuration
public class DataBaseConfig {
#Bean(name = "mysqlDb")
#Primary
#ConfigurationProperties(prefix="spring.datasource")
public DataSource mysqlDataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "sqliteDb")
#ConfigurationProperties(prefix="spring.secondDatasource")
public DataSource sqliteDataSource() {
return DataSourceBuilder.create().build();
}
#Bean(name = "cli")
public JdbcTemplate slaveJdbcTemplate(#Qualifier("mysqlDb") DataSource datasource) {
return new JdbcTemplate(datasource);
}
#Bean(name = "usr")
#Primary
public JdbcTemplate masterJdbcTemplate(#Qualifier("sqliteDb") DataSource secondDatasource) {
return new JdbcTemplate(secondDatasource);
}
}
Console errors :
Unable to create initial connections of pool
HHH000342: Could not obtain connection to query metadata : ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaAutoConfiguration.class]: Invocation of init method failed; nested exception is org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]
Create the connection pools programmatically rather than letting Spring Boot autoconfigure them for you. Then you can handle any errors in building the datasource in your code. See:
Configure DataSource programmatically in Spring Boot
Alternatively create a single connection datasource at runtime rather than at boot time, and add retry logic in the event of an error (have a look at Spring Retry)

Spring Boot 1.2.5.RELEASE & Spring Security 4.0.2.RELEASE - How to load configuration file before security context initialization?

I'm facing a problem with Spring: I'm migrating from Spring Security ver. 3.2.7.RELEASE to 4.0.2.RELEASE. Everything was working fine in older version, however a problem occured when it came to loading DataSource.
Let me describe the architecture:
Application is secured with both SAML and LDAP mechanisms (SAML configuration is pretty similar to config given here: https://github.com/vdenotaris/spring-boot-security-saml-sample/blob/master/src/main/java/com/vdenotaris/spring/boot/security/saml/web/config/WebSecurityConfig.java).
They both need to connect to database in order to get some required data. We use MyBatis with Spring Mybatis to get needed data. That's, where the problem begins.
My DAO configuration class looks like this:
#Configuration
#EnableConfigurationProperties
#MapperScan(basePackages = { "pl.myapp" })
public class DaoConfiguration {
#Bean
#Primary
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#Primary
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(dataSource());
}
#Bean
#Primary
public SqlSessionFactoryBean sqlSessionFactoryBean() {
SqlSessionFactoryBean sqlSessionFactoryBean = new SqlSessionFactoryBean();
sqlSessionFactoryBean.setDataSource(dataSource());
// some stuff happens here
return sqlSessionFactoryBean;
}
#Bean
#Primary
public DataSourceTransactionManager transactionManager() {
return new DataSourceTransactionManager(dataSource());
}
#Bean
#ConfigurationProperties(prefix = "liquibase.datasource")
#ConditionalOnProperty(name="liquibase.enabled")
public DataSource liquibaseDataSource() {
DataSource liquiDataSource = DataSourceBuilder.create().build();
return liquiDataSource;
}
}
In previous version it worked like a charm, but now it has a problem loading mappers, resulting in Bean creation exception on FactoryBean type check: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'someMapper' defined in file [<filename>]: Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Property 'sqlSessionFactory' or 'sqlSessionTemplate' are required
over and over again (it's not my problem, it's a known Spring/MyBatis bug).
I did some debugging and discovered something interesting: it looks like DaoConfiguration is not treated like a configuration here! I mean: if I add
#Bean
public SqlSessionFactory sqlSessionFactory() throws Exception {
return sqlSessionFactoryBean().getObject();
}
to this config, "normal" call of #Bean annotated method should result in calling proper interceptor, here it lacks this funcionality.
My prediction is that: this config class has not been properly wrapped yet and Spring Security already needs beans produced by it.
Is there any solution to properly load this configuration before Spring Security is initialized? Or am I just wrong and missing something (maybe not so) obvious?

Spring boot - Issue with multiple DataSource

I have a ReST service that needs to fetch data from two different DBs (Oracle and MySQL) and merge this data in the response.
I have below configuration.
Config for DB 1:
#Configuration
public class DbConfig_DB1{
#Bean(name="siebelDataSource")
public EmbeddedDatabase siebelDataSource(){
return new EmbeddedDatabaseBuilder().
setType(EmbeddedDatabaseType.H2).
addScript("schema.sql").
addScript("test-data.sql").
build();
}
#Autowired
#Qualifier("siebelDataSource")
#Bean(name = "siebelJdbcTemplate")
public JdbcTemplate siebelJdbcTemplate(DataSource siebelDataSource) {
return new JdbcTemplate(siebelDataSource);
}
}
Config for DB2:
#Configuration
public class DbConfig_DB2{
#Bean(name="brmDataSource")
public EmbeddedDatabase brmDataSource(){
return new EmbeddedDatabaseBuilder().
setType(EmbeddedDatabaseType.H2).
addScript("schema-1.sql").
addScript("test-data-1.sql").
build();
}
#Autowired
#Qualifier("brmDataSource")
#Bean(name = "brmJdbcTemplate")
public JdbcTemplate brmJdbcTemplate(DataSource brmDataSource) {
return new JdbcTemplate(brmDataSource);
}
}
Data Access:
#Repository
public class SiebelDataAccess {
protected final Logger log = LoggerFactory.getLogger(getClass());
#Autowired
#Qualifier("siebelJdbcTemplate")
protected JdbcTemplate jdbc;
public String getEmpName(Integer id) {
System.out.println(jdbc.queryForObject("select count(*) from employee", Integer.class));
Object[] parameters = new Object[] { id };
String name = jdbc.queryForObject(
"select name from employee where id = ?", parameters,
String.class);
return name;
}
}
I am not able to start the app as I below error:
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private javax.sql.DataSource org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration.dataSource;
nested exception is org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type [javax.sql.DataSource] is defined: expected single matching bean but found 2: brmDataSource,siebelDataSource
The issue is with two DataSource beans in the context. How to resolve this?
You could mark one of them as #Primary so Spring Boot auto configuration for transactions manager would know which one to pick. If you need to manage transactions with both of them then I'm afraid you would have to setup transactions management explicitly.
Please refer to the Spring Boot documentation
Creating more than one data source works the same as creating the first one. You might want to mark one of them as #Primary if you are using the default auto-configuration for JDBC or JPA (then that one will be picked up by any #Autowired injections).

Resources