Tests not working for multiple datasource with spring boot liquibase - spring-boot

I am creating unittesting for spring boot application having multiple datasources. (Configuration is heavily inspired from this answer)
#Configuration
public class DatasourceConfig {
#Primary
#Bean
#ConfigurationProperties(prefix = "datasource.primary")
public DataSource primaryDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix = "datasource.primary.liquibase")
public LiquibaseProperties primaryLiquibaseProperties() {
return new LiquibaseProperties();
}
#Bean
public SpringLiquibase primaryLiquibase() {
return springLiquibase(primaryDataSource(), primaryLiquibaseProperties());
}
#Bean
#ConfigurationProperties(prefix = "datasource.secondary")
public DataSource secondaryDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix = "datasource.secondary.liquibase")
public LiquibaseProperties secondaryLiquibaseProperties() {
return new LiquibaseProperties();
}
#Bean
public SpringLiquibase secondaryLiquibase() {
return springLiquibase(secondaryDataSource(), secondaryLiquibaseProperties());
}
private static SpringLiquibase springLiquibase(DataSource dataSource, LiquibaseProperties properties) {
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setDataSource(dataSource);
liquibase.setChangeLog(properties.getChangeLog());
liquibase.setContexts(properties.getContexts());
liquibase.setDefaultSchema(properties.getDefaultSchema());
liquibase.setDropFirst(properties.isDropFirst());
liquibase.setShouldRun(properties.isEnabled());
liquibase.setLabels(properties.getLabels());
liquibase.setChangeLogParameters(properties.getParameters());
liquibase.setRollbackFile(properties.getRollbackFile());
return liquibase;
}
...
}
and
datasource:
primary:
url: jdbc:mysql://localhost/primary
username: username
password: password
liquibase:
change-log: classpath:/db/changelog/db.primary.changelog-master.xml
secondary:
url: jdbc:mysql://localhost/secondary
username: username
password: password
liquibase:
change-log: classpath:/db/changelog/db.secondary.changelog-master.xml
The application is running properly.
The problem is unit testing is failing with this configuration. I have used h2 as embedded DB. Added h2 as test dependency and added h2 in datasource URL as well in application.yaml for testing.
application.yaml for testing
management:
endpoints.web.exposure.include: "*"
security.enabled: false
spring:
zipkin:
discoveryClientEnabled: false
sender:
type: kafka
#type: web
liquibase:
enabled: false
datasource:
url: "jdbc:h2:mem:testdb"
jdbc-url: "jdbc:h2:mem:testdb"
username: sa
password:
secondary-datasource:
url: "jdbc:h2:mem:testdb"
jdbc-url: "jdbc:h2:mem:testdb"
username: sa
password:
datasource:
primary-liquibase:
liquibase:
url: "jdbc:h2:mem:testdb"
username: sa
password:
secondary-liquibase:
liquibase:
url: "jdbc:h2:mem:testdb"
username: sa
password:
liquibase:
enable: false
Unit testing file
package com.foo.bar.car;
import com.foo.bar.car.utils.KSUIDGenerator;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.springframework.boot.jdbc.EmbeddedDatabaseConnection;
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
#SpringBootTest
#AutoConfigureMockMvc
#AutoConfigureTestDatabase
// https://stackoverflow.com/a/58786742/1534925
//#AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
//#AutoConfigureTestDatabase(connection = EmbeddedDatabaseConnection.H2)
class CarApplicationTests {
#Test
public void contextLoads() {
String h = "Hello World!";
Assertions.assertEquals(h, "Hello World!");
}
#Test
public void testKSUIDGeneration() {
Assertions.assertNotNull(KSUIDGenerator.generateKSUID());
}
}
I have created a repository to demonstrate this.
When I do gradlew clean check it gives me following error
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [liquibase.integration.spring.SpringLiquibase]: Factory method 'primaryLiquibase' threw exception; nested exception is java.lang.IllegalArgumentException: No visible constructors in class org.springframework.boot.test.autoconfigure.jdbc.TestDatabaseAutoConfiguration$EmbeddedDataSourceFactoryBean
Not sure what configuration change I am missing.

There are several adjustments to be done to make it work:
Exclude Autoconfiguration for DB and Liquibase in your CarApplication.java
#SpringBootApplication(exclude = { DataSourceAutoConfiguration.class,
LiquibaseAutoConfiguration.class })
Get rid of sping prefix in your configuration (both in app and test properties)
Add liquibase changelog location to test properties
Double check with the setup below in case it wont work after all listed above adjustments (I changed things here and there the way I usually code to make it comfortable to me):
CarApplication.java
#SpringBootApplication(exclude = { DataSourceAutoConfiguration.class,
LiquibaseAutoConfiguration.class })
public class CarApplication {
public static void main(String[] args) {
SpringApplication.run(CarApplication.class, args);
}
}
application.yaml
server:
port: 9999
datasource:
url: jdbc:postgresql://localhost:8888/foo
jdbcUrl: jdbc:postgresql://localhost:5432/foo
username: username
password: password
driverClassName: org.postgresql.Driver
liquibase:
change-log: classpath:db-changelog/primary.xml
secondary-datasource:
url: jdbc:postgresql://localhost:8888/bar
jdbcUrl: jdbc:postgresql://localhost:5432/bar
username: username
password: password
driverClassName: org.postgresql.Driver
liquibase:
change-log: classpath:db-changelog/secondary.xml
DatabaseConfig.java
#Configuration
public class DatabaseConfig {
#Bean
#Primary
#ConfigurationProperties(prefix = "datasource")
public DataSource primaryDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix = "secondary-datasource")
public DataSource secondaryDataSource() {
return DataSourceBuilder.create().build();
}
#Bean
#ConfigurationProperties(prefix = "datasource.liquibase")
public LiquibaseProperties primaryLiquibaseProperties() {
return new LiquibaseProperties();
}
#Bean("liquibase")
public SpringLiquibase primaryLiquibase() {
return springLiquibase(primaryDataSource(), primaryLiquibaseProperties());
}
#Bean
#ConfigurationProperties(prefix = "secondary-datasource.liquibase")
public LiquibaseProperties secondaryLiquibaseProperties() {
return new LiquibaseProperties();
}
#Bean
public SpringLiquibase secondaryLiquibase() {
return springLiquibase(secondaryDataSource(), secondaryLiquibaseProperties());
}
private static SpringLiquibase springLiquibase(DataSource dataSource, LiquibaseProperties properties) {
SpringLiquibase liquibase = new SpringLiquibase();
liquibase.setDataSource(dataSource);
liquibase.setChangeLog(properties.getChangeLog());
liquibase.setContexts(properties.getContexts());
liquibase.setDefaultSchema(properties.getDefaultSchema());
liquibase.setDropFirst(properties.isDropFirst());
liquibase.setShouldRun(properties.isEnabled());
liquibase.setLabels(properties.getLabels());
liquibase.setChangeLogParameters(properties.getParameters());
liquibase.setRollbackFile(properties.getRollbackFile());
return liquibase;
}
}
test application.yaml
datasource:
url: "jdbc:h2:mem:testdb"
jdbc-url: "jdbc:h2:mem:testdb"
username: sa
password:
liquibase:
change-log: classpath:db-changelog/primary.xml
secondary-datasource:
url: "jdbc:h2:mem:testdb"
jdbc-url: "jdbc:h2:mem:testdb"
username: sa
password:
liquibase:
change-log: classpath:db-changelog/secondary.xml
CarApplicationTest.java
#SpringBootTest
class CarApplicationTests {
#Test
public void contextLoads() {
String h = "Hello World!";
Assertions.assertEquals(h, "Hello World!");
}
#Test
public void testKSUIDGeneration() {
Assertions.assertNotNull(KSUIDGenerator.generateKSUID());
}
}

Related

Microservices router locator dont work with euroka urls

everyone
I just making a microservices application with gateway and eureka
(I used spring 3.0.2)
Eureka server
#SpringBootApplication
#EnableEurekaServer
public class EurekaServerApplication {
public static void main(String[] args) {
SpringApplication.run(EurekaServerApplication.class, args);
}
}
application.yml
server:
port: 8761
eureka:
client:
fetch-registry: false
register-with-eureka: false
Gateway
#SpringBootApplication
#EnableDiscoveryClient
public class GateWayApplication {
public static void main(String[] args) {
SpringApplication.run(GateWayApplication.class, args);
}
}
Cloud config
#Configuration
#Slf4j
#AllArgsConstructor
public class CloudConfig {
private final DiscoveryClient discoveryClient;
private final APIRouter APIRouter;
private URI getURI(String service) throws URISyntaxException {
// log.info("service url - {}", discoveryClient.getInstances(service).get(0).getUri());
// return discoveryClient.getInstances(service).get(0).getUri();
return new URI("lb://" + service + '/');
}
// #Bean
// #LoadBalanced
// public RestTemplate restTemplate() {
// return new RestTemplate();
// }
#Bean
#LoadBalanced
public WebClient.Builder loadBalancedWebClientBuilder() {
return WebClient.builder();
}
#Bean
public RouteLocator gatewayRoutes(RouteLocatorBuilder builder) {
var builderRoutes = builder.routes();
for (var api : APIRouter.getApis()) {
for (var path : api.getRoutes()) {
builderRoutes.route(r ->
{
try {
return r.path("/api" + "/" + api.getVersion() + "/" + path.getUrl() + "/**")
.filters(f -> f.stripPrefix(3))
.uri(getURI(path.getServiceName()));
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
});
}
}
return builderRoutes.build();
}
}
application.yml
server:
port: 5000
spring:
main:
web-application-type: reactive
allow-bean-definition-overriding: true
mvc:
pathmatch:
matching-strategy: ant_path_matcher
application:
name: gateway
datasource:
url: jdbc:postgresql://localhost:5432/jwt_security
username: postgres
password: 123456
driver-class-name: org.postgresql.Driver
jpa:
hibernate:
ddl-auto: update
show-sql: true
properties:
hibernate:
format_sql: true
database: postgresql
database-platform: org.hibernate.dialect.PostgreSQLDialect
api-router:
apis:
- version: v1
routes:
- url: kleiderkammer
service-name: KLEIDER-KAMMER
eureka:
client:
service-url:
defaultZone: http://localhost:8761/eureka
Kleiderkammer
#RestController
#RequestMapping("public/")
public class PublicController {
#GetMapping("/test")
String test() {
return "Hello from public";
}
}
#SpringBootApplication
#EnableDiscoveryClient
public class KleiderkammerApplication {
public static void main(String[] args) {
SpringApplication.run(KleiderkammerApplication.class, args);
}
}
server:
port: 8700
spring:
application:
name: kleider-kammer
datasource:
url: jdbc:postgresql://localhost:5432/kleiderkammer
username: postgres
password: 123456
driver-class-name: org.postgresql.Driver
jpa:
hibernate:
ddl-auto: update
show-sql: true
properties:
hibernate:
format_sql: true
database: postgresql
database-platform: org.hibernate.dialect.PostgreSQLDialect
eureka:
client:
service-url:
defaultZone: http://localhost:8761/eureka
instance:
prefer-ip-address: true
But when i make request
http://localhost:5000/api/v1/kleiderkammer/public/test
i just get nothing
enter image description here
And i have not an error message
I have tried to change lb to http(than i have an error)
I tried to use RestTemplate instead of WebClient.Builder(for load balancer)

micronaut multiple datasource encypted password

I created a micronaut application that has access to multiple datasources through jdbctemplate. I configured the jdbctemplates like so:
#Factory
#Requires(beans = DatasourceFactory.class)
public class JdbcTemplateFactory {
#Context
#EachBean(DataSource.class)
JdbcTemplate jdbcTemplate(DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
}
This uses io.micronaut.configuration.jdbc.tomcat.DatasourceFactory which consumes my configuration yml:
datasources:
datasource111111:
url: url
username: username
password: password
driverClassName: org.h2.Driver
datasource222222:
url: url
username: username
password: password
driverClassName: org.h2.Driver
The issue is I want to somehow decrypt the password coming from configuration. My first attempt is to "replace" the DatasourceConfiguration bean that the factory is using but no luck, it gives me an error (io.micronaut.context.exceptions.DependencyInjectionException) that doesn't make sense.
Here's my replace attempt:
#Replaces(DatasourceConfiguration.class)
#EachProperty(value = BasicJdbcConfiguration.PREFIX, primary = "default")
public class EncryptedDatasourceConfiguration extends DatasourceConfiguration {
public EncryptedDatasourceConfiguration(String name) {
super(name);
}
#Override
public String getPassword() {
return "encrypted password";
}
}
Any idea what I'm doing wrong?? Thanks!
Answer:
Had to listen to bean creation as suggested
#Singleton
public class DatasourceInitiliazer implements BeanCreatedEventListener<DatasourceConfiguration> {
#Override
public DatasourceConfiguration onCreated(BeanCreatedEvent<DatasourceConfiguration> event) {
final DatasourceConfiguration datasource = event.getBean();
datasource.setPassword("encryptedPassword");
return datasource;
}
}
You're most likely better off creating a BeanCreatedEventListener that reads the password, decrypts it, and sets it back on the configuration

Spring boot multiple data sources - Connectivity Error Handling

I'm working with multiple data sources (Oracle and SQL-Server) in spring boot rest application. In this application, I have more than 25+ end-points exist to process client requests. But when one of the databases is down like Oracle or SQL-server is not available for some reason, my application is unable to start the server.
Looked couple examples on google and stack overflow but they're different what I'm looking for...
package com.foobar;
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories
(
entityManagerFactoryRef = "entityManagerFactory",
basePackages = { "com.foobar.foo.repo" }
)
public class FooDbConfig
{
#Primary
#Bean(name = "dataSource")
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Primary
#Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean
entityManagerFactory(
EntityManagerFactoryBuilder builder,
#Qualifier("dataSource") DataSource dataSource
) {
return builder
.dataSource(dataSource)
.packages("com.foobar.foo.domain")
.persistenceUnit("foo")
.build();
}
#Primary
#Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(
#Qualifier("entityManagerFactory") EntityManagerFactory
entityManagerFactory)
{
return new JpaTransactionManager(entityManagerFactory);
}
}
same configuration for 2nd data-source but with different properties.
I'm using below example as a base code reference to implement my requirements
Example link
I'm looking for a solution if one DB server is available out of N application should start and process client requests and whenever 2nd DB server is available then it connects automatically and processes other endpoints requests
I've created a solution recently for multitenancy with datasources and using liquibase, but if not use the liquibase, just remove that works too!
Example of application.yml
spring:
dataSources:
- tenantId: db1
url: jdbc:postgresql://localhost:5432/db1
username: postgres
password: 123456
driver-class-name: org.postgresql.Driver
liquibase:
enabled: true
default-schema: public
change-log: classpath:db/master/changelog/db.changelog-master.yaml
- tenantId: db2
url: jdbc:postgresql://localhost:5432/db2
username: postgres
password: 123456
driver-class-name: org.postgresql.Driver
- tenantId: db3
url: jdbc:postgresql://localhost:5432/db3
username: postgres
password: 123456
driver-class-name: org.postgresql.Driver
DataSourceConfiguration
#Configuration
#EnableTransactionManagement
#EntityScan(basePackages = { "br.com.dijalmasilva.springbootmultitenancyliquibase" })
#EnableJpaRepositories(basePackages = { "br.com.dijalmasilva.springbootmultitenancyliquibase" })
public class DataSourceConfiguration {
#Bean(name = "dataSources")
#Primary
public Map<Object, Object> getDataSources(DataSourceProperties dataSourceProperties) {
return dataSourceProperties.getDataSources().stream().map(dataSourceProperty -> {
DataSource dataSource = DataSourceBuilder.create()
.url(dataSourceProperty.getUrl())
.username(dataSourceProperty.getUsername())
.password(dataSourceProperty.getPassword())
.driverClassName(dataSourceProperty.getDriverClassName())
.build();
return new TenantIdDataSource(dataSourceProperty.getTenantId(), dataSource);
}).collect(Collectors.toMap(TenantIdDataSource::getTenantId, TenantIdDataSource::getDataSource));
}
#Bean(name = "tenantRoutingDataSource")
#DependsOn("dataSources")
public DataSource dataSource(Map<Object, Object> dataSources) {
AbstractRoutingDataSource tenantRoutingDataSource = new TenantRoutingDataSource();
tenantRoutingDataSource.setTargetDataSources(dataSources);
tenantRoutingDataSource.setDefaultTargetDataSource(dataSources.get("db1"));
tenantRoutingDataSource.afterPropertiesSet();
return tenantRoutingDataSource;
}
#Data
#AllArgsConstructor
private class TenantIdDataSource {
private Object tenantId;
private Object dataSource;
}
}
TenantRoutingDataSource
public class TenantRoutingDataSource extends AbstractRoutingDataSource {
#Override
protected Object determineCurrentLookupKey() {
return TenantContext.getCurrentTenant();
}
}
DataSourceProperties
#Data
#Component
#ConfigurationProperties(prefix = "spring")
public class DataSourceProperties {
private List<DataSourceProperty> dataSources = new ArrayList<>();
}
DataSourceProperty
#Data
public class DataSourceProperty {
private String tenantId;
private String url;
private String username;
private String password;
private String driverClassName;
private LiquibaseProperties liquibase;
}
See the complete code, maybe help you!
Link of project: https://github.com/dijalmasilva/spring-boot-multitenancy-datasource-liquibase

Spring data - If primary datasource fails the second does not come up

I am using Spring boot and Spring data and I want to use primarily a MySQL datasource but if fails to connect go to a H2 datasource.
So far, I do the change just moving the #Primary in the configurations, but if I put the #Primary in the MySQL (main data source) and stop the MySQL server in my pc, the other bean does not come up... What do I need?
application.yml:
# Main properties
spring:
application:
name: app
jpa:
database: default
show-sql: false
hibernate:
ddl-auto: update
properties:
hibernate:
format_sql: false
current_session_context_class: org.springframework.orm.hibernate5.SpringSessionContext
# Main database: MySQL
main.datasource:
url: jdbc:mysql://localhost:3306/app?useSSL=false
driver-class-name: com.mysql.jdbc.Driver
username: sa
password: sa
# Backup database: H2
backup.datasource:
url: jdbc:h2:${project.directory}/app;DB_CLOSE_ON_EXIT=FALSE
driver-class-name: org.h2.Driver
username: sa
password: sa
Main data source
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories("org.app")
#EntityScan("org.app")
public class MainDataSourceConfig {
#Primary
#Bean(name = "mainDataSource")
#ConfigurationProperties(prefix = "main.datasource")
public DataSource mainDataSource() {
return DataSourceBuilder.create().build();
}
}
Backup data source:
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories("org.app")
#EntityScan("org.app")
public class BackupDataSourceConfig {
#Bean(name = "backupDataSource")
#ConfigurationProperties(prefix = "backup.datasource")
public DataSource backupDataSource() {
return DataSourceBuilder.create().build();
}
}
Thanks!
I figure out how to do it. Hope this can help anyone:
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories("org.app")
#EntityScan("org.app")
public class DataSourceConfig {
private static final String USERNAME = "sa";
private static final String PASSWORD = "sa";
#Bean
#Primary
public DataSource dataSource() {
DataSource dataSource;
try {
dataSource = getMainDataSource();
dataSource.getConnection().isValid(500);
} catch (Exception e) {
log.error("Main database not valid.", e);
dataSource =getBackupDataSource();
}
return dataSource;
}
private DataSource getMainDataSource() {
return DataSourceBuilder.create()
.driverClassName("com.mysql.jdbc.Driver")
.username(USERNAME)
.password(PASSWORD)
.url("jdbc:mysql://localhost:3306/app?useSSL=false")
.build();
}
private DataSource getBackupDataSource() {
return DataSourceBuilder.create()
.driverClassName("org.h2.Driver")
.username(USERNAME)
.password(PASSWORD)
.url("jdbc:h2:/app;DB_CLOSE_ON_EXIT=FALSE")
.build();
}
}
Just on bean.

Spring Hibernate Connection Leak

I'm getting below error after 5-6 requests.
org.springframework.dao.DataAccessResourceFailureException
Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Using below code working perfectly, except it exhausts the connection pool after few requests.
I'm new to Spring framework, and made up all these using online samples. I have tried a few variants and all failed. Any help would be appreciated. Thanks.
application.yml
spring:
datasource:
type: com.zaxxer.hikari.HikariDataSource
dataSourceClassName: com.mysql.jdbc.jdbc2.optional.MysqlDataSource
jdbcUrl: jdbc:mysql://localhost:3306/db_name?autoReconnect=true&useSSL=false&useUnicode=true&characterEncoding=utf8
catalog: db_name
username: myusername
password: mypassword
testOnBorrow: true
validationQuery: SELECT 1
testWhileIdle: true
timeBetweenEvictionRunsMillis: 3600000
jpa:
show_sql: true
hibernate:
dialect: org.hibernate.dialect.MySQLDialect
show_sql: false
format_sql: true
connection:
provider_class: com.zaxxer.hikari.hibernate.HikariConnectionProvider
release_mode: after_transaction
...
ApplicationConfiguration.java
#Configuration
#PropertySource("classpath:application.yml")
#EnableTransactionManagement
#EnableSwagger2
#EntityScan("com...dal.data")
public class ApplicationConfiguration extends WebMvcConfigurerAdapter {
#Configuration
#ConfigurationProperties(prefix="spring.datasource")
public class JpaConfig extends HikariConfig {}
#Autowired
private JpaConfig jpaConfig;
#Bean(destroyMethod = "close")
public DataSource dataSource() {
return new HikariDataSource(jpaConfig);
}
#Bean
public SessionFactory sessionFactory() {
LocalSessionFactoryBuilder factoryBuilder = new LocalSessionFactoryBuilder(dataSource());
factoryBuilder.addAnnotatedClasses(
com...dal.data.MyEntity.class, ...
);
return factoryBuilder.buildSessionFactory();
}
TestDaoImpl.java
#Repository
#Scope(value = "request", proxyMode = ScopedProxyMode.TARGET_CLASS)
public class TestDaoImpl implements TestDao {
private static final Logger logger = LoggerFactory.getLogger(TestDaoImpl.class);
#PersistenceContext
private EntityManager em;
#SuppressWarnings("unchecked")
#Override
public List<MyEntity> getEntities() {
return em.unwrap(Session.class)
.createCriteria(MyEntity.class, "myEntity")
.list();
}
#Override
#Transactional
public void saveTest(MyEntity test) throws OperationException {
try {
em.persist(test);
} catch (Exception e) {
logger.error("ERROR saving test", e);
throw new OperationException("PS-SERVER");
}
}
This Code is working good.
The issue was with another #Repository class in the project was doing
#Inject
private SessionFactory sessionFactory;
which was eating up the connection, even when the code will not be called in the test service. I'm still not sure how that works, but once i replace that code with
#PersistenceContext
private EntityManager em;
it worked.

Resources