Multi-tenancy: Managing multiple datasources with Spring Data JPA - spring-boot

I need to create a service that can manage multiple datasources.
These datasources do not necessarily exist when the app when first running the app, actually an endpoint will create new databases, and I would like to be able to switch to them and create data.
For example, let's say that I have 3 databases, A, B and C, then I start the app, I use the endpoint that creates D, then I want to use D.
Is that possible?
I know how to switch to other datasources if those exist, but I can't see any solutions for now that would make my request possible.
Have you got any ideas?
Thanks

To implement multi-tenancy with Spring Boot we can use AbstractRoutingDataSource as base DataSource class for all 'tenant databases'.
It has one abstract method determineCurrentLookupKey that we have to override. It tells the AbstractRoutingDataSource which of the tenant datasource it has to provide at the moment to work with. Because it works in the multi-threading environment, the information of the chosen tenant should be stored in ThreadLocal variable.
The AbstractRoutingDataSource stores the info of the tenant datasources in its private Map<Object, Object> targetDataSources. The key of this map is a tenant identifier (for example the String type) and the value - the tenant datasource. To put our tenant datasources to this map we have to use its setter setTargetDataSources.
The AbstractRoutingDataSource will not work without 'default' datasource which we have to set with method setDefaultTargetDataSource(Object defaultTargetDataSource).
After we set the tenant datasources and the default one, we have to invoke method afterPropertiesSet() to tell the AbstractRoutingDataSource to update its state.
So our 'MultiTenantManager' class can be like this:
#Configuration
public class MultiTenantManager {
private final ThreadLocal<String> currentTenant = new ThreadLocal<>();
private final Map<Object, Object> tenantDataSources = new ConcurrentHashMap<>();
private final DataSourceProperties properties;
private AbstractRoutingDataSource multiTenantDataSource;
public MultiTenantManager(DataSourceProperties properties) {
this.properties = properties;
}
#Bean
public DataSource dataSource() {
multiTenantDataSource = new AbstractRoutingDataSource() {
#Override
protected Object determineCurrentLookupKey() {
return currentTenant.get();
}
};
multiTenantDataSource.setTargetDataSources(tenantDataSources);
multiTenantDataSource.setDefaultTargetDataSource(defaultDataSource());
multiTenantDataSource.afterPropertiesSet();
return multiTenantDataSource;
}
public void addTenant(String tenantId, String url, String username, String password) throws SQLException {
DataSource dataSource = DataSourceBuilder.create()
.driverClassName(properties.getDriverClassName())
.url(url)
.username(username)
.password(password)
.build();
// Check that new connection is 'live'. If not - throw exception
try(Connection c = dataSource.getConnection()) {
tenantDataSources.put(tenantId, dataSource);
multiTenantDataSource.afterPropertiesSet();
}
}
public void setCurrentTenant(String tenantId) {
currentTenant.set(tenantId);
}
private DriverManagerDataSource defaultDataSource() {
DriverManagerDataSource defaultDataSource = new DriverManagerDataSource();
defaultDataSource.setDriverClassName("org.h2.Driver");
defaultDataSource.setUrl("jdbc:h2:mem:default");
defaultDataSource.setUsername("default");
defaultDataSource.setPassword("default");
return defaultDataSource;
}
}
Brief explanation:
map tenantDataSources it's our local tenant datasource storage which we put to the setTargetDataSources setter;
DataSourceProperties properties is used to get Database Driver Class name of tenant database from the spring.datasource.driverClassName of the 'application.properties' (for example, org.postgresql.Driver);
method addTenant is used to add a new tenant and its datasource to our local tenant datasource storage. We can do this on the fly - thanks to the method afterPropertiesSet();
method setCurrentTenant(String tenantId) is used to 'switch' onto datasource of the given tenant. We can use this method, for example, in the REST controller when handling a request to work with database. The request should contain the 'tenantId', for example in the X-TenantId header, that we can retrieve and put to this method;
defaultDataSource() is build with in-memory H2 Database to avoid using the default database on the working SQL server.
Note: you must set spring.jpa.hibernate.ddl-auto parameter to none to disable the Hibernate make changes in the database schema. You have to create a schema of tenant databases beforehand.
A full example of this class and more you can find in my repo.
UPDATED
This branch demonstrates an example of using the dedicated database to store tenant DB properties instead of property files (see the question of #MarcoGustavo below).

Related

Merging common Hikari properties with data source specific properties?

I am creating a spring boot app which has multiple data sources (7 in total) and whilst properties like dbUrl, username and password are data source specific, a great many are common. I obviously don't want to duplicate the properties for each data source and am trying to work out how I can create a HikariConfig instance for each datasource using a blended set of common properties.
The properties have the following format
spring.oracle.datasource.driverClassName
spring.oracle.datasource.autoCommit
spring.oracle.datasource.instance[0].dbUrl
spring.oracle.datasource.instance[0].username
spring.oracle.datasource.instance[0].password
spring.oracle.datasource.instance[1].dbUrl
spring.oracle.datasource.instance[1].username
spring.oracle.datasource.instance[1].password
spring.oracle.datasource.instance[2].dbUrl
spring.oracle.datasource.instance[2].username
spring.oracle.datasource.instance[2].password
spring.oracle.datasource.instance[n].dbUrl
spring.oracle.datasource.instance[n].username
spring.oracle.datasource.instance[n].password
I did try using configuration properties with a class that had the format below (lombok annotations ommitted)
public class DataSourceProperties extends HikariConfig {
public List<Instance> instance;
public static class Instance {
public String dbUrl;
public String username;
public String password;
}
}
But although the object is populated correctly I cannot figure out how I then create the n HikariConfig instances using the properties in the pojo. Clone or BeanUtils does not work, as it copies across all of the null fields which are rejected by the HikarConfig setters.
Anyone know a possible solution without duplicating the common properties and resorting to manual creation of the HikariConfig instances?
In the end, I used Spring's BeanUtils and wrote a function which found the null fields so they could be excluded.
Spring BeanUtils API Doc

Spring AbstractRoutingDataSource is caching Datasource

I have a Spring MVC application deployed as Multi-Tenant (Single Instance) on Tomcat. All users login to the same application.
User belongs to a Region, and each Region has a separate Database instance.
We are using Dynamic DataSource Routing using Spring AbstractRoutingDataSource".
This works correctly only for the first time, when User_1 from Region_1 logs into the application, Datasource_1 is correctly assigned.
But subsequently when User_2 from Reqion_2 logs into the application, AbstractRoutingDataSource never gets called and Datasource_1 gets assigned.
It looks like Spring AbstractRoutingDataSource is caching the Datasouce.
Is there a way to change this behaviour of AbstractRoutingDataSource and get it working correctly?
You should provide more details for a better understanding.
I think the problem might be related to changing the tenant identifier. You may have a ThreadLocal storage to store the tenant identifier.
public class ThreadLocalStorage {
private static ThreadLocal<String> tenant = new ThreadLocal<>();
public static void setTenantName(String tenantName) {
tenant.set(tenantName);
}
public static String getTenantName() {
return tenant.get();
}
}
AbstractRoutingDataSource should use this to retrieve the tenantId
public class TenantAwareRoutingDataSource extends AbstractRoutingDataSource {
#Override
protected Object determineCurrentLookupKey() {
return ThreadLocalStorage.getTenantName();
}
}
And you should set the tenantId on each request for the current thread that is handling the request so that the following operations will be done on the correct database. For you may add a Spring Security filter to extract the tenant identifier from JWT token or from host subdomain.

Spring Data JPA and JDBC template

I'm working on Spring Boot Application and I use Spring Data, HikariCP and JDBC, but I have a problem.
Inside one method I get a particular User from the database using a Spring Data repository. After I get the User from the DB I use JdbcTemplate.query to get some other information from the DB with username of above User but the application freezes and after some time it throws
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30006ms.
When I debug the code I see that jdbctemplate is using hikariCP as datasource.
This is the code I'm using:
public User getUser() {
User user = userRepository.findByUsernameAndEnabledTrue("username");
List<String> roles= getUserRoles(user.getUsername())
return user;
}
private List<String> getUserRoles(String username) {
List<String> roles = this.jdbcTemplate.query("SELECT ga.authority FROM group_authorities ga INNER JOIN group_members gm ON gm.group_id = ga.group_id INNER JOIN users u ON gm.username=u.username WHERE u.username=?;",
new Object[]{username},new ResultSetExtractor<List<String>>() {
#Override
public List<String> extractData(ResultSet resultSet) throws SQLException, DataAccessException {
List<String> roles = new ArrayList<>();
while (resultSet.next()) {
roles.add(resultSet.getString("authority"));
}
return roles;
}
});
return roles;
}
I made a research how to use those together and share same transaction or something like but unfortunately can't fix it.
Your problem seems to be that the JdbcTemplate uses a different connection than your repository. And since the connection pool makes only one connection available and that is already used by the repository, you run into the timeout.
Increasing the capacity of the connection pool would fix that immediate problem, but the repository and the JdbcTemplate would use different connections and therefore transactions, which you probably don't want.
You don't show where your JdbcTemplate gets it's connection from, but that is probably where things go wrong. To fix it get the EntityManager injected. Then get the Connection from it. How to do that is JPA implementation dependent. Here are the versions for Eclipse Link and for Hibernate. Then use that Connection to create your JdbcTemplate.
It is possible just inject DataSource or JdbcTemplate in a custom repository, for example. And if JPA and JDBC calls are inside one transaction (generated by #Transactional, for example), Spring is smart enough to use JPATransactionManager for both cases with the same transaction and connection.
https://billykorando.com/2019/05/06/jpa-or-sql-in-a-spring-boot-application-why-not-both/

Spring Boot equivalent to XML multi-database configuration

I would like to port two projects to Spring Boot 1.1.6. The are each part of a larger project. They both need to make SQL connections to 1 of 7 production databases per web request based region. One of them persists configuration setting to a Mongo database. They are both functional at the moment but the SQL configuration is XML based and the Mongo is application.properties based. I'd like to move to either xml or annotation before release to simplify maintenance.
This is my first try at this forum, I may need some guidance in that arena as well. I put the multi-database tag on there. Most of those deal with two connections open at a time. Only one here and only the URL changes. Schema and the rest are the same.
In XML Fashion ...
#Controller
public class CommonController {
private CommonService CommonService_i;
#RequestMapping(value = "/rest/Practice/{enterprise_id}", method = RequestMethod.GET)
public #ResponseBody List<Map<String, Object>> getPracticeList(#PathVariable("enterprise_id") String enterprise_id){
CommonService_i = new CommonService(enterprise_id);
return CommonService_i.getPracticeList();
}
#Service
public class CommonService {
private ApplicationContext ctx = null;
private JdbcTemplate template = null;
private DataSource datasource = null;
private SimpleJdbcCall jdbcCall = null;
public CommonService(String enterprise_id) {
ctx = new ClassPathXmlApplicationContext("database-beans.xml");
datasource = ctx.getBean(enterprise_id, DataSource.class);
template = new JdbcTemplate(datasource);
}
Each time a request is made, a new instance of the required service is created with the appropriate database connection.
In the spring boot world, I've come across one article that extended TomcatDataSourceConfiguration.
http://xantorohara.blogspot.com/2013/11/spring-boot-jdbc-with-multiple.html That at least allowed me to create a java configuration class however, I cannot come up with a way to change the prefix for the ConfigurationProperties per request like I am doing with the XML above. I can set up multiple configuration classes but the #Qualifier("00002") in the DAO has to be a static value. //The value for annotation attribute Qualifier.value must be a constant expression
#Configuration
#ConfigurationProperties(prefix = "Region1")
public class DbConfigR1 extends TomcatDataSourceConfiguration {
#Bean(name = "dsRegion1")
public DataSource dataSource() {
return super.dataSource();
}
#Bean(name = "00001")
public JdbcTemplate jdbcTemplate(DataSource dsRegion1) {
return new JdbcTemplate(dsRegion1);
}
}
On the Mongo side, I am able to define variables in the configurationProperties class and, if there is a matching entry in the appropriate application.properties file, it overwrites it with the value in the file. If not, it uses the value in the code. That does not work for the JDBC side. If you define a variable in your config classes, that value is what is used. (yeah.. I know it says mondoUrl)
#ConfigurationProperties(prefix = "spring.mongo")
public class MongoConnectionProperties {
private String mondoURL = "localhost";
public String getMondoURL() {
return mondoURL;
}
public void setMondoURL(String mondoURL) {
this.mondoURL = mondoURL;
}
There was a question anwsered today that got me a little closer. Spring Boot application.properties value not populating The answer showed me how to at least get #Value to function. With that, I can set up a dbConfigProperties class that grabs the #Value. The only issue is that the value grabbed by #Value is only available in when the program first starts. I'm not certain how to use that other than seeing it in the console log when the program starts. What I do know now is that, at some point, in the #Autowired of the dbConfigProperties class, it does return the appropriate value. By the time I want to use it though, it is returning ${spring.datasource.url} instead of the value.
Ok... someone please tell me that #Value is not my only choice. I put the following code in my controller. I'm able to reliably retrieve one value, Yay. I suppose I could hard code each possible property name from my properties file in an argument for this function and populate a class. I'm clearly doing something wrong.
private String url;
//private String propname = "${spring.datasource.url}"; //can't use this
#Value("${spring.datasource.url}")
public void setUrl( String val) {
this.url = val;
System.out.println("==== value ==== " + url);
}
This was awesome... finally some progress. I believe I am giving up on changing ConfigurationProperties and using #Value for that matter. With this guy's answer, I can access the beans created at startup. Y'all were probably wondering why I didn't in the first place... still learning. I'm bumping him up. That saved my bacon. https://stackoverflow.com/a/24595685/4028704
The plan now is to create a JdbcTemplate producing bean for each of the regions like this:
#Configuration
#ConfigurationProperties(prefix = "Region1")
public class DbConfigR1 extends TomcatDataSourceConfiguration {
#Bean(name = "dsRegion1")
public DataSource dataSource() {
return super.dataSource();
}
#Bean(name = "00001")
public JdbcTemplate jdbcTemplate(DataSource dsRegion1) {
return new JdbcTemplate(dsRegion1);
}
}
When I call my service, I'll use something like this:
public AccessBeans(ServletRequest request, String enterprise_id) {
ctx = RequestContextUtils.getWebApplicationContext(request);
template = ctx.getBean(enterprise_id, JdbcTemplate.class);
}
Still open to better ways or insight into foreseeable issues, etc but this way seems to be about equivalent to my current XML based ways. Thoughts?

Set default ResultSet Type to TYPE_SCROLL_INSENSITIVE in Spring JDBC

From what I can see Spring JDBC sets the default ResultSet type to ResultSet.TYPE_FORWARD_ONLY in PreparedStatementCreatorFactory. I'd like to change it so all my Spring DAOs get ResultSet.TYPE_SCROLL_INSENSITIVE for all queries.
Is the best approach to simply extend PreparedStatementCreatorFactory with the new default, and then have all my DAOs use it?
Thanks.
I know this question is quite old but just in case someone gets into the same issue and lands here.
The way I solved this problem is simply by extending the NamedParameterJdbcTemplate
Example
public class ScrollNamedParameterJdbcTemplate extends NamedParameterJdbcTemplate {
public ScrollNamedParameterJdbcTemplate(DataSource dataSource) {
super(dataSource);
}
#Override
protected PreparedStatementCreatorFactory getPreparedStatementCreatorFactory(
ParsedSql parsedSql, SqlParameterSource paramSource) {
String sqlToUse = NamedParameterUtils.substituteNamedParameters(parsedSql, paramSource);
List<SqlParameter> declaredParameters =
NamedParameterUtils.buildSqlParameterList(parsedSql, paramSource);
PreparedStatementCreatorFactory preparedStatementCreatorFactory =
new PreparedStatementCreatorFactory(sqlToUse, declaredParameters);
preparedStatementCreatorFactory.setResultSetType(ResultSet.TYPE_SCROLL_INSENSITIVE);
return preparedStatementCreatorFactory;
}
}
Then create a bean definition for this specific template
#Bean("scrollNamedParameterJdbcTemplate")
NamedParameterJdbcOperations scrollNamedParameterJdbcTemplate(DataSource dataSource) {
return new ScrollNamedParameterJdbcTemplate(dataSource);
}
Then you can just inject this template if you need the scroll type on your result sets

Resources