Unable to dynamically provision a GORM-capable data source in Grails 3 - spring

We are implementing a multitenant application (database per tenant) and would like to include dynamic provisioning of new tenants without restarting the server. This is Grails 3.2.9 / GORM 6.
Among other things this involves creating a dataSource at runtime, without it being configured in application.yml at the application startup.
According to the documentation (11.2.5. Adding Tenants at Runtime) there exists ConnectionSources API for adding tenants at runtime, but the ConnectionSource created this way doesn't seem to be properly registered with Spring (beans for the dataSource, session and transaction manager) and Grails complain about missing beans when we try to use the new datasource.
We expect that when we use the ConnectionSources API to create a connection source for a new database, Grails should initialise it with all the tables according to the GORM Domains in our application, execute Bootstrap.groovy, etc., just like it does for the sources statically configured in application.yml This is not happening either though.
So my question is whether the ConnectionSources API is intended for a different purpose than we are trying to use it for, or it is just not finished/tested yet.

I meant to come back to you. I did manage to figure out a solution. Now this is for schema per customer, not database per customer, but I suspect it would be easy to adapt. I first create the schema using a straight Groovy Sql object as follows:
void createAccountSchema(String tenantId) {
Sql sql = null
try {
sql = new Sql(dataSource as DataSource)
sql.withTransaction {
sql.execute("create schema ${tenantId}" as String)
}
} catch (Exception e) {
log.error("Unable to create schema for tenant $tenantId", e)
throw e
} finally {
sql?.close()
}
}
Then I run the same code as the Liquibase plugin uses, with some simple defaults, as follows:
void updateAccountSchema(String tenantId) {
def applicationContext = Holders.applicationContext
// Now try create the tables for the schema
try {
GrailsLiquibase gl = new GrailsLiquibase(applicationContext)
gl.dataSource = applicationContext.getBean("dataSource", DataSource)
gl.dropFirst = false
gl.changeLog = 'changelog-m.groovy'
gl.contexts = []
gl.labels = []
gl.defaultSchema = tenantId
gl.databaseChangeLogTableName = defaultChangelogTableName
gl.databaseChangeLogLockTableName = defaultChangelogLockTableName
gl.afterPropertiesSet() // this runs the update command
} catch (Exception e) {
log.error("Exception trying to create new account schema tables for $tenantId", e)
throw e
}
}
Finally, I tell Hibernate about the new schema as follows:
try {
hibernateDatastore.addTenantForSchema(tenantId)
} catch (Exception e) {
log.error("Exception adding tenant schema for ${tenantId}", e)
throw e
}
Anywhere you see me referring to 'hibernateDatastore' or 'dataSource' I have those injected by Grails as follows:
def hibernateDatastore
def dataSource
protected String defaultChangelogTableName = "databasechangelog"
protected String defaultChangelogLockTableName = "databasechangeloglock"
Hope this helps.

Related

Getting too many connection for role when using DataSource

I have a Rest service and when it gets it has to do some insertion and updation to almost 25 database. So when I tried like the below code, it was working in my localhost but when I deploy to my staging server I was getting FATAL: too many connections for role "user123"
List<String> databaseUrls = null;
databaseUrls.forEach( databaseUrl -> {
DataSource dataSource = DataSourceBuilder.create()
.driverClassName("org.postgresql.Driver")
.url(databaseUrl)
.username("user123")
.password("some-password")
.build();
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
});
As per my understanding DataSource need not to be closed because it is never opened.
Note:
A DataSource implementation need not be closed, because it is never
“opened”. A DataSource is not a resource, is not connected to the
database, so it is not holding networking connections nor resources on
the database server. A DataSource is simply information needed when
making a connection to the database, with the database server's
network name or address, the user name, user password, and various
options you want specified when a connection is eventually made.
Can someone tell why I am getting this issue
The problem is in DataSourceBuilder, it actually creates of the connection pools which spawns some number of connections and keeps them running:
private static final String[] DATA_SOURCE_TYPE_NAMES = new String[] {
"org.apache.tomcat.jdbc.pool.DataSource",
"com.zaxxer.hikari.HikariDataSource",
"org.apache.commons.dbcp.BasicDataSource" };
Javadoc says:
/**
* Convenience class for building a {#link DataSource} with common implementations and
* properties. If Tomcat, HikariCP or Commons DBCP are on the classpath one of them will
* be selected (in that order with Tomcat first). In the interest of a uniform interface,
* and so that there can be a fallback to an embedded database if one can be detected on
* the classpath, only a small set of common configuration properties are supported. To
* inject additional properties into the result you can downcast it, or use
* <code>#ConfigurationProperties</code>.
*/
Try to use e.g. SingleConnectionDataSource, then your problem will gone:
List<String> databaseUrls = null;
Class.forName("org.postgresql.Driver");
databaseUrls.forEach( databaseUrl -> {
SingleConnectionDataSource dataSource;
try {
dataSource = new SingleConnectionDataSource(
databaseUrl, "user123", "some-password", true /*suppressClose*/);
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
} catch (Exception e) {
log.error("Failed to run queries for {}", databaseUrl, e);
} finally {
// release resources
if (dataSource != null) {
dataSource.destroy();
}
}
});
First thing it is very bad architecture decision to have single application managing 50 database. Anyway instead of creating DataSource in for loop, you should make use of Factory Design pattern to create DataSource for each DB. You should add some connection pooling mechanism to your system . HijariCP and TomcatPool are most widely used. Analyse logs of failure thread for any further issues.

Spring Boot - Firebase cloud storage configuration and CRUD operations

I am trying to find any simple and good examples of configuration and CRUD operations of Firebase cloud storage using Spring-boot but it seems there is not any good, simple and well-explained example for it.
I would like someone help me find a good example or help me with some code of configuration and CRUD operations.
Till now :
#PostConstruct
public void initialize() {
try {
FileInputStream serviceAccount =
new FileInputStream("./serviceAccount.json");
FirebaseOptions options = new FirebaseOptions.Builder()
.setCredentials(GoogleCredentials.fromStream(serviceAccount))
.setDatabaseUrl("databaseUrl")
.setStorageBucket("bucket")
.build();
FirebaseApp.initializeApp(options);
Firestore db = FirestoreClient.getFirestore();
} catch (Exception e) {
e.printStackTrace();
}
}
I was able to do CRUD operations in firebase database until added .setStorageBucket(bucket)
Error : Constructor threw exception; nested exception is java.lang.IllegalStateException: FirebaseApp with name [DEFAULT] doesn't exist.

Can we use a second database for javers audit related tables

I will mention my problem.
I need two databases.
DB1 for my application tables
DB2 for saving only the audited tables jv_...
To solve the problem I did the following
`#Bean
public ConnectionProvider jpaConnectionProvider() {
OtherConnectionProvider other = new OtherConnectionProvider();
try {
other.setConnection(dataSource().getConnection());
} catch (SQLException e) {
e.printStackTrace();
}
return scp;
}`
OtherConnectionProvider is a implementation of org.javers.repository.sql.ConnectionProvider.
dataSource() is the normal javax.sql.Datasource.
After using this, spring ignores the database properties mentioned in application.properties and creates the schema and javers related tables in this new schema since I have the following in my application.properties.
spring.jpa.hibernate.ddl-auto=create
Setting dedicated database for Javers audit data is easy for MongoDB (see https://javers.org/documentation/spring-boot-integration/#starter-repository-configuration), but there is no out-of-the box solution for SQL. The main problem is coordinating transactions in two independent SQL databases.
See How to configure transaction management for working with 2 different db in Spring?
Thanks for the reply. I fixed it the following way. See https://www.baeldung.com/spring-data-jpa-multiple-databases. But in the url, it has mentioned about two database configuration. One of the configuration which is Primary should be picked up from application.properties. Second database configuration can be picked up from spring configuration as mentioned in the URL https://javers.org/documentation/spring-integration/#jpa-entity-manager-integration. The solution is tricky enough since the standard properties of spring.datasource are not applicable here. Moreover, addition of commit properties using javers will help. This will act as tenant information.
Following is the typical code where CustomJpaHibernateConnectionProvider is the implementation of org.javers.repository.sql.ConnectionProvider
#Bean
public ConnectionProvider jpaConnectionProvider() {
CustomJpaHibernateConnectionProvider scp = new
CustomJpaHibernateConnectionProvider();
try {
scp.setConnection(dataSource().getConnection());
} catch (SQLException e) {
e.printStackTrace();
}
return scp;
}
and the datasource would like this.
#Bean
#ConfigurationProperties(prefix="spring.javers-datasource")
public DataSource dataSource(){
return DataSourceBuilder.create().build();
}
The data sources properties are not standard Spring boot properties.
spring.datasource.jdbcUrl = jdbc:postgresql://localhost/test
spring.datasource.username = postgres
spring.datasource.driverClassName=org.postgresql.Driver

Spring export schema DDL to the database in runtime

I'm using Dynamic DataSource in my Spring Boot application.
The problem is I need to generate tables from my entities. There is a way with
spring.jpa.hibernate.ddl-auto=update
but it doesn't work for me since I need to connect to Database in run-time.
What I need to know is can I call some method to do same things as Spring does on application startup with mentioned option.
Okey after some research I found the answer. All you need is to ask sessionFactoryBuilder to generate update scripts for your database and execute than with JdbcTemplate.
LocalSessionFactoryBuilder sessionFactory = new LocalSessionFactoryBuilder(dataSource);
sessionFactory.scanPackages("su");
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
try{
List<SchemaUpdateScript> scripts = sessionFactory.generateSchemaUpdateScriptList(new PostgreSQL9Dialect(),
new DatabaseMetadata(dataSource.getConnection(), new PostgreSQL9Dialect(), sessionFactory));
log.info("Schema update scripts["+scripts.size()+"]");
for (SchemaUpdateScript script:scripts ) {
log.info(script.getScript());
jdbcTemplate.execute(script.getScript());
}
}catch (Exception e){
log.error("error updating schema",e);
}

Orientdb partitioned graph java implementation

I've got a backend Spring application and Orientdb graph database. I use Tinkerpop Frames to map orientdb vertices to java objects and OPS4J for spring transaction management. Now I want to implement there a multitenancy where several customers (tenants) uses this one application instance. This application completely works on REST principles and it is opened to several Angular applications - each per customer. So there's as many frontend Angular applications as our customers and only one backend REST Spring application. Backend recognize the tenant from a HTTP request.
Now I'm not sure about the best solution...
First solution
When I read the Orientdb documentation, I found there a way how to implement multitenancy in orientdb - http://orientdb.com/docs/2.1/Partitioned-Graphs.html. However I don't know how to use it through the Java API unless I don't want to create a new database connection for each request. Because right now the spring transaction manager takes connections from connection pool which is centrally set in Spring transaction management configuration. I didn't find any Java example to this.
Spring transaction management config:
#Configuration
#EnableTransactionManagement
public class TransactionConfig {
#Bean
#Qualifier("graphDbTx")
public OrientTransactionManager graphDbTransactionManager() {
OrientTransactionManager bean = new OrientTransactionManager();
bean.setDatabaseManager(graphDatabaseFactory());
return bean;
}
#Bean
public OrientBlueprintsGraphFactory graphDatabaseFactory() {
OrientBlueprintsGraphFactory dbf = new OrientBlueprintsGraphFactory();
dbf.setMaxPoolSize(6);
dbf.setUrl(DbConfig.DATABASE_URL);
dbf.setUsername("admin");
dbf.setPassword("admin");
return dbf;
}
#Bean
public FramedGraphFactory framedGraphFactory() {
return new FramedGraphFactory(new JavaHandlerModule());
}
}
Getting connection:
protected FramedGraph<OrientGraph> framedGraph() {
return framedGraphFactory.create(gdbf.graph());
}
Second solution
Another solution is to use the Tinkerpop
PartitionGraph
class which works on Orientdb but I didn't find any sentence about this possibility in Orientdb documentation. Just this in Tinkerpop - https://github.com/tinkerpop/blueprints/wiki/Partition-Implementation. It works but in the end it just creates a not indexed property in every orientdb vertex so I'm afraid about performance of querying here.
Does anyone have any experiences with this? Any suggestion?
Using the Java API to create a partitioned DB (if I understand what you're interested in) macro steps are:
get connection (using the pool the istance of db are reused);
modify class V and E; create new user enable to write;
when you log in the db, user1 can write Vertices, invisible to the
user2 and contrary;
//WRITE IN YOUR CONTROLLER: CREATE USER ENABLE TO WRITE ON DB ..............
Connection con = new Connection();
OrientGraph noTx = con.getConnection();
//create partition
noTx.begin();
noTx.command(new OCommandSQL("ALTER CLASS V superclass orestricted")).execute();
noTx.command(new OCommandSQL("ALTER CLASS E superclass orestricted")).execute();
noTx.commit();
//create different users
noTx.begin();
String ridRule = "";
Iterable<Vertex> rule = noTx.command(new OCommandSQL("select from ORole where name = 'writer'")).execute();
ridRule = rule.iterator().next().getId().toString();
noTx.command(new OCommandSQL("INSERT INTO ouser SET name = 'user1', status = 'ACTIVE', password = 'user1', roles = ["+ridRule+"]")).execute();
noTx.command(new OCommandSQL("INSERT INTO ouser SET name = 'user2', status = 'ACTIVE', password = 'user2', roles = ["+ridRule+"]")).execute();
noTx.commit();
//will not close the graph instance, but will keep open and available for the next requester
noTx.shutdown();
//finally To release all the instances and free all the resources
con.clodeAllConnect();
//WRITE IN YOUR CONTROLLER: LOGIN WITH USER APPROPRIATE .....................
//CODE to login with user1 or user2, CREATE VERTEX SET label = 'food', name = 'Pizza' etc....
}
//beans
public static class Connection {
private OrientGraphFactory factory = null;
public Connection() {
//recyclable pool of instances
factory = new OrientGraphFactory("remote:localhost/blog").setupPool(1, 10);
}
//return the connection
public OrientGraph getConnection() {
OrientGraph txGraph = factory.getTx();
return txGraph;
}
public void clodeAllConnect(){
factory.close();
}
}
To adapt these steps and insert them in Spring might be useful this link that is OrientDB - spring implementation. it isn't much but I hope will be of help.

Resources