Orientdb partitioned graph java implementation - spring

I've got a backend Spring application and Orientdb graph database. I use Tinkerpop Frames to map orientdb vertices to java objects and OPS4J for spring transaction management. Now I want to implement there a multitenancy where several customers (tenants) uses this one application instance. This application completely works on REST principles and it is opened to several Angular applications - each per customer. So there's as many frontend Angular applications as our customers and only one backend REST Spring application. Backend recognize the tenant from a HTTP request.
Now I'm not sure about the best solution...
First solution
When I read the Orientdb documentation, I found there a way how to implement multitenancy in orientdb - http://orientdb.com/docs/2.1/Partitioned-Graphs.html. However I don't know how to use it through the Java API unless I don't want to create a new database connection for each request. Because right now the spring transaction manager takes connections from connection pool which is centrally set in Spring transaction management configuration. I didn't find any Java example to this.
Spring transaction management config:
#Configuration
#EnableTransactionManagement
public class TransactionConfig {
#Bean
#Qualifier("graphDbTx")
public OrientTransactionManager graphDbTransactionManager() {
OrientTransactionManager bean = new OrientTransactionManager();
bean.setDatabaseManager(graphDatabaseFactory());
return bean;
}
#Bean
public OrientBlueprintsGraphFactory graphDatabaseFactory() {
OrientBlueprintsGraphFactory dbf = new OrientBlueprintsGraphFactory();
dbf.setMaxPoolSize(6);
dbf.setUrl(DbConfig.DATABASE_URL);
dbf.setUsername("admin");
dbf.setPassword("admin");
return dbf;
}
#Bean
public FramedGraphFactory framedGraphFactory() {
return new FramedGraphFactory(new JavaHandlerModule());
}
}
Getting connection:
protected FramedGraph<OrientGraph> framedGraph() {
return framedGraphFactory.create(gdbf.graph());
}
Second solution
Another solution is to use the Tinkerpop
PartitionGraph
class which works on Orientdb but I didn't find any sentence about this possibility in Orientdb documentation. Just this in Tinkerpop - https://github.com/tinkerpop/blueprints/wiki/Partition-Implementation. It works but in the end it just creates a not indexed property in every orientdb vertex so I'm afraid about performance of querying here.
Does anyone have any experiences with this? Any suggestion?

Using the Java API to create a partitioned DB (if I understand what you're interested in) macro steps are:
get connection (using the pool the istance of db are reused);
modify class V and E; create new user enable to write;
when you log in the db, user1 can write Vertices, invisible to the
user2 and contrary;
//WRITE IN YOUR CONTROLLER: CREATE USER ENABLE TO WRITE ON DB ..............
Connection con = new Connection();
OrientGraph noTx = con.getConnection();
//create partition
noTx.begin();
noTx.command(new OCommandSQL("ALTER CLASS V superclass orestricted")).execute();
noTx.command(new OCommandSQL("ALTER CLASS E superclass orestricted")).execute();
noTx.commit();
//create different users
noTx.begin();
String ridRule = "";
Iterable<Vertex> rule = noTx.command(new OCommandSQL("select from ORole where name = 'writer'")).execute();
ridRule = rule.iterator().next().getId().toString();
noTx.command(new OCommandSQL("INSERT INTO ouser SET name = 'user1', status = 'ACTIVE', password = 'user1', roles = ["+ridRule+"]")).execute();
noTx.command(new OCommandSQL("INSERT INTO ouser SET name = 'user2', status = 'ACTIVE', password = 'user2', roles = ["+ridRule+"]")).execute();
noTx.commit();
//will not close the graph instance, but will keep open and available for the next requester
noTx.shutdown();
//finally To release all the instances and free all the resources
con.clodeAllConnect();
//WRITE IN YOUR CONTROLLER: LOGIN WITH USER APPROPRIATE .....................
//CODE to login with user1 or user2, CREATE VERTEX SET label = 'food', name = 'Pizza' etc....
}
//beans
public static class Connection {
private OrientGraphFactory factory = null;
public Connection() {
//recyclable pool of instances
factory = new OrientGraphFactory("remote:localhost/blog").setupPool(1, 10);
}
//return the connection
public OrientGraph getConnection() {
OrientGraph txGraph = factory.getTx();
return txGraph;
}
public void clodeAllConnect(){
factory.close();
}
}
To adapt these steps and insert them in Spring might be useful this link that is OrientDB - spring implementation. it isn't much but I hope will be of help.

Related

Getting too many connection for role when using DataSource

I have a Rest service and when it gets it has to do some insertion and updation to almost 25 database. So when I tried like the below code, it was working in my localhost but when I deploy to my staging server I was getting FATAL: too many connections for role "user123"
List<String> databaseUrls = null;
databaseUrls.forEach( databaseUrl -> {
DataSource dataSource = DataSourceBuilder.create()
.driverClassName("org.postgresql.Driver")
.url(databaseUrl)
.username("user123")
.password("some-password")
.build();
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
});
As per my understanding DataSource need not to be closed because it is never opened.
Note:
A DataSource implementation need not be closed, because it is never
“opened”. A DataSource is not a resource, is not connected to the
database, so it is not holding networking connections nor resources on
the database server. A DataSource is simply information needed when
making a connection to the database, with the database server's
network name or address, the user name, user password, and various
options you want specified when a connection is eventually made.
Can someone tell why I am getting this issue
The problem is in DataSourceBuilder, it actually creates of the connection pools which spawns some number of connections and keeps them running:
private static final String[] DATA_SOURCE_TYPE_NAMES = new String[] {
"org.apache.tomcat.jdbc.pool.DataSource",
"com.zaxxer.hikari.HikariDataSource",
"org.apache.commons.dbcp.BasicDataSource" };
Javadoc says:
/**
* Convenience class for building a {#link DataSource} with common implementations and
* properties. If Tomcat, HikariCP or Commons DBCP are on the classpath one of them will
* be selected (in that order with Tomcat first). In the interest of a uniform interface,
* and so that there can be a fallback to an embedded database if one can be detected on
* the classpath, only a small set of common configuration properties are supported. To
* inject additional properties into the result you can downcast it, or use
* <code>#ConfigurationProperties</code>.
*/
Try to use e.g. SingleConnectionDataSource, then your problem will gone:
List<String> databaseUrls = null;
Class.forName("org.postgresql.Driver");
databaseUrls.forEach( databaseUrl -> {
SingleConnectionDataSource dataSource;
try {
dataSource = new SingleConnectionDataSource(
databaseUrl, "user123", "some-password", true /*suppressClose*/);
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
} catch (Exception e) {
log.error("Failed to run queries for {}", databaseUrl, e);
} finally {
// release resources
if (dataSource != null) {
dataSource.destroy();
}
}
});
First thing it is very bad architecture decision to have single application managing 50 database. Anyway instead of creating DataSource in for loop, you should make use of Factory Design pattern to create DataSource for each DB. You should add some connection pooling mechanism to your system . HijariCP and TomcatPool are most widely used. Analyse logs of failure thread for any further issues.

Testing synchronized method/blocks in spring

I have the following method:
#Transactional
#Override
public Ticket buyTicketFor(String user) {
Ticket ticket = null;
synchronized (MUTEX) {
List<Ticket> availableTickets = ticketRepo.findAllTickets();
Collections.shuffle(availableTickets);
ticket = availableTickets.get(0);
}
return ticket;
}
I want to create a test that allows me to test two users getting two tickets, but they must not access the DB at the same time. How can i create a test to allow me to do this. I wanted to use mockito and junit 5, but im thinking that it will only be possible using integration testing, where i spin up an actual server and DB.

Include newly added data sources into route Data Source object without restarting the application server

Implemented Spring's AbstractRoutingDatasource by dynamically determining the actual DataSource based on the current context.
Refered this article : https://www.baeldung.com/spring-abstract-routing-data-source.
Here on spring boot application start up . Created a map of contexts to datasource objects to configure our AbstractRoutingDataSource. All these client context details are fetched from a database table.
#Bean
#DependsOn("dataSource")
#Primary
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
The problem is if i add a new client details, I cannot get that in routeDataSource. Obvious reason is that these values are set on start up.
How can I achieve to add new client context and I had to re intialize the routeDataSource object.
Planning to write a service to get all the client context newly added and reset the routeDataSource object, no need to restart the server each time any changes in the client details.
A simple solution to this situation is adding #RefreshScope to the bean definition:
#Bean
#Primary
#RefreshScope
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
Add Spring Boot Actuator as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Then trigger the refresh endpoint POST to /actuator/refresh to update the DataSource (actually every refresh scoped bean).
So this will depend on how much you know about the datasources to be added, but you could set this up as a multi-tenant project. Another example of creating new datasources:
#Autowired private Map <String, Datasource> mars2DataSources;
public void addDataSourceAtRuntime() {
DataSourceBuilder dataSourcebuilder = DataSourcebuilder.create(
MultiTenantJPAConfiguration.class.getclassloader())
.driverclassName("org.postgresql.Driver")
.username("postgres")
.password("postgres")
.url("Jdbc: postgresql://localhost:5412/somedb");
mars2DataSources("tenantX", datasourcebuilder.build())
}
Given that you are using Oracle, you could also use its database change notification features.
Think of it as a listener in the JDBC driver that gets notified whenever something changes in your database table. So upon receiving a change, you could reinitialize/add datasources.
You can find a tutorial of how to do this here: https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820
Though, depending on your organization database notifications need some extra firewall settings for the communication to work.
Advantage: You do not need to manually call the REST Endpoint if something changes, (though Marcos Barberios answer is perfectly valid!)

Spring configuration / properties different per request

I need to figure out if the following scenario is possible in Spring.
If we have different services / databases per region, can Spring facilitate directing calls to those services / databases per request from a single deployment? To give an example, all requests from user X will be directed to services / databases in the EAST region while all requests from user Y will be directed to services / databases in the WEST region.
Obviously connections to each database will use connection pooling, so the configuration will need to differ, not just properties. When other services are initialized, there is authentication done, so it's not just about databases connections.
This being Spring, I'd like to avoid having to pass implementations around. Can I direct Spring to use a specific configuration per request? Is there a better way to accomplish this?
-- Edit --
Technically it can be done like this, though this isn't exactly easily maintainable.
#Configuration
#PropertySource("classpath:region1.properties")
public class TestIndependentConfigurationRegion1Configuration {
#Bean
public String sampleServiceUrl(#Value("${sample.service.url}") String value) {
return value;
}
#Bean
public TestIndependentConfigurationSampleService testSampleService() {
return new TestIndependentConfigurationSampleService();
}
}
#Configuration
#PropertySource("classpath:region2.properties")
public class TestIndependentConfigurationRegion2Configuration {
#Bean
public String sampleServiceUrl(#Value("${sample.service.url}") String value) {
return value;
}
#Bean
public TestIndependentConfigurationSampleService testSampleService() {
return new TestIndependentConfigurationSampleService();
}
}
#Controller
public class TestIndependentConfigurationController {
protected ApplicationContext testRegion1ApplicationContext = new AnnotationConfigApplicationContext(TestIndependentConfigurationRegion1Configuration.class);
protected ApplicationContext testRegion2ApplicationContext = new AnnotationConfigApplicationContext(TestIndependentConfigurationRegion2Configuration.class);
#RequestMapping("/sample/service")
#ResponseBody
public String testSampleService() {
TestIndependentConfigurationSampleService testSampleService = null;
if(/* region 1 */) {
testSampleService = (TestIndependentConfigurationSampleService) testRegion1ApplicationContext.getBean("testSampleService");
}
if(/* region 2 */) {
testSampleService = (TestIndependentConfigurationSampleService) testRegion2ApplicationContext.getBean("testSampleService");
}
testSampleService.executeSampleService();
return "SUCCESS";
}
}
I don't think you can do that with properties. BUT, you should look at (netflix) ribbon client that is integrated with spring. Some of the ribbon's features allow you to load balance request's between regions. You could customize the ribbon client to do what you want.
Some readings here :
https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-ribbon.html

Spring Boot JPA and HikariCP maintaining active connections

Brief:
Is there a way to ensure that a connection to the database is returned to the pool?
Not-brief:
Data flow:
I have some long running tasks that could be sent to the server in large volume bursts.
Each of the requests is recorded in the DB that the submission was started. Then send that request off for processing.
If failure or success the request is recorded after the task is completed.
The issue is that after the submission is recorded all the way through the long running task, the connection pool uses an "active" connection. This could potential use up any size pool I have if the burst was large enough.
I am using spring boot with the following structure:
Controller - responds at "/" and has the "service" autowired.
Service - Contains all the JPA repositories and #Transactional methods to interact with the database.
When every the first service method call is made from the controller it opens an active connection and doesn't release it until the controller method returns.
So, Is there a way to return the connection to the pool after each service method?
Here is the service class in total:
#Service
#Slf4j
class SubmissionService {
#Autowired
CompanyRepository companyRepository;
#Autowired
SubmissionRepository submissionRepository;
#Autowired
FailureRepository failureRepository;
#Autowired
DataSource dataSource
#Transactional(readOnly = true)
public Long getCompany(String apiToken){
if(!apiToken){
return null
}
return companyRepository.findByApiToken(apiToken)?.id
}
#Transactional
public void successSubmission(Long id) {
log.debug("updating submission ${id} to success")
def submissionInstance = submissionRepository.findOne(id)
submissionInstance.message = "successfully analyzed."
submissionInstance.success = true
submissionRepository.save(submissionInstance)
}
#Transactional
public long createSubmission(Map properties) {
log.debug("creating submission ${properties}")
dataSource.pool.logPoolState()
def submissionInstance = new Submission()
for (key in properties.keySet()) {
if(submissionInstance.hasProperty(key)){
submissionInstance."${key}" = properties.get(key)
}
}
submissionInstance.company = companyRepository.findOne(properties.companyId)
submissionRepository.save(submissionInstance)
return submissionInstance.id
}
#Transactional
public Long failureSubmission(Exception e, Object analysis, Long submissionId){
//Track the failures
log.debug("updating submission ${submissionId} to failure")
def submissionInstance
if (submissionId) {
submissionInstance = submissionRepository.findOne(submissionId)
submissionRepository.save(submissionInstance)
}
def failureInstance = new Failure(submission: submissionInstance, submittedJson: JsonOutput.toJson(analysis), errorMessage: e.message)
failureRepository.save(failureInstance)
return failureInstance.id
}
}
It turns out that #M.Deinum was onto the right track. Spring Boot JPA automatically turns on the "OpenEntityManagerInViewFilter" if the application property spring.jpa.open_in_view is set to true, which it is by default. I found this in the JPA Configuration Source.
After setting this to false, the database session wasn't held onto, and my problems went away.

Resources