We plan to change our multitenant ordering system on the intranet.
All products of the product catalog are retrieved through web services. This back-end architecture can not be replaced. Today, however, we are facing performance problems that should be eliminated with the new solution.
Therefore, we plan to use one caching db per tenant and we have made first tests with RavenDB.
The product catalog is relatively static, and we mainly will read data from the cache.
Only at the intermediate storage of the shopping cart data is also written.
We plan to regenerate each database once per hour, and then replace the existing database with the new one. We hope that this simplifies the update of the caching databases with the new product catalog.
There are, however, doubts whether this is contrary to the architecture of RavenDB. (existing Indexes, References)
Is our approach at all possible?
Has anyone found a good solution in a similar situation?
Thank you for your help
MS007,
Using RavenDB as as persistent view model storage is very common.
But I don't see why you want to actually refresh the RavenDB databases on an hourly basis. It would be much cheaper to simply refresh the changed data, and you don't have to worry about what is going on in the system while you are dropping a database and creating a new one.
Related
It seems that in the traditional microservice architecture, each service gets its own database with a different understanding of the data (described here). Sometimes it is considered permissible for databases to duplicate data. For instance, the "Users" service might know essentially everything about a user, whereas the "Posts" service might just store primary keys and usernames (so that the author of a post can have their name displayed, for instance). This page talks about eventual consistency, sources of truth, and other related concepts when data is duplicated. I understand that microservice architectures sometimes include a shared database, but most places I look suggest that this is a rare strategy.
As for why each service typically gets its own database, all I've seen so far is "so that each service owns its own resources," but I'm not convinced that a) the service layer in any way "owns" the persisted resources accessed through the database to begin with, or that b) services even need to own the resources they require rather than accessing necessary subsets of the master resources through a shared database.
So what are some of the justifications that each service in a microservice architecture should get its own database?
There are a few reasons why it does make sense to use a separate database per micro-service. Some of them are:
Scaling
Splitting your domain in micro-services is fine. You can scale your particular micro-service on the deployed web-server on demand or scale out as needed. That it obviously one of the benefits when using micro-services. More importantly you can have micro-service-1 running for example on 10 servers as it demands this traffic but micro-service-2 only requires 1 web-server so you deploy it on 1 server. The good thing is that you control this and you can manage your computing resources like in order to save money as Cloud providers are not cheap.
Considering this what about the database?
If you have one database for multiple services you could not do this. You could not scale the databases individually as they would be on one server.
Data partitioning to reduce size
Automatically as you split your domain in micro-services with each containing 1 database you split the amount of data that is stored in each database. Ideally if you do this you can have smaller database servers with less computing power and/or RAM.
In general paying for multiple small servers is cheaper then one large one.
So in this case you could make use of this fact and save some resources as well.
If it happens that the already spited by domain database have large amount of data techniques like data sharding or data partitioning could be applied additional, but this is another topic.
Which db technology fits the business requirement
This is very important pro fact for having multiple databases. It would allow you to pick the database technology which fits your Business requirement best in order to get the best performance or usage of it. For example some specific micro-service might have some Read-heavy operations with very complex filter options and a full text search requirement. Using Elastic Search in this case would be a good choice. Some other micro-service might use SQL Server as it requires SQL specific features like transnational behavior or similar. If for some reason you have one database for all services you would be stuck with the particular database technology which might not be so performant for those requirement. It is a compromise for sure.
Developer discipline
If for some reason you would have a couple micro-services which would share their database you would need to deal with the human factor. The developers would need to be disciplined to not cross domains and access/modify the other micro-services database(tables, collections and etc) which would be hard to achieve and control. In large organisations with a lot of developers this could be a serious problem. With a hard/physical split this is not an issue.
Summary
There are some arguments for having database per micro-service but also some against it. In general the guidelines and suggestions when using micro-services are to have the micro-service together with its data autonomous in order to work independent in Ideal case(this is not the case always). It is defiantly a compromise as well as using micro-services in general. As always the rule is the rule but there are exceptions to it. Micro-services architecture is flexible and very dependent of your Domain needs and requirements. If you and your team identify that it makes sense to merge multiple micro-service databases to 1 and that it solves a lot of your problems then go for it.
Microservices
Microservices advocate design constraints where each service is developed, deployed and scaled independently. This philosophy is only possible if you have database per service. How can i continue my business if i have DB failure and what steps i can take to mitigate this?DB is essential part of any enterprise application. I agree there are different number of challenges when services has its own databases.
Why Independent database?
Unlike other approaches this approach not only keeps your code-base clean and extendable but you truly omit the single point of failure in your business. To achieve this services sometimes can have duplicated data as well, as long as my service is autonomous and services can only be autonomous if i have database per service.
From business point of view, Lets take eCommerce application. you have microserivces like Booking, Order, Payment, Recommendation , search and so on. Database is shared. What happens if the DB is down ? All your services are down ! and there is no point using Microservies architecture other than you have clean code base.
If you have each service having it's own database , i don't mind if my recommendation service is not working but i can still search and book the order and i haven't lost the customer. that's the whole point.
It comes at cost and challenges, but in longer run it pays off.
SQL / NoSQL
Each service has it's own needs. To get the best performance I can use SQL for payment service (transaction) and I can use (I should) NoSQL for recommendation service. Shared database wouldn't help me in this case. In modern cloud Architectures like CQRS, Event Sourcing, Materialized views, we sometimes use 2 different databases for same service to get the performance out of it.
Again Database per service is not only about resources or how much data should it own. But we really have to see the bigger picture. Yes we have certain practices how much data and duplication is good or bad but that's another debate.
Hope that helps !
I'm trying to work out of I can take advantage of a caching layer in my web application or not (and if so which technology).
Our web app has and internal and external component and I would like if possible to add an in-memory cache tier between the Web App and DB Tier for the public external component. We are suffering DB performance issues and I want to alleviate stress on the DB as much as possible (plus make our public facing site of the component lightening fast).
The external component offers a location search facility based on a post code. E.g enter post code for an area and you get 50 results back each time (the data is relatively stale) the DB might change (new record added 1 per day) so I was thinking if a cache tier was possible then I could invalidate the cache nightly and then load it again (as opposed to the cache aside pattern).
Question:
Based on my overview above e.g. postcode mapping to multiple records (JSON or serializable objects) can I use a cache tier to store the data in-memory (total size of data ~100 MG, heaps of RAM free) and retrieve multiple records back per post code based on a caching technology "key-value data stores"?
If number 1 above is feasible, what caching technology, we are using a PHP front end, Zend server has an im-memory cache but it doesn't look mature, I would prefer Redis over Memcached for caching, thoughts?
If pre-loading the cache nightly is not achievable, thoughts on a better approach to utilise the cache?
If in-memory caching is not achievable at all (based on my requirement) then should I look at opmtiising the DB (it's SQL Server), e.g. loading the search table into SQL cache on SQL Server start-up?
Other, something I'm missing?
Thanks in advance, all comments welcome!
Cheers,
I have Oracle as my main RDBMS for read and write, but I want to use couchbase as caching layer as it has map-reduce as can be used as memcache. Any idea as to how i can implement that, and how to transfer and update data in the caching layer, when Oracle is updated or inserted etc.
You are not telling anything about your current performance issues.
I have seen too many applications which do not really take advantage of RDBMS/SQL features, especially if an ORM sits in between.
The cure is to put another cache on top of a database, and to synchronize this in a cluster manually using IP multicasts (SwarmCache for example), message queues (JMS) or nightly import jobs. It could create more problems in the end. And it increases system complexity.
So my answer to your question is: I would not do it, as long as there is room for improvement regarding your data model and/or queries.
I believe your question is about Database synchronization. This can be done through a combination of using DB dependencies and "right-thru" features that I am not too sure about whether couchbase offers. So with DB dependency you have cached items dependent upon Db items and if the DB items are updated or deleted the corresponding dependent item in the cache is removed and at the same time you can write a "right-thru" handler executed at the server level; and the main purpose of this handler is loading fresh copies of the removed items in the cache. So, basically, you'll write the handler once and registerit with the cache server and the cache server will execute it when needed to sync. new items in the DB with the cache. This reading on Db synchronization can be useful . Its based on a product Ncache.
So your question is not directly related to Couchbase, but as other stated more about how you can be alerted when data are changing into your Oracle instance.
One thing that is not well known is the Oracle Database Change Notification feature that is quite cool for this:
http://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm
So you can create an application that is listening to your changes and pushes the data into Couchbase.
I have a website developed with ASP.NET MVC, Entity Framework Code First and SQL Server.
The website has entities that each have a history of statuses that we defined (NEW, PACKED, SHIPPED etc.)
The DB contains a table in which a completely separate system inserts parcel tracking data.
I have to read this data tracking data and, following certain business rules, add to the existing status history of my entities.
The best way I can think of is to write an independent Windows service to poll the tracking data every so often and update my entity statuses from that. However, that makes me concerned about DB concurrency issues.
Please could someone advise me on the best strategy for this scenario?
Many thanks
There are different ways to do it. It also depends on the response time you need. If you need to update your system as soon as the tracking system updates the record then a trigger is the preferred way. Alternative way is to schedule a job which will run every 15/30mins and sync the 2 systems.
As for the concurrency issue you can use a concurrency token field. Entity framework has support for this.
We are designing our new product, which will include multi-tenancy. It will be written in ASP.NET and C#, and may be hosted on Windows Azure or some other Cloud hosting solution.
We’ve been looking at MVC and other technologies and, to be honest, we’re getting bogged down in various acronyms (MVC, EF, WCF etc. etc.).
A particular requirement of our application is causing a headache – the users will be able to add fields to the database, or even create a whole new module.
As a result, each tenant would have a database with a different structure to every other tenant using the system. We envisage that every tenant will have their own database, rather than sharing a database.
(Adding fields etc. to the system will be accomplished using a web interface).
All well and good, but the problem comes when creating a data model for MVC. Modifying a data model programmatically to add a field to a table seems to be impossible, according to this link:
Create EDM during runtime?
This is a major headache for us. Even if we don’t use MVC, I think we’d still want to create a data model (perhaps for used with LINQ to SQL).
We’re considering having a table with loads of fields in it, and instead of adding fields to the database we allocate an existing field in the table when the user wants to add a field to his form. Not sure I like that idea, though.
Of course, we don’t have to use MVC or Entity Framework, but it appears to me that these are the kind of technologies that Microsoft would steer us towards for future development.
Any thoughts? I’m assuming that we’re not the first people in the world to consider this idea of a user-customisable application.
I'd make sure that you have fully explored the option of creating 'Name-Value Pair' type tables as described here http://msdn.microsoft.com/en-us/library/aa479086.aspx#mlttntda_nvp
before you start looking at a customizable schema. Also don't forget that you are going to have to grant much higher permissions to your sql accounts in order for them to create tables on the fly.
A customizable schema means that your sql accounts will also need much higher permissions. It wouldnt be advisable to assign these higher permissions to a tenants account, but to a separate provisioning account which can perform these tasks.
Also before investing effort into EF - try googling 'EF Vote of No Confidence'. It was raised (i believe) mainly in reaction to earlier versions but its definately worth reading up on. nHibernate is an alternative worth investigating.
Just off the top of my head it sounds like a bad idea to allow users to change the database schema. I think you are missing a layer of abstraction. In my mind, it would be more correct to use the database to hold data that describes the format of a customer's data. The actual data would then be saved in a text column as xml, including version information.
This solution may not fit your needs, but I don't know the details of your project. So just consider it my 5 cents.
Most modern SQL databases today supports the 'jsonb' type for key/value storage as a field. Other types (hstor for postgres) exists too. Forget about XML, that's yesterday and no application with respect for itself implements XML unless it is for importing/converting old data.