I'm new to AWS.
scenario 1: I have database A and database B and I want to join multiple tables and transfer data from A to B realtime. could anyone please suggest what the best practice to do that?
scenario 2: I have two databases which are in different instance and want to join multiple tables and transfer data from one instance to another realtime could anyone please suggest the best practice to do that?
Related
I have a system with a microservice architecture. It has two services: Service A and Service B each with it's own database like in the following diagram.
As far as I understand having a separate database for each service is a better approach. In this design each service is the owner of its data, it's responsible for creating, updating, deleting and enforcing constraints.
In order to have Service A data in Database B I was thinking of using JDBC Kafka Connector, but I am not sure if Table1 and Table2 in Database B should enforce constraints from Database A.
If the constraint, like the foreign key from Table2 to Table1 should exist in Database B then, is there a way to have the connector know about this?
What are other common or better ways to sync data or solve this problem?
The easiest solution seems to sync per table without any constraints in Database B. That would make things easier but it could also lead to a situation where Service's A data in Service B is inconsistent. For example having entries in Table2 that point to a non-existing entry in Table1
If the constraint, like the foreign key from Table2 to Table1 should
exist in Database B then, is there a way to have the connector know
about this?
No unfortunately the "Kafka JDBC Connector" does not know about constraints.
Based on your question I assume that Table1 and Table2 are duplicated tables in Database B which exist in Database A. In Database A you have constraints which you are not sure you should add in Database B?
If that is the case then I am not sure if using "Kafka JDBC Connector" to sync data is the best choice.
You have a couple options:
Enforce the usage of Constraints like Foreign Keys in Database B but you would need to update it from your application level and not through "Kafka JDBC Connector". So for this option you can not use "Kafka JDBC Connector". You would need to write some small service/worker to read the data from that Kafka topic and populate your database tables. This way you control what is saved to the db and you can validate the constraints even before trying to save to your database. But the question here is do you really need to have the Constraints? They are important in micro-service-A but do you really need them in micro-service-B as it is just a copy of the data?
Not use constraints and allow temporary inconsistency. This is common in micro-services world. When working with Distributed systems you always have to think about the CAP Theorem. So you take into account that some data might at some point be inconsistent but you have to make sure that you will eventually bring it back to consistent state. This means you would need to develop on your application level some cleanup/healing mechanism which will recognize this data and correct it. So Db constraints do not necessary have to be enforced on data which the micro-service does not own and is considered as External data to that micro-service Domain.
Rethink your design. Usually we duplicate data in micro-service-B from micro-service-A in order to avoid coupling between the services so that the service micro-service-B can live and operate even when the micro-service-A is down or not running for some reason. We also do it to reduce load from micro-service-B to micro-service-A for every operation which needs data from Table1 and Table2. Table1 and Table2 are owned by micro-service-A and micro-service-A is the only source of truth for this data. Micro-service-B is using a duplicate of that data for its operations.
Looking at your databases design following questions might help you figuring out what would be the best option for you system:
Is it necessary to duplicate the data in micro-service-B?
If I duplicate the data do I need both tables and do I need all their columns/data in micro-service-B? Usually you just store/duplicate only a subset of the Entity/Table that you need.
Do I need the same table structure in micro-service-A as in micro-service-A? You have to decide this based on your Domain but very often you Denormalize your tables and change them in order to fit the needs of micro-service-B operations. As usually all these design decisions depend on your application Domain and use case.
Lets say we have a microservice A and a B. B has its own database. However B has to be horizontally scaled, thus we end up having 3 instances of B. What happens to the database? Does it scale accordingly, does it stays the same (centralized) database for the 3 B instances, does it become a distributed database, what happens?
The answer is based on the which kind of data should be shared from 3 B instances. Some occasions:
The B is just read data without write anything, the DB can use replicate methodology, and three B instance just read data from different DB instance, and DB was replicated.
The B instance can read/write data without interrupt other B instance, that mean every B instance can have designated data, and no data sharing between instances, the database was changed to three databases with same schema but totally different data;
The B instances should share the most of data, and every instance can occasion write the data back to the DB. So B instance should use one DB and some DB lock to avoid conflict between the instances.
In other some different situation, there will be many other approaches to solve the issue such as using memory DB like redis, queue service like rabbitMQ for B instance.
using one database by mutliple service instances is ok when you are using data partitioning.
As explained by Chris Richardson in pattern database per service,
Instances of the same service should share the same database
I have two oracle databases. Database A and Database B. Database B should be in sync with Database A. Data within DB- B wont be altered, it is only for view purpose. All the data change in DB- A should reflect in DB- B. After googling, I found db link and Materialized view
could help but I am not clear how to use them. Please give any idea.
I think you need to read the following:
http://docs.oracle.com/cd/B19306_01/backup.102/b14191/rcmdupdb.htm
http://docs.oracle.com/cd/E11882_01/server.112/e22490/dp_overview.htm#SUTIL100
A materialized view can be used for replication purposes but what you are referring to is duplication not replication.
If you all have a DBA in your org most definitely hand this task over to them. These are the kind of problems they eat for breakfast.
Best of luck.
I have two separate database in two separate servers. Both these database have same table . I just wanted to compare similar tables wrt to the data contained.
Also to access one database from other database do I need to create a DBLink
Have you tried to find anything in Google? There are millions of posts for this topic.
Use this documentation to learn about dbms_comparison package
I'm building a project combined of number of sites with common subject.
The sites rely on one central database that holds the common info for all of them.
In addition, each site has another database that holds its unique info (I will refer to it as unique-db in the next lines so I won't be misunderstood).
For example, the Languages table sits in the central db. That said, I suddenly noticed that I need to use the Languages table in one of my unique-db in order for the table to act as a FK so I don't have to create the same table again in the unique-db.
Do I have to create the same table again this time in the unique-db? Or is there a way to connect tables from separate databases?
In addition, we decided using linq2entity and soon we're gonna run some complex queries against the different databases. Will I have a problem with this matter?
How should I go on with that? Was it wise to split the data into a few databases?
I really appreciate all the help I can get!
One thing that might make your life easier is to create views of the central tables in each unique db. Linq to Entities will pick up views as if they were tables.