I have started learning and building micro service based project, but I always stuck into scoping situation and end up creating a sort of monolith. in my below foo-bar example, please suggest what should be the scoping and how achieve desired output.
Services or tables
Employees
Department
Employee-Department-Mapping
Assumption
Employee or Department does not have any cross reference for each other all relationships are maintained in 3rd table Employee-Department-Mapping
Relationship could be One-To-Many or Many-To-Many based on business to business, in this example it is One-To-Many (Department with Employee)
Requirement
I want to get total salaries paid department wise. similar to below query, here I am making simple joins on 3 tables. which is only possible if all are in the same database and single micro service.
Select d.DepartmentName, SUM(e.salary)
from Employee e, Department d, Employee-Department-Mapping c
where d.DepartmentName == c.DepartmentName
AND e.Employee == c.Employee
Group By c.DepatmentId
constraints
employee table has salary information
employee department relationship is maintained by 3rd table.
I am not looking for exact answer but an approach to solve such problems.
wonder how would you design your micro service if you need aggregated outputs, would you bring all these tables in the single microservice ? I do not want pull millions of records from everymicroservice and do aggregation in memory.
Microservice boundries -if you don't have any other scalability reason- should be defined by the business. And in this particular case -without knowing any other requirements- I would say you should go for one microservice which manage those 2 entities. Having said that I experienced enough to know that in the real world this solution is not always feasible and possible. Fortunately there are some patterns you can follow to fix the situation you described. For example CQRS could be a solution https://dzone.com/articles/microservices-with-cqrs-and-event-sourcing.
You have to change your thinking to microservices here. what I can propose here is
You will have to do multiple calls to different microservices and return one result you can use API Gateway pattern. You will have to write an aggregator service to aggregate the output of the two tables via two API services.
You will have to denormalize the database and have the dept name in your Employees table if you don't like the above option. That's why we use NoSQL in microservices.
Related
We are trying to enable collaborative chats in an educational application. The general idea is that the users (students) will start an exercise, this will make them join an specific waiting room. And somehow the system should be able to decide it has enough students to create a group of n students (depending on the exercise) according to a given strategy and will send a message to those students to join a chatroom with a generated ID so that they can collaborate.
Right now we are totally blank and cannot decide on how to make the server decide wether to try and create groups or to wait for more students. Our stack is in Spring Boot, Redis and Postgress. Our initial idea was to add the students into a waiting room in Redis and launch a Spring event every time a student joined the waitlist. However, we understand that approach might generate many race conditions, which should be avoided.
Create an exercise_students table which has a SERIAL column on it, call it arrival_order or something. Another column for group_id. As students sign in for the exercise, insert them into this table. By nature SERIAL is atomically auto-incremented, so you avoid race conditions. Regularly query the table for students with no group_id (I assume you have a exercise_group table of some sort that defines how many students are part of a group). When the count reaches n, update them with the group_id and create a new group in the exercise_group table for the next group.
Relational databases are pretty good at this sort of thing. Atomic updating of state is pretty straightforward stuff.
I want to use the many to one and other DB Relationship in micro-service architecture. In monolithic architecture we can create the entity relationship easily as they belongs to same project but in micro-service architecture how we can achieve the same.
Example:
There is one userDeatil service and other is productDetail service.Now there is third service called orderDetail and an order will have userID and ProductIDs associated with it. So how can we manage the relationship between 'user and order' and 'order and product'.
I have searched over net but didn't able to get the fair idea.There is another thread having same query but not having the clear answer. Link
In my opinion your case is about how you specify your services especially how you define the bounded context of each service !!
According to the situation above-mentioned I don't see any reason why the product service should know anythings about orders (even if it s just the order-id) and backwards. One more problem I see in this case: your services will not work if one service is not available (E.g when the product service it not online, your order service will not be able to work, because he needs the product details from the product service...).
I Think you should rethink the bounded contexts of your microservices. You should keep in mind:
ensure a loose coupling of the microservices
a microservice has always to work even other Microservices are not available (Resilience / Reliability).
The DDD (domain-driven-design) paradigm with its tools provides her a great help to assist you, during the definition process of your services, you encourage these qualities.
So, the following is JUST an idea (it s not a recommendation and you should review whether it matters for your business case) :
It seems like the "order" process is your core business domain. So you have to focus on it.
The user service (i hope you mean here the customer and not a user in terms of authentication/authorization) has only the responsibility to manage the customers, and may be their adresses, bank-Accountings etc.. It should not know anything about your orders or products.
The same is valid for the product service. It owns only data about products. It has no relation either to the customer nor to the order-service.
The order service by itself deals only with orders and should own only data that belong to an order (like ship Adress and some information about the product-items ordered). I think the customer-Id is also important here to keep the relation between the order and the customer. This way you can e.g get all orders made by a certain customer-id....
We are planning to migrate our product to micro service architecture using spring boot.
I have few doubts/concerns for "each service should have separate database".
If I created separate database for each service, then how to handle the table dependency.
For example: I have 2 micro services.
1. Customer Service: it has "customer" table (fields: customerId, customerName, etc...)
2. Order Service: it has "order" table (fields: orderId, orderName, customerId, etc...)
Note: for simplicity I mentioned only 1 table in each service here.
When I tried to access the order details based on orderId, I may want to retrieve the details of customer also.
In this case, I have two options in my mind:
Using RestTemplate call the "customer service" GET of customerById API and fetch the required details and set it in the response of OrderDetails.
Create customer table also in Order Service and using database tool replicate the data from customer service to order service's customer table. i.e., any changes done to customer table in customer service database will reflect on order service's customer table.
So, I can easily fetch relevant data from order service itself.
Please suggest which one is preferable. If both doesn't make sense, please suggest best way to handle this.
Thanks.
I have two scenarios that I want to support but I don’t know the best way to design relations in the elasticsearch. I read the entire elasticsearch documentation but I couldn’t find the best way to design types for my scenarios.
Multiple one to many.
Let’s assume that I have the following tables in my relational database that I want to transfer to the elasticsearch:
Transaction table Id User1Id User2Id ….
User table Id Name
Transaction contains two references to User. As far as I know I cannot use the parent->child relation specifying two parents? I need to store transaction and user in separate types because they can be changed separately. I need to be able to search transaction through user details and return users connected with transactions. Any idea how to design such structure in the elastic search?
Many to many
Let’s assume that we have the following tables:
Order Id …
OrderLine OrderId UserId Amount …
User Id Name
Order line is always saved with the order so I thought that I can store order with order lines as a nested object relation but the user must be in the separate table. Is there any way how can I connected multiple users from order line with user type? I assume that I can use application side join but I need to retrieve order and order line always together and be able to search order by user data.
I can use grandparent and grandchildren relations but then I need to make joins in the application. Any idea how to design it in the best way?
We use CRM 4.0 at our institution and have no plans to upgrade presently as we've spend the last year and a half customising and extending the CRM to work with our processes.
A tiny part of model is a simply hierarchy, we have a group of learning rooms that has a one-to-many relationship with another entity that describes the courses available for that learning room.
Another entity has a list of all potential and enrolled students who have expressed an interest in whichever course.
That bit's all straightforward and works pretty well and is modelled into 3 custom entities.
Now, we've got an Admin application that reads the rooms and then wants to show the courses for that room, but only where there are enrolled students.
In SQL this is simplified to:
SELECT DISTINCT r.CourseName, r.OtherInformation
FROM Rooms r
INNER JOIN Students S
ON S.CourseId = r.CourseId
WHERE r.RoomId = #RoomId
And this indeed is very close to the eventual SQL that CRM generates.
We use a Crm QueryEntity, a Filter and a LinkEntity to represent this same structure.
The problem now is that the CRM normalizes the a customize entity into a Base Table which has the standard CRM entity data that all share, and then an ExtensionBase Table which has our customisations. To Give a flattened access to this, it creates a view that merges both tables.
This view is what is used by the Generated SQL.
Now the base tables have indices but the view doesn't.
The problem we have is that all we want to do is return Courses where the inner join is satisfied, it's enough to prove there are entries and CRM makes it SELECT DISTINCT, so we only get one item back for Room.
At first this worked perfectly well, but now we have thousands of queries, it takes well over 30 seconds and of course causes a timeout in anything but SMS.
I'm given to believe that we can create and alter indices on tables in CRM and that's not considered to be an unsupported modification; but what about Views ?
I know that if we alter an entity then its views are recreated, which would of course make us redo our indices when this happens.
Is there any way to hint to CRM4.0 that we want a specific index in place ?
Another source recommends that where you get problems like this, then it's best to bring data closer together, but this isn't something I'd feel comfortable in trying to engineer into our solution.
I had considered putting a new entity in that only has RoomId, CourseId and Enrolment Count in to it, but that smacks of being incredibly hacky too; After all, an index would resolve the need to duplicate this data and have some kind of trigger that updates the data after every student operation.
Lastly, whilst I know we're stuck on CRM4 at the moment, is this the kind of thing that we could expect to have resolved in CRM2011 ? It would certainly add more weight to the upgrading this 5 year old product argument.
Since views are "dynamic" (conceptually, their contents are generated on-the-fly from the base tables every time they are used), they typically can't be indexed. However, SQL Server does support something called an "indexed view". You need to create a unique clustered index on the view, and the query analyzer should be able to use it to speed up your join.
Someone asked a similar question here and I see no conclusive answer. The cited concerns from Microsoft are Referential Integrity (a non-issue here) and Upgrade complications. You mention the unsupported option of adding the view and managing it over upgrades and entity changes. That is an option, as unsupported and hackish as it is, it should work.
FetchXml does have aggregation but the query execution plans still uses the views: here is the SQL generated from a simple select count from incident:
'select
top 5000 COUNT(*) as "rowcount"
, MAX("__AggLimitExceededFlag__") as "__AggregateLimitExceeded__" from (select top 50001 case when ROW_NUMBER() over(order by (SELECT 1)) > 50000 then 1 else 0 end as "__AggLimitExceededFlag__" from Incident as "incident0" ...
I dont see a supported solution for your problem.
If you are building an outside admin app and you are hosting CRM 4 on-premise you could go directly to the database for your query bypassing the CRM API. Not supported but would allow you to solve the problem.
I'm going to add this as a potential answer although I don't believe its a sustainable or indeed valid long-term solution.
After analysing the indexes that CRM had defined automatically, I realised that selecting more information in my query would be enough to fulfil the column requirements of an Index and now the query runs in less then a second.