Upgrading skills from Azure Service bus to Rabbit MQ + Mass Transit. Please bear with me.
I have convoluted understanding in mind about Saga and State Machine (Automatonymous). Are they synonym of each other.
Does the name SAGA originates from integrated state machine functionalities
in service bus? Can we say Saga is superset of service bus?
As Azure does not have integrated state machine in their service bus.
In my opinion, Sagas are use for the transactions, for example among microservices, to maintain consistency between them.
You have applied the Database per Service pattern. Each service has
its own database. Some business transactions, however, span multiple
service so you need a mechanism to ensure data consistency across
services. For example, lets imagine that you are building an
e-commerce store where customers have a credit limit. The application
must ensure that a new order will not exceed the customer’s credit
limit. Since Orders and Customers are in different databases the
application cannot simply use a local ACID transaction.
Reference: http://microservices.io/patterns/data/saga.html
Related
I started reading microservices architecture and got confused with one below point.
Each service should have a separate database.
Does each service mean a single web(rest) api having its own database?
For example if I take motor insurance claim operation as a business scenario where I modelled business domain services in 3 part Insurance claim services , partner (automobile service providers) services and customer services.
Insurance claim settlement operation in claim api will require other information like incident , survey done by an inspector, policy detail , documents etc.
Now I can create 5 web(rest) api in Insurance claim services and will store its data in common db and other services like partner and customer service will have their own web apis and db
What is correct ?
All web api (claimAPI, PolicyAPI, IncidentAPI, SurveyAPI and DocumentAPI) in claim insurance services should have their own db or they can keep data in single data base ?
Thanks.
To follow microservice best practice, it is correct that they should each have their own database and be exposed solely by APIs. This is because every service in your architecture should be independent and de-coupled from other services. If 2+ services share a database, then there may arise problems in operation or upgrade.
One big issue with a shared database is each service would need to trust that another service doesn't modify it's information. But since they all have access to the same database, one of the others could in fact modify the underlying data and make things unstable or insecure.
Further, with 2+ services relying on a shared database, then you're forced to use the exact same database/version with all. You lose the freedom to independently use MySQL for one and MongoDB for another. Even if the same tool is used for all, when you do maintenance or migration on one you're forced to do it for the rest. All this adds up to some coupled services that make them harder to maintain and scale.
you can have common database for all microservices, it is one of the microservices patterns:
https://microservices.io/patterns/data/shared-database.html
https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-data-persistence/shared-database.html
check those links to see advantages and disadvantages of this approach.
I am new to microservices and trying to break up a big monolithic application into microservices. While scoping the microservice I am unable to decide whether I should go for a data duplication between services or ignore SRP by clubbing all requiring the same data into 1 service. Following is the scenario.
I have a service which receives Customer order say build a car with these parts and features. Now I have other 2 functionalities which uses the Parts and features to derive some runtime value say ;
If the order contains part A and Feature A then perform X operation. As each of these functionalities have there respective UI for configuration and runtime engine to derive the output and most of the time changes only comes in these respective function blocks, I thought of creating the separate microservices.
Creating the separate microservice would need data(Parts and Features) to be duplicated. Another option could be given each of these service uses the same data is clubbing all of them into 1, but with that I again create a big service which if goes down will stop all 3 functionalities and is against SRP. Another option could be when the data is required by the other 2 services make a call and get it from Order Service, but that is making it highly dependent and getting the data over network for each operation.
Can anyone suggest what would be ideal to do in such case.
Microservice should communicate via events using publisher/subscriber model and topics. See the tri-lateral design pattern. In a pub/sub system the producer writes the event to the topic and a broker makes it available to each subscriber, aka microservice. This is one advantage of eventing, one producer, and many consumers.
Order A has a BOM that expresses parts and features an order represents. When the order is placed, other services need that information such as the feature/part microservice and UI components and their respective datastores. For example, define two topics parts-ordered and features-ordered.
The order microservice writes to the parts and features topics respectively. The feature, parts, and UX microservices read and act upon those events.
You need to make a choice about your system of record. You can write to an orders DB before publishing the events, or let the pub/sub it. In your example, it sounds like you are talking about three different tables; order, parts, and events. That decomposes cleaning into the microservices you describe where each maintains the data it is responsible for. I'd be inclined to just use the pub/sub as your audit trail.
If you are writing in Go we have blueprints for the most common design patterns.
First, you mentioned that you are trying to convert the monolithic application into microservices. You can create/caters the microservices on basis of domain data, we can be called it domain-driven architecture.
Suppose you have the business functionality for customer data, customer order, customer order handling, and customer payment. And currently, it's part of a monolithic application. So you can create the subdomain for each functionality like Customer domain, Order domain, order handle domain, and payment domain respectively. Each domain contains several microservices depends on the business requirement.
For e.g you can check the Amazon website, In personal/customer data, you see the customer name, phone number, address, billing account information, delivery address type(office/home). In this case the under customer domain, there will be 3 microservices required(It totally depends on your domain design). One for customer(handles customer name, phone number, reference of Billing account id, reference of address id), second for Billing account(Billing account number, billing account information, reference if customer id), third for Address data(customer office address, preferable address). And for each microservice, there will be a dedicated database/buckets, Only that microservice can change/add the data. If any other microservice wants to add/update/get data, it needs to be get by calling that microservices HTTP endpoint over the network.
Updating the data in other microservice::
Now coming to your question about data duplication, Let's consider the above example.
If Customer microservice wants to store/ cache the billing account data for some purpose, that microservice can store that data in the database but again Customer microservice needs to make sure that, the current data of the billing account is always real one and not the old one. For this customer, microservice needs to listen to the event whenever there is update in billing account data, so old data in billing account gets purged and customer microservice always has the latest data of billing.
you can read here about event driven architecture.
https://en.wikipedia.org/wiki/Event-driven_architecture#:~:text=Event%2Ddriven%20architecture%20(EDA),sale%22%20to%20%22sold%22.
You can read more about this at the below links about Domain driven design.
https://www.thoughtworks.com/insights/blog/domain-driven-design-services-architecture
https://en.wikipedia.org/wiki/Domain-driven_design
This is my free book :)
https://github.com/vaquarkhan/microservices-recipes-a-free-gitbook
If you want to create microservice then need to follow microservice guideline.
Now come to real world :) really difficult to meet all microservice requirements as database has own licensing cost etc. so you can choose pragmatic microservices. You can get started with them faster and pick and choose the pieces that make sense for your team.
Design Domain driven design oriented microservice : DDD talks about problems as domains. It describes independent problem areas as Bounded Contexts and each Bounded Context correlates to a microservice.
Where to draw the boundaries is the key task when designing and defining a microservice.
DDD patterns help you understand the complexity in the domain, the domain model for each Bounded Context, you identify and define the entities, value objects, and aggregates that model your domain. You build and refine a domain model that is contained within a boundary that defines your context. And that is explicit in the form of a microservice. The components within those boundaries end up being your microservices.
https://martinfowler.com/bliki/AnemicDomainModel.html
https://github.com/vaquarkhan/Domain-driven-design
https://github.com/vaquarkhan/ddd-by-examples.github.io/blob/master/ddd-factory.pdf
Now you can create layers on top of you microservice and build complex logic using orchestration and choreography.
Example :
Gateway Customer order Application layer microservice --domain model layer microservice infrastructure layer
There are various ways to decompose an application into services.
1.Decompose by business capability
2.Decompose by subdomain
And there are various ways for data management also
SAGA
API Composition
Database per service
Please go through with the link for more details click
Suppose that I have a microservice for messaging. The microservice knows how to send emails. The service have templates of emails that have some sort of "template engine" like pugjs, and can replace data in the body of the message.
I have an user service (used for authentication/authorization for example), and a bank account service (each user have one). Between the User microservice and Bank Account microservice it's clear that we don't have to duplicate any data than de user's uuid.
But I want now to send every day a message to each user with their account statement. The Messaging microservice needs data from the User microservice and the Bank Account microservice.
Okay... This is a small case of the real world. Now I know that to have the benefits of decoupled microservices I must follow some rules:
I can't share databases between microservices
I can't make synchronous requests between microservices
Okay... I can use a broker and each time a new user is created/updated the Messaging microservice can store that data. But really, this is a stupid thing:
I don't want to have inconsistency with this data, and keeping things sync is hard
The development time and complexity of the Messaging Microservice must now consider: listen and extract the relevant data from events, keep data consistent about other domains/services, managing the saved data on database
And think about a Messaging microservice. Really I must store all the data needed to parse the templates?
I read a lot about microservices and people creating rules for their simple examples. But I never really saw a good explanation and real-world examples like above.
So how to have the microservices above without data duplication?
In your domain example I would not let the message service know anything about bank or user details. Instead the message service should just receive instructions to send messages to recipients along with the given content. I would use a dedicated scheduled job (maybe implemented as an account notification service) that performs the work of acquiring the user and account data from the corresponding services, compiles the information for the message service and instructs it to actually send the messages. This introduces another "higher level, business purpose entity/service" but allows you to keep a clear separation of concerns.
In general it will happen frequently that your "basic" domain services are used by another service that represents a specific business purpose and requires their data. Dependency in itself is not a bad thing as long as concerns are seperated clearly and interfaces versioned, changes communicated etc.
Don't forget the whole idea of microservices is for allowing teams to have dedicated responsibilities with clear interfacing. It is about organization as much as it is about architecture.
I'm managing a very large enterprise application in that I've implemented microservice architecture. Standalone microservices have been created based on business entities & operations.
For example,
User Operations Service
Product Operations Service
Finance Operations Service
Please note that each service implemented using an n-tier architecture with WCF. i.e have separate tiers(which is independently deployable to separate server) for business and data access.
There is a centralized database which is accessed by all the microservices. There are a couple of common entities like 'user' accessed by all the services, so we have redundant database calls in multiple services. More efforts required due to database access from many places(i.e a column rename requires deployment of all the apps)
To reduce & optimize code, I'm planning to create separate microservice and move all the database operations into it. i.e services can call "Database Operations Service" for any database operations like add/update/select.
I want to know if there are any hidden challenges that I'm not aware of. Whether should I go with this thought? What can I consider as improvements in this concept?
I'm planning to create separate microservice and move all the database operations into it
That's how you will lose all benefits from microservice architecture. One service is down — the whole application is down. Unless you have replication on several nodes.
If your app does not work if one service went down(not implying that it's that service that connects to database), then it's still bad architecture and you are not using benefits of microservice architecture.
Correct for of communication would be if service would have their own databases. Or at least that every service that wants, for example, entity User, will not fetch it from DB, but will fetch it from appropriate service. And that appropriate service could fetch it from common DB at the beginning.
Next step (improvement) in the process of accommodation to microservice architecture would be creation of separate databases for each service. And by “separate” I mean that temporal fault of one service or temporal fault of one database will allow the rest of the app to be alive and functioning.
Generally, there are no hidden challenges in your approach. It just does not give any benefits, as an intermediate form between monolith application and microservice-based.
Can anyone give me some suggestions of a business scenarios where I can implement Java Messaging Services (JMS). The message can be sent either by queue(point-to-point) or topic (regular/durable subscription).
I will be using JMS (enabled through TIBCO Enterprise Messaging Services).
The business scenarios must involve atleast 3 IT systems/applications.
The classic use case is that of an Enterprise Service Bus with JMS as one of the available transports. In this case any number of IT systems can request a service invocation by placing a message on a well-known queue. The service provider listening on that queue dynamically determines the reply based on the JMS message's Reply-To fields. An example of a typical service is to inquire on or update customer demographic information. For purposes of inquiry, this definitely meets your requirement of involving at least 3 IT systems since pretty much everything dealing with customers would need to request this service.
Another example with broad application is logging. I have several customers using JMS messages to capture log records from across the network and forward them to a hub of central servers. Because it is JMS, the central hub can be highly available by using redundant servers and can scale horizontally to absorb seasonal loads.
For pub/sub an example I really liked is from an insurance company. They publish events on topics that are subscribed in various call centers, internal news tickers and to business partners. During a hurricane a few years back, these events included updates on landfall predictions and then after the storm passed the updates included locations of mobile claims adjusters and other support services. Pub/Sub was a great way to coordinate this massive mobilization of personnel and communicate back to ground support back at headquarters.
A more mundane pub/sub use case with broad applicability is systems management. Instrumented applications can publish their status and interested parties can receive those notifications. If something is acting weird in Production, the administrator can dynamically enable a subscription to a stream of diagnostics. Ordinarily with no subscribers, the diagnostics are not produced. However, without any interruption in the running system, simply by subscribing, diagnostic messages from the app are produced on demand.
It's actually harder to find examples where JMS messaging should not be used. The most common contraindications are truly synchronous messaging and a requirement to process messages in strict sequence. All JMS providers I'm aware of make allowances for these requirements to varying degrees and I'm aware of many deployments of systems with these requirements. However the ideal use cases for JMS messaging are truly asynchronous or pseudo-synchronous communication and messages that are atomic (that is to say messages have no dependencies on each other or to specific broker instances).
Here are some of the scenarios where we (food retailer) use messaging:
-connection systems between remote locations, in our case POS and inventory management systems in stores, and central ERP and forecast systems: master data changes are sent as XML messages from the central ERP system to the store systems. the store systems send changes in inventory, orders and sales to the central systems. This is completely PTP based, as the master data is unique for each store.
-usage as a central messaging backbone, either directly for systems that are capable to do messaging, or via some adapter functionality for databases, files, SAP systems or HTTP. Here the messaging system builds the base for our ESB.