As I heard most of the time that in micro services architecture, for every single micro service we have to create individual database.
But if I have to maintain foreign key constraint across the different databases which is not possible. Like I have a user table in authentication micro service and I want to use it in my catalog service(userid column from user table)
So how can it be resolve.
Thanks in Advance
You can maintain a shadow copy (with only useful information for eg. just the userid column) of user table in catalog service via event sourcing(for e.g. you can use rabbit MQ or apache kafka for async messaging).
Catalog service will use the user information in read only mode. This solution is however effective only when user information doesn't change frequently. Otherwise async communication can be inefficient and costly.
In that case you can implement API calls from catalog service to user service for any validations to be done on user data.
Use the Saga Pattern to maintain data consistency across services.
A saga is a sequence of local transactions. Each local transaction
updates the database and publishes a message or event to trigger the
next local transaction in the saga. If a local transaction fails
because it violates a business rule then the saga executes a series of
compensating transactions that undo the changes that were made by the
preceding local transactions.
Related
Martin Fowler's description of the Event Collaboration pattern (https://martinfowler.com/eaaDev/EventCollaboration.html) appears to imply that requisite external data (data from other services) that is needed for a service to function should be replicated and maintained within the service.
This seems to imply that we should not resort issuing explicit queries.
For example:
Say you have a communications service that is responsible for sending emails to clients and is dependent order information (that lives in the order service) to send an order confirmation email.
With Event Collaboration, the communications service will have some internal representation of all orders that it will have built up by consuming relevant order creation/modification events.
In this example a query to retrieve order details will not be necessary to generate the confirmation email.
Are there any instances in which we would use explicit query messages rather than data replication when adopting the Event Collaboration pattern?
i think even in this case, what i would have done is create a consumer of OrderPlaced event in Order Microservice Only. That event processor will read all the details from order create a MailToBeSent event and write it on a Topic or Queue , which CommunicationService should listen and send the email.
Communication Service should not understand , how to create a email based on order(as core purpose of cummunication service is to send emails).
Design wise also communication service should not require to change every time you add a new service which want a mail sending functionality.
We are working on a system that is supposed to 'run' jobs on distributed systems.
When jobs are accepted they need to go through a pipeline before they can be executed on the end system.
We've decided to go with a micro-services architecture but there one thing that bothers me and i'm not sure what would be the best practice.
When a job is accepted it will first be persisted into a database, then - each micro-service in the pipeline will do some additional work to prepare the job for execution.
I want the persisted data to be updated on each such station in the pipeline to reflect the actual state of the job, or the its status in the pipeline.
In addition, while a job is being executed on the end system - its status should also get updated.
What would be the best practice in sense of updating the database (job's status) in each station:
Each such station (micro-service) in the pipeline accesses the database directly and updates the job's status
There is another micro-service that exposes the data (REST) and serves as DAL, each micro-service in the pipeline updates the job's status through this service
Other?....
Help/advise would be highly appreciated.
Thanx a lot!!
To add to what was said by #Anunay and #Mohamed Abdul Jawad
I'd consider writing the state from the units of work in your pipeline to a view (table/cache(insert only)), you can use messaging or simply insert a row into that view and have the readers of the state pick up the correct state based on some logic (date or state or a composite key). as this view is not really owned by any domain service it can be available to any readers (read-only) to consume...
Consider also SAGA Pattern
A Saga is a sequence of local transactions where each transaction updates data within a single service. The first transaction is initiated by an external request corresponding to the system operation, and then each subsequent step is triggered by the completion of the previous one.
http://microservices.io/patterns/data/saga.html
https://dzone.com/articles/saga-pattern-how-to-implement-business-transaction
https://medium.com/#tomasz_96685/saga-pattern-and-microservices-architecture-d4b46071afcf
If you would like to code the workflow:
Micorservice A which accepts the Job and command for update the job
Micorservice B which provide read model for the Job
Based on JobCreatedEvents use some messaging queue and process and update the job through queue pipelines and keep updating JobStatus through every node in pipeline.
I am assuming you know things about queues and consumers.
Myself new to Camunda(workflow engine), that might be used not completely sure
accessing some shared database between microservices is highly not recommended as this will violate the basic rule of microservices architecture.
microservice must be autonomous and keep it own logic and data
also to achive a good microservice design you should losely couple your microservices
Multiple microservices accessing the database is not recommended. Here you have the case where each of the service needs to be triggered, then they update the data and then some how call the next service.
You really need a mechanism to orchestrate the services. A workflow engine might fit the bill.
I would however suggest an event driven system. I might be going beyond with a limited knowledge of the data that you have. Have one service that gives you basic crud on data and other services that have logic to change the data (I would at this point would like to ask why you want different services to change the state, if its a biz req, its fine) Once you get the data written just create an event to which services can subscribe and react to it.
This will allow you to easily add more states to your pipeline in future.
You will need a service to manage the event queue.
As far as logging the state of the event was concerned it can be done easily by logging the events.
If you opt for workflow route you may use Amazon SWF or Camunda or really there quite a few options out there.
If going for the event route you need to look into event driven system in mciroservies.
I'm new to the microservices architecture and am seeing that it is possible under the model to call a microservice from another via HTTP request. However, I am reading that if a service is down all other services should still operate.
My question is, how is this generally achieved?
Example, a microservice that handles all Car record manipulation may need access to the service which handles the Vehicle data. How can the Car Microservice complete it's operations if that service is down or doesn't respond?
You should generally consider almost zero sync communication between microservices(if still you want sync comminucation try considering circuit breakers which allow your service to be able to respond but with logical error message , if no circuit breaking used dependent services will also go down completly).This could be achieved by questioning the consistency requirement of the micorservice.
Sometimes these things are not directly visible , for eg: lets say there are two services order service and customer service and order service expose a api which say place a order for customer id. and business say you cannot place a order for a unknown customer
one implementation is from the order service you call the customer service in sync ---- in this case customer service down will impact your service, now lets question do we really need this.
Because a scenario could happen where customer just placed an order and somebody deleted that customer from customer service, now we have a order which dosen't belong to customer.Consistency cannot be guaranteed.
In the new sol. we are saying allow the order service to place the order without checking the customer id and do one of the following:
Using ProcessManager check the customer validity and update the status of the order as invalid and when customer get deleted using ProcessManager update the order status as invalid or perform business logic
Do not check at all , because placing a order dosen't count a thing, when this order will be in the process of dispatch that service will anyway check the customer status
Statement here is try to achieve more async communication between microservices , mostly you will be able find the sol. in the consistency required by the business. But in case your business wants to check it 100% you have to call other service and if other service is down , your service will give logical errors.
I am currently working with a legacy system that consists of several services which (among others) communicate through some kind of Enterprise Service Bus (ESB) to synchronize data.
I would like to gradually work this system towards the direction of micro services architecture. I am planning to reduce the dependency on ESB and use more of message broker like RabbitMQ or Kafka. Due to some resource/existing technology limitation, I don't think I will be able to completely avoid data replication between services even though I should be able to clearly define a single service as the data owner.
What I am wondering now, how can I safely do a database backup restore for a single service when necessary? Doing so will cause the service to be out of sync with other services that hold the replicated data. Any experience/suggestion regarding this?
Have your primary database publish events every time a database mutation occurs, and let the replicated services subscribe to this event and apply the same mutation on their replicated data.
You already use a message broker, so you can leverage your existing stack for broadcasting the events. By having replication done through events, a restore being applied to the primary database will be propagated to all other services.
Depending on the scale of the backup, there will be a short period where the data on the other services will be stale. This might or might not be acceptable for your use case. Think of the staleness as some sort of eventual consistency model.
I am implementing an event-driven microservice architecture. Imagine the following scenario:
Chat service: Ability to see conversations and send messages. Conversations can have multiple participants.
Registration-login service: Deals with the registration of new users, and login.
User service: Getting/updating user profiles.
The registration-login service emits the following event with the newly created user object:
registration-new
login-success
logout-success
The chat service then listens on registration-new and stores some fields of user in its own redis cache. It also listens on login-success and stores the token, and on logout-success to delete the token.
The user service has the following event: user-updated. When this is fired, a listener in the chat service updates the data corresponding to the user id in redis. Like the chat service, the user service also listens on login-success and logout-success and does the same thing as what the chat service does.
My question is the following: is this a good way to do this? It feels a bit counterintuitive to be sharing data everywhere. I need some advice on this. Thank you!
Seems that there's no other way. Microservices architecture puts lots of stress in avoiding data sharing so as to not create dependencies. That means that each microservice will have some data duplicated. That also means that there must exist a way of getting data from other contexts. The preferred methods strive for eventual consistency, such as sending messages to event sourcing or AMQP systems and subscribing to them. You can also use synchronous methods (RPC calls, distributed transactions). That creates additional technologic dependencies, but if you cannot accept eventual consistency it could be the only way.