Let's say that we have microservice A (MS A) and Microservice B (MS B).
MS B has data about Products. MS A needs the productnames of MS B.
Each time a product is added, updated or deleted, MS B puts a message on a message queue.
MS A is subscribed to that queue, so it can updated it's own internal state.
Now my question:
How do we fill the internal state of MS A when we deploy it to production the first time?
I couldn't find any documentation about the pros and cons of the possible solutions.
I could think of:
Export/import on database level.
Pros: not much work.
Cons: can miss data if during export/import changes to the data of MS A are made.
Implement calls for GetData and GetDataChangedSince
Pros: failsafe
Cons: a lot of work
Are there any other options? Are there any other pros/cons?
You could use the following workflow:
prepare the microservice B to push the events to the queue or stop it if it is already pushing to the queue; instead, it pushes to a circular buffer (a buffer that is rewritten when full) and waits for a signal from microservice A
deploy the microservice A into production servers but you don't reference it from nowhere; it just runs, waiting for events in the queue
run a script that get all product names from microservice B and push them into the queue as a simulated event; when it finishes the product names it signals the microservice B (optionally telling the date or sequence number or whatever de-duplication technique you have to detect duplicate events)
microservice B then copy the events from the buffer newer that the last pushed by microservice A (or it finds out itself from the queue what is the last one) into the queue and then ignores the buffer and continue to work as normally.
It sounds like there is a service/API call missing from you architecture. Moving a service into production should be no different than recovering from a a failure and should not require any additional steps. Perhaps the messages should be consumed from the queue by another service that can then be queried for the complete list of products.
Related
We are working on an IOT platform, which ingests many device parameter
values (time series) every second from may devices. Once ingested the
each JSON (batch of multiple parameter values captured at a particular
instance) What is the best way to track the JSON as it flows through
many microservices down stream in an event driven way?
We use spring boot technology predominantly and all the services are
containerised.
Eg: Option 1 - Is associating UUID to each object and then updating
the states idempotently in Redis as each microservice processes it
ideal? Problem is each microservice will be tied to Redis now and we
have seen performance of Redis going down as number api calls to Redis
increase as it is single threaded (We can scale this out though).
Option 2 - Zipkin?
Note: We use Kafka/RabbitMQ to process the messages in a distributed
way as you mentioned here. My question is about a strategy to track
each of this message and its status (to enable replay if needed to
attain only once delivery). Let's say a message1 is being by processed
by Service A, Service B, Service C. Now we are having issues to track
if the message failed getting processed at Service B or Service C as
we get a lot of messages
Better approach will be using Kafka instead of Redis.
Create a topic for every microservice & keep moving the packet from
one topic to another after processing.
topic(raw-data) - |MS One| - topic(processed-data-1) - |MS Two| - topic(processed-data-2) ... etc
Keep appending the results to same object and keep moving it down the line, untill every micro-service has processed it.
I have doubt related to MicroServices. Suppose there are 5 Micro-Services, lets say M1, M2, M3, M3, M4 and M5. There are 4 databases which are connected/accessed by 4 micro-services.
For example, M2 connected to MySQL, M3 connected to Cassandra, M4 connected to MongoDB and M5 connected to Oracle.
Now
Step-1: M1 making a call to M2 to update some user data in MySQL and it updated successfully then finally it got success response from M2
Step-2: M1 making a call to M3 to update some data in Cassandra and it updated successfully then finally it got success response from M3
Step-3: M1 making a call to M4 to update some data in MongoDB and it failed due to some DB server problem or any other problem.
Here my requirement is, I want to rollback DB changes that happened to previous micro-services(M2 and M3)
What should we need to do to achieve this kind of rollback scenario?
This is a typical case of distributed transaction. Regardless of the fact that you use separate technology for your database or the same on different server you perform an operation which is transactional.
In order to handle a rollback on that type of transaction you can not relay on the database technology mechanism for transactions and rollbacks. You have to do it on your own.
Saga Pattern
Common solution for distributed transaction scenarios in micro-service architecture is the Saga pattern.
Distributed sagas is a pattern for managing failures in scenarios as the one that you have described.
Saga are created based on business process for example "Buy a Product in online shop". This process can involve multiple actions on multiple micro-services. Saga will control and manage this process execution and if one of the steps fail it will trigger actions to revert the actions done before the failing action.
There are multiple ways to implement sagas. It depends on your architecture and the way your micro-services communicate with each other. Do you use Commands and/or Events?
Example
"Buy a Product in online shop" business process. Lets say this business process has 3 simple steps done by 3 different micro-services:
Action 1 - Reserve Product in products-inventory-micro-service
Action 2 - Validate payment in payment-micro-service
Action 3 - Order a product in orders-micro-service
Using Events:
You can publish events to perform some action(or actions) and if one of the action fails you can publish a revert(or delete) event for that event. For the above business process lets say the 1. Action succeeded and the 2. Action failed. In this case in order to rollback the 1. Action you would publish an event like "RemoveReservationFromProduct" in order to remove the reservation and revert the state back to the state as it was before the transaction for that Business process started. This event would be picked up by a event handler which would go and revert that state in your database. Since it is an event you can implement retry mechanism for failures or just reapply it later if there is some bug in the code.
Using commands:
If you have direct calls to your micro-services as commands using some kind of rest api you could run some delete or update endpoints to revert the changes that you have done. For the above business process lets say the 1. Action succeeded and the 2. Action failed. In this case in order to rollback the 1. Action you would call the delete api to delete the reservation for a particular product in order to remove the reservation and revert the state back to the state as it was before the transaction for that Business process started.
You can take a look at this example how to implement the Saga pattern.
From what I understand, a Saga is what you are looking for.
The idea is to provide for every state altering operation an undo-operation, that has to be called if things went bad down stream.
You can make sure that you have #Transactional enabled in this entire sequence of Invocation.
Consider invocation of all microservices from M1 as single transaction.
Expose a rollback in following way:
While updating DB in M2, M3 and M4, place the values in Spring cache as well along
with DB.
Upon invoking /rollback in M2, M3 or M4, get the values from Spring Cache and undo
them from DB.
In the fallbackMethod of hysterix command, when M1 replies with error or some default output, invoke /rollback of other services.
This may not be a perfect solution, as it introduces another fault point as /rollback handling, but fastest one that can be implemented.
to answer your question lets add some business requirements
Case 1. M1 is doing all interaction with other microservices based on an event recieved like Order Placed
Now in this case M2 ... M5 update ,
requirement 1: if all of them are independent of each other.
first create 5 event from one event and then
in such a case you could add this event in a table mark this event as unprocessed and some timer reads unprocessed event and tries to do all the tasks in a Idempotent way, also you could have reporting if such tasks are failing and your team can check them and manually resolve them.
(you could implement a similar logic by using a failover queue - which sends the same event back to the original queue after some time)
requirement 2: if all are not independent
use a single event and still the same solution.
in the above solution the main benefit is even if your system restart in between the transactions you will alwayss eventually have the consistent system
Case 2. if the M1 api is invoked and M1 needs to do all tasks from multiple microservice and then give response to user.
we could create a started event in M1 microservice DB (sync_event_table)
try to do update in all microservice
after all complete , update the sync event table with completed
for those cases which are not completed - run a timer which checks for job which are not completed for > X min and then do the undo actions or whatever required,.
Essence:
So if you see all solutions suggests your system to turn all the diff. microservice update
by creating a job
checking job status
writing a undo/redo job feature
We have a Node.js application running loopback, the main purpose of which is to process orders received from the client. Currently the entire order process is handled during the single http request to make the order, including the payment, insertion into the database and sending confirmation emails etc.
We are finding that this method, whilst working at the moment, lacks scalability - the application is going to need to process, potentially, thousands of orders per minute as it grows. In addition, our order process currently writes data to our own database, however we are now looking at third party integrations (till systems) over which we have no control of the speed or availability.
In addition, we also currently have a potential race condition; we have to assign a 'short code' to each order for easy reference by the client - these need to rotate, so if the starting number is 1 and the maximum is 100, the 101st order must be assigned the number 1. At the moment we are looking at the previous order and either incrementing the previous reference by 1 or setting it back to the start - obviously this is fine at the moment due to the low traffic - however as we scale this could result in multiple orders being assigned the same reference number.
Therefore, we want to implement a queue to manage all of this. Our app is currently deployed on Heroku, where we already use a worker process for some of the monthly number crunching our app requires. Whilst having read some of the Heroku articles on implementing a queue (https://devcenter.heroku.com/articles/asynchronous-web-worker-model-using-rabbitmq-in-node, https://devcenter.heroku.com/articles/background-jobs-queueing) it is not clear how, over multiple worker dynos, we would ensure the order in which these queued items are processed and that the same job is not processed more than once by multiple dynos. The order of processing is not so important, however the lack of repetition is extremely important as if two orders are processed concurrently we run the risk of the above race condition.
So essentially my question is this; how do we avoid the same queue job being processed more than once when scaled across multiple dynos on Heroku?
What you need is already provided by RabbitMQ, the message broker used by the CloudAMQP add-on of Heroku.
You don't need to worry about the race condition of multiple workers. A job placed onto the queue is stored until a consumer retrieves it. When a worker consumes a job from the queue, no other workers will be able to consume it.
RabbitMQ manages all such aspects of message queing paradigm.
A couple of links useful for your project:
What is RabbitMQ?
Getting started with RabbitMQ and Node.js
I'm designing a multi-tenant Azure Service Fabric application in which we'll be storing event data in Azure Append-Only blobs.
There'll be two kinds of blobs; merge blobs (one per tenant); and instance blobs (one for each "object" owned by a tenant - there'll be 100K+ of these per tenant)
There'll be a single writer per instance blob. This writer keeps track of the last written blob position and can thereby ensure (using conditional writes) that no other writer has written to the blob since the last successful write. This is an important aspect that we'll use to provide strong consistency per instance.
However, all writes to an instance blob must also eventually (but as soon as possible) reach the single (per tenant) merge blob.
Under normal operation I'd like these merge writes to take place within ~100 ms.
My question is about how we best should implement this guaranteed double-write feature:
The implementation must guarantee that data written to an instance blob will eventually also be written to the corresponding merge blob exactly once.
The following inconsistencies must be avoided:
Data is successfully written to an instance blob but never written to the corresponding merge blob.
Data is written more than once to the merge blob.
Most easiest way as for me is to use events: Service Bus or Event Hubs or any other provider to guaranty that an event will be stored and reachable at least somewhere. Plus, it will give a possibility to write events to Blob Storage in batches. Also, I think it will significantly reduce pressure on Service Fabric and will allow to process events at desired timing.
So you could have a lot of Stateless Services or just Web Workers that will pick up new messages from a queue and in batch send them to a Statefull Service.
Let's say that it will be a Merge service. You would need to partition these services and the best way to send a batch of events grouped by one partition is to make such Stateless Service or Web Worker.
Than you can have a separate Statefull Actor for each object. But on your place I would try to create 100k actors or any other real workload and see how expensive it would be. If it is too expensive and you cannot afford such machines, then everything could be handled in another partitioned Stateless Service.
Okay, now we have the next scheme: something puts logs into ESB, something peaks these evetns from ESB in batches or very frequently, handling transactions and processing errors. After that something peaks bunch of events from a queue, it sends it to a particular Merge service that stores data in its state and calls particular actor to do the same thing.
Once actor writes its data to its state and service does the same, then such sevent in ESB can be marked as processed and removed from the queue. Then you just need to write stored data from Merge service and actors to Blob storage once in a while.
If actor is unable to store event, then operation is not complete and Merge service should not store data too. If Blob storage is unreachable for actors or Merge services, it will become reachable in the future and logs will be stored as they are saved in state or at least they could be retrieved from actors/service manually.
If Merge service is unreachable, I would store such event in a poison message queue for later processing, or try to write logs directly to Blob storage but it is a little bit dangerous though chances to write at that moment only to one kind of storage are pretty low.
You could use a Stateful Actor for this. You won't need to worry about concurrency, because there is none. In the state of the Actor you can keep track of which operations were successfully completed. (write 1, write 2)
Still, writing 'exactly once' in a distributed system (without a DTC) is never 100% waterproof.
Some more info about that:
link
link
I'm trying to implement the actor model pattern (Somewhat mashed toghter with producer consumer) by using spring's application events and threadpoolexecutors
my main objective is to decouple each of the layers .
my architecture is as follows :
i have a war deployed that trough a rest API receives requests for business transactions , at any given moment there can be X number of transactions alive
where X is configureable number , the actual execution must be asynchronous and each transaction must be in a different thread .
the requests themselves are treated in a FIFO manner but there is some complexity to it as some requests must wait for other to complete before they can be processed but that does not mean other requests can't be processed , e.g. : don't process a withdrawal from account 2 if a deposit to account 2 was requested before it , so if i get hits for :
deposit(2)
withdrawal(2)
deposit(3)
where the numbers are account numbers i want to process them in this order:
deposit(2)
deposit(3)
withdrawal(2)
i've built the architecture is this way :
i have a rest api that gets the hits and writes them to the DB (distributed system that has to have the state in DB)and publishes
a clientrequestevent in the application context
i have singleton bean that is in charge of publishing the producer events and monitoring how many events he has sent (i.e. : he is in charge of limiting the number of concurrent processes and implementing the above logic)
and i have a few other listeners each for an action (withdrawal deposit etc..) that listen to the events published by the latter and publish a done event.
every thing works great everything is done is different threads and all flows great but i have
a problem with the middle layer the one in charge of determing whether or not there is a free slot
i don't want to have a synchronous method nor do i want to do some tricks around an atomiclong or something like that i would rather use some blockingqueue to publish the events but i can't find a nice way to determine when an event is done so i can put a new one back in .
the biggest problem is that for requesting a new work i have to go the DB which is a heavy task , as this system should work under heavy load .
i would like to somehow utilize a blockingqueue and a threadpool so that will take from a size bounded queue with threads the minute one slot is free
what would be a good way to handle this ?
thanks in advance