I am trying to solving a problem where the spring state machine have actions which invoke external long running processes via RabbitMQ. Here are the steps:
A state machine event is issued
The associated Action send a message to an external microservice via RabbitMQ
The external microservice takes 1 hour to process the request and send the response back to the State machine
The state machine picks up the message and update the state.
The issue I am having is how to block the state machine and wait for the response from that remote service before updating the state. I would greatly appreciate any help regarding this
You do not need explicit blocking of the state machine.
The machine will remain in the same target state (pre step 1) unless the next event (in your case response from the long running process is received), for that instance of a machine.
If to track, you can have another intermediate state "WAITINGFORMESSAGE", and transition from this stage to the next stage on receiving of message.
Related
I have few micro services, they accept the data from customer and process the request asynchronously, later customer can come and check the status, to make my platform my robust, I am planning to providing HA with running same setup services (atleat 2) and also registering with eureka, and also all my services are behind load balancer
Now I stuck at providing a solution if node failure or services goes down after accepting the request.
Lets say I have Service-A1 and Service-A2, both have same capability.
Now Service-A1 accepts the request and gave accepted response to customer then started processing the job and updating its intermediate results in db, now due to some node failure or service crash it could not complete the job.
In this case I want other service to auto detect(get notified) to continue, so it can read the job status and continue the job request for completion.
Is there any feature in Spring Eureka or zookeeper to watch and notify other to continue. ?
I've created two services.
One of them (scheduler) only requests to the other (backoffice) for performing some "large" operations.
When backoffice receives a request:
first creates a mark (key on redis) in order to set that the process has started.
Each time a request is reached:
backoffice checks if the mark exist.
When it exists means that the previous process has not yet finished, and escape it.
Perform the large process.
When process is finished, the previous key in redis is removed.
It would be something like this:
if (key exists)
return;
make long process... (1);
remove key;
The problem arises when service is destroyed when the process has not already finished and then it doesn't removes the mark on redis. It means the process will never run again.
Is there any way to solve this kind of problems?
The way to solve this problem is use an existing engine as building custom scalable and robust solution for reliable service orchestration is really hard.
I recommend looking at Uber Cadence Workflow which would allow to convert your pseudocode into a real production application with minor changes.
You can fire a background job that updates timestamp under the key, e.g. every minute.
When service attempts to start the process it must verify key existence (as it does now) + timestamp under the key. If it is more than 1 minute ago then the previous attempt is stale and you can start over.
Sounds like you should be using a messaging queue to schedule tasks for the back office service. Queuing solutions like RabbitMQ allow you to manually acknowledge (or “ack”) that the process is complete. Whenever a subscriber crashes, the queue detects that the connection dropped without acknowledgement and will re-enqueue the same task which will be picked up by the next available subscriber. Here’s another thread talking about this problem specifically focused on messaging queues:
What happens to fetched messages when RabbitMQ consumer crashes?
I am thinking of using Spring State Machine for a TCP client. The protocol itself is given and based on proprietary TCP messages with message id and length field. The client sets up a TCP connection to the server, sends a message and always waits for the response before sending the next message. In each state, only certain responses are allowed. Multiple clients must run in parallel.
Now I have the following questions related to Spring State machine.
1) During the initial transition from disconnected to connected the client sets up a connection via java.net.Socket. How can I make this socket (or the DataOutputStream and BufferedReader objects got from the socket) available to the actions of the other transitions?
In this sense, the socket would be some kind of global resource of the state machine. The only way I have seen so far would be to put it in the message headers. But this does not look very natural.
2) Which runtime environment do I need for Spring State Machine?
Is a JVM enough or do I need Tomcat?
Is it thread-safe?
Thanks, Wolfgang
There's nothing wrong using event headers but those are not really global resources as header exists only for duration of a event processing. I'd try to add needed objects into an machine's extended state which is then available for all actions.
You need just JVM. On default machine execution is synchronous so there should not be any threading issues. Docs have notes if you want to replace underlying executor asynchronous(this is usually done if multiple concurrent regions are used).
The ZeroMQ guide describes in the Getting an Out-of-Band Snapshot section that
The client first subscribes to updates and then makes a state request. This guarantees that the state is going to be newer than the oldest update it has.
How does making the subscription first guarantee that the client will receive all updates newer than the snapshot state? For example
Client subscribes to state updates
Client requests the state snapshot
Client receives the state snapshot
State changes happen at the server
Client's subscription to state changes is complete
So the client would miss the state changes happening on step 4. Is this scenario possible?
Allow me to describe the process a little more fully:
Server publishes updates as they occur
New client subscribes to updates (client SUB to server PUB)
New client requests current state from server (client DEALER to server ROUTER) - (IMPORTANT: it's assumed it will take longer for this request to reach the server and begin building the snapshot this than it takes for the SUB socket to finish connecting and subscribe to updates - this is generally a reasonable assumption, but note it)
Server builds snapshot of current state to respond to request
Server continues to publish updates as they occur
New client queues all of these updates that they are subscribed to - does not process them yet (this is part of ZMQ "for free")
Server sends back current state (IMPORTANT: If the state request from the client occurred after the subscription completed, then one of two scenarios is true: either (A) there were no new updates after the new client joined, and so the state is just the history before the new client joined, or (B) there were new updates that are both in the state and queued in the client's SUB socket. (A) is trivially correct, so we'll focus on (B).)
New client processes the state - this brings it up to current.
New client begins to process the messages in the SUB socket. If there are any we check them against the history we now have. If we already have this update (from the state), we discard it. If we don't, it's a new message and we deal with it.
New client continues to process the messages as normal, all caught up to date and processing all new messages.
... even though in the example code, the SUB socket doesn't start to recv() messages until after it receives the state, it's still getting them from the publisher and queuing them until it's ready to process them, so there's no scenario where an update is missed, instead the opposite scenario where messages are duplicated is planned for and handled.
I have a Mass Transit Service Bus that is listening to several queues and processing the messages. I would like to somehow pause the processing of new requests and wait for the current requests to complete so that I can run some housekeeping tasks.
A few of my own thoughts:
I have investigated the service bus BeforeConsumingMessage handler and although this would allow me to check for a 'Pause processing' flag in my database, I unsure how I would then actually pause the processing!
We are using RabbitMQ - could I use this to put the queues in a suspend state?
I have found so little on this subject that I wonder if it is an 'anti-pattern' and I should just stop my Mass Transit services if I want to run some housekeeping jobs and trust in any partially complete sagas to be picked up when the service bus starts back up. (Rather not go for this option, though).
So my question is: Is there a way to instruct the service bus to finish processing the current sagas but do not take any more messages from the queue?
Cleanly shutting down a MT service will wait for any messages in process to completely finished. Why sometimes it takes a little while for a service to shutdown. Shutting down the service is the best way to handle this, you are sure MT is not pulling any new messages.
If your sagas are serialized to a backing store, e.g. NHibernate, then the state will be saved until the service is restarted and the sagas will pick up in the state they were left after the last message was processed. You should be in good shape. We do this all the time for any maintenance periods.
If you REALLY must leave the service running, call Dispose on the IServiceBus instance. This will do the same thing, letting the current consumers finish then releasing all your resources. Once you have done maintenance you can create a new IServiceBus instance as needed.