I have a simple integration flow that poll data based on a cron job from database, publish on a DirectChannel, then do split and transformations, and publish on another executor service channel, do some operations and finally publish to an output channel, its written using dsl style.
Also, I have an endpoint where I might receive an http request to trigger this flow, at this point I send the messages one of the mentioned channels to trigger the flow.
I want to make sure that the manual trigger doesn’t happen if the flow is already running due to either the cron job or another request.
I have used the isRunning method of the StandardIntegrationFlow, but it seems that it’s not thread safe.
I also tried using .wireTap(myService) and .handle(myService) where this service has an atomicBoolean flag but it got set per every message, which is not a solution.
I want to know if the flow is running without much intervention from my side, and if this is not supported how can I apply the atomic boolean logic on the overall flow and not on every message.
How can I simulate the racing condition in a test in order to make sure my implementation prevent this?
The IntegrationFlow is just a logical container for configuration phase. It does have those lifecycle methods, but only for an internal framework logic. Even if they are there, they don't help because endpoints are always running if you want to do them something by some event or input message.
It is hard to control all of that since it is in an async state as you explain. Even if we can stop a SourcePollingChannelAdapter in the beginning of that flow to let your manual call do do something, it doesn't mean that messages in other threads are not in process any more. The AtomicBoolean cannot help here for the same reason: even if you set it to true in the MessageSourceMutator.beforeReceive() and reset back to false in its afterReceive() when message is null, it still doesn't mean that messages you pushed down in other thread are already processed.
You might consider to use an aggregator for AtomicBoolean resetting in the end of batch since you mention that you pull data from DB, so perhaps there is a number of records per poll you can track downstream. This way your manual call could be skipped until aggregator collects results for that batch.
You also need to think about stopping a SourcePollingChannelAdapter at the moment when manual action is permitted, so there won't be any further race conditions with the cron.
Related
I have the need to access the saga repository from within a consumer to read the current status of the saga correlated to the message being consumed.
Scenario:
I have an external service, when this service consumes an event coming from the saga I want to see if the saga is still in the correct state because if meanwhile the saga changed its state the consumer must skip the event.
How: I surely could query the saga repository implementation chosen by using its the native framework, but I would like to use an abstraction, an interface, to load the saga state from within the consumer, in order to be able to switch to a different repository implementation in the future.
Any help is appreciated.
If the saga initiated the command, sending it to the consumer, why would the consumer need to check the saga's state? Is there a long delay between the time the command is sent and the consumer is able to process it?
The type of check you are asking about sort of goes against what a system would generally do when processing commands. If you do need to do this type of check, I'd actually suggest a request/response interaction using the request client to which the saga would respond if the command is still valid. That way, the logic (and locking) of the saga repository remains under the saga's control.
If needed, a separate endpoint could be used for that request to ensure it isn't backed up behind other messages targeting the saga. If that is desired, post a comment and I'll update the answer.
I've created two services.
One of them (scheduler) only requests to the other (backoffice) for performing some "large" operations.
When backoffice receives a request:
first creates a mark (key on redis) in order to set that the process has started.
Each time a request is reached:
backoffice checks if the mark exist.
When it exists means that the previous process has not yet finished, and escape it.
Perform the large process.
When process is finished, the previous key in redis is removed.
It would be something like this:
if (key exists)
return;
make long process... (1);
remove key;
The problem arises when service is destroyed when the process has not already finished and then it doesn't removes the mark on redis. It means the process will never run again.
Is there any way to solve this kind of problems?
The way to solve this problem is use an existing engine as building custom scalable and robust solution for reliable service orchestration is really hard.
I recommend looking at Uber Cadence Workflow which would allow to convert your pseudocode into a real production application with minor changes.
You can fire a background job that updates timestamp under the key, e.g. every minute.
When service attempts to start the process it must verify key existence (as it does now) + timestamp under the key. If it is more than 1 minute ago then the previous attempt is stale and you can start over.
Sounds like you should be using a messaging queue to schedule tasks for the back office service. Queuing solutions like RabbitMQ allow you to manually acknowledge (or “ack”) that the process is complete. Whenever a subscriber crashes, the queue detects that the connection dropped without acknowledgement and will re-enqueue the same task which will be picked up by the next available subscriber. Here’s another thread talking about this problem specifically focused on messaging queues:
What happens to fetched messages when RabbitMQ consumer crashes?
I have the problem that I have to run very long running processes on my Webservice and now I'm looking for a good way to handle the result. The scenario : A user executes such a long running process via UI. Now he gets the message that his request was accepted and that he should return some time later. So there's no need to display him the status of his request or something like this. I'm just looking for a way to handle the result of the long running process properly. Since the processes are external programms, my application server is not aware of them. Therefore I have to wait for these programms to terminate. Of course I don't want to use EJBs for this because then they would block for the time no result is available. Instead I thought of using JMS or Spring Batch. Does anyone ever had the same problem or an advice which solution would be better?
It really depends on what forms of communication your external programs have available. JMS is a very good approach and immediately available in your app server but might not be the best option if your external program is a long running DB query which dumps the result in a text file...
The main advantage of Spring Batch over "just" using JMS as an aynchronous communcations channel is the transactional properties, allowing the infrastructure to retry failed jobs, group jobs together and such. Without knowing more about your specific setup, it is hard to give detailed advise.
Cheers,
I had a similar design requirement, users were sending XML files and I had to generate documents from them. Using JMS in this case is advantageous since you can always add new instances of these processes which can consume and execute the jobs in parallel.
You can use a timer task to check status or monitor these processes. Also, you can publish a message to a JMS queue once the processes are completed.
I'm pretty new to MSMQ 4.0. I got stuck with below scenario;
Service A takes User Details and Returns an User ID.
Then Service B takes Billing detials with User ID.
Now I have to Queue these steps. I'm planning to use Transaction Queue.
Could some one please help me with
1)Get the ID from first message and include it in the second message.
2)If at least one step failed I have to rollback(transaction Queue does it for me) retry or 5 times and if it still failed then move it to VerifyAdminQueue for verification by Admin.I dont like using DeadLetter Queue etc.,
Thanks in advance.
Services built with MSMQ queues are truly one-way. This means that there is no built in concept of a response. There are many ways you can implement a request-response communication pattern using MSMQ but with all of them you will need to construct and send the response back to the caller yourself.
With one way actions, rollback is very simple, and indeed MSMQ will rollback any failed steps in the transmission of a message. More complex operations such as request-response however lack any concept of a transaction in MSMQ and so any rollback across more than one message transmission steps will require you to write compensatory code.
I'm using a message listener to process some messages from MQ based on Spring's DefaultMessageListenerContainer. After I receive a message, I have to make a Web Service (WS) call. However, I don't want to do this in the onMessage method because it would block the onMessage method until the invocation of WS is successful and this introduces latency in dequeuing of messages from the queue. How can I decouple the invocation of the Web Service by calling it outside of the onMesage method or without impacting the dequeuing of messages?
Thanks,
I think you might actually want to invoke the web service from your onMessage. Why do you want to dequeue messages quickly, then delay further processing? If you do what you're saying, you'd probably have to introduce another level of queueing, or some sort of temporary "holding" collection, which is redundant. The point of the queue is to hold messages, and your message listener will pull them off and process them as quickly as possible.
If you are looking for a way to maximize throughput on the queue, you might think about making it multi-threaded, so that you have multiple threads pulling messages off the queue to invoke the web service. You can easily do this by setting the "concurrentConsumers" configuration on the DefaultMessageListenerContainer. If you set concurrentConsumers to 5, you'll have 5 threads pulling messages off the queue to process. It does get tricky if you have to maintain ordering on the messages, but there may be solutions to that problem if that's the case.
I agree with answer provided before me , however I can see a usecase similar to this very common in practice. I'm adding my two cents It might be valid in some cases that you don't want to do time consuming work in your onMessage Thread (which is pulling message from Q)
We have something similar in one workflow, where if user selects some XYZ option on GUI that means at server we need to connect to another external webservice to get ABCD in this case we do not make call to webservice in onMessage Thread and use ThreadPool to dispatch and handle that call.
If something wrong happens during webservice call we broadcast that to GUI as separate Message , there is concept of request id which is preserved across messages so that GUI can relate error messages. You can use ExecutorService implementation to submit task.
hope it helps.