Let's imagine the situation, then we have distributed transactions in microservices, managed by SAGA approach . Out transaction exists of calling service A, then service B, then service C.
The scenario which fails:
Service A completed the task successfully and sent the message to B
service
Service B completed the task successfully and sent the message to C
service
Service C fails and sends a message to service B to undo the
transaction.
Service B tries to undo the transaction but fails.
As the result, we have inconsistent data, which is unacceptable.
The question is, how to deal with the case when the transaction rollback fails in SAGA pattern?
Service B retries immediately but fails
Service B retries after few seconds but fails
Service B retries after few minutes but fails
Message is moved to "dead letter queue" because it wasn't marked as processed by Service B
Engineer fixes the bug on service B
Engineer resends message from "dead letter queue"
Service B handles message successfully
Read more on this topic https://codeopinion.com/handling-failures-in-message-driven-architecture/
Related
In process A, I'm using
Bus.CreateRequestClient<TRequest>
to create client then using
client.GetResponse<TResponse1, TResponse12>(request)
to get response from process B.
In process B I respond to the request using consumeContext.RespondAsync(#event) as well as publishing the event with consumeContext.Publish(#event)
I have a process C subscribe to the published event.
So I have process A, B and C up and running, then I stop process C.
I send the request in process A and the queue for process C receives a message, but after a few seconds, the message disappear from the queue.
I've check the config the queue is configured with durable=true and autodelete=false.
If I use fire and forget (send the message to end point) instead of request/response in A, and only publish the event in B. then the message stays on the queue for C.
Is this by design?
Also I noticed that if I get a message on the queue by using fire and forget, the all the messages come after stays on the queue regardless if it was created using request/response or not.
See below screenshot, during highlighted period, the messages are taken off the queue without consumer ack.
So, as it turns out, this is a bug. The request TimeToLive is being applied to every message produced by the request consumer, which is why it eventually disappears from the queue.
Fixed
Will be released as 6.3.1 probably tonight.
We have a application iconnect,iconnect( Receiver ) application is expecting the message from the RTGS(Sender) application,RTGS is transferred the around 40000 messages. iconnect application is received the all messages except one messages. we have checked with RTGS application for the same,they have successfully sent the messages to targeted queue.
{ 1)We have checked all channel status its working fine,
2)We have checked the xmit queue and local queue curdepth.no messages are pilled up in queue.
3)We have checked the uncom messages in queue level no uncom messages are available.
4)And also we have checked application logs and DB logs,we haven't found the transaction. }
we unable to find the one transaction can any one help me for why the transaction got failed.and where we can find the failed transactions transactions.
#Subbarao: thanks for replay we have already tried the below command getting popup "no transaction found"
dspmqtrn -m queue manager name.
is there any process to we can find the losses transaction.
I'm using MassTransit 3.4 / RabbitMQ 3.6.5 and I love it, but have come across a problem. I have 2 services, A and B. When service A starts, it immediately calls service B via request/response pattern. If B isn't running at the time, the message will timeout which is ok - service A is still set as running. Later, when service B comes online, if service A makes the same request/response call, I'm expecting that it's going to work, but it's not. The request is made, service B picks it up and does a RespondAsync with the proper message, but the response ends up in the service A's bus Skipped queue. Why? Is this scenario not supported? How can I make sure that I can start up my services in any order and ensure that they'll work?
It's working fine if I start service B first, and then start service A, but I don't want to have to rely on this kind of hacky start sequence.
Each time I do the request/response, I'm getting the IRequestClient as it says in the docs like so...
IRequestClient<TRequest, TResponse> client = this.busControl.CreateRequestClient<TRequest, TResponse>(uri, TimeSpan.FromSeconds(timeout));
return await client.Request(message);
Thanks,
Andy
We have an HA application which runs on a few servers pointing to the same queue manager. Because the message sequencing is important we use a semaphore queue to make sure not two apps are reading messages from a business queue even if there are multiple messages waiting to be processed on that queue.
The work flow is something like below:
The business queue has a MQTT_DEPT trigger set on it to fire up when the queue dept is equals to one
The business queue receives one or multiple massages.
The trigger fires up when the queue dept becomes one and put a trigger message into an initiation queue.
Being a queue dept trigger it should switch itself OFF after delivering the trigger message into the initiation queue
There is a MDB listening to that initiation queue which gets the trigger messaage
Because sequence is crucial for us we put a limit on the initiation queue to have the maximum queue dept of one. This way only one MDB can get a message at a time
When the trigger message arrives the MDB goes and reads all messages from the business queue and when done it sends a trigger activate message to a dedicated queue.
Once this is done it commits everything. The trigger generated message gets discarded. At this point the trigger activation message becomes available for processing.
A trigger activating MDB gets the activation message and go and switches the queue dept trigger back on on the business queue so the whole process can kick off again.
In most of the cases all works fine. However now and then we get some messages in the dead letter queue which causes support calls by our monitoring tools. Inspecting the messages in the DLQ we noticed they are trigger generated messages and the reason for being sent to the DLQ is that the initiation queue was full (remember our limit of maximum one message).
We had no explanation about how this is possible. The IBM documentation says the triggering is switched off after sending the first trigger generated message but however from what we are experimenting it looks like wile the first trigger generated message gets processed and such is in an uncommitted state a second trigger generated message gets created and sent to the initiation queue. Not being able to deliver it (because of the max dept = 1 limit) it sends it to the dead letter queue.
Any inputs about why this would be possible and how to get around it would be very welcome. False support calls at 2:00AM are for sure not funny.
Thank you in advance for your inputs.
The explanation for our problem was on the IBM MQ triggering documentation which we should read more carefully IBM note:
Note: If you stop and restart the queue manager, the TriggerInterval timer is reset. There is a small window during which it is possible to produce two trigger messages. The window exists when the trigger attribute of the queue is set to enabled at the same time as a message arrives and the queue was not previously empty (MQTT_FIRST) or had TriggerDepth or more messages (MQTT_DEPTH).
Once we had the explanation the solution was simple. Rather than configuring the business queue to send the trigger generated message straight to the initiation queue we configured it to send to a different queue whit ho maximum queue dept restriction set on it. Then we implemented another MDB to get these trigger generated messages and redirect them to the initiation queue. This MDB was implemented two swallow any exception caused by an unable to deliver trigger generated message to the initiation queue because the existing queue dept limitation.
In other words with this new MDB we took the role of delivering the trigger generated message from the MQ manager in our hands and made the process ignore any delivery failure caused by the queue dept.
Assume some spring bean has transactional method and it does two things:
asynchronously send JMS messages
updates DB
The problem is that when the message is consumed, the DB transaction sometimes not have committed, and handler failed to find in DB expected value.
The evident solution - is to send in transaction, but I use async send available on ActiveMQ to increase throughput. If I make send transactional, I forfeit asynchrocity and also risk getting OuOfMemory.
How would you solve this problem?
You need to use an XA-enabled transaction manager, and send the JMS message and do the update to the DB in the same, distributed transaction.
This doesn't change anything to asynchronicity: the send call will return before the receiver receives the message. But when it receives it, the transaction will be committed, and the receiver will find the updates in the database.
I don't see what OutOfMemoryErrors have to do with this.