Application 1(A1) sends messages to A2 over MQ. A2 uses XA transactions so that a message dropped on the queue is picked by A2, processed and written to the DB and the whole transaction is committed at once.
I would like to test whether A2 correctly maintains system consistency if the transaction fails mid-way and whether XA has been implemented correctly.
I would like to stop the DB as soon as A2 picks up the message. But I am not sure whether I will have enough time to stop the DB and whether I will know for sure that the message has been picked.
Any other suggestions for testing this?
Thanks,
Yash
I am assuming you are using Java here, otherwise, some of this won't be applicable.
The quick, pragmatic solution is to inject a delay into your process which will give you time to take your transactionally destructive action. The easiest way to do this would be to run the app in a debugger. Place a breakpoint at some suitable location (perhaps after the message has been received and the DB write is complete but not committed) and kill the DB when the debugger pauses the thread. Alternatively, add a test hook to your code whereby the thread will sleep if the MQ message has a header titled something unlikely like 'sleeponmessagereceived'.
A more complex but sophisticated technique is to use error injection via some AOP tool. I would definitely look at Byteman. It allows you to inject bytecode at runtime and was originally written to test XA scenarios like yours for the Arjuna transaction manager. You can inject code procedurally, or you can annotate unit testing procedures. Some of the advantages of this approach is that you can direct Byteman to trigger an error condition based on a variety of other conditions, such as nth invocation, or if a method arg is X. Also, depending on how detailed your knowledge of your transaction manager is, you can recreate a wider set of scenarios to produce some more tricky XA outcomes like Heuristic exceptions. There's some examples here that demonstrate how to use Byteman scripts to validate MQ XA Recovery. This project is intended to help reproducing XA failure and recovery. It is JBoss specific, but I would imagine you would be able to adapt to your environment.
Related
I have a simple integration flow that poll data based on a cron job from database, publish on a DirectChannel, then do split and transformations, and publish on another executor service channel, do some operations and finally publish to an output channel, its written using dsl style.
Also, I have an endpoint where I might receive an http request to trigger this flow, at this point I send the messages one of the mentioned channels to trigger the flow.
I want to make sure that the manual trigger doesn’t happen if the flow is already running due to either the cron job or another request.
I have used the isRunning method of the StandardIntegrationFlow, but it seems that it’s not thread safe.
I also tried using .wireTap(myService) and .handle(myService) where this service has an atomicBoolean flag but it got set per every message, which is not a solution.
I want to know if the flow is running without much intervention from my side, and if this is not supported how can I apply the atomic boolean logic on the overall flow and not on every message.
How can I simulate the racing condition in a test in order to make sure my implementation prevent this?
The IntegrationFlow is just a logical container for configuration phase. It does have those lifecycle methods, but only for an internal framework logic. Even if they are there, they don't help because endpoints are always running if you want to do them something by some event or input message.
It is hard to control all of that since it is in an async state as you explain. Even if we can stop a SourcePollingChannelAdapter in the beginning of that flow to let your manual call do do something, it doesn't mean that messages in other threads are not in process any more. The AtomicBoolean cannot help here for the same reason: even if you set it to true in the MessageSourceMutator.beforeReceive() and reset back to false in its afterReceive() when message is null, it still doesn't mean that messages you pushed down in other thread are already processed.
You might consider to use an aggregator for AtomicBoolean resetting in the end of batch since you mention that you pull data from DB, so perhaps there is a number of records per poll you can track downstream. This way your manual call could be skipped until aggregator collects results for that batch.
You also need to think about stopping a SourcePollingChannelAdapter at the moment when manual action is permitted, so there won't be any further race conditions with the cron.
You have a command/operation which means you both need to save something in database end send an event/message to another system. For example you have an OrderService and when a new order is created you want to publish an "OrderCreated"-event for another system/systems to react on (either direct message or using a message broker) and do something.
The easiest (and naive) implementation is to save in db and if successful then send message. But of course this is not bullet proof because the other service/message broker is down or your service crash before sending message.
One (and common?) solution is to implement "outbox pattern", i.e. instead of publish messages directly you save the message to an outbox table in your local database as part of your database transaction (in this example save to outbox table as well as order table) and have a different process (polling db or using change data capture) reading the outbox table and publish messages.
What is your solution to this dilemma, i.e. "update database and send message or do neither"? Note: I am not talking about using SAGAs (could be part of a SAGA though but this is next level).
I have in the past used different approaches:
"Do nothing", i.e just try to send the message and hope it will be sent. Which might be fine in some cases especially with a stable message broker running on same machine.
Using DTC (in my case MSDTC). Beside all the problem with DTC it might not work with your current solution.
Outbox pattern
Using an orchestrator which will retry process if you have not got a "completed" event.
In my current project it is not handled well IMO and I want to change it to be more resilient and self correcting. Sometimes when a service is calling another service and it fails the user might retry and it might work ok. But some operations might require out support to fix it (if it is even discovered).
ATM it is not a Microservice solution but rather two large (legacy) monoliths communicating and is running on same server but moving to a Microservice architecture in the near future and might run on multiple machines.
I would like to use the same service classes in both the publisher (which will be a REST API) and consumer. Since sending messages can be a part of these service classes, they have an instance of IBus injected into them so they can publish/send messages. This is fine on the REST API side, but the MassTransit documentation states the following:
Once you have consumers you will ALWAYS use ConsumeContext to interact with the bus, and never the IBus.
What's the reason behind this? Is it just performance related or does using IBus have any other consequences? And what are the alternatives to doing this? Would injecting IPublishEndpoint and ISendEndpointProvider be the accepted solution here, or does that not really change anything?
The reason why I want to do this is because some actions can be done either synchronously by using the API, or happen automatically in the background by using a message, and having to duplicate the business logic would be very inconvenient and hard to maintain.
Bonus question: The documentation states the same thing for TransactionalBus:
Never use the TransactionalBus or TransactionalEnlistmentBus when writing consumers. These tools are very specific and should be used only in the scenarios described.
However, if I want to support transactions in the above mentioned services, I will probably have to use TransactionalBus, but is it safe to do so in consumers? I do know about the in-memory outbox, but I have 2 problems with it:
It can only be used on the consumer side, so the publisher would not support transactions
It does not support "partial transactions" - the codebase that I'm working on has certain places where transactions don't wrap the entire API call, but rather only parts of it, so cases where some entities are successfully written to the database before the transaction is even started can happen, and in these cases the corresponding messages would need to be sent/published as well. This could easily be done by calling Release on the TransactionalBus at the right time, but couldn't be done when using the outbox since it's all or nothing (if an exception happens, nothing will be sent).
This bonus question isn't that important since I could probably work around it, but is still something I'm curious about, as it could be resolved by using TransactionalBus (if that won't cause any issues in consumers).
You should be using IPublishEndpoint or ISendEndpointProvider to publish or send messages from your components and/or services. There is almost never a reason to use IBus.
IPublishEndpoint and ISendEndpointProvider are registered a scoped, so a valid scope is required. In a service that normally doesn't have a scope, one can easily be created using provider.CreateScope(). Scopes should also be disposed of when they are no longer used.
Note that current versions should use provider.CreateAsyncScope() instead, and to make it easy just assign it using:
await using var scope = provider.CreateAsyncScope()
var publishEndpoint = scope.ServiceProvider.GetService<IPublishEndpoint>();
For any components, consumers, etc. simply use constructor injection for either of those two types, and they will resolve the proper services depending upon the context.
Also, don't use ITransactionBus. The new outbox is a better solution, as it's actually in the transaction. I will eventually remove ITransactionBus from MassTransit.
I am using mulesoft ESB with Anypoint studio for a project. In one of my flows I am using one-way message exchange pattern to dispatch from VM (persistence file store VM connector) to JMS, both xa transaction enabled to avoid losing messages.
Consider a scenario where we send a message every time user updates his/her last name to ESB. For example, let's say user changes last name to 'A', but quickly changes to 'B', so final result is expected to be 'B'.
1) Is it likely that message 'B' gets processed before message 'A' in my case? and thus last name being set to 'A' instead of 'B'?
2) How do I avoid that apart from using 'request-response' MEP?
3) Is there a way to write unit tests for making sure order of messages being processed is maintained from VM (one-way, xa enabled) to JMS (one-way, xa enabled)?
4) How do I go about testing that manually?
Thank you in advance. Any pointers/help will be appreciated.
It's not likely, since your system would normally react way quicker than a user can submit requests. However, that may be the case during a load peak.
To really ensure message order, you really need a single bottleneck (a single instance/thread) in your solution to handle all requests. That is, you need to make sure your processing strategy in Mule is synchronous and that you only have a single consumer on the VM queue. If you have a HA setup with multiple Mule servers, you may have potential to get messages out of order. In that case, and if the user initially is connected using HTTP, you can get around most of the problem using a load balancer with a sticky session strategy.
A perhaps more robust and scalable solution is to make sure the user submits it's local timestamp on each request with high resolution. Then you can make sure to discard any "obsolete" updates when storing the information into a database. However, that is not in the mule VM/JMS layer, but rather in the database.
For testability - no, I don't think there is a truly satisfying way to be 100% sure messages won't come out of order during any condition by just writing integration tests or performing manual tests. You need to verify the message path theoretically to make sure there is no part where one message can bypass another.
I have the problem that I have to run very long running processes on my Webservice and now I'm looking for a good way to handle the result. The scenario : A user executes such a long running process via UI. Now he gets the message that his request was accepted and that he should return some time later. So there's no need to display him the status of his request or something like this. I'm just looking for a way to handle the result of the long running process properly. Since the processes are external programms, my application server is not aware of them. Therefore I have to wait for these programms to terminate. Of course I don't want to use EJBs for this because then they would block for the time no result is available. Instead I thought of using JMS or Spring Batch. Does anyone ever had the same problem or an advice which solution would be better?
It really depends on what forms of communication your external programs have available. JMS is a very good approach and immediately available in your app server but might not be the best option if your external program is a long running DB query which dumps the result in a text file...
The main advantage of Spring Batch over "just" using JMS as an aynchronous communcations channel is the transactional properties, allowing the infrastructure to retry failed jobs, group jobs together and such. Without knowing more about your specific setup, it is hard to give detailed advise.
Cheers,
I had a similar design requirement, users were sending XML files and I had to generate documents from them. Using JMS in this case is advantageous since you can always add new instances of these processes which can consume and execute the jobs in parallel.
You can use a timer task to check status or monitor these processes. Also, you can publish a message to a JMS queue once the processes are completed.