I have an backlogged topic of ActiveMQ messages for customer data in a production environment, and I need to write a script that dequeues it and does whatever business logic is necessary.
If something goes wrong in the business logic, but I've already read the message (via JMS probably), that would presumably mean all the messages are gone - and I'd be in huge trouble.
Is there a way to read the messages without deleting them?
This is a common pattern in messaging. If you're using the JMS API you have a couple of options:
Use CLIENT_ACKNOWLEDGE mode when you create your session and acknowledge the message once the business logic is complete. If the business logic fails don't acknowledge the message and it won't be removed from the queue.
Use a transacted session. If the business logic completes successfully then commit the transaction. If the business fails then rollback the transaction.
Both of these are very common and you can find more information about all the proper API calls, etc. using your favorite search engine.
Related
I have REST API gateway which calls one of the microservices with MassTransit request client. This request is not durable and is meant to live for a short time - essentially it's just replacement of "traditional" synchronous (via HTTP/GRPC/etc) gateway-microservice communication.
On microservice side I have consumer which under the hood uses DbContext and Transaction (EFC) to perform some work in database. After the work is done it should publish "WorkDoneEvent" (to be consumed later by other microservices) and return result of the work to api gateway. Event must be published atomically along with transaction used to perform the work. It does not matter if ApiGateway will receive response / will retry request - as soon as transaction is commited both work result and sending "WorkDoneEvent" must be guaranteed.
Normally this is done with transactional outbox which first saves published event to database within same transaction as the work is done. (And then some process constantly "polls" outbox and tries send message to the broker, when done it removes message from outbox). As far as I know.
MassTransit seems to have transactional outbox built in: https://masstransit-project.com/advanced/middleware/transactions.html#transactional-bus.
However in docs it clearly states:
Never use the TransactionalBus or TransactionalEnlistmentBus when writing consumers. These tools are very specific and should be used only in the scenarios described.
And this is exactly what I want to do...
Why I should not do it?
I'd suggest using the InMemoryOutbox, which is part of MassTransit. It's significantly lighter weight, is designed to work in a consumer, and will not publish your events until after the consumer has completed (but prior to acknowledging the message at the broker). The only consideration is that your consumer should be idempotent (which needs to be the case in your approach as well) and if the operation was already performed on a retry, it should republish the events.
There are videos, articles, and a sample to go along with it.
Is it possible to keep a history of messages (with message content would be perfect) that have already been retrieved and are no longer on a queue?
In the application I can see when the sender attempts to put the message in the queue and when the receiver attempts to pick the messages up, but I'd like to see when the message really arrived into the queue and when the messages were really received.
Does MQ Explorer have this function? How would I use it?
What you are looking for is a message tracking/auditing software for IBM MQ. You can find a list of what is available here.
It is possible to use an API exit to make copies of messages in a queue or to audit both PUT and GET operations.
It is also possible to put messages to a topic, then create as many administrative subscriptions to destination queues as required. Something can then GET and log messages from one of those destination queues. The problem with this is that MQ changes the message ID between publication and consumption whereas in a queue it remains static.
There is no native MQ function to capture messages. It's possible to use linear logs and later scrape the logs but these do not necessarily capture all messages due to optimization. (A message PUT to a waiting getter outside of syncpoint for example.) However there is at least one commercial product to scrape linear transaction logs to audit message activity.
The philosophy of MQ in general is that it is the delivery mechanism and deals with envelope data to route and deliver but does not deal with payload data. WAS, IIB and other broker/transformation engines are where IBM has put all of the functions that deal with message payloads.
It looks like ATG's implementation of JMS is achieved through scheduler whereby it polls database in specific interval through SqlJmsProvider component. I do agree that ATG's DMS does provide with all JMS feature like Queue, Topic, Durable Subscribers, Retry etc... but isin't using a scheduler to poll the DB to send ATG Order to Fulfillment an overkill ? (too many queries fired)
From the ATG Documentation it explains that there are two JMS providers available within ATG:
SQL JMS
Local JMS
The difference between the two are:
Local JMS is Synchronous and extremely fast. It runs within a single transaction and is bound to a single process, thus when sending a message, it blocks while waiting for an acknowledgement. SQL JMS on the otherhand, is Asynchronous and can be used across processes (thus an order submission on Commerce can be processed on Fulfillment). SQL JMS is non-blocking so once the message is put on the queue, the requesting process can continue. This also means that Commerce can continue running, even if Fulfillment is down. The messages are also persisted in SQL JMS while they are stateless and lost during a restart for Local JMS.
Using the scheduler to poll the queue is an acceptable solution and most of the older asynchronous message queues implemented this solution. In IBM MQ Version 7 performance has been improved by reducing the amount of polling while using IBM WAS Version 6 for example the solution is also based on regular polling of the queue.
So no, polling the database on a scheduled interval is not an overkill.
I am wondering how we can ensure message durability when using websphere MQ and WCF. I want to be able to have my WCF process pick messages off of the queue and if there is an issue that the applciation encounters (power outage, etc) I don't lose the messages. I also would like to not have to use a transaction if at all possible because I want to eliminate distributed transactions.
Thanks,
S
Well, there's transactions and there's distributed transactions. The "right" answer is to use the WMQ 1-phase commit here. That doesn't have the complexity of XA transactions but it does give you the ability to roll back a message without losing it. In fact, when using clients you really should be using at least 1-phase commit just to prevent loss of messages.
Short of that there is always the "browse-with-lock, delete-message-under-cursor" method. I'm pretty sure everything you need to do the browseing, locking and deleting is exposed under .NET but perhaps Shashi will comment and confirm.
WebSphere MQ WCF custom channel has a feature "Assured Delivery" that guarantees that a service request or reply is actioned and not lost. This is the 1-phase commit (also known as SYNC_POINT in) WMQ.
"Assuered Delivery" is a service contract attribute. Here are more details about the feature.
I’m writing a server/client game, a typical scenario looks like this: one client (clientA) send a message to the server, there is a MessageDrivenBean in server to handle such messages. After the MDB finished its job, it sends the result message back to another client (clientB).
In my opinion I only need two queues for such communication, one for input the other for output. Creating new queue for each connection is not a good idea, right?
The Input queue is relative clear, if more clients are sending message at the same time, the messages are just waiting in the queue, while there are more MDB instances in server, that should not a big performance issue.
But on the other side I am not quite clear about the output queue, should I use a topic instead of a queue? Every client is listening the output queue, one of them gets the new message and checks the property to determine if the message is to it, if not, it rollback the transaction, the message goes back to queue and be ready for other client … It should work but must be very slow. If I use topic instead, every client gets a copy of the message, if it’s not to it, just ignores the message. It should be better, right?
I’m new about message system. Is there any suggestion about my implementation? Thanks!
To begin with, choosing JMS as a gaming platform is, well, unusual — businesses use JMS brokers for delivery reliability and transaction support. Do you really need this heavy lifiting in a game? Shouldn't you resort to your own HTTP-based protocol, for example?
That said, two queues are a standard pattern for point-to-point communication. Creating a queue for a new connection is definitely not OK — message-driven beans are attached to queues at deployment time, so you won't be able to respond to queue creation events. Besides, queues are not meant to be created and destroyed in short cycles, they're rather designed to be long-living entities. If you need to deliver a message to one precise client, have the client listen on the server response queue with a message selector set to filter only the messages intended for this client (see javax.jms.Message API).
With topics it's exactly as you noted — each connected client will get a copy of the message — so again, it's not a good pattern to send to n clients a message that has to be discarded by n-1 clients.
MaDa;
You could stick one output queue (or topic) and simply tag the message with a header that identifies the intended client. Then, clients can listen on the queue/topic using a selector. Hopefully your JMS implementation has efficient server-side listener evaluation.