I want to make an application where:
Users will send a message to queue
Listener will listen messages of
queue After processing of message in
listener, listener
will create a new response message
and send it to another success or
failure queue.
Questions:
Should I use activemq as storage of processed messages?
Will I be able to retrieve all messages of one queue without listing to them?
Do we have any other solution for keeping all processed messages? I want to make a report of all processed messages present in success and failure queue.
You can find a minimalistic sample here and here for using ActiveMQ with Spring. About persistence options, read ActiveMQ docs. Also you might want to check out these slides, to get a general overview on Spring JMS with ActiveMQ.
ActiveMQ is not a storage facility, it is a message-passing facility.
If you want to store the messages after processing them then use a database. For example, create a table that has a status flag for success or failure, then for reporting query that table.
Related
Is it possible to keep a history of messages (with message content would be perfect) that have already been retrieved and are no longer on a queue?
In the application I can see when the sender attempts to put the message in the queue and when the receiver attempts to pick the messages up, but I'd like to see when the message really arrived into the queue and when the messages were really received.
Does MQ Explorer have this function? How would I use it?
What you are looking for is a message tracking/auditing software for IBM MQ. You can find a list of what is available here.
It is possible to use an API exit to make copies of messages in a queue or to audit both PUT and GET operations.
It is also possible to put messages to a topic, then create as many administrative subscriptions to destination queues as required. Something can then GET and log messages from one of those destination queues. The problem with this is that MQ changes the message ID between publication and consumption whereas in a queue it remains static.
There is no native MQ function to capture messages. It's possible to use linear logs and later scrape the logs but these do not necessarily capture all messages due to optimization. (A message PUT to a waiting getter outside of syncpoint for example.) However there is at least one commercial product to scrape linear transaction logs to audit message activity.
The philosophy of MQ in general is that it is the delivery mechanism and deals with envelope data to route and deliver but does not deal with payload data. WAS, IIB and other broker/transformation engines are where IBM has put all of the functions that deal with message payloads.
Hi I am new to Camel and have a design question related to JMS queues.
I am receiving set of data. These data have a reference date. These data are sent every 15 minutes by a batch process.
I have to process the received data and forward them to another route.
If a given data cannot be processed, I need to reprocess it. And I have to ensure it is processed before the next data set is processed.
So I was thinking of creating a JMS route to receive these data before processing. Then process it. Then send it to another queue.
FTP --> Process data rows (A) --> JMS Queue --> Processor (B) --> direct:call
If processor B fails I want the data to be processed before the next data set is sent by FTP. (because second data set may contain an update of the data of the first dataset)
So I was thinking using a queue, to make sure they are always processed in the order they are being received.
But my experience with JMS, without Camel, is that once the object is consumed from the queue it is not in the queue anymore.
Is it also the case with Camel?
In this case to I have to retry to process the data, or put them back in the queue?
This "recovery" part is not clear to me and I'd like to understand the patterns that do support this.
Many thanks for your help
Gilles
This part "once the object is consumed from the queue it is not in the queue anymore." is not fully correct. Actually, when you are subscribing to the queue and getting a message you need to process it and send acknowledge back to the JMS broker. If acknowledge is successful then the message will be removed from the queue. But if acknowledge will be not successful or if your process will die and connection to the broker will break then the message will not be removed from the queue and will be passed to another consumer.
Often most of the JMS libraries are using mode when acknowledgement is sent right when message was received by consumer but you always have possibility to change this mode and send acknowledgement manually when your processing part will be finished successfully.
What about camel jms (http://camel.apache.org/jms.html) you can use endpoint option "acknowledgementModeName" which has some different possible values like:
AUTO_ACKNOWLEDGE (default) - acknowledgement will be sent right after corresponded "from" in your route
CLIENT_ACKNOWLEDGE - allows the application to control when the acknowledgment is sent and if there are no exceptions will be thrown during exchange processing then message will be acknowledged and removed from queue.
Let's suppose I have several subscribers consuming from a topic. After a message has been delivered to all the subscribers I'd like to trigger a job that would use this message in input.
So the easy way to do that would be to move messages that have been succesfully delivered to all the sucscribers to a queue from which my job would consume messages.
Is it part of JMS?
Is there any message broker able to do that directly?
If not is there a simple solution to solve this problem?
You should be able to do this using activemq's advisories.
See here for more about advisory messages: http://activemq.apache.org/advisory-message.html
So what you want to do, for the topic in question, is track:
the number of consumers
when a message is dispatched to them
when the message has been ack'd by each of the consumers
to get the number of consumers, listen to the "ActiveMQ.Advisory.Consumer.Topic." advisory topic
to get when a message is dispatched, listen to the "ActiveMQ.Advisory.MessageDelivered.Topic."
to get when a message has been ack'd, listen to "ActiveMQ.Advisory.MessageConsumed.Topic."
you could easily use Apache Camel to help out with this (listening to the topics) and aggregating whether or not all consumers have processed (ack'd) the message.. then that could kick off your further processing..
You could just create another durable subscription to route the message from the topic to queue directly. From that queue your job can consume messages. This is much easier than creating a trigger to route the messages to a queue.
So the easy way to do that would be to move messages that have been
succesfully delivered to all the sucscribers to a queue from which my
job would consume messages. Is it part of JMS?
No, this is not part of JMS specification.
I have a question regarding the behavior of a jdbc message store. I notice that if a failure occurs for instance if the following sequence is followed:
1)inbound-channel-adapter - Creates a message payload and sends it to a channel
2)The channel is polled for the payload by a jms:outbound-channel-adapter which adds an entry in the JMS queue. I am using ActiveMQ.
When I test for a situation like shutting down the MOM I would like to persist the message in the message-store, but this is not the default behavior. Looks like SI processes the payload in the channel and then pushes it out (deletes it from the store). I can see the message being inserted in the message-store in the database.
Am I wrong in understanding the function of a message-store? I thought that the message would persist until a successful run. Thanks for any feedback.
You need to set up durable subscriptions for that. I think most of the SI JMS components do support that.
I have been reading on JMS, ActiveMQ and Camel.
From my understanding, if we need to integrate multiple system where integration involves only routing of messages in same format, activeMQ is sufficient else we can opt for camel.
Now incase of message redelivery expiration caused by some exception or transaction rollback, activeMQ sends the failed message to dead letter queue.
We are not using Apache camel as activeMQ is sufficient for our requirements but should I opt for camel if I want to monitor and audit the failed messages or will it add an overhead ?
I have not used camel before so if someone can direct me how to monitor dead letter messages. We are thinking of having dead letter queue for each of our topic or queue and provide audit logs for client.
Depends. What do you want to do with DLQ messages? How many errors do you anticipate?
If the error rate is very low and it's hard to predict what to do with errors, maybe some alert could trigger a human to look at the DLQ message on DLQ to figure out what to do with it. Retry? File a bug report? Call someone?
To monitor DLQs you could use JMX or it's http equivalent, Jolokia, to monitor the queues and send an alert if the queue size is more than zero.
You can of course use Apache Camel to somehow process the dead message. Like send it in an email, save it to an error reporting system or take other action to alert a user, use different logic to process the message or what not. This logic you can write in any framework, not just Apache Camel. But Camel is a decent choice.
Anyhow, there is no silver bullet for DLQ management. You must build a system that (manual or automatic) that fits your needs and limits.