I have a question regarding the behavior of a jdbc message store. I notice that if a failure occurs for instance if the following sequence is followed:
1)inbound-channel-adapter - Creates a message payload and sends it to a channel
2)The channel is polled for the payload by a jms:outbound-channel-adapter which adds an entry in the JMS queue. I am using ActiveMQ.
When I test for a situation like shutting down the MOM I would like to persist the message in the message-store, but this is not the default behavior. Looks like SI processes the payload in the channel and then pushes it out (deletes it from the store). I can see the message being inserted in the message-store in the database.
Am I wrong in understanding the function of a message-store? I thought that the message would persist until a successful run. Thanks for any feedback.
You need to set up durable subscriptions for that. I think most of the SI JMS components do support that.
Related
Hi I am new to Camel and have a design question related to JMS queues.
I am receiving set of data. These data have a reference date. These data are sent every 15 minutes by a batch process.
I have to process the received data and forward them to another route.
If a given data cannot be processed, I need to reprocess it. And I have to ensure it is processed before the next data set is processed.
So I was thinking of creating a JMS route to receive these data before processing. Then process it. Then send it to another queue.
FTP --> Process data rows (A) --> JMS Queue --> Processor (B) --> direct:call
If processor B fails I want the data to be processed before the next data set is sent by FTP. (because second data set may contain an update of the data of the first dataset)
So I was thinking using a queue, to make sure they are always processed in the order they are being received.
But my experience with JMS, without Camel, is that once the object is consumed from the queue it is not in the queue anymore.
Is it also the case with Camel?
In this case to I have to retry to process the data, or put them back in the queue?
This "recovery" part is not clear to me and I'd like to understand the patterns that do support this.
Many thanks for your help
Gilles
This part "once the object is consumed from the queue it is not in the queue anymore." is not fully correct. Actually, when you are subscribing to the queue and getting a message you need to process it and send acknowledge back to the JMS broker. If acknowledge is successful then the message will be removed from the queue. But if acknowledge will be not successful or if your process will die and connection to the broker will break then the message will not be removed from the queue and will be passed to another consumer.
Often most of the JMS libraries are using mode when acknowledgement is sent right when message was received by consumer but you always have possibility to change this mode and send acknowledgement manually when your processing part will be finished successfully.
What about camel jms (http://camel.apache.org/jms.html) you can use endpoint option "acknowledgementModeName" which has some different possible values like:
AUTO_ACKNOWLEDGE (default) - acknowledgement will be sent right after corresponded "from" in your route
CLIENT_ACKNOWLEDGE - allows the application to control when the acknowledgment is sent and if there are no exceptions will be thrown during exchange processing then message will be acknowledged and removed from queue.
Is there any way you can control the order delivery of messages in a topic in EMS or Tibco (using a JMS Topic Subscriber)? Something like the message selector, but instead of filtering to do ordering.
I would like to use a header like JMSXDeliveryCount, so that new messages will get a higher priority. I know that there's RedeliveryDelay, but that works only for queues, not topics.
Even JMSPriority could be an option if I can set it after getting the message with a topic subscriber. Can I do that? Maybe with a Java Code activity?
The broker will deliver messages in order as it receives them, but re-ordering messages is a distributed computing / EIP problem (see sequencer). The issue is the broker does't know how many messages are coming or how long to keep the window open for re-ordering messages. This activity is best done outside the broker in your application where that information is understood.
EIP frameworks and ESB's have done a nice job of standardizing a handler for this type of workflow.
ref: EIP Sequencer http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageSequence.html
JMSPriority is indeed an option, but it cannot be set on an already received message. The solution is to confirm the message and republished it with a different priority.
Currently I'm using bluelock's camel-spring-amqp component for my application.
What I want to achieve is:
Pull a message from RabbitMQ server.
Persist it to a database on successful processing / Send it to another "Error" queue on Exception
Tell the original queue that it is now safe to remove the message from the queue.
As of this writing, I'm able to pull from rabbit and persist to database using camel routes. What I don't really know how to do is acknowledge that my processing is done to the original queue. Here is my current route:
from("spring-amqp:EXCHANGE:queuename?autodelete=false&durable=true&type=direct&routingKey=")
.bean(Transform.class, "transform(byte[])")
.to("jpa:com.my.company.models.MyModel?entityType=java.util.ArrayList")
I realize I can set the acknowledgmentMode to NONE. But I don't know how to "manually" acknowledge once I have persisted my message.
Thanks for any help!
I'm new in Camel but I know a thing or two about RabbitMQ.
With RabbitMQConsumer the message is acknowledged if the processor doesn't throw any exception (line 133 at RabbitMQConsumer source).
So I suppose if you let your processor propagate the exception, the message won't be acknowledged. I haven't used spring-amqp but I guess it should have a similar behaviour.
I am using weblogic 10.3 .
I am trying to configure a durable subscription with persistent messaged backed by a jdbc store (in Oracle DB). I have a topic to which an MDB is listening as a durable subscriber.
Under scenario-1 : If I send the message, it hits the MDB.
Under scenario-2 : I suspend the MDB hoping that the messages send to the topic will stay around as long as they don’t get consumed by the MDB (which is the only registered durable subscriber). But when I do send the message to the topic, it briefly shows up there and then it goes away (i see it using HermesJMS).
I was under the impression that the messages since they are not being consumed by the MDB will get logged into the JDBC store in this case but the the WLSTORE table in oracle db does not get any messages persisted in it either.
I later found out that the messages do show up in the topic > durable subscribers > Show messages in the admin console.
So apparantly what’s happening is topic doesn’t keep the message, but the durable subscription that’s registered under it, keeps it until the message does get consumed.
Question -1) But the fact that messages are not going to the Oracle based jdbc store, still doesn’t make sense ? What am I doing wrong here ?
Question -2) Even when I do resume the MDB to start listening for the messages, the topic keeps showing all the message still intact under the durable subscribers (in admin console) - I was hoping they will get removed from there as they got processed by the only registered durable subscriber.
Messages do not show up in the topic, since topics and queues are different communication models. Assume you have a durable topic with 2 (durable) subscribers: A and B. You want both of them to get the message. To ensure that, A and B both have to confirm they received the message.
This is also the reason why you get all message redelivered, after reconnecting, your MDB has to call commit() on the message, to tell the server it is done processing.
This also explains why the topic itself does not store messages, they are stored per durable consumer. Because A might commit the message, but B will not (might be "offline"). So you need a copy for each consumer.
I found a couple of interesting things in further testing -
for Question 1) _ Even if I don't configure the jdbc store for the JMS server in weblogic 10, it has its own default file store which is always working without any configuration. This file store is used for storing the persistent messages for durable subscriptions and that store will keep the messages around even across server restarts.
For more reading - http://docs.oracle.com/cd/E17904_01/web.1111/e13701/store.htm#i1130575
for Question 2)_ My MDB was expecting XML messages in certain format, during my testing to keep things simple, i started sending small text messages and forgot that they will not get processed successfully in the MDB. so the MDB was failing on all those messages, and rolling back the transaction which is why the messages were never getting removed when I resumed the MDB.
This pretty much answers both my questions.
I want to make an application where:
Users will send a message to queue
Listener will listen messages of
queue After processing of message in
listener, listener
will create a new response message
and send it to another success or
failure queue.
Questions:
Should I use activemq as storage of processed messages?
Will I be able to retrieve all messages of one queue without listing to them?
Do we have any other solution for keeping all processed messages? I want to make a report of all processed messages present in success and failure queue.
You can find a minimalistic sample here and here for using ActiveMQ with Spring. About persistence options, read ActiveMQ docs. Also you might want to check out these slides, to get a general overview on Spring JMS with ActiveMQ.
ActiveMQ is not a storage facility, it is a message-passing facility.
If you want to store the messages after processing them then use a database. For example, create a table that has a status flag for success or failure, then for reporting query that table.