I work on a task that requires:
consuming data from JMS;
processing it;
loading it into a database.
As the documentation suggests:
I start with <int-jms:message-driven-channel-adapter channel="CHANNEL1" ... /> to send new JMS messages to the CHANNEL1 channel;
I apply the transformer that converts messages from the CHANNEL1 channel to JobLaunchRequest with a job that inserts data to the database and the payload that contains original JMS message's payload;
The transformed messages go to the CHANNEL2 channel;
<batch-int:job-launching-gateway request-channel="CHANNEL2"/> starts a new job execution when a new message appears in the channel;
The problem is that I start a new database transaction each time a new jms messages received.
The question: how should I handle such a flow? What is the common pattern for this?
UPDATE
I start the job for each message. One message contains one piece of data. If I resort in just using spring-batch then I will have to manage some sort of a poller (correct me if I am wrong), but I would like to apply a message driven approach like (either one):
Grace period: when a new message appears I wait for 10 more messages or start processing everything I received 10 seconds after the first message is received.
I simply read everything the JMS queue contains after I got notified that the queue contains a new message.
Of course, I would like the solution to be transnational: the order of message processing does not matter.
The BatchMessageListenerContainer can be used in your use case. It enables the batching of messages within a single transaction.
Note this class is not part of the main framework, it’s actually a test class, but you can use it if it fits your needs.
Hope this helps.
Related
I can't find this information anywhere. I have two queues, #JmsListener(destination = "p1"), #JmsListener(destination = "p2"). How can I make sure I only process 1 message at a time, even though I am listening to 2 queues, and also how do I configure the polling of what queue I get messages from first, that is after processing a message I want to poll p1 first. Or do weighted polling: p1:90%, p2:10%. Etc.
Basically I am asking how to implement priority processing of messages for Spring. I'm using SQS which doesn't support priorities.
Use one of the JmsTemplate receive() or receiveAndConvert() methods instead of the message-driven model.
Use transactions if you want to ensure no message loss.
I need help with a problem with gcp pub/sus. I have a process that send 100 messages with filters to pubsub and another application (in spring boot) receive these messages. When spring boot application receive message from pubsub (not pull), process 100 messages but, into the process, receive more messages, in diferents times receive diferents numbers of messages, any times receive 120, another 140, and the others more than 200. I wasn't found any solution of this, this is my code:
#Bean
#ServiceActivator(inputChannel = "pubsubInputChannel")
public MessageHandler messageReceiver() {
return message -> {
System.out.println("Message arrived! Payload: " + new String((byte[]) message.getPayload()));
//other process of app (call other api)
AckReplyConsumer consumer = (AckReplyConsumer) message.getHeaders().get(GcpPubSubHeaders.ACKNOWLEDGEMENT);
consumer.ack();
};
}
please help me!!!
Duplicate messages can happen for different reasons in Google Cloud Pub/Sub. One thing to keep in mind is that Cloud Pub/Sub offers at-least-once delivery meaning that some amount of duplicates is always possible, so your application must be resilient to them. That many duplicates does seem a bit high, though. In general duplicates can generally happen for the following reasons:
Messages are being sent by the publisher more than once. This can happen if the publisher got disconnected from Cloud Pub/Sub and sent the same message again. If this type of duplication occurs, then the messages will have different message IDs.
The subscriber is taking too long to acknowledge messages. In your code, you have //other process of app (call other api). How long does this process take? If it is longer than the deadline for acknowledging the message, then the message will be redelivered. Keep in mind that if this other process requires locks be grabbed for all messages, there could be a contention issue with too many requests trying to get those locks at the same time, resulting in processing delays. By default, the ack deadline for a message is ten seconds. When using the Java client library, the deadline is automatically extended by the maxAckExtensionPeriod, which defaults to one hour. This property can be set in the DefaultSubscriberFactory for Spring as well.
Messages are not acked at all. If an exception prevents the call to ack or there is deadlock resulting in that line of code never being reached, then the message will be redelivered.
The use case is one of a large backlog of small messages. In this situation, buffers are prone to fill up in the client in a way that results in redelivery of messages.
We are using IBM WebSphere MQ as JMS provider with Spring MDP (Message Driven POJO).
Is there any way in JMS where we can configure time related properties in message so that message can be consumed at particular defined time only?
For example, if I am sending three messages into queue M1, M2 and M3. Where, I can configure M2 message property let say 3 AM. And consumer side, consumer can only pick this message # 3 AM only. If time is not defined, messages should be consumed in a way that JMS Receiver does.
JMS 2.0 specification has defines Delivery Delay. With this feature a message producer can specify that a message must not be delivered until after a specified time interval. The message will be available for delivery after the specified time. But this may not help you as you want to a message to be consumed at a specified time. Typically messaging applications are designed to consume messages as soon as they are made available by the messaging provider.
If you want to process messages at a specified time only, you could create another queue "queue_3am", and schedule a reader to run exactly at 3am.
A variation is to set the timestamp as a message property. So one queue could contain messages to be processed at different points in time. The reader could use message selectors to get relevant messages only.
But you should use a "message pickup timeframe" by adding two timestamps as message properties, for eaxmple set the window to 1 or 5 minutes.
The receiver can use a message selector: A selector is a condition using message properties.
Have a look at this
I have a JMS queue published by a third party.
I want to setup multiple consumers on different machines, with only one particular machine's consumer, acknowledging messages on that queue. In short, if a particular machine's consumer does not receive the message, then that message should not be removed from queue.
Is this achievable ?
Okay, you might have your reasons for this setup and it's easy to achieve.
I would go with local session transactions. It is rather easy to commit or rollback the transactions acording to some critera, such as which server is consuming the message. If rolled back, the message will end up first in the queue again.
Sample code might look like this:
public class MyConsumer implements MessageListener{
Session sess;
public void init(Connection conn, Destination dest){
// connection and destination from JNDI, or some other method.
sess = conn.createSession(true, Session.AUTO_ACKNOWLEDGE);
MessageConsumer cons = sess.createConsumer(dest);
cons.setMessageListener(this);
conn.start();
}
#Override
public void onMessage(Message msg) {
// Do whatever with message
if(isThisTheSpecialServer()){
sess.commit();
}else{
sess.rollback();
}
}
private boolean isThisTheSpecialServer(){
// figure out if this server should delete messages or not
}
}
If you are doing this inside a Java EE container with JTA and you are using UserTransactions, you could just call UserTransaction.setRollBack();
or if you are using declarative transactions you could just throw a Runtime exception to make the transaction fail and rollback the message to the queue, once you have read the message and done things. Note that database changes will roll back as well with this approach (if you are using JTA and not local JMS transactions).
UPDATE:
You should really do this using transactions, not acknowledgement.
A summary of this topic (for ActiveMQ, but written generally for JMS) is found here.
http://activemq.apache.org/should-i-use-transactions.html
I don't know if this behaviour is consistent with all JMS implementations, but for ActiveMQ if you try to use a non transacted session with Session.CLIENT_ACKNOWLEDGEMENT, then it will not really behave as you expect. A message that has been read, but not acknowledged, is still on the queue, but will not get "released" and delivered to other JMS consumers until the connection is broken to the first consumer (i.e. connection.close(), a crash or similar).
Using local transactions, you can controll this by session.commit() and session.rollback() explicitly. I see no real point in not using transactions. Acknowledgement is just there to guarantee delivery.
Another way to look at this is in the case of a forwarding queue. You could apply it to your design by doing the following:
Create a consumer on the published queue from the third party.
This consumer has one job - distribute every message to other queues.
Create additional queues that your real subscribers will listen to.
Code your message listener to take each message and forward it to the various destinations.
Change each of your listeners to read from their specific queue.
By doing this, you ensure that every listener sees every message, every transaction works as expected, and you don't make any assumptions about how the message is being sent (for example, what if the publisher side is doing AUTO_ACKNOWLEDGE ?)
We have quartz process that polls a ActiveMQ JMS queue.
We know that we could get several messages a minute would like to only respond to the most current message at a configured polling rate of a minute or more.
We don't need to process any of the previous messages.
Is there a way to configure the queue to get this behavior?
Its seems like a topic has the ability to do this via the subscription recovery policy using a count of 1. We would like to do this using a queue to guarantee (more or less) a single delivery of the message.
Or is there a conceptual flaw in our assumptions...
Thanks
In my opinion there is no standard operation for this, so you will have to write some code....
One possible solution would be to use a QueueBrowser together with a QueueReceiver:
Through the QueueReceiver you would get an Enumeration of the messages in the queue. For each message you can now perform a receive with a MessageSelector on the JMSMessageID as long as hasMoreElements() returns true. The last message will be the one you want to have....
When using activemq, you can use "image caching" on topics. One of the settings there is to always keep the last mesage sent..
Take a look at the Subscription recovery Policy settings:
http://activemq.apache.org/subscription-recovery-policy.html