I can't find this information anywhere. I have two queues, #JmsListener(destination = "p1"), #JmsListener(destination = "p2"). How can I make sure I only process 1 message at a time, even though I am listening to 2 queues, and also how do I configure the polling of what queue I get messages from first, that is after processing a message I want to poll p1 first. Or do weighted polling: p1:90%, p2:10%. Etc.
Basically I am asking how to implement priority processing of messages for Spring. I'm using SQS which doesn't support priorities.
Use one of the JmsTemplate receive() or receiveAndConvert() methods instead of the message-driven model.
Use transactions if you want to ensure no message loss.
Related
I have a RM queue test-queue. In there I need to handle 3 separate messages (message-1, message-2, message-3) for 3 separate processes in 3 separate services.
I use #RabbitListener like below to access the message
#RabbitListener(queues = "test-queue")
public void getMessage1(Message message) {
System.out.println(message);
}
but I need to access specific message ex:message-1 only in this function.
Any heads up?
That’s wrong design for AMQP protocol . You need to think about 3 different queues for those messages and correct bindings for them from a single exchange with the proper routing. Then you can easy have 3 consumers from those queues.
My point is that queue entity is a consumer responsibility. The producer just dump a message into an exchange. So, you just dictate from your consumer application how you’d like to get produced messages.
Out of subject you can investigate a Spring Integration router pattern implementation if you really can’t change your RabbitMQ structure : https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#messaging-routing-chapter
I'm trying to configure a queue that is aware of the events that are being processed.
Questions
Does this make sense? :)
Is it possible to configure/customize ActiveMQ?
Are there any other library that can be "easily" configured to handle such cases? Kafka?
Problem
The queue contains events. Each event is associated with an object. A consumer takes the event from the queue and performs a task. Each event should be taken only by exactly one consumer.
Constraints
Events for the same object cannot be processed concurrently.
But events for different objects should be processed in parallel.
Example
The queue is
ObjectA-Event1
ObjectA-Event2
ObjectB-Event1
ObjectC-Event1
The Consumer1 should receive ObjectA-Event1 from the queue. The Consumer2 should receive ObjectB-Event1 from the queue and not the ObjectA-Event2. The ObjectA-Event2 should be available for consumers only when the first consumer completes the task for the ObjectA-Event1.
It looks to me like you should use message groups. Messages for each object should be in the same group so that they are received by the same consumer and processed serially. Messages in different groups are free to be processed by different consumers.
What can be the best way to aggregate messages from many different sources (actually queues/topics) into a single queue/topic and then consume it. I am trying to design an application to receive messages from different topics in JMS using weblogic.
You could write your own "aggregator" as a stand-alone Java application:
For each queue/topic have a reader in its own thread.
Each reader sends its received message again on a "aggregate queue".
Have another thread to listen on the "aggregate queue".
As a variation, you could use a JVM Queue (like java.util.concurrent.ArrayBlockingQueue) as the "aggregate queue". This is faster, does not require another MQ queue, does not need network bandwidth, but it's not persistent.
Another idea is to use a "Message driven bean (MDB)" for each incoming queue/topic:
Again, each of these MDBs just reads the message and resends it to the "aggregate queue".
Have another MDB listening on the "aggregate queue".
A few suggestions on quality requirements. I belive you have to consider them.
They will be highly relate with your technical solution.
is that message loss acceptable?
client ack could be considered.
e.g. A memory queue sit in middle, e.g. incoming queue1...n -> ArrayBlockingQueue in memory -> outgoing queue. The data in the ArrayBlockingQueue , will lost when app crash.
is that message duplicate acceptable for the single outgoing queue?
I would suggest yes.
Set applicable level PossibleDuplicateFlag to make the client aware of that.
how fast the incoming messages per second on the diff incoming queue?
one queue session has only a uniqe thread. Performance has to be considered in advance.
I have a JMS queue published by a third party.
I want to setup multiple consumers on different machines, with only one particular machine's consumer, acknowledging messages on that queue. In short, if a particular machine's consumer does not receive the message, then that message should not be removed from queue.
Is this achievable ?
Okay, you might have your reasons for this setup and it's easy to achieve.
I would go with local session transactions. It is rather easy to commit or rollback the transactions acording to some critera, such as which server is consuming the message. If rolled back, the message will end up first in the queue again.
Sample code might look like this:
public class MyConsumer implements MessageListener{
Session sess;
public void init(Connection conn, Destination dest){
// connection and destination from JNDI, or some other method.
sess = conn.createSession(true, Session.AUTO_ACKNOWLEDGE);
MessageConsumer cons = sess.createConsumer(dest);
cons.setMessageListener(this);
conn.start();
}
#Override
public void onMessage(Message msg) {
// Do whatever with message
if(isThisTheSpecialServer()){
sess.commit();
}else{
sess.rollback();
}
}
private boolean isThisTheSpecialServer(){
// figure out if this server should delete messages or not
}
}
If you are doing this inside a Java EE container with JTA and you are using UserTransactions, you could just call UserTransaction.setRollBack();
or if you are using declarative transactions you could just throw a Runtime exception to make the transaction fail and rollback the message to the queue, once you have read the message and done things. Note that database changes will roll back as well with this approach (if you are using JTA and not local JMS transactions).
UPDATE:
You should really do this using transactions, not acknowledgement.
A summary of this topic (for ActiveMQ, but written generally for JMS) is found here.
http://activemq.apache.org/should-i-use-transactions.html
I don't know if this behaviour is consistent with all JMS implementations, but for ActiveMQ if you try to use a non transacted session with Session.CLIENT_ACKNOWLEDGEMENT, then it will not really behave as you expect. A message that has been read, but not acknowledged, is still on the queue, but will not get "released" and delivered to other JMS consumers until the connection is broken to the first consumer (i.e. connection.close(), a crash or similar).
Using local transactions, you can controll this by session.commit() and session.rollback() explicitly. I see no real point in not using transactions. Acknowledgement is just there to guarantee delivery.
Another way to look at this is in the case of a forwarding queue. You could apply it to your design by doing the following:
Create a consumer on the published queue from the third party.
This consumer has one job - distribute every message to other queues.
Create additional queues that your real subscribers will listen to.
Code your message listener to take each message and forward it to the various destinations.
Change each of your listeners to read from their specific queue.
By doing this, you ensure that every listener sees every message, every transaction works as expected, and you don't make any assumptions about how the message is being sent (for example, what if the publisher side is doing AUTO_ACKNOWLEDGE ?)
I was wondering what is the difference between a JMS Queue and JMS Topic.
ActiveMQ page says
Topics
In JMS a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues
A JMS Queue implements load balancer semantics. A single message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
I want to have 'something' what will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker.
Any thoughts?
That means a topic is appropriate. A queue means a message goes to one and only one possible subscriber. A topic goes to each and every subscriber.
It is simple as that:
Queues = Insert > Withdraw (send to single subscriber) 1:1
Topics = Insert > Broadcast (send to all subscribers) 1:n
Topics are for the publisher-subscriber model, while queues are for point-to-point.
A JMS topic is the type of destination in a 1-to-many model of distribution.
The same published message is received by all consuming subscribers. You can also call this the 'broadcast' model. You can think of a topic as the equivalent of a Subject in an Observer design pattern for distributed computing. Some JMS providers efficiently choose to implement this as UDP instead of TCP. For topic's the message delivery is 'fire-and-forget' - if no one listens, the message just disappears. If that's not what you want, you can use 'durable subscriptions'.
A JMS queue is a 1-to-1 destination of messages. The message is received by only one of the consuming receivers (please note: consistently using subscribers for 'topic client's and receivers for queue client's avoids confusion). Messages sent to a queue are stored on disk or memory until someone picks it up or it expires. So queues (and durable subscriptions) need some active storage management, you need to think about slow consumers.
In most environments, I would argue, topics are the better choice because you can always add additional components without having to change the architecture. Added components could be monitoring, logging, analytics, etc.
You never know at the beginning of the project what the requirements will be like in 1 year, 5 years, 10 years. Change is inevitable, embrace it :-)
Queues
Pros
Simple messaging pattern with a transparent communication flow
Messages can be recovered by putting them back on the queue
Cons
Only one consumer can get the message
Implies a coupling between producer and consumer as it’s an one-to-one relation
Topics
Pros
Multiple consumers can get a message
Decoupling between producer and consumers (publish-and-subscribe pattern)
Cons
More complicated communication flow
A message cannot be recovered for a single listener
As for the order preservation, see this ActiveMQ page. In short: order is preserved for single consumers, but with multiple consumers order of delivery is not guaranteed.
If you have N consumers then:
JMS Topics deliver messages to N of N
JMS Queues deliver messages to 1 of N
You said you are "looking to have a 'thing' that will send a copy of the message to each subscriber in the same sequence as that in which the message was received by the ActiveMQ broker."
So you want to use a Topic in order that all N subscribers get a copy of the message.
TOPIC:: topic is one to many communication... (multipoint or publish/subscribe)
EX:-imagine a publisher publishes the movie in the youtub then all its subscribers will gets notification....
QUEVE::queve is one-to-one communication ...
Ex:-When publish a request for recharge it will go to only one qreciever ...
always remember if request goto all qreceivers then multiple recharge happened so while developing analyze which is fit for a application
Queue is JMS managed object used for holding messages waiting for subscribers to consume. When all subscribers consumed the message , message will be removed from queue.
Topic is that all subscribers to a topic receive the same message when the message is published.