Attempting to implement a way to time my consumer to receive messages from a queue every 30 minutes or so.
For context, I have 20 messages in my error queue until x minutes have passed, then my route consumes all messages on queue and proceeds to 'sleep' until another 30 minutes has passed.
Not sure the best way to implement this, I've tried spring #Scheduled, camel timer, etc and none of it is doing what I'm hoping for. I've been trying to get this to work with route policy but no dice in the correct functionality. It just seems to immediately consume from queue.
Is route policy the correct path or is there something else to use?
The route that reads from the queue will always read any message as quickly as it can.
One thing you could do is start / stop or suspend the route that consumes the messages, so have this sort of set up:
route 1: error_q_reader, which goes from('jms').
route 2: a timed route that fires every 20 mins
route 2 can use a control bus component to start the route.
from('timer?20mins') // or whatever the timer syntax is...
.to("controlbus:route?routeId=route1&action=start")
The tricky part here is knowing when to stop the route. Do you leave it run for 5 mins? Do you want to stop it once the messages are all consumed? There's probably a way to run another route that can check the queue depth (say every 1 min or so), and if it's 0 then shutdown route 1, you might get it to work, but I can assure you this will get messy as you try to deal with a number of async operations.
You could also try something more exotic, like a custom QueueBrowseStrategy which can fire an event to shutdown route 1 when there are no messages on the queue.
I built a customer bean to drain a queue and close, but it's not a very elegant solution, and I'd love to find a better one.
public class TriggeredPollingConsumer {
private ConsumerTemplate consumer;
private Endpoint consumerEndpoint;
private String endpointUri;
private ProducerTemplate producer;
private static final Logger logger = Logger.getLogger( TriggeredPollingConsumer.class );
public TriggeredPollingConsumer() {};
public TriggeredPollingConsumer( ConsumerTemplate consumer, String endpoint, ProducerTemplate producer ) {
this.consumer = consumer;
this.endpointUri = endpoint;
this.producer = producer;
}
public void setConsumer( ConsumerTemplate consumer) {
this.consumer = consumer;
}
public void setProducer( ProducerTemplate producer ) {
this.producer = producer;
}
public void setConsumerEndpoint( Endpoint endpoint ) {
consumerEndpoint = endpoint;
}
public void pollConsumer() throws Exception {
long count = 0;
try {
if ( consumerEndpoint == null ) consumerEndpoint = consumer.getCamelContext().getEndpoint( endpointUri );
logger.debug( "Consuming: " + consumerEndpoint.getEndpointUri() );
consumer.start();
producer.start();
while ( true ) {
logger.trace("Awaiting message: " + ++count );
Exchange exchange = consumer.receive( consumerEndpoint, 60000 );
if ( exchange == null ) break;
logger.trace("Processing message: " + count );
producer.send( exchange );
consumer.doneUoW( exchange );
logger.trace("Processed message: " + count );
}
producer.stop();
consumer.stop();
logger.debug( "Consumed " + (count - 1) + " message" + ( count == 2 ? "." : "s." ) );
} catch ( Throwable t ) {
logger.error("Something went wrong!", t );
throw t;
}
}
}
You configure the bean, and then call the bean method from your timer, and configure a direct route to process the entries from the queue.
from("timer:...")
.beanRef("consumerBean", "pollConsumer");
from("direct:myRoute")
.to(...);
It will then read everything in the queue, and stop as soon as no entries arrive within a minute. You might want to reduce the minute, but I found a second meant that if JMS as a bit slow, it would time out halfway through draining the queue.
I've also been looking at the sjms-batch component, and how it might be used with with a pollEnrich pattern, but so far I haven't been able to get that to work.
I have solved that by using my application as a CronJob in a MicroServices approach, and to give it the power of gracefully shutting itself down, we may set the property camel.springboot.duration-max-idle-seconds. Thus, your JMS consumer route keeps simple.
Another approach would be to declare a route to control the "lifecycle" (start, sleep and resume) of your JMS consumer route.
I would strongly suggest that you use the first approach.
If you use ActiveMQ you can leverage the Scheduler feature of it.
You can delay the delivery of a message on the broker by simply set the JMS property AMQ_SCHEDULED_DELAY to the number of milliseconds of the delay. Very easy in the Camel route
.setHeader("AMQ_SCHEDULED_DELAY", 60000)
It is not exactly what you look for because it does not drain a queue every 30 minutes, but instead delays every individual message for 30 minutes.
Notice that you have to enable the schedulerSupport in your broker configuration. Otherwise the delay properties are ignored.
<broker brokerName="localhost" dataDirectory="${activemq.data}" schedulerSupport="true">
...
</broker>
You should consider Aggregation EIP
from(URI_WAITING_QUEUE)
.aggregate(new GroupedExchangeAggregationStrategy())
.constant(true)
.completionInterval(TIMEOUT)
.to(URI_PROCESSING_BATCH_OF_EXCEPTIONS);
This example describes the following rules: all incoming in URI_WAITING_QUEUE objects will be grouped into List. constant(true) is a grouping condition (wihout any). And every TIMEOUT period (in millis) all grouped objects will be passed into URI_PROCESSING_BATCH_OF_EXCEPTIONS queue.
So the URI_PROCESSING_BATCH_OF_EXCEPTIONS queue will deal with List of objects to process. You can apply Split EIP to split them and to process one by one.
Related
I would like to have a Retry Policy only on a specific listener that listen to a specific queue (DLQ in the specific case).
#RabbitListener(queues = "my_queue_dlq", concurrency = "5")
public void listenDLQ(Message dlqMessage) {
// here implement logic for resend message to original queue (my_queue) for n times with a certain interval, and after n times... push to the Parking Lot Queue
}
BUT if I am not misunderstood when I specify a Retry Policy (for ex. on the application.yaml) all #RabbitListeners use it.
I suppose the only way would be to create a dedicated container factory, but this last one would be identical to the default one with ONLY one more Retry policy ... and it doesn't seem to me like the best to do so.
Something like that :
#RabbitListener(containerFactory = "container-factory-with-retrypolicy", queues = "myDLQ", concurrency = "5")
public void listenDLQ(Message dlqMessage) {
// here implement logic for resend message to original queues for n times with a certain interval
}
Do you see alternatives ?
Thank you in advance.
The ListenerContainer instances are registered to the RabbitListenerEndpointRegistry. You can obtain a desired one by the #RabbitListener(id) value. There you can get access to the MessageListener (casting to the AbstractAdaptableMessageListener) and set its retryTemplate property.
Or you can implement a ContainerCustomizer<AbstractMessageListenerContainer>, check its getListenerId() and do the same manipulation against its getMessageListener().
I'm stucked with that kind of a problem. I use kafka as transport between services. Tried to draw sequence diagram
First of all planning service get main task and handling it, planning service pass it to few services then. My main problem is: I musn't pick another main task, until f.e. second service send result to kafka and planning service will process the result.
My main listener have this structure
#KafkaListener(
containerFactory = "genFactory",
topics = "${main}")
public void listenStartGeneratorTopic( GeneratorMessage message, Acknowledgment acknowledgment){
//do some logic
//THEN send message to first service, and then in that listener new task sends to second
sendTaskToQueue(task);
acknowledgment.acknowledge();
log.info("All done in method");
}
As I understood, I need aknowledge() after all my logic with result from second service will be done. So I tried to add boolean flag in CompletableFuture, setting it in true when my planning service get response from second service. And do blocking get() in main listener to continue after.
private CompletableFuture<Boolean> isMessageProcessed = new CompletableFuture<>();
#KafkaListener(topics = "${report}")
public void listenReport(ReportMessage reportMessage) {
isMessageProcessed = CompletableFuture.completedFuture(true);
}
}
#KafkaListener(
containerFactory = "genFactory",
topics = "${main}")
public void listenStartGeneratorTopic( GeneratorMessage message, Acknowledgment acknowledgment){
//do some logic
//THEN send message to first service, and then in that listener new task sends to second
sendTaskToQueue(task);
isMessageProcessed.join();
log.info("message is ready for commit");
acknowledgment.acknowledge();
}
That's looks strange enough and that idea doesn't bring me result.
So, can you give me advice, what can I do in that situation?
Why not using 6 topics? I believe this is better separation of duties and might allow you better scale,
Guess I would check KStream as well in your case...
My idea goes like this:
PLANNING SERVICE read from topic1.start do work send to topic2 ,
FIRST SERVICE read from topic2 do work and send to topic3
PLANNING SERVICE (another instance) read from topic3 do work and write to topic4
SECOND SERVICE reads topic4 do work send to topic5
PLANNING SERVICE (another instance) read from topic5 and write to topic6.done
I know that this error was mentioned in other posts as (https://github.com/spring-projects/spring-amqp/issues/999 and https://github.com/spring-projects/spring-amqp/issues/853 ) but I haven't found the solution for me.
My project has defined a set of microservices that publishing and consuminng messages by queue. when I
run my stress tests with 200 Transaction per seconds I get this error:
The channelMax limit is reached. Try later
This error is only shown in one microservice in the rest no.
I am using in my project:
spring-boot-starter-amqp.2.3.4.RELEASE
spring-rabbit:2.2.11
and my setting of rabbit is :
public ConnectionFactory publisherConnectionFactory() {
final CachingConnectionFactory connectionFactory = new
CachingConnectionFactory(rabbitMQConfigProperties.getHost(), rabbitMQConfigProperties.getPort());
connectionFactory.setUsername(rabbitMQConfigProperties.getUser());
connectionFactory.setPassword(rabbitMQConfigProperties.getPass());
connectionFactory.setPublisherReturns(true);
connectionFactory.setPublisherConfirms(true);
connectionFactory.setConnectionNameStrategy(connecFact -> rabbitMQConfigProperties.getNameStrategy());
connectionFactory.setRequestedHeartBeat(15);
return connectionFactory;
}
#Bean(name = "firstRabbitTemplate")
public RabbitTemplate firstRabbitTemplate(MessageDeliveryCallbackService messageDeliveryCallbackService) {
final RabbitTemplate template = new RabbitTemplate(publisherConnectionFactory());
template.setMandatory(true);
template.setMessageConverter(jsonMessageConverter());
template.setReturnCallback((msg, i, s, s1, s2) -> {
log.error("Publisher Unable to deliver the message {} , Queue {}: --------------", s1, s2);
messageDeliveryCallbackService.returnedMessage(msg, i, s, s1, s2);
});
template.setConfirmCallback((correlationData, ack, cause) -> {
if (!ack) {
log.error("Message unable to connect Exchange, Cause {}: ack{}--------------", cause,ack);
}
});
return template;
}
My questions are :
Should I set up the ChannelCacheSize and setChannelCheckoutTimeout?. I did a test increasing the channelCacheSize to 50 but the issue is still happenning. What would it be the best value for these parameters as per I mentioned it earlier?. I read about channelCheckoutTimeout should be higher than 0 but I don't know what value i must set up.
Right now i am processing around 200 Transaction per second but this number will be increased progressly
Thank you in advance.
channel_max is negotiated between the client and server and applies to connections. The default is 2047 so it looks like you broker has imposed a lower limit.
https://www.rabbitmq.com/channels.html#channel-max
When using publisher confirms, returning channels to the cache is delayed until the confirm is received; hence more channels are generally needed when the volume is high.
You can either reconfigure the broker to allow more channels, or change the CacheMode to CONNECTION instead of the default (CHANNEL).
https://docs.spring.io/spring-amqp/docs/current/reference/html/#cachingconnectionfactory
I have a simple shell script that connect to GCP and try to pull Pub/Sub messages from a topic.
When launched, it check if any message exist, does a simple action if so, then ack the message and loop .
It looks like that :
while [ 1 ]
do
gcloud pubsub subscriptions pull...
// Do something
gcloud pubsub subscriptions ack ...
done
Randomly it does not pull the messages : they stay in the queue and are not pulled.
So we tried to add a while loop when getting the message with something like 5 re-try in order to avoid those issues work better but not perfectly. I also think that is a bit shabby...
This issue happened on other project that where migrated from a script shell to Java (for some other reasons) where we used a pull subscription and it work perfectly on those projects now !
We must probably do something wrong but I don't know what...
I have read that sometimes gcloud pull less messages than what's really on the pubsub queue :
https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull
But it must at least pull one ... In our case no messages are pulled but randomly.
Is there something to improve here ?
In general, relying on a shell script that uses gcloud to retrieve messages and do something with them is not going to be an efficient way to use Cloud Pub/Sub. It is worth noting that the lack of messages being returned in pull is not indicative of a lack of messages; it just means that messages could not be returned before the pull request's deadline. The gcloud subscriptions pull command sets the returnImmediately property (see info in pull documentation) to true, which basically means that if there aren't messages already quickly accessible in memory, then no messages are going to be returned. This property is deprecated and should not be set to true, so that is probably something that we need to explore changing in gcloud.
You would be better off writing a subscriber using the client libraries that sets up a stream and continuously retrieves messages. If your intention is to run this only periodically, then you could write a job that reads messages and waits some time after messages have not been received and shuts down. Again, this would not guarantee that all messages would be consumed that are available, but it would be true in most cases.
A version of this in Java would look like this:
import com.google.cloud.pubsub.v1.AckReplyConsumer;
import com.google.cloud.pubsub.v1.MessageReceiver;
import com.google.pubsub.v1.ProjectSubscriptionName;
import com.google.pubsub.v1.PubsubMessage;
import java.util.concurrent.atomic.AtomicLong;
import org.joda.time.DateTime;
/** A basic Pub/Sub subscriber for purposes of demonstrating use of the API. */
public class Subscriber implements MessageReceiver {
private final String PROJECT_NAME = "my-project";
private final String SUBSCRIPTION_NAME = "my-subscription";
private com.google.cloud.pubsub.v1.Subscriber subscriber;
private AtomicLong lastReceivedTimestamp = new AtomicLong(0);
private Subscriber() {
ProjectSubscriptionName subscription =
ProjectSubscriptionName.of(PROJECT_NAME, SUBSCRIPTION_NAME);
com.google.cloud.pubsub.v1.Subscriber.Builder builder =
com.google.cloud.pubsub.v1.Subscriber.newBuilder(subscription, this);
try {
this.subscriber = builder.build();
} catch (Exception e) {
System.out.println("Could not create subscriber: " + e);
System.exit(1);
}
}
#Override
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
// Process message
lastReceivedTimestamp.set(DateTime.now().getMillis());
consumer.ack();
}
private void run() {
subscriber.startAsync();
while (true) {
long now = DateTime.now().getMillis();
long currentReceived = lastReceivedTimestamp.get();
if (currentReceived > 0 && ((now - currentReceived) > 30000)) {
subscriber.stopAsync();
break;
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("Error while waiting for completion: " + e);
}
}
System.out.println("Subscriber has not received message in 30s. Stopping.");
subscriber.awaitTerminated();
}
public static void main(String[] args) {
Subscriber s = new Subscriber();
s.run();
System.exit(0);
}
}
About SEDA component in Camel, anybody knows if a router removes the Exchange object from the queue when routing it? My router is working properly, but I'm afraid it keeps the Exchange objects in the queue, so my queue will be continuously growing...
This is my router:
public class MyRouter extends RouteBuilder {
#Override
public void configure() {
from("seda:input")
.choice()
.when(someValue)
.to("bean:someBean?method=whatever")
.when(anotherValue)
.to("bean:anotherBean?method=whatever");
}
}
If not, does anybody know how to remove the Exchange object from the queue once it has been routed or processed (I am routing the messages to some beans in my application, and they are working correctly, the only problem is in the queue).
Another question is, what happens if my input Exchange does not match any of the choice conditions? Is it kept in the queue as well?
Thanks a lot in advance.
Edited: after reading Claus' answer, I have added the end() method to the router. But my problem persists, at least when testing the seda and the router together. I put some messages in the queue, mocking the endpoints (which are receiving the messages), but the queue is getting full every time I execute the test. Maybe I am missing something. This is my test:
#Test
public void test() throws Exception {
setAdviceConditions(); //This method sets the advices for mocking the endpoints
Message message = createMessage("text", "text", "text"); //Body for the Exchange
for (int i = 0; i < 10; i++) {
template.sendBody("seda:aaa?size=10", message);
}
template.sendBody("seda:aaa?size=10", message); //java.lang.IllegalStateException: Queue full
}
Thanks!!
Edited again: after checking my router, I realised of the problem, I was writing to a different endpoint than the one the router was reading from (facepalm)
Thank you Claus for your answer.
1)
Yes when a Exchange is routed from a SEDA queue its removed immediately. The code uses poll() to poll and take the top message from the SEDA queue.
SEDA is in-memory based so yes the Exchanges is stored on the SEDA queue in-memory. You can configure a queue size so the queue can only hold X messages. See the SEDA docs at: http://camel.apache.org/seda
There is also JMX operations where you can purge the queue (eg empty the queue) which you can use from a management console.
2)
When the choice has no predicates that matches, then nothing happens. You can have an otherwise to do some logic in these cases if you want.
Also mind that you can continue route after the choice, eg
#Override
public void configure() {
from("seda:input")
.choice()
.when(someValue)
.to("bean:someBean?method=whatever")
.when(anotherValue)
.to("bean:anotherBean?method=whatever")
.end()
.to("bean:allGoesHere");
}
eg in the example above, we have end() to indicate where the choice ends. So after that all the messages goes there (also the ones that didnt match any predicates)