There is a ActiveMQ queue (QueueA). A service (MyService) subscribes to messages, processes it and sends the message to another ActiveMQ queue (QueueB).
QueueA -> MyService -> QueueB
Cosider a scenario where thousands of messages are in QueueA. At the same time, QueueB is down. I want to stop processing if a certain number of messages (say 100) messages are consecutively failing while sending messages to QueueB. It should test for a rolling window in certain time period (say, 100 consecutive messages failed in 60 seconds) and stop consuming from QueueA. It should then test if the service is up after 15 minutes or so by sending one more message. If it still fails, again stop consuming from QueueA for another 15 minutes.
Right now, what is happening is that all the messages are erroring out and we have to reprocess every message again. There is a recovery mechanism, but the recovery mechanism is getting overloaded because of the limitations of the current archituecture.
Is there any pattern for this? Is it the same circuit breaker (I am aware of it in synchronous context). If so, not sure if there is a solution in Java / Spring Boot / Apache Camel. Yes, that is the technology stack we are currently on. Any guidelines for the pattern also will help even if you may not have the knowledge of this specific technology platform.
I have also read the following question in StackOverflow.
Is circuit breaker pattern applicable for asynchronous requests also?
Thanks and appreciate your time in helping me with this.
Have a look on the Camel RoutePolicy of type ThrottlingExceptionRoutePolicy which is based on the CircuitBreakerLoadBalancer.
Using this policy should allow you to stop consuming from the endpoint when the circuit is in the open state (to compare with the standard circuit behahiour : bypass the service call, and fallback to another response).
#Bean
public ThrottlingExceptionRoutePolicy myCustomPolicy() {
// Important: do not open circuit for this kind of exceptions
List<Class<?>> handledExceptions = Arrays.asList(MyException.class);
return new ThrottlingExceptionRoutePolicy(failureThreshold, failureWindow, halfOpenAfter, handledExceptions);
}
from("jms:queue:QueueA")
.routePolicy(myCustomPolicy)
.to("mock:MyService")
Related
I know the DefaultMessageListenerContainer polls by design. And that the receiveTimeout which sets the polling interval defaults to 1 second.
The way I understand it is that the DMLC will issue a get, and waits the 'receiveTimeout' defined interval (1 second) before it times out and issues another get.
From what I have read, we can set this receiveTimout value to a larger value and have NO effect on messages getting picked up from the MQ because the active 'get' will sit on the listener until a message arrives... and once/if the timeout interval expires it will just submit another get which remains active on the queue until a message arrives.
So my questions is, what is the benefit of a smaller receiveTimout interval? If we are always going to process a message when it arrives, why on earth would we want to poll the queue every second?
We are running many large applications, and the polling is simply running the CPU usage/bill through the roof, and I cannot find a justification for this.
Yes - the 1 second receive timeout can be very CPU intensive with a large number of queues.
The general idea for the DefaultMessageListenerContainer was to wait for a bit (1 second seems to be a very short wait period), and then, if you don't get a message, it actually tears everything down and does a full reconnect. This is kind of a poor-mans error handling. "If I haven't heard from the broker, assume that something is broken, drop everything and reconnect". If the reconnect were not so expensive, it might not be a bad strategy. Or if you have only one queue. Or maybe you are expecting 10 messages a second and do want to reconnect if a second goes by. If you have a reasonable number of destinations, the reconnect traffic can get downright abusive.
For IBM MQ, failures on the JMS connection/session are reliably picked up. You don't have the, "it just sits there not getting any messages for some reason" scenario. So setting the timeout to 10 minutes (whatever) would be fine.
Note that if you are running in a JEE application server, and your JMS connections are managed by the JCA, then that layer is responsible for detecting bad connections and you don't have to worry about it up in the application layer.
With Camel and for SpringBoot GitHub might be useful.
I am new to Spring Boot #kafkaListener. My application receiving almost 200K message per second on topic. I want to separate message listener and processing of the message.
How can I use java.util.concurrent.BlockingQueue with #kafkaListener? Can I use it by using CompletableFuture?
Any sample code will help more.
I believe you want to have your consumer with pipelining implemented. Its not uncommon for one to implement this in a scenario like yours. Why? Well, the KafkaConsumer lacks in that decompressing / deserializing can be time consuming without considering the time it takes to do processing. Since these operations are stacked behind one thread, it would be ideal to separate the polling from the processing, which is achieved through a couple of buffers.
One way to do this: your EventReceiver spins up a thread for the polling. That thread would do the same thing you always do, but instead of firing off the listeners for each event, you'd pass the event to a receivedEvents buffer which could be BlockingQueue<RecieveEvent>. So in the for loop, you pass each record to the blocking queue. This thread would leverage another buffer once the for loop is over, like Queue<Map<TopicPartition, OffsetAndMetadata>> -- and it would commit the offsets that the processingThread has successfully processed.
Next, your EventReceiver spins up another thread - processingThread. This would handle pulling records from the buffer, firing the event to all the listeners for this receiver, and then update the Queues state for the pollingThread to commit.
Why doesn't the processingThread just commit the events instead of passing it back to the pollingThread? This is bc KafkaConsumer requires that the same thread that calls .poll() should be the one that calls consumer.commitAsync(...) or else you'll get a concurrency exception.
This approach doesn't work with auto commit enabled.
In terms of how one can do this using Spring Kafka, I'm not completely sure. However, I do know Spring Kafka separates EventReceiver from EventListener (#KafkaListener) which is separating the low-level kafka work from the business logic. In theory, you'd have to tune their implementation, but I think implementing this one without Spring Kafka library would be easier.
I need help with a problem with gcp pub/sus. I have a process that send 100 messages with filters to pubsub and another application (in spring boot) receive these messages. When spring boot application receive message from pubsub (not pull), process 100 messages but, into the process, receive more messages, in diferents times receive diferents numbers of messages, any times receive 120, another 140, and the others more than 200. I wasn't found any solution of this, this is my code:
#Bean
#ServiceActivator(inputChannel = "pubsubInputChannel")
public MessageHandler messageReceiver() {
return message -> {
System.out.println("Message arrived! Payload: " + new String((byte[]) message.getPayload()));
//other process of app (call other api)
AckReplyConsumer consumer = (AckReplyConsumer) message.getHeaders().get(GcpPubSubHeaders.ACKNOWLEDGEMENT);
consumer.ack();
};
}
please help me!!!
Duplicate messages can happen for different reasons in Google Cloud Pub/Sub. One thing to keep in mind is that Cloud Pub/Sub offers at-least-once delivery meaning that some amount of duplicates is always possible, so your application must be resilient to them. That many duplicates does seem a bit high, though. In general duplicates can generally happen for the following reasons:
Messages are being sent by the publisher more than once. This can happen if the publisher got disconnected from Cloud Pub/Sub and sent the same message again. If this type of duplication occurs, then the messages will have different message IDs.
The subscriber is taking too long to acknowledge messages. In your code, you have //other process of app (call other api). How long does this process take? If it is longer than the deadline for acknowledging the message, then the message will be redelivered. Keep in mind that if this other process requires locks be grabbed for all messages, there could be a contention issue with too many requests trying to get those locks at the same time, resulting in processing delays. By default, the ack deadline for a message is ten seconds. When using the Java client library, the deadline is automatically extended by the maxAckExtensionPeriod, which defaults to one hour. This property can be set in the DefaultSubscriberFactory for Spring as well.
Messages are not acked at all. If an exception prevents the call to ack or there is deadlock resulting in that line of code never being reached, then the message will be redelivered.
The use case is one of a large backlog of small messages. In this situation, buffers are prone to fill up in the client in a way that results in redelivery of messages.
I am trying to implement a kind of Priority Channel with spring integration but I am blocked and didn't find a solution on the web.
I would like to read multiples channel (6) alternatively with a service activator. Each channel is for a priority level (CRITICAL, HIGHEST, HIGH, NORMAL, LOW, LOWEST). Message come from RabbitMQ and are distributed on correct channel with a Router.
The problem is that I would like to create a Service Activator who read alternatively in the channels using a round robin based on time.
For example, CRITICAL should have a 5 secondes running time, and then the service switch to HIGHEST for 3 seconds, and then to HIGH for 1 second, ...
Is it possible to do it properly with spring integration ?
Maybe I don't use the correct component to do it ?
Regards
The Priority Channel pattern works a bit different way.
It is a queue with sort support. When a new message arrives to the queue it is sorted to the proper place according to its priority. That absolutely does matter how your consumer of this channel works. The priority happens only in the channel. The consumer just polls messages from that queue like they are ordered for it: the CRITICAL, than HIGHEST, if CRITICAL aren't present and so on.
On the other hand, if you distribute messages by priority do different channels, why just don't have separate Service Activators for each of those channels? And each priority will be read by its own process.
There is no such a solution based on the "time to run". It just doesn't seem with a good fit for messaging architecture. Although you might can implement via scheduled task cancel() or Quartz to "perform task until...".
UPDATE
Regarding time control, I think you can come up with the solution which in the infinite loop really start()s different service activators and stop()s them after an appropriate scheduled time. All those service activators should listen to different queue channels.
I'm trying to implement the actor model pattern (Somewhat mashed toghter with producer consumer) by using spring's application events and threadpoolexecutors
my main objective is to decouple each of the layers .
my architecture is as follows :
i have a war deployed that trough a rest API receives requests for business transactions , at any given moment there can be X number of transactions alive
where X is configureable number , the actual execution must be asynchronous and each transaction must be in a different thread .
the requests themselves are treated in a FIFO manner but there is some complexity to it as some requests must wait for other to complete before they can be processed but that does not mean other requests can't be processed , e.g. : don't process a withdrawal from account 2 if a deposit to account 2 was requested before it , so if i get hits for :
deposit(2)
withdrawal(2)
deposit(3)
where the numbers are account numbers i want to process them in this order:
deposit(2)
deposit(3)
withdrawal(2)
i've built the architecture is this way :
i have a rest api that gets the hits and writes them to the DB (distributed system that has to have the state in DB)and publishes
a clientrequestevent in the application context
i have singleton bean that is in charge of publishing the producer events and monitoring how many events he has sent (i.e. : he is in charge of limiting the number of concurrent processes and implementing the above logic)
and i have a few other listeners each for an action (withdrawal deposit etc..) that listen to the events published by the latter and publish a done event.
every thing works great everything is done is different threads and all flows great but i have
a problem with the middle layer the one in charge of determing whether or not there is a free slot
i don't want to have a synchronous method nor do i want to do some tricks around an atomiclong or something like that i would rather use some blockingqueue to publish the events but i can't find a nice way to determine when an event is done so i can put a new one back in .
the biggest problem is that for requesting a new work i have to go the DB which is a heavy task , as this system should work under heavy load .
i would like to somehow utilize a blockingqueue and a threadpool so that will take from a size bounded queue with threads the minute one slot is free
what would be a good way to handle this ?
thanks in advance