Application is not stopped with missingTopicFatal and manually started listeners - spring

I'm using spring and spring-kafka.
When the application starts, I'm loading compacted topics into memory then when the topics are fully read, the application is started.
In order to do that I created #KafkaListeners with autostartup to false and a SmartLifeCycle bean with AbstractMessageListenerContainer.DEFAULT_PHASE - 1 phase which is doing listener.start() on all those listeners (which read compacted topics) then wait for them to be finished.
This is working great but if I set spring.kafka.listener.missing-topics-fatal = true with a missing topic, there is an error :
org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is java.lang.IllegalStateException: Topic(s) [mytopic] is/are not present and missingTopicsFatal is true
It's the expected behavior, but the application is not shutdown, my manually started listeners keeps running and sending errors :
java.lang.IllegalStateException: org.springframework.context.annotation.AnnotationConfigApplicationContext#502e1410 has not been refreshed yet
How can I make the application exit in this case ?

Catch the exception and shut down the JVM:
#SpringBootApplication
public class So60036945Application {
public static void main(String[] args) {
try {
SpringApplication.run(So60036945Application.class, args);
}
catch (ApplicationContextException ace) {
if (ace.getCause() instanceof IllegalStateException) {
ace.printStackTrace();
System.exit(1);
}
}
}
#KafkaListener(id = "so60036945", topics = "so60036945a")
public void listen(String in) {
System.out.println(in);
}
}
But, as I said in Gitter, it's better to auto-start the compacted topic listeners and start the other listeners manually (the other way around to what you are doing now).

Related

Spring AMQP: Stopping a SimpleConsumer from DirectMessageListenerContainer

I have a use case where I am dynamically registering and removing a queue to and from a container based on some predicate. I am using a DirectMessageListenerContainer based on the advice given in the documentation as per my needs.
Those dynamic queues are temporary ones that should get deleted if they have no messages and are not in use. Right now I have a Scheduler running periodically which deregisters the queue from the container if the predicate is true.
For me, the problem is even after removing the queue from the container the consumer bound to the queue is not getting released/stopped and thus the queue is not getting eligible for delete(due to the in-use policy by the consumer).
Is there a way to release or stop a consumer without restarting the container?
When you remove a queue, its consumer(s) are canceled.
This works as expected:
#SpringBootApplication
public class So72540658Application {
public static void main(String[] args) {
SpringApplication.run(So72540658Application.class, args);
}
#Bean
DirectMessageListenerContainer container(ConnectionFactory cf) {
DirectMessageListenerContainer dmlc = new DirectMessageListenerContainer(cf);
dmlc.setQueueNames("foo", "bar");
dmlc.setMessageListener(msg -> {});
return dmlc;
}
#Bean
ApplicationRunner runner(DirectMessageListenerContainer container) {
return args -> {
System.out.println("Hit enter to remove bar");
System.in.read();
container.removeQueueNames("bar");
};
}
}
Hit enter; then:
Perhaps your consumer thread is stuck someplace? Try taking a thread dump. If you can't figure it out; post an MCRE somplace.

MessageListener.onMessage is getting called continuously on RabbitMQ with Spring Boot

I have MessageListener.onMessage with a thread sleep. I'm simulating actual processing time the onMessage
method will take by the above mentioned Thread sleep. However what I have noticed is that it is getting called multiple times consecutively for the remaining messages till they get processed by the onMessage method. I see this as an inefficiency.
Actual message count in to queue : 1000
Output of running number for hits
onMessage<<15656
onMessage<<15657
onMessage<<15658
onMessage<<15659
onMessage<<15660
onMessage<<15661
onMessage<<15662
onMessage<<15663
Code block
#Service
class ThreadPooledMessageListener implements MessageListener {
#Autowired
TaskExecutor threadPoolTaskExecutor;
AtomicInteger processedCount = new AtomicInteger();
#Override
public void onMessage(Message message) {
System.out.println("onMessage<<" + processedCount.incrementAndGet());
threadPoolTaskExecutor.execute(new MessageProcessor(message));
}
}
class MessageProcessor implements Runnable {
Message processingMessage;
public MessageProcessor(Message message) {
this.processingMessage = message;
}
#Override
public void run() {
System.out.println("================================"+ Thread.currentThread().getName());
System.out.println(processingMessage);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("================================");
}
}
What are the possible fixes for this.
As #Gary Russell has pointed out; Issue was that I have used non-spring managed container SimpleMessageListenerContainer in my code. Fixed it with spring managed bean and defined concurrency there. Works as expected.
Fixed code segment
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueues(queue);
container.setMessageListener(threadPooledMessageListener);
container.setConcurrentConsumers(4);
container.start();
return container;
}
>I see this as an inefficiency.
It's not clear what you mean. Since you are handing off the processing of a message to another thread, the listener exits immediately and, of course, the next message is delivered.
This will risk message loss in the event of a failure.
If you are trying to achieve concurrency; it's better to set the container concurrentConsumers property and not do your own thread management in the listener. The container will manage the consumers for you.

Spring rabbitlistner stop listening to queue using annotation syntax

A colleague and I are working on an application using Spring which needs to get a message from a RabbitMQ queue. The idea is to do this using (the usually excellent) spring annotation system to make the code easy to understand. We have the system working using the #RabbitListner annotation but we want to get a message on demand. The #RabbitListner annotation does not do this, it just receives messages when they are available. The demand is determined by the "readiness" of the client i.e. a client should "get" a message from te queue stop listing and process the message. Then determine if it is ready to receive a new one and reconnect to the queue.
We have been looking into doing this by hand just using the spring-amqp/spring-rabbit modules and while this is probably possible we would really like to do this using spring. After many hours of searching and going through the documentation, we have not been able to find an answer.
Here is the recieving code we currently have:
#RabbitListener(queues = "jobRequests")
public class Receiver {
#Autowired
private JobProcessor jobProcessor;
#RabbitHandler
public void receive(Job job) throws InterruptedException, IOException {
System.out.println(" [x] Received '" + job + "'");
jobProcessor.processJob(job);
}
}
Job processor:
#Service
public class JobProcessor {
#Autowired
private RabbitTemplate rabbitTemplate;
public boolean processJob(Job job) throws InterruptedException, IOException {
rabbitTemplate.convertAndSend("jobResponses", job);
System.out.println(" [x] Processing job: " + job);
rabbitTemplate.convertAndSend("processedJobs", job);
return true;
}
}
In other words, when the job is received by the Receiver it should stop listening for new jobs and wait for the job processor to be done and then start listing for new messages.
We have re-created the null pointer exception here is the code we use to send from the server side.
#Controller
public class MainController {
#Autowired
RabbitTemplate rabbitTemplate;
#Autowired
private Queue jobRequests;
#RequestMapping("/do-job")
public String doJob() {
Job job = new Job(new Application(), "henk", 42);
System.out.println(" [X] Job sent: " + job);
rabbitTemplate.convertAndSend(jobRequests.getName(), job);
return "index";
}
}
And then the receiving code on the client side
#Component
public class Receiver {
#Autowired
private JobProcessor jobProcessor;
#Autowired
private RabbitListenerEndpointRegistry rabbitListenerEndpointRegistry;
#RabbitListener(queues = "jobRequests")
public void receive(Job job) throws InterruptedException, IOException, TimeoutException {
Collection<MessageListenerContainer> messageListenerContainers = rabbitListenerEndpointRegistry.getListenerContainers();
for (MessageListenerContainer listenerContainer :messageListenerContainers) {
System.out.println(listenerContainer);
listenerContainer.stop();
}
System.out.println(" [x] Received '" + job + "'");
jobProcessor.processJob(job);
for (MessageListenerContainer listenerContainer :messageListenerContainers) {
listenerContainer.start();
}
}
}
And the updated job processor
#Service
public class JobProcessor {
public boolean processJob(Job job) throws InterruptedException, IOException {
System.out.println(" [x] Processing job: " + job);
return true;
}
}
And the stacktrace
[x] Received 'Job{application=com.olifarm.application.Application#aaa517, name='henk', id=42}'
[x] Processing job: Job{application=com.olifarm.application.Application#aaa517, name='henk', id=42}
Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.NullPointerException
2015-12-18 11:17:44.494 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:838)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:93)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1301)
at java.lang.Thread.run(Thread.java:745)
WARN 325899 --- [cTaskExecutor-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it
java.lang.NullPointerException: null
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:838) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:93) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
The stopping of the listener works and we do receive a new job but when it try's to start it again the NPE is thrown. We checked the rabbitMQ log and found that the connection is closed for about 2 seconds and then re-opened automatically even if we put the thread in sleep in the job processor. This might be the source of the problem? The error doesn't break the program however and after it is thrown the receiver is still able to receive new jobs. Are we abusing the mechanism here or is this valid code?
To get messages on-demand, it's generally better to use rabbitTemplate.receiveAndConvert() rather than a listener; that way you completely control when you receive messages.
Starting with version 1.5 you can configure the template to block for some period of time (or until a message arrives). Otherwise it immediately returns null if there's no message.
The listener is really designed for message-driven applications.
If you can block the thread in the listener until the job completes, no more messages will be delivered - by default the container has only one thread.
If you can't block the thread until the job completes, for some reason, you can stop()/start() the listener container by getting a reference to it from the Endpoint Registry.
It's generally better to stop the container on a separate thread.

Spring AWS SQS Reconnect After Losing Connection

I am using Spring Cloud AWS (1.0.1.RELEASE) with Spring Boot to run a SQS consumer. The application runs fine, but when it looses network connection (for instance if I switch my WIFI off on my laptop when it runs on it), I see errors on the console and the application never recovers. It just hangs there and does not reconnect after the network becomes available. I have to kill it and bring it up. How do I force it to recover by itself?
// Spring Boot entry point:
public static void main(String[] args) {
SpringApplication.run(MyConsumerConfiguration.class, args);
}
// Message Listener (A different class)
#MessageMapping(value = "myLogicalQueueName" )
public void receive(MyPOJO object) {
}
The error I see at console:
Exception in thread "simpleMessageListenerContainer-1" com.amazonaws.AmazonClientException: Unable to execute HTTP request: sqs.us-east-1.amazonaws.com
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:473)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:297)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2422)
at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1130)
at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1678)
at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745
I just figured out the problem why SQS is not able to reconnect after network connection lost.
Actually seems to be a problem in current Spring AWS implementation of org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.java
private class AsynchronousMessageListener implements Runnable {
private final QueueAttributes queueAttributes;
private final String logicalQueueName;
private AsynchronousMessageListener(String logicalQueueName, QueueAttributes queueAttributes) {
this.logicalQueueName = logicalQueueName;
this.queueAttributes = queueAttributes;
}
#Override
public void run() {
while (isRunning()) {
ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
CountDownLatch messageBatchLatch = new CountDownLatch(receiveMessageResult.getMessages().size());
for (Message message : receiveMessageResult.getMessages()) {
if (isRunning()) {
MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
getTaskExecutor().execute(new SignalExecutingRunnable(messageBatchLatch, messageExecutor));
} else {
break;
}
}
try {
messageBatchLatch.await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
Above code spins up a new thread which does the polling to SQS queue to grab messages. Once network connection is dropped getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest()) throws UnknownHostException, which is not handled in the code and causes thread termination.
So when network connection is established later on, there is no thread polling the queue to retrieve the data.
I have already raised a issue with Spring for this. Following is the link: https://github.com/spring-cloud/spring-cloud-aws/issues/82
Hope this explains it all.

Refreshing Spring context when JMS message delivered

I'd like to refresh my application context when system receives JMS message. In order to do it, I set up Spring Integration jms:message-driven-channel-adapter which forwards message to service activator implementing ApplicationContextAware. This activator (ConfigurationReloader class) invokes ConfigurableApplicationContext#refresh() method.
Below is sample code snippet:
<jms:message-driven-channel-adapter id="jmsDriverConfigurationAdapter"
destination="configurationApplyQueue" channel="jmsConfigurationInboundChannel" />
<channel id="jmsConfigurationInboundChannel"/>
<service-activator input-channel="jmsConfigurationInboundChannel" ref="configurationReloader" method="refresh"/>
And my activator:
public final class ConfigurationReloader implements ApplicationContextAware {
private ConfigurableApplicationContext applicationContext;
public void refresh() {
this.applicationContext.refresh();
}
#Override
public void setApplicationContext(
final ApplicationContext applicationContext) throws BeansException {
if (applicationContext instanceof ConfigurableApplicationContext) {
this.applicationContext =
(ConfigurableApplicationContext) applicationContext;
}
}
}
In case of delivering such message, context start shutdown operation but stuck on DefaultMessageListenerContainer bean shutdown:
2011-11-14 15:42:52,980 [org.springframework.jms.listener.DefaultMessageLis tenerContainer#0-1] DEBUG org.springframework.jms.listener.DefaultMessageLis tenerContainer - Shutting down JMS listener container
2011-11-14 15:42:52,980 [org.springframework.jms.listener.DefaultMessageLis tenerContainer#0-1] DEBUG org.springframework.jms.listener.DefaultMessageLis tenerContainer - Waiting for shutdown of message listener invokers
2011-11-14 15:42:55,104 [org.springframework.jms.listener.DefaultMessageLis tenerContainer#0-1] DEBUG org.springframework.jms.listener.DefaultMessageLis tenerContainer - Still waiting for shutdown of 1 message listener invokers
Invoking this operation over JMS is crucial for me since new configuration parameters are delivered along with message.
It is standard Spring MVC application with DispatcherServlet on the front based on the latest SpringCore and Spring Integration. Also I am sure that it's JMS related issue, because invoking ConfigurationLoader through controller works fine.
As I've debugged, it stucks after DefaultMessageListenerContainer#538 line invocation (wait() method on lifecycleMonitor):
/**
* Destroy the registered JMS Sessions and associated MessageConsumers.
*/
protected void doShutdown() throws JMSException {
logger.debug("Waiting for shutdown of message listener invokers");
try {
synchronized (this.lifecycleMonitor) {
while (this.activeInvokerCount > 0) {
if (logger.isDebugEnabled()) {
logger.debug("Still waiting for shutdown of " + this.activeInvokerCount +
" message listener invokers");
}
this.lifecycleMonitor.wait(); // <--- line 538
}
}
}
catch (InterruptedException ex) {
// Re-interrupt current thread, to allow other threads to react.
Thread.currentThread().interrupt();
}
}
...there is nobody to call notify / notifyAll on monitor so maybe it's some kind of bug?
Thank you for any hints!
Can you please explain why do you need such sophisticated architecture? Reloading application context when JMS message is received? Sounds crazy (or maybe ingenious?)
Nevertheless, I am not 100% sure but the information you provided is pretty clear: you are trying to shutdown an application context while consuming JMS message. But since the consumer is Spring-managed, context cannot be destroyed because it waits for all beans to finish - including yours ConfigurationReloader required by Spring Integration message consumer. And ConfigurationReloader cannot finish because it waits for context to be destroyed (refresh() is blocking).
Simply put - you have introduced a cyclic dependency and a deadlock.
The solution is simple - delay the context refresh so that it happens after the JMS message consumption. The easiest way would be:
public void refresh() {
Thread destroyThread = new Thread() {
#Override
public void run() {
this.applicationContext.refresh();
}
};
destroyThread.start();
}
Not pretty but I'm almost sure this will work.

Resources