I am using spring boot 2.1.1 release and spring-boot-starter-amqp , the ##RabbitListener stopped consuming messages although it was working fine. When i restarted the consumer API , it starts to work fine.
Also from the management UI, it shows no consumers on this queue
Java 1.8 version
RabbitMQ 3.7.11 cluster (3 nodes)
Kubernetes
No Exception at the java client side or rabbitmq server side.
Heartbeates and keepalive with default values
I tried to re-synchronize the queue via rabbitmqctl, but it still not working.
#Component
public class Receiver {
Logger logger = Logger.getLogger(Receiver.class);
#RabbitListener(queues="test")
public void recievedMessage(String msg) {
logger.info("Recieved Message: " + msg);
}
}
The most common cause, by far, for problems like this is the container thread is "stuck" somewhere in user code - either in the listener, or code called by the listener; e.g. a deadlock.
First step is to take a thread dump the next time it happens to see what the listener container thread(s) are doing.
Related
I have a Spring boot application based on CommandLineRunner, when it starts it does some calculations, sends the metrics to Prometheus and shuts down.
I am using Prometheus Push Gateway with Micrometer, mainly based on this tutorial:
https://luramarchanjo.tech/2020/01/05/spring-boot-2.2-and-prometheus-pushgateway-with-micrometer.html
This works fine if I leave the application running however with my particular Spring boot application, it looses the metrics sent just before the shutdown.
I have had similar issue with CloudWatch however it was clear with the Registry implementation CloudWatchMeterRegistry, that it starts a thread in the background and it uses the property cloudwatch.step to schedule the dispatch of the collected metrics. I am struggling to see how PrometheusMeterRegistry is working and not sending the metrics before the application shutsdown.
I have tried to add meterRegistry.close(); just before the shutdown, however it still has the same issue!
After some investigation resolved this by calling the shutdown() method on PrometheusPushGatewayManager.
#SpringBootApplication
public class SpringBootConsoleApplication implements CommandLineRunner {
#Autowired
PrometheusPushGatewayManager prometheusPushGatewayManager;
#PreDestroy
public void onExit() {
System.out.println("Exiting..");
prometheusPushGatewayManager.shutdown();
}
...
And add following in the application.properties:
management.metrics.export.prometheus.pushgateway.shutdown-operation=PUSH
I am using Netty server for a Spring boot application. Is there anyway to monitor the Netty server queue size so that we will come to know if the queue is full and server is not able to accept any new request? Also, Is there any logging by netty server if the queue is full or unable to accept a new request?
Netty does not have any logging for that purpose but I implemented a way to find pending tasks and put some logs according to your question. here is a sample log from my local
you can find all code here https://github.com/ozkanpakdil/spring-examples/tree/master/reactive-netty-check-connection-queue
About code which is very explanatory from itself but NettyConfigure is actually doing the netty configuration in spring boot env. at https://github.com/ozkanpakdil/spring-examples/blob/master/reactive-netty-check-connection-queue/src/main/java/com/mascix/reactivenettycheckconnectionqueue/NettyConfigure.java#L46 you can see "how many pending tasks" in the queue. DiscardServerHandler may help you how to discard if the limit is full. You can use jmeter for the test here is the jmeter file https://github.com/ozkanpakdil/spring-examples/blob/master/reactive-netty-check-connection-queue/PerformanceTestPlanMemoryThread.jmx
if you want to handle netty limit you can do it like the code below
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
totalConnectionCount.incrementAndGet();
if (ctx.channel().isWritable() == false) { // means we hit the max limit of netty
System.out.println("I suggest we should restart or put a new server to our pool :)");
}
super.channelActive(ctx);
}
You should check https://stackoverflow.com/a/49823055/175554 for handling the limits and here is another explanation about "isWritable" https://stackoverflow.com/a/44564482/175554
One more extra, I put actuators in the place http://localhost:8080/actuator/metrics/http.server.requests is nice to check too.
I have a Spring Boot app (Jhipster) that uses STOMP over WebSockets to communicate information from the server to users.
I recently added an ActiveMQ server to handle scaling the app horizontally, with an Amazon auto-scaling group / load-balancer.
I make use the convertAndSendToUser() method, which works on single instances of the app to locate the authenticated users' "individual queue" so only they receive the message.
However, when I launch the app behind the load balancer, I am finding that messages are only being sent to the user if the event is generated on the server that their websocket-proxy connection (to the broker) is established on?
How do I ensure the message goes through ActiveMQ to whichever instance of the app that the user is actually "connected too" regardless of which instance receives, say an HTTP Request that executes the convertAndSendToUser() event?
For reference here is my StompBrokerRelayMessageHandler:
#Bean
public AbstractBrokerMessageHandler stompBrokerRelayMessageHandler() {
StompBrokerRelayMessageHandler handler = (StompBrokerRelayMessageHandler) super.stompBrokerRelayMessageHandler();
handler.setTcpClient(new Reactor2TcpClient<>(
new StompTcpFactory(orgProperties.getAws().getAmazonMq().getStompRelayHost(),
orgProperties.getAws().getAmazonMq().getStompRelayPort(), orgProperties.getAws().getAmazonMq
().getSsl())
));
return handler;
}
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableStompBrokerRelay("/queue", "/topic")
.setSystemLogin(orgProperties.getAws().getAmazonMq().getStompRelayHostUser())
.setSystemPasscode(orgProperties.getAws().getAmazonMq().getStompRelayHostPass())
.setClientLogin(orgProperties.getAws().getAmazonMq().getStompRelayHostUser())
.setClientPasscode(orgProperties.getAws().getAmazonMq().getStompRelayHostPass());
config.setApplicationDestinationPrefixes("/app");
}
I have found the name corresponding to the queue that is generated on ActiveMQ by examining the headers in the SessionSubscribeEvent, that is generated in the listener when a user subscribes to a user-queue, as simpSessionId.
#Override
#EventListener({SessionSubscribeEvent.class})
public void onSessionSubscribeEvent(SessionSubscribeEvent event) {
log.debug("Session Subscribe Event:" +
"{}", event.getMessage().getHeaders().toString());
}
Corresponding queues' can be found in ActiveMQ, in the format: {simpDestination}-user{simpSessionId}
Could I save the sessionId in a key-value pair and just push messages onto that topic channel?
I also found some possibilities of setting ActiveMQ specific STOMP properties in the CONNECT/SUBSCRIBE frame to create durable subscribers if I set these properties will Spring than understand the routing?
client-id & subcriptionName
Modifying the MessageBrokerReigstry config resolved the issue:
config.enableStompBrokerRelay("/queue", "/topic")
.setUserDestinationBroadcast("/topic/registry.broadcast")
Based on this paragraph in the documentation section 4.4.13:
In a multi-application server scenario a user destination may remain
unresolved because the user is connected to a different server. In
such cases you can configure a destination to broadcast unresolved
messages to so that other servers have a chance to try. This can be
done through the userDestinationBroadcast property of the
MessageBrokerRegistry in Java config and the
user-destination-broadcast attribute of the message-broker element in
XML
I did not see any documentation on "why" /topic/registry.broadcast was the correct "topic" destination, but I am finding various iterations of it:
websocket sessions sample doesn't cluster.. spring-session-1.2.2
What is MultiServerUserRegistry in spring websocket?
Spring websocket - sendToUser from a cluster does not work from backup server
What is happening to me is that the MDB receives messages and tries to process them and even my server has not started completely
any idea how to solve this?
You can find out if your server startup is completed by one of the following two techniques:
use ServletContextListener, once your application deployment is complete, server would call ServletContextListener.contextInitialized method
Use mbean support from wildfly, you can query mBean via JMX interface of wildfly and figure out if the server state is 'started'. But mind you, your code would be tied down to wildfly only in this case.
Once you decide the option to figure out the server startup state, you need to check for it in your MDB's postconstruct method and go ahead only if the server is started.
#MessageDriven(...)
public class MyMdb implements MessageListener {
#PostConstruct
public void init() {
// check if server has started here
//if server is not started, sleep and re-check again.
}
public void onMessage(Message message) {
}
}
I made a simple Jms project with 2 java files names are MessageSender.java,MessageConsumer.java.one for sending messages to Activemq:Queue and another for consuming messages from Activemq:Queue.Deployed this project in Apache Tomcat.following code was consumer code.
ActiveMQConnectionFactory connectionFactory=new ActiveMQConnectionFactory("admin","admin","tcp://localhost:61617?jms.prefetchPolicy.queuePrefetch=1");
Connection connection=connectionFactory.createConnection();
final Session session=connection.createSession(true, Session.CLIENT_ACKNOWLEDGE);
Queue queue=session.createQueue("ThermalMap");
javax.jms.MessageConsumer consumer=session.createConsumer(queue);
//anonymous class
MessageListener listener = new MessageListener() {
#Override
public void onMessage(Message msg) {
// My business code
}
};
Later If I want to change consumer code,I don't want to stop Tomcatbecause If I stop Tomcat entire jms project should not work. So clients can't able to sent messages to Activemq:Queue.So I don't want to follow this way.
I am thinking, If I stop consumers through Activemq console page.I don't need to stop Tomcat So clients can able to send messages normally.For this I check AMQ console page,I didn't seen any consumers.
Is it correct way to do this.
If it is correct way, How can I do this.
can anyone suggest me.
Thanks.
Call the .close() method on your MessageConsumer.