Report metrics during shutdown of spring-boot app - spring-boot

I have a shutdownhook which is successfully executed, but the metrics is not reported. Any advice is appreciated! I guess the issues can be
StatsDMetricWriter might be disposed before the shutdown hook? How can I verify? Or is there a way to ensure the ordering of the configured singletons?
The time gap between metric generation and app shutdown < configured delay. I tried spawning a new Thread with Thread.sleep(20000). But it didn't work
The code snippets are as follows:
public class ShutDownHook implements DisposableBean {
#Autowired
private MetricRegistry registry;
#Override
public void destroy() throws Exception {
registry.counter("appName.deployments.count").dec();
//Spawned new thread here with high sleep with no effect
}
}
My Metrics Configuration for dropwizard is as below:
#Bean
#ExportMetricReader
public MetricRegistryMetricReader metricsDWMetricReader() {
return new MetricRegistryMetricReader(metricRegistry);
}
#Bean
#ExportMetricWriter
public MetricWriter metricWriter() {
return new StatsdMetricWriter(app, host, port);
}
The reporting time delay is set as 1 sec:
spring.metrics.export.delay-millis=1000
EDIT:
The problem is as below:
DEBUG 10452 --- [pool-2-thread-1] o.s.b.a.m.statsd.StatsdMetricWriter : Failed to write metric. Exception: class java.util.concurrent.RejectedExecutionException, message: Task com.timgroup.statsd.NonBlockingUdpSender$2#1dd8867d rejected from java.util.concurrent.ThreadPoolExecutor -- looks like ThreadPoolExecutor is shutdown before the beans are shutdown.
Any Suggestions please?
EDIT
com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsPoller.getCommandJson() has the following piece of code
json.writeNumberField("reportingHosts", 1); // this will get summed across all instances in a cluster
I'm not sure how/why the numbers will add up? Where can I find that logic?

Related

Sprint boot async - does not use max-size

I am trying out Spring Boot's Async feature, but I am having some trouble getting it to work as I need.
This is my application yml
spring:
task:
execution:
pool:
max-size: 100
queue-capacity: 5
keep-alive: "10s"
core-size: 10
Application class
#SpringBootApplication
#EnableAsync
public class ServiceApplication {
public static void main(String[] args) {
SpringApplication.run(ServiceApplication.class, args);
}
}
Service class:
for (int i=0;i< 40; i++) {
CompletableFuture.runAsync(()-> {
try {
System.out.println("------------------Starting thread------------------");
//do some action here
System.out.println("------------------Ending thread------------------");
} catch (Exception e) {
e.printStackTrace();
}
});
}
I am expecting to see the System.out print out 40 times. The operations in between take long enough, and I have tried adding Thread.sleep(), but I do not see the sysouts printed more than 8 times. Is there something wrong with my config, or does it not work the way I expect?
Completable future has no idea about the pool that is used by Spring.
From docs of runAsync() method:
Returns a new CompletableFuture that is asynchronously completed by a
task running in the ForkJoinPool.commonPool() after it runs the given
action. Params: runnable – the action to run before completing the
returned CompletableFuture Returns: the new CompletableFuture
So, those tasks are being run on ForkJoinPool, not on executor used by Spring.
About executor used by Spring with #EnableAsync:
By default, Spring will be searching for an associated thread pool
definition: either a unique TaskExecutor bean in the context, or an
Executor bean named "taskExecutor" otherwise. If neither of the two is
resolvable, a SimpleAsyncTaskExecutor will be used to process async
method invocations. Besides, annotated methods having a void return
type cannot transmit any exception back to the caller. By default,
such uncaught exceptions are only logged.
You could try autowire that executor and pass it as an argument to
public static CompletableFuture<Void> runAsync(Runnable runnable, Executor executor)
Returns a new CompletableFuture that is asynchronously completed by a
task running in the given executor after it runs the given action.
Params: runnable – the action to run before completing the returned
CompletableFuture executor – the executor to use for asynchronous
execution

Activiti Escalation Listener Configuration

I am using activiti 5.18.
Behind the scenes : There are few task which are getting routed though a workflow. Some of these tasks are eligible for escalation. I have written my escalation listener as follows.
#Component
public class EscalationTimerListener implements ExecutionListener {
#Autowired
ExceptionWorkflowService exceptionWorkflowService;
#Override
public void notify(DelegateExecution execution) throws Exception {
//Process the escalated tasks here
this.exceptionWorkflowService.escalateWorkflowTask(execution);
}
}
Now when I start my tomcat server activiti framework internally calls the listener even before my entire spring context is loaded. Hence exceptionWorkflowService is null (since spring hasn't inejcted it yet!) and my code breaks.
Note : this scenario only occurs if my server isn't running at the escalation time of tasks and I start/restart my server post this time. If my server is already running during escalation time then the process runs smoothly. Because when server started it had injected the service and my listener has triggered later.
I have tried delaying activiti configuration using #DependsOn annotation so that it loads after ExceptionWorkflowService is initialized as below.
#Bean
#DependsOn({ "dataSource", "transactionManager","exceptionWorkflowService" })
public SpringProcessEngineConfiguration getConfiguration() {
final SpringProcessEngineConfiguration config = new SpringProcessEngineConfiguration();
config.setAsyncExecutorActivate(true);
config.setJobExecutorActivate(true);
config.setDataSource(this.dataSource);
config.setTransactionManager(this.transactionManager);
config.setDatabaseSchemaUpdate(this.schemaUpdate);
config.setHistory(this.history);
config.setTransactionsExternallyManaged(this.transactionsExternallyManaged);
config.setDatabaseType(this.dbType);
// Async Job Executor
final DefaultAsyncJobExecutor asyncExecutor = new DefaultAsyncJobExecutor();
asyncExecutor.setCorePoolSize(2);
asyncExecutor.setMaxPoolSize(50);
asyncExecutor.setQueueSize(100);
config.setAsyncExecutor(asyncExecutor);
return config;
}
But this gives circular reference error.
I have also tried adding a bean to SpringProcessEngineConfiguration as below.
Map<Object, Object> beanObjectMap = new HashMap<>();
beanObjectMap.put("exceptionWorkflowService", new ExceptionWorkflowServiceImpl());
config.setBeans(beanObjectMap);
and the access the same in my listener as :
Map<Object, Object> registeredBeans = Context.getProcessEngineConfiguration().getBeans();
ExceptionWorkflowService exceptionWorkflowService = (ExceptionWorkflowService) registeredBeans.get("exceptionWorkflowService");
exceptionWorkflowService.escalateWorkflowTask(execution);
This works but my repository has been autowired into my service which hasn't been initialized yet! So it again throws error in service layer :)
So is there a way that I can trigger escalation listeners only after my entire spring context is loaded?
Have you tried binding the class to ApplicationListener?
Not sure if it will work, but equally I'm not sure why your listener code is actually being executed on startup.
Try to set the implementation type of listeners using Java class or delegate expression and then in the class implement JavaDelegate instead of ExecutionListener.

spring cloud contract verification at deployment

I have extensively gone through SpringCloudContract. It is very effective TDD. I want to verify the contract during actual deployment. I have n number of micro-services (Spring stream:Source/Processor/Sink) and want to allow user to link them when they define a stream (kafka)in dataflow server dashboard. I am passing certain Object in the stream which act as
input/out for micro-service. I want to check the compatibility for micro-services and warn the user accordingly. SpringCloudContract facilitate to verify the contract during the develpment time and not a run time.
Kindly help.
I am new to Spring cloud contract, but I have found a way to start StubRunner but when it trigger the certificate I get following.
2017-04-26 16:14:10,373 INFO main c.s.s.ContractTester:36 - ContractTester : consumerMessageListener >>>>>>>>>>>>>>>>>>>>>>>>>>>>org.springframework.cloud.contract.stubrunner.BatchStubRunner#5e13f156
2017-04-26 16:14:10,503 ERROR main o.s.c.c.v.m.s.StreamStubMessages:63 - Exception occurred while trying to send a message [GenericMessage [payload={"name":"First","description":"Valid","value":1}, headers={id=49c6cc5c-93c8-2498-934a-175f60f42c03, timestamp=1493203450482}]] to a channel with name [verifications]
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.input'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload={"name":"First","description":"Valid","value":1}, headers={id=49c6cc5c-93c8-2498-934a-175f60f42c03, timestamp=1493203450482}]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:93)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
at org.springframework.cloud.contract.verifier.messaging.stream.StreamStubMessages.send(StreamStubMessages.java:60)
at org.springframework.cloud.contract.verifier.messaging.stream.StreamStubMessages.send(StreamStubMessages.java:
The same work fine with Maven install, but not with main class.
...
#RunWith(SpringRunner.class)
#AutoConfigureMessageVerifier
#EnableAutoConfiguration
#EnableIntegration
#Component
#DirtiesContext
public class ContractTester {
private static Logger logger = LoggerFactory.getLogger(ContractTester.class);
#Autowired StubTrigger stubTrigger;
#Autowired ConsumerMessageListener consumerMessageListener;
#Bean
public boolean validSimpleObject() throws Exception {
logger.info("ContractTester : consumerMessageListener >>>>>>>>>>>>>>>>>>>>>>>>>>>>"+stubTrigger);
stubTrigger.trigger("accepted_message");
if(consumerMessageListener ==null) {
logger.info("ContractTester : consumerMessageListener >>>>>>>>>>>>>>>>>>>>>>>>>>>>");
}
logger.info("ContractTester >>>>>>>>>>>>>>>>>>>>>>>>>>>>" +consumerMessageListener.toString());
SimpleObject simpleObject = (SimpleObject) consumerMessageListener.getSimpleObject();
logger.info("simpleObject >>>>>>>>>>>>>>>>>>>>>>>>>>>>" +simpleObject.toString());
assertEquals(1, simpleObject.getValue());
//then(listener.eligibleCounter.get()).isGreaterThan(initialCounter);
return true;
}
}

Spring rabbitlistner stop listening to queue using annotation syntax

A colleague and I are working on an application using Spring which needs to get a message from a RabbitMQ queue. The idea is to do this using (the usually excellent) spring annotation system to make the code easy to understand. We have the system working using the #RabbitListner annotation but we want to get a message on demand. The #RabbitListner annotation does not do this, it just receives messages when they are available. The demand is determined by the "readiness" of the client i.e. a client should "get" a message from te queue stop listing and process the message. Then determine if it is ready to receive a new one and reconnect to the queue.
We have been looking into doing this by hand just using the spring-amqp/spring-rabbit modules and while this is probably possible we would really like to do this using spring. After many hours of searching and going through the documentation, we have not been able to find an answer.
Here is the recieving code we currently have:
#RabbitListener(queues = "jobRequests")
public class Receiver {
#Autowired
private JobProcessor jobProcessor;
#RabbitHandler
public void receive(Job job) throws InterruptedException, IOException {
System.out.println(" [x] Received '" + job + "'");
jobProcessor.processJob(job);
}
}
Job processor:
#Service
public class JobProcessor {
#Autowired
private RabbitTemplate rabbitTemplate;
public boolean processJob(Job job) throws InterruptedException, IOException {
rabbitTemplate.convertAndSend("jobResponses", job);
System.out.println(" [x] Processing job: " + job);
rabbitTemplate.convertAndSend("processedJobs", job);
return true;
}
}
In other words, when the job is received by the Receiver it should stop listening for new jobs and wait for the job processor to be done and then start listing for new messages.
We have re-created the null pointer exception here is the code we use to send from the server side.
#Controller
public class MainController {
#Autowired
RabbitTemplate rabbitTemplate;
#Autowired
private Queue jobRequests;
#RequestMapping("/do-job")
public String doJob() {
Job job = new Job(new Application(), "henk", 42);
System.out.println(" [X] Job sent: " + job);
rabbitTemplate.convertAndSend(jobRequests.getName(), job);
return "index";
}
}
And then the receiving code on the client side
#Component
public class Receiver {
#Autowired
private JobProcessor jobProcessor;
#Autowired
private RabbitListenerEndpointRegistry rabbitListenerEndpointRegistry;
#RabbitListener(queues = "jobRequests")
public void receive(Job job) throws InterruptedException, IOException, TimeoutException {
Collection<MessageListenerContainer> messageListenerContainers = rabbitListenerEndpointRegistry.getListenerContainers();
for (MessageListenerContainer listenerContainer :messageListenerContainers) {
System.out.println(listenerContainer);
listenerContainer.stop();
}
System.out.println(" [x] Received '" + job + "'");
jobProcessor.processJob(job);
for (MessageListenerContainer listenerContainer :messageListenerContainers) {
listenerContainer.start();
}
}
}
And the updated job processor
#Service
public class JobProcessor {
public boolean processJob(Job job) throws InterruptedException, IOException {
System.out.println(" [x] Processing job: " + job);
return true;
}
}
And the stacktrace
[x] Received 'Job{application=com.olifarm.application.Application#aaa517, name='henk', id=42}'
[x] Processing job: Job{application=com.olifarm.application.Application#aaa517, name='henk', id=42}
Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.NullPointerException
2015-12-18 11:17:44.494 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:838)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:93)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1301)
at java.lang.Thread.run(Thread.java:745)
WARN 325899 --- [cTaskExecutor-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it
java.lang.NullPointerException: null
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:838) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:93) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
The stopping of the listener works and we do receive a new job but when it try's to start it again the NPE is thrown. We checked the rabbitMQ log and found that the connection is closed for about 2 seconds and then re-opened automatically even if we put the thread in sleep in the job processor. This might be the source of the problem? The error doesn't break the program however and after it is thrown the receiver is still able to receive new jobs. Are we abusing the mechanism here or is this valid code?
To get messages on-demand, it's generally better to use rabbitTemplate.receiveAndConvert() rather than a listener; that way you completely control when you receive messages.
Starting with version 1.5 you can configure the template to block for some period of time (or until a message arrives). Otherwise it immediately returns null if there's no message.
The listener is really designed for message-driven applications.
If you can block the thread in the listener until the job completes, no more messages will be delivered - by default the container has only one thread.
If you can't block the thread until the job completes, for some reason, you can stop()/start() the listener container by getting a reference to it from the Endpoint Registry.
It's generally better to stop the container on a separate thread.

Refreshing Spring context when JMS message delivered

I'd like to refresh my application context when system receives JMS message. In order to do it, I set up Spring Integration jms:message-driven-channel-adapter which forwards message to service activator implementing ApplicationContextAware. This activator (ConfigurationReloader class) invokes ConfigurableApplicationContext#refresh() method.
Below is sample code snippet:
<jms:message-driven-channel-adapter id="jmsDriverConfigurationAdapter"
destination="configurationApplyQueue" channel="jmsConfigurationInboundChannel" />
<channel id="jmsConfigurationInboundChannel"/>
<service-activator input-channel="jmsConfigurationInboundChannel" ref="configurationReloader" method="refresh"/>
And my activator:
public final class ConfigurationReloader implements ApplicationContextAware {
private ConfigurableApplicationContext applicationContext;
public void refresh() {
this.applicationContext.refresh();
}
#Override
public void setApplicationContext(
final ApplicationContext applicationContext) throws BeansException {
if (applicationContext instanceof ConfigurableApplicationContext) {
this.applicationContext =
(ConfigurableApplicationContext) applicationContext;
}
}
}
In case of delivering such message, context start shutdown operation but stuck on DefaultMessageListenerContainer bean shutdown:
2011-11-14 15:42:52,980 [org.springframework.jms.listener.DefaultMessageLis tenerContainer#0-1] DEBUG org.springframework.jms.listener.DefaultMessageLis tenerContainer - Shutting down JMS listener container
2011-11-14 15:42:52,980 [org.springframework.jms.listener.DefaultMessageLis tenerContainer#0-1] DEBUG org.springframework.jms.listener.DefaultMessageLis tenerContainer - Waiting for shutdown of message listener invokers
2011-11-14 15:42:55,104 [org.springframework.jms.listener.DefaultMessageLis tenerContainer#0-1] DEBUG org.springframework.jms.listener.DefaultMessageLis tenerContainer - Still waiting for shutdown of 1 message listener invokers
Invoking this operation over JMS is crucial for me since new configuration parameters are delivered along with message.
It is standard Spring MVC application with DispatcherServlet on the front based on the latest SpringCore and Spring Integration. Also I am sure that it's JMS related issue, because invoking ConfigurationLoader through controller works fine.
As I've debugged, it stucks after DefaultMessageListenerContainer#538 line invocation (wait() method on lifecycleMonitor):
/**
* Destroy the registered JMS Sessions and associated MessageConsumers.
*/
protected void doShutdown() throws JMSException {
logger.debug("Waiting for shutdown of message listener invokers");
try {
synchronized (this.lifecycleMonitor) {
while (this.activeInvokerCount > 0) {
if (logger.isDebugEnabled()) {
logger.debug("Still waiting for shutdown of " + this.activeInvokerCount +
" message listener invokers");
}
this.lifecycleMonitor.wait(); // <--- line 538
}
}
}
catch (InterruptedException ex) {
// Re-interrupt current thread, to allow other threads to react.
Thread.currentThread().interrupt();
}
}
...there is nobody to call notify / notifyAll on monitor so maybe it's some kind of bug?
Thank you for any hints!
Can you please explain why do you need such sophisticated architecture? Reloading application context when JMS message is received? Sounds crazy (or maybe ingenious?)
Nevertheless, I am not 100% sure but the information you provided is pretty clear: you are trying to shutdown an application context while consuming JMS message. But since the consumer is Spring-managed, context cannot be destroyed because it waits for all beans to finish - including yours ConfigurationReloader required by Spring Integration message consumer. And ConfigurationReloader cannot finish because it waits for context to be destroyed (refresh() is blocking).
Simply put - you have introduced a cyclic dependency and a deadlock.
The solution is simple - delay the context refresh so that it happens after the JMS message consumption. The easiest way would be:
public void refresh() {
Thread destroyThread = new Thread() {
#Override
public void run() {
this.applicationContext.refresh();
}
};
destroyThread.start();
}
Not pretty but I'm almost sure this will work.

Resources