How can I shutdown Spring boot thread pool project amicably which is 24x7 running - spring

I have created spring boot thread pool project which has thread that needs to run 24x7 once spawned but when I need to stop the app in server for some maintenance it should shutdown after completing its current task and not taking up any new task.
My code for the same is:
Config class
#Configuration
public class ThreadConfig {
#Bean
public ThreadPoolTaskExecutor taskExecutor(){
ThreadPoolTaskExecutor executorPool = new ThreadPoolTaskExecutor();
executorPool.setCorePoolSize(10);
executorPool.setMaxPoolSize(20);
executorPool.setQueueCapacity(10);
executorPool.setWaitForTasksToCompleteOnShutdown(true);
executorPool.setAwaitTerminationSeconds(60);
executorPool.initialize();
return executorPool;
}
}
Runnable class
#Component
#Scope("prototype")
public class DataMigration implements Runnable {
String name;
private boolean run=true;
public DataMigration(String name) {
this.name = name;
}
#Override
public void run() {
while(run){
System.out.println(Thread.currentThread().getName()+" Start Thread = "+name);
processCommand();
System.out.println(Thread.currentThread().getName()+" End Thread = "+name);
if(Thread.currentThread().isInterrupted()){
System.out.println("Thread Is Interrupted");
break;
}
}
}
private void processCommand() {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public void shutdown(){
this.run = false;
}
}
Main class:
#SpringBootApplication
public class DataMigrationPocApplication implements CommandLineRunner{
#Autowired
private ThreadPoolTaskExecutor taskExecutor;
public static void main(String[] args) {
SpringApplication.run(DataMigrationPocApplication.class, args);
}
#Override
public void run(String... arg0) throws Exception {
for(int i = 1; i<=20 ; i++){
taskExecutor.execute(new DataMigration("Task " + i));
}
for (;;) {
int count = taskExecutor.getActiveCount();
System.out.println("Active Threads : " + count);
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
if (count == 0) {
taskExecutor.shutdown();
break;
}
}
System.out.println("Finished all threads");
}
}
I need help to understand if I need to stop my spring boot application it should stop all the 20 threads running which runs (24x7) otherwise after completing there current loop in while loop and exit.

I would propose couple of changes in this code to resolve the problem
1) since in your POC processCommand calls Thread.sleep, when you shutdown the executor and it interrupts workers InterruptedException get called but is almost ignored in your code. After that there is if(Thread.currentThread().isInterrupted()) check which will return false for the reason above. Similar problem is outlined in the post below
how does thread.interrupt() sets the flag?
the following code change should fix the problem:
private void processCommand() {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
shutdown();
}
}
2) Also because of ThreadConfig::taskExecutor executorPool.setWaitForTasksToCompleteOnShutdown(true) Spring will call executor.shutdown instead of executor.shutdownNow. According to javadoc ExecutorService.shutdown
Initiates an orderly shutdown in which previously submitted tasks are
executed, but no new tasks will be accepted.
So I would recommend to set
executorPool.setWaitForTasksToCompleteOnShutdown(false);
Other things to improve in this code: although DataMigration is annotated as a component the instances of this class are creared not by Spring. You should try using factory method similar to ThreadConfig::taskExecutor in order to make Spring initiate instances of DataMigration for example to inject other bean into DataMigration instances.
In order to shutdown executor when running jar file on linux environment you can for example add actuator module and enable shutdown endpoint:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
in application.properties:
endpoints.shutdown.enabled=true
It will enable JMX shutdown endpoint and you can call shutdown on it.
If you want current job cycle of the task to be finished you should set
executorPool.setWaitForTasksToCompleteOnShutdown(true);
In order to connect to your jvm process on linux env remotely you have to specify an RMI Registry port.
Here is a detailed article:
How to access Spring-boot JMX remotely
If you just need to connect to JMX from local env you can run jsoncole or command-line tools : Calling JMX MBean method from a shell script
Here is an example uf using one of these tools - jmxterm
$>run -d org.springframework.boot: -b org.springframework.boot:name=shutdownEndpoint,type=Endpoint shutdown
#calling operation shutdown of mbean org.springframework.boot:name=shutdownEndpoint,type=Endpoint with params []
#operation returns:
{
message = Shutting down, bye...;
}

Related

Kafka Consumer with Circuit Breaker, Retry Patterns using Resilience4j

I need some help in understanding how I can come up with a solution using Spring boot, Kafka, Resilence4J to achieve a microservice call from my Kafka Consumer. Let's say if the Microservice is down then I need to notify my Kafka consumer using a circuit breaker pattern to stop fetching the messages/events until the Microservice is up and running.
With Spring Kafka, you could use the pause and resume methods depending on the CircuitBreaker state transitions. The best way I found for this is to define it as "supervisor" with an #Configuration Annotation. Resilience4j is also used.
#Configuration
public class CircuitBreakerConsumerConfiguration {
public CircuitBreakerConsumerConfiguration(CircuitBreakerRegistry circuitBreakerRegistry, KafkaManager kafkaManager) {
circuitBreakerRegistry.circuitBreaker("yourCBName").getEventPublisher().onStateTransition(event -> {
switch (event.getStateTransition()) {
case CLOSED_TO_OPEN:
case CLOSED_TO_FORCED_OPEN:
case HALF_OPEN_TO_OPEN:
kafkaManager.pause();
break;
case OPEN_TO_HALF_OPEN:
case HALF_OPEN_TO_CLOSED:
case FORCED_OPEN_TO_CLOSED:
case FORCED_OPEN_TO_HALF_OPEN:
kafkaManager.resume();
break;
default:
throw new IllegalStateException("Unknown transition state: " + event.getStateTransition());
}
});
}
}
This is what I used in combination with a KafkaManager annotated with #Component.
#Component
public class KafkaManager {
private final KafkaListenerEndpointRegistry registry;
public KafkaManager(KafkaListenerEndpointRegistry registry) {
this.registry = registry;
}
public void pause() {
registry.getListenerContainers().forEach(MessageListenerContainer::pause);
}
public void resume() {
registry.getListenerContainers().forEach(MessageListenerContainer::resume);
}
}
In addition my consumer service looks like this:
#KafkaListener(topics = "#{'${topic.name}'}", concurrency = "1", id = "CBListener")
public void receive(final ConsumerRecord<String, ReplayData> replayData, Acknowledgment acknowledgment) throws
Exception {
try {
httpClientServiceCB.receiveHandleCircuitBreaker(replayData);
acknowledgement.acknowledge();
} catch (Exception e) {
acknowledgment.nack(1000);
}
}
And the #CircuitBreaker Annotation:
#CircuitBreaker(name = "yourCBName")
public void receiveHandleCircuitBreaker(ConsumerRecord<String, ReplayData> replayData) throws
Exception {
try {
String response = restTemplate.getForObject("http://localhost:8081/item", String.class);
} catch (Exception e ) {
// throwing the exception is needed to trigger the Circuit Breaker state change
throw new Exception();
}
}
And this is additionally supplemented by the following application.properties
resilience4j.circuitbreaker.instances.yourCBName.failure-rate-threshold=80
resilience4j.circuitbreaker.instances.yourCBName.sliding-window-type=COUNT_BASED
resilience4j.circuitbreaker.instances.yourCBName.sliding-window-size=5
resilience4j.circuitbreaker.instances.yourCBName.wait-duration-in-open-state=10000
resilience4j.circuitbreaker.instances.yourCBName.automatic-transition-from-open-to-half-open-enabled=true
spring.kafka.consumer.enable.auto.commit = false
spring.kafka.listener.ack-mode = MANUAL_IMMEDIATE
Also have a look at https://resilience4j.readme.io/docs/circuitbreaker
If you are using Spring Kafka, you could maybe use the pause and resume methods of the ConcurrentMessageListenerContainer class.
You can attach an EventListener to the CircuitBreaker which listens on state transitions and pauses or resumes processing of events. Inject the CircuitBreakerRegistry into you bean:
circuitBreakerRegistry.circuitBreaker("yourCBName").getEventPublisher().onStateTransition(
event -> {
switch (event.getStateTransition()) {
case CLOSED_TO_OPEN:
container.pause();
case OPEN_TO_HALF_OPEN:
container.resume();
case HALF_OPEN_TO_CLOSED:
container.resume();
case HALF_OPEN_TO_OPEN:
container.pause();
case CLOSED_TO_FORCED_OPEN:
container.pause();
case FORCED_OPEN_TO_CLOSED:
container.resume();
case FORCED_OPEN_TO_HALF_OPEN:
container.resume();
default:
}
}
);

Can not run few methods sequentially when Spring Boot starts

I have to run a few methods when Application starts, like the following:
#SpringBootApplication
public class Application implements CommandLineRunner {
private final MonitoringService monitoringService;
private final QrReaderServer qrReaderServer;
#Override
public void run(String... args) {
monitoringService.launchMonitoring();
qrReaderServer.launchServer();
}
However, only the first one is executed! And the application is started:
... Started Application in 5.21 seconds (JVM running for 6.336)
... START_MONITORING for folder: D:\results
The second one is always skipped!
If change the call order - the only the second one will be executed.
Could not find any solution for launching both at the beginning - tried #PostConstruct, ApplicationRunner, #EventListener(ApplicationReadyEvent.class)...
Looks like they are blocking each other somehow. Despite the fact that both have void type.
Monitoring launch implementation:
#Override
public void launchMonitoring() {
log.info("START_MONITORING for folder: {}", monitoringProperties.getFolder());
try {
WatchKey key;
while ((key = watchService.take()) != null) {
for (WatchEvent<?> event : key.pollEvents()) {
WatchEvent.Kind<?> kind = event.kind();
if (kind == ENTRY_CREATE) {
log.info("FILE_CREATED: {}", event.context());
// some delay for fully file upload
Thread.sleep(monitoringProperties.getFrequency());
String fullFileName = getFileName(event);
String fileName = FilenameUtils.removeExtension(fullFileName);
processResource(fullFileName, fileName);
}
}
key.reset();
}
} catch (InterruptedException e) {
log.error("interrupted exception for monitoring service", e);
} catch (IOException e) {
log.error("io exception while processing file", e);
}
}
QR Reader start (launch TCP server with Netty configuration):
#Override
public void launchServer() {
try {
ChannelFuture serverChannelFuture = serverBootstrap.bind(hostAddress).sync();
log.info("Server is STARTED : port {}", hostAddress.getPort());
serverChannel = serverChannelFuture.channel().closeFuture().sync().channel();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
shutdownQuietly();
}
}
How to solve this issue?
Start launchMonitoring() asynchronously.
The easiest way to do this is to enable Async by adding #EnableAsync on your Application
and then annotate launchMonitoring() with #Async
Not sure if launchServer() should also be started asynchronously.
EDIT: completed Answer
No task executor bean found for async processing: no bean of type TaskExecutor and no bean named 'taskExecutor' either
By default Spring will create a SimpleAsyncTaskExecutor, but you can provide your taskExecutor
Example:
#EnableAsync
#Configuration
public class AsyncConfig implements AsyncConfigurer {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.set... // your custom configs
executor.initialize();
return executor;
}
...
}

odd behaviour - websocket spring - send message to user using listen / notify postgresql

I am experiencing an odd behavior of my spring boot websocket set-up.
Sometimes it works, sometimes it doesn't, it just feels random.
I have tried the several setups, none proved solid: I moved the last piece of code in a commandlinerunner inside the primary class of the application and the last choice was a different class with #Component annotation.
My setup is the following: I use a jdbc driver (pgjdbc-ng) to use the listen notify function of postgres.I have a function and a trigger that listens to a specific postgres table for inserations. If any occur, notifications are sent through the websocket. The other and is an angular app that uses ng2-stompjs to listen to /topic/notificari for notifications. I am not posting the code because the notifications don't get out of spring, the angular is not the problem.
Kind regards,
This is my WebSocketConfiguration
Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
registry.enableSimpleBroker("/topic", "/queue", "/user", "/notificari");
registry.setApplicationDestinationPrefixes("/app");
registry.setUserDestinationPrefix("/user");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/socket").setAllowedOrigins("*")
.setHandshakeHandler(new CustomHandshakeHandler());
}
I am using a class ListenNotify and the JDBC driver pgjdbc-ng to connect to the postgresql db and use listen notify functionality
public class ListenNotify {
private BlockingQueue queue = new ArrayBlockingQueue(20);
PGConnection connection;
public ListenNotify() {
PGNotificationListener listener = new PGNotificationListener() {
#Override
public void notification(int processId, String channelName, String payload) {
queue.add(payload);
}
};
try {
PGDataSource dataSource = new PGDataSource();
dataSource.setHost("localhost");
dataSource.setDatabase("db");
dataSource.setPort(5432);
dataSource.setUser("user");
dataSource.setPassword("pass");
connection = (PGConnection) dataSource.getConnection();
connection.addNotificationListener(listener);
Statement statement = connection.createStatement();
statement.execute("LISTEN n_event");
statement.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
public BlockingQueue getQueue() {
return queue;
}
}
And finally this is the code that instantiate the ListenNotify object and listens to postgres for events that might trigger notifications that have to be send using websocket.
#Component
public class InstantaNotificari {
#Autowired
SimpMessagingTemplate template;
#EventListener(ApplicationReadyEvent.class)
public void runn() {
System.out.println("invocare met");
ListenNotify ln = new ListenNotify();
BlockingQueue queue = ln.getQueue();
System.out.println("the que ies "+ queue);
while (true) {
try {
String msg = (String) queue.take();
System.out.println("msg " + msg);
template.convertAndSend("/topic/notificari", msg);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
I didn't use Spring so I can't test your code. Here is my tested version. I think this summarizes the differences -
Change to a try with resources block. This will close the connection on destruction of the class.
Move your while(true) into the try block on the Listener so that the
lines inside the try block doesn't ever get out of execution scope.
The while(true) is blocking, so it needs to be on another thread. ListenNotify extends Thread
I'm sure there are other ways of implementing and welcome corrections to any of my assumptions.
My tested, running code is in this answer JMS Websocket delayed delivery.

spring restservice asynchronous logging functionality

I had written rest services in spring that is running perfectly fine.
Now, I need to add perform some db transactions before returning response to user.
This db transaction is independent to response retrieved.
For example,
#PostMapping("login")
public TransactionResponse loginAuthentication(#Valid #RequestBody LoginRequestBody loginRequest) {
TransactionResponse transactionResponse = new TransactionResponse();
try {
transactionResponse = loginService.validateUser(loginRequest);
//independent transaction needs to be executed in a separate thread
loginSerice.addLoginLog(transactionResponse);
//return below response without waiting to compelete above log transaction
return transactionResponse;
}
catch (Exception e) {
return CommonUtils.setErrorResponse(transactionResponse, e);
}
}
I read upon async controller in spring mvc link. Although controller
executes respective functionality in a separate thread but I don't want to wait for db transaction to be completed. After getting response from service layer, it should be forwarded to user without any delay.
Any Suggestions !!
Spring version is 4.3
I posted this answer to help the fellow developers with same kind of requirement (to execute a void function in a separate thread).
Since I am not experienced in multithreading/asynchronous environment, I want to keep it simple by using spring asynchronous methods.
So, First I created the Thread Pool
#Configuration
#EnableAsync
public class ThreadConfig {
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(4);
executor.setMaxPoolSize(4);
executor.setThreadNamePrefix("WEBAPP");
executor.initialize();
return executor;
}
}
Then I created a service that will execute my code in a separate thread.
#Async
#Service
#Transactional
public class LoggingService {
public void logintransaction() throws Exception{
System.out.println("start login loggin");
Thread.sleep(5000);
System.out.println("exit");
}
}
Lastly, I called the above service on my controller. As I can see Total Time Taken is printed first, then "start login loggin" is printed. This means my new method is executed in a new thread.
#Autowired
private LoggingService loggingService;
#PostMapping("login")
public TransactionResponse loginAuthentication(#Valid #RequestBody LoginRequestBody loginRequest) {
long startTime = System.currentTimeMillis();
TransactionResponse transactionResponse = new TransactionResponse();
try {
transactionResponse = loginService.validateUser(loginRequest);
//independent transaction needs to be executed in a separate thread
//loginSerice.addLoginLog(transactionResponse);
loggingService.logintransaction();
//return below response without waiting to compelete above log transaction
System.err.println("Total Time Taken=>"+(System.currentTimeMillis() - startTime));
return transactionResponse;
}
catch (Exception e) {
return CommonUtils.setErrorResponse(transactionResponse, e);
}
}
Thanks

OSGI Integration testing, Pax Exam probe executing tests before container bundles start

I'm currently trying to write an OSGI integration tests using PAX-EXAM+KARAF and faced a problem that pax exam trying to execute test methods before dependent bundles are actually started/initialized in container. What's strange, sometimes test succeeds, and in that case all bundles/contexts are started and logged, but the majority of times it does not. Delays in methods won't help:( Anyone could help with this issue?
I'm using
PAX-EXAM 2.6.0,
org.apache.karaf.tooling.exam.container 2.3.0,
apache-karaf 2.3.0.
Code:
#Inject
BundleContext bundleContext;
#Inject
EntityManagerFactoryService entityManagerFactoryService;//Service exposed trough OSGI
protected EntityManager entityManager;
#Before
public void init() throws InterruptedException {
entityManager = entityManagerFactoryService.getEntityManagerFactory().createEntityManager();
}
#Configuration
public static Option[] configuration() throws Exception {
return new Option[] {
karafDistributionConfiguration().frameworkUrl(maven().groupId("org.apache.karaf").artifactId("apache-karaf").type("zip").versionAsInProject())
.karafVersion("2.3.0").name("Apache Karaf"),
mavenBundle("com.google.guava", "guava", "13.0.1").startLevel(30),
mavenBundle("com.mysql.jdbc", "com.springsource.com.mysql.jdbc", "5.1.6").startLevel(30),
mavenBundle("javax.persistence", "com.springsource.javax.persistence", "2.0.0").startLevel(30),
mavenBundle("org.apache.commons", "com.springsource.org.apache.commons.lang", "2.6.0").startLevel(30),
...the rest of bundles
junitBundles(), };
Test method:
#Test
public void contextNotNull() {
Assert.assertNotNull(entityManagerFactoryService);
}
The log:
java.lang.ClassNotFoundException: com.startjg.crp.core.dao.service.EntityManagerFactoryService not found by PAXEXAM-PROBE-749fa717-8bdc-4d9a-9803-bdaf6d4edac0 [144]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Full log:
https://www.dropbox.com/s/v12r15sbmtu9svp/log.txt
Also tried with no luck:
protected <T> Object getService(Class<T> serviceClass) {
int maxCount = 5;
int delay = 5000;
for (int i = 0; i <= maxCount; i++) {
if (bundleContext.getServiceReference(serviceClass) != null) {
ServiceReference<T> serviceReference = bundleContext.getServiceReference(serviceClass);
return bundleContext.getService(serviceReference);
} else {
try {
Thread.sleep(delay);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
return null;
}
#Before
public void init() throws InterruptedException {
EntityManagerFactoryService emfs = (EntityManagerFactoryService) getService(EntityManagerFactoryService.class);
entityManager = entityManagerFactoryService.getEntityManagerFactory().createEntityManager();
}
Are you sure your EntityManager is working and available as Service?
You are most probably missing some dependencies, that's why your bundle containing the service isn't fully started.
The current pax exam 3.3.0 now fully supports karaf as a container. The former karaf exam code now fully moved to pax exam.

Resources