I am use spring boot 2.0.2, and use Async to execute tasks.
I register a TaskExecutor to control thread pool size, like below:
public class ThreadConfig {
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(2);
executor.setMaxPoolSize(3);
executor.setQueueCapacity(3);
executor.setThreadNamePrefix("anniversary");
executor.initialize();
System.out.println("******* name " + executor.getThreadNamePrefix());
System.out.println("********** core pool size " + executor.getCorePoolSize());
return executor;
}
}
But during running, there are more than 20 threads are created,
*********Thread[SimpleAsyncTaskExecutor-1,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-2,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-3,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-4,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-5,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-6,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-7,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-8,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-9,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-10,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-11,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-12,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-13,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-14,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
Related
I am currently developing a SpringBoot version 2.6. 3 application.The app runs ffmpeg processes on thread pool executors like:
#Bean(name = "ffmpeg")
public TaskExecutor getAncaExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(10);
executor.setMaxPoolSize(10);
executor.setQueueCapacity(100);
executor.setThreadNamePrefix("ffmpeg-");
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setAwaitTerminationSeconds(120);
executor.initialize();
return executor;
}
#Autowired("ffmpeg")
TaskExecutor taskExecutor;
public void transcode() {
CompletableFuture.supplyAsync(() -> {
Process process = Runtime.getRuntime().exec("ffmpeg -i input -y.......");
result = process.waitFor();
return result;
}, taskExecutor).thenAccept(result -> handleResult());
});
}
in my application.properties I have
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=120s
When the app receives SIGINIT
ffmpeg gets killed of with 255 status code ( SIGINIT)
handle method is executed
Is there a way to wait a certain amount of time on those running ffmpeg processes to end, why does Spring Boot kills them off if taskExecutor has been with setWaitForTasksToCompleteOnShutdown(true);setAwaitTerminationSeconds(120);
I have some questions about using Spring's #Scheduled and #Async functionality together.
Lets say my requirement is to process 5 rows from a DB every 1 second, so in 1 pass of the scheduler thread would create 5 asynchronous threads
to process each row from a Database.
My questions are as follows:
1) Is there an alternative way to creating 5 ascynchonis threads instead of using a while loop within the scheduled method?
One problem I see with this approach is the thread pools active count may not equal the max pool size and therefore the loop will not break before 1 second has passed.
2) In some cases the log statement in the AsyncService i.e. Executing dbItem on the following asyncExecutor : task-scheduler-1 displays task-scheduler-1 as the thread name and not async_thread_ as i would always expect?
3) If my scheduler thread takes longer than 1 second to run, what happens the subsequent pass of the scheduler?
The asyncExecutor:
#Override
#Bean(name="asyncExecutor")
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(5);
threadPoolTaskExecutor.setQueueCapacity(5);
threadPoolTaskExecutor.setMaxPoolSize(10);
threadPoolTaskExecutor.setThreadNamePrefix("async_thread_");
threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(true);
return threadPoolTaskExecutor;
}
which is injected into a class with a scheduled method:
#Autowired
#Qualifier("asyncExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Autowired
AsyncService asyncService;
#Autowired
private DaoService daoService;
#Scheduled(fixedRateString = "1000")
public void schedulerMetod() {
try {
while (threadPoolTaskExecutor.getActiveCount() < threadPoolTaskExecutor.getMaxPoolSize()) {
DbItem dbItem = daoService.retrieveNewItemFromDB();
if (dbItem != null){
asyncService.processNewItem(dbItem);
}
}
} catch (ObjectOptimisticLockingFailureException ole){
log.info(ole.getMessage());
} catch (Exception ex){
log.error(ex.getMessage());
}
}
#Service
public class AsyncServiceImpl implements AsyncService {
#Autowired
private TaskService taskService;
#Override
#Transactional
#Async("asyncExecutor")
public void processNewItem(DbItem dbItem) {
log.debug("Executing dbItem on the following asyncExecutor : " + Thread.currentThread().getName());
taskService.processNewItem(dbItem);
}
}
Using ThreadPoolTaskScheduler bean you will have twofold purpose as it implements both TaskExecutor (supports #Async) and TaskScheduler (supports #Scheduled) interfaces. You can configure like this:
#Bean
public ThreadPoolTaskScheduler taskScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
threadPoolTaskScheduler.setPoolSize(5);
threadPoolTaskScheduler.setThreadNamePrefix("async_thread_");
threadPoolTaskScheduler.initialize();
return threadPoolTaskScheduler;
}
If you give another executor name such as taskScheduler1 then you should provide executor name as an attribute in #Async("taskScheduler1").
By this implementation you already have answers for your questions 1 & 2.
About your 3rd question, if all threads in your thread pool are busy then your schedulers do nothing until a thread released then it executes the scheduling task. You may think to give the thread pool a bigger size.
Last but not least, only usage of #Scheduled is enough as it works asynchronously.
I am using Spring Cloud library to poll SQS. How can I set poll interval?
#Bean
#Primary
public AmazonSQSAsync amazonSQSAsync() {
return AmazonSQSAsyncClientBuilder.standard().
withCredentials(awsCredentialsProvider()).
withClientConfiguration(clientConfiguration()).
build();
}
#Bean
#ConfigurationProperties(prefix = "aws.queue")
public SimpleMessageListenerContainer simpleMessageListenerContainer(AmazonSQSAsync amazonSQSAsync) {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer();
simpleMessageListenerContainer.setAmazonSqs(amazonSQSAsync);
simpleMessageListenerContainer.setMessageHandler(queueMessageHandler());
simpleMessageListenerContainer.setMaxNumberOfMessages(10);
simpleMessageListenerContainer.setTaskExecutor(threadPoolTaskExecutor());
return simpleMessageListenerContainer;
}
#Bean
public QueueMessageHandler queueMessageHandler() {
QueueMessageHandlerFactory queueMessageHandlerFactory = new QueueMessageHandlerFactory();
queueMessageHandlerFactory.setAmazonSqs(amazonSQSAsync());
QueueMessageHandler queueMessageHandler = queueMessageHandlerFactory.createQueueMessageHandler();
return queueMessageHandler;
}
#Bean
public ThreadPoolTaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(10);
executor.setMaxPoolSize(10);
executor.setThreadNamePrefix("oaoQueueExecutor");
executor.initialize();
return executor;
}
Call setWaitTimeOut(N) function of the base class AbstractMessageListenerContainer in package org.springframework.cloud.aws.messaging.listener. N is the long poll timeout in seconds.
For example if you want to wait for 5 sec before it returns, use below line of code in your queueMessageHandler() function. The default is 1 sec if you don't call this function. The maximum long polling timeout is 20 sec, so the max you can give to this function is 20 which means "wait for 20 seconds"
simpleMessageListenerContainer.setWaitTimeOut (5);
The source code is here: https://github.com/spring-cloud/spring-cloud-aws/blob/master/spring-cloud-aws-messaging/src/main/java/org/springframework/cloud/aws/messaging/listener/AbstractMessageListenerContainer.java
/**
* Configures the wait timeout that the poll request will wait for new message to arrive if the are currently no
* messages on the queue. Higher values will reduce poll request to the system significantly.
*
* #param waitTimeOut
* - the wait time out in seconds
*/
public void setWaitTimeOut(Integer waitTimeOut) {
this.waitTimeOut = waitTimeOut;
}
I have the following ThreadPoolTaskExecutor thats gets created with the expected core/max pool size configurations.
#Slf4j
#Configuration
public class ThreadPoolConfig {
#Value("${corePoolSize}")
private Integer corePoolSize;
#Value("${queueCapacity}")
private Integer queueCapacity;
#Value("${maxPoolSize}")
private Integer maxPoolSize;
#Bean(name="myThreadPoolTaskExecutor")
public ThreadPoolTaskExecutor myThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setBeanName("myThreadPoolTaskExecutor");
executor.setCorePoolSize(corePoolSize);
executor.setQueueCapacity(queueCapacity);
executor.setMaxPoolSize(maxPoolSize);
executor.setThreadNamePrefix("my_thread_");
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.initialize();
log.debug("threadPoolTaskExecutor CorePoolSize is : " + executor.getCorePoolSize());
log.debug("threadPoolTaskExecutor MaxPoolSize is : " + executor.getMaxPoolSize());
return executor;
}
}
When my #scheduled method runs the max pool size is set to the DEFAULT value of 2147483647 and I don't understand why it's not using the configured ThreadPoolTaskExecutor above:
#EnableScheduling
public class SchedulingConfig {
}
#Component
public class Scheduler {
#Autowired
#Qualifier("myThreadPoolTaskExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Scheduled(fixedRateString = "${fixedRate}")
public void invokeScheduledThread() {
while (threadPoolTaskExecutor.getActiveCount() <= threadPoolTaskExecutor.getMaxPoolSize()) {
log.debug("Active Thread Pool count is : " + threadPoolTaskExecutor.getActiveCount() + ", Max Thread Pool count is : " + threadPoolTaskExecutor.getMaxPoolSize() + " on the scheduled Thread : " + Thread.currentThread().getName());
//call a service to retrieve some items to process
threadPoolTaskExecutor.execute(Some Object that implements runnable);
}
}
}
Output:
Active Thread Pool count is : 0, Max Thread Pool count is : 2147483647 on the scheduled Thread : task-scheduler-1
I put a break point into the initialise() method of org.springframework.scheduling.concurrent.ExecutorConfigurationSupport
and it looks like the method is getting invoked 3 times, twice with a ThreadName Prefix of "my_thread_"
which is expected and finally once for a Bean called "taskScheduler" with a ThreadName Prefix of "task-scheduler-".
Does anyone know why I can't use my own ThreadPoolTaskExecutor within the Scheduler class?
I wanted to use a default #Scheduler to run on a single thread every x number of seconds and create X number of Threads using my own ThreadPoolTaskExecutor.
Use ThreadPoolTaskScheduler instead of ThreadPoolTaskExecutor.
For example:
#Configuration
public class SpringSchedulerConfig {
private static final int THREAD_POOL_SIZE = 5;
#Bean
public ThreadPoolTaskScheduler getScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
//we want every Job in a separate Thread.
threadPoolTaskScheduler.setPoolSize(THREAD_POOL_SIZE);
return threadPoolTaskScheduler;
}
}
I'm setting up a kafka listener in a spring boot application and I can't seem to get the listener running in a pool using an executor. Here's my kafka configuration:
#Bean
ThreadPoolTaskExecutor messageProcessorExecutor() {
logger.info("Creating a message processor pool with {} threads", numThreads);
ThreadPoolTaskExecutor exec = new ThreadPoolTaskExecutor();
exec.setCorePoolSize(200);
exec.setMaxPoolSize(200);
exec.setKeepAliveSeconds(30);
exec.setAllowCoreThreadTimeOut(true);
exec.setQueueCapacity(0); // Yields a SynchronousQueue
exec.setThreadFactory(ThreadFactoryFactory.defaultNamingFactory("kafka", "processor"));
return exec;
}
#Bean
public ConsumerFactory<String, PollerJob> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.GROUP_ID_CONFIG, consumerGroup);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
DefaultKafkaConsumerFactory<String, PollerJob> factory = new DefaultKafkaConsumerFactory<>(props,
new StringDeserializer(),
new JsonDeserializer<>(PollerJob.class));
return factory;
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, PollerJob> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, PollerJob> factory
= new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(Integer.valueOf(kafkaThreads));
factory.getContainerProperties().setListenerTaskExecutor(messageProcessorExecutor());
factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL);
return factory;
}
The ThreadFactoryFactory used by the ThreadPoolTaskExecutor just makes sure the thread is named like 'kafka-1-processor-1'.
The ConsumerFactory has the ENABLE_AUTO_COMMIT_CONFIG flag set to false and I'm using manual mode for the acknowledgement which is required to use executors according to the documentation.
My listener looks like this:
#KafkaListener(topics = "my_topic",
group = "my_group",
containerFactory = "kafkaListenerContainerFactory")
public void listen(#Payload SomeJob job, Acknowledgment ack) {
ack.acknowledge();
logger.info("Running job {}", job.getId());
....
}
Using the Admin Server I can inspect all the threads and only one kafka-N-processor-N threads is being created but I expected to see up to 200. The jobs are all running one at a time on the that one thread and I can't figure out why.
How can I get this setup to run the listeners using my executor with as many threads as possible?
I'm using Spring Boot 1.5.4.RELEASE and kafka 0.11.0.0.
If your topic has only one partition, according the consumer group policy, only one consumer is able to poll that partition.
The ConcurrentMessageListenerContainer indeed creates as much target KafkaMessageListenerContainer instances as provided concurrency. And it does that only in case it doesn't know the number of partitions in the topic.
When the rebalance in consumer group happens only one consumer gets partition for consuming. All the work is really done there in a single thread:
private void startInvoker() {
ListenerConsumer.this.invoker = new ListenerInvoker();
ListenerConsumer.this.listenerInvokerFuture = this.containerProperties.getListenerTaskExecutor()
.submit(ListenerConsumer.this.invoker);
}
One partition - one thread for sequential records processing.