I have the following ThreadPoolTaskExecutor thats gets created with the expected core/max pool size configurations.
#Slf4j
#Configuration
public class ThreadPoolConfig {
#Value("${corePoolSize}")
private Integer corePoolSize;
#Value("${queueCapacity}")
private Integer queueCapacity;
#Value("${maxPoolSize}")
private Integer maxPoolSize;
#Bean(name="myThreadPoolTaskExecutor")
public ThreadPoolTaskExecutor myThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setBeanName("myThreadPoolTaskExecutor");
executor.setCorePoolSize(corePoolSize);
executor.setQueueCapacity(queueCapacity);
executor.setMaxPoolSize(maxPoolSize);
executor.setThreadNamePrefix("my_thread_");
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.initialize();
log.debug("threadPoolTaskExecutor CorePoolSize is : " + executor.getCorePoolSize());
log.debug("threadPoolTaskExecutor MaxPoolSize is : " + executor.getMaxPoolSize());
return executor;
}
}
When my #scheduled method runs the max pool size is set to the DEFAULT value of 2147483647 and I don't understand why it's not using the configured ThreadPoolTaskExecutor above:
#EnableScheduling
public class SchedulingConfig {
}
#Component
public class Scheduler {
#Autowired
#Qualifier("myThreadPoolTaskExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Scheduled(fixedRateString = "${fixedRate}")
public void invokeScheduledThread() {
while (threadPoolTaskExecutor.getActiveCount() <= threadPoolTaskExecutor.getMaxPoolSize()) {
log.debug("Active Thread Pool count is : " + threadPoolTaskExecutor.getActiveCount() + ", Max Thread Pool count is : " + threadPoolTaskExecutor.getMaxPoolSize() + " on the scheduled Thread : " + Thread.currentThread().getName());
//call a service to retrieve some items to process
threadPoolTaskExecutor.execute(Some Object that implements runnable);
}
}
}
Output:
Active Thread Pool count is : 0, Max Thread Pool count is : 2147483647 on the scheduled Thread : task-scheduler-1
I put a break point into the initialise() method of org.springframework.scheduling.concurrent.ExecutorConfigurationSupport
and it looks like the method is getting invoked 3 times, twice with a ThreadName Prefix of "my_thread_"
which is expected and finally once for a Bean called "taskScheduler" with a ThreadName Prefix of "task-scheduler-".
Does anyone know why I can't use my own ThreadPoolTaskExecutor within the Scheduler class?
I wanted to use a default #Scheduler to run on a single thread every x number of seconds and create X number of Threads using my own ThreadPoolTaskExecutor.
Use ThreadPoolTaskScheduler instead of ThreadPoolTaskExecutor.
For example:
#Configuration
public class SpringSchedulerConfig {
private static final int THREAD_POOL_SIZE = 5;
#Bean
public ThreadPoolTaskScheduler getScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
//we want every Job in a separate Thread.
threadPoolTaskScheduler.setPoolSize(THREAD_POOL_SIZE);
return threadPoolTaskScheduler;
}
}
Related
I have InboundChannelAdapter configured with S3StreamingMessageSource.
I forced Poller to use taskExecutor with only 1 thread. But I see the same file is being picked up by the same thread 3 times with 3-4 seconds interval. Even though poller interval is 10 seconds. I've specified Composite filter which consists of pattern filter and acceptoncefilter. But no result, file is always picked up 3 times.
String prefix = "some_prefix";
String channel = "some_channel"
Pattern filePattern = Pattern.compile(
"^" + prefix + "some_file_name_pattern");
#Bean
#InboundChannelAdapter(value = channel,
poller = #Poller(fixedDelay = "10000", taskExecutor = "threadPoolTaskExecutor"))
public MessageSource<InputStream> createS3InboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template());
messageSource.setRemoteDirectory(bucketName);
CompositeFileListFilter<S3ObjectSummary> compositeFileListFilter = new ChainFileListFilter<>();
compositeFileListFilter.addFilter(new S3PersistentAcceptOnceFileListFilter(
new SimpleMetadataStore(), prefix));
compositeFileListFilter.addFilter(new S3RegexPatternFileListFilter(filePattern));
messageSource.setFilter(compositeFileListFilter);
return messageSource;
}
#Transformer(inputChannel = channel,"another_channel")
public Message<S3ObjectInputStream> enrich(Message<S3ObjectInputStream> s3ObjectInputStreamMessage) {
S3ObjectInputStream s3ObjectInputStream = s3ObjectInputStreamMessage.getPayload();
URI zipUri = s3ObjectInputStream.getHttpRequest().getURI();
LOGGER.info("Picking up file : {}", zipUri.getPath());
...
}
private S3RemoteFileTemplate template() {
S3SessionFactory sessionFactory = new S3SessionFactory(amazonS3);
return new S3RemoteFileTemplate(sessionFactory);
}
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setMaxPoolSize(1);
executor.setThreadNamePrefix("single_thread_task_executor");
executor.initialize();
return executor;
}
I see that the app comes to #Transformer 3 times. Would really appreciate any help.
I have some questions about using Spring's #Scheduled and #Async functionality together.
Lets say my requirement is to process 5 rows from a DB every 1 second, so in 1 pass of the scheduler thread would create 5 asynchronous threads
to process each row from a Database.
My questions are as follows:
1) Is there an alternative way to creating 5 ascynchonis threads instead of using a while loop within the scheduled method?
One problem I see with this approach is the thread pools active count may not equal the max pool size and therefore the loop will not break before 1 second has passed.
2) In some cases the log statement in the AsyncService i.e. Executing dbItem on the following asyncExecutor : task-scheduler-1 displays task-scheduler-1 as the thread name and not async_thread_ as i would always expect?
3) If my scheduler thread takes longer than 1 second to run, what happens the subsequent pass of the scheduler?
The asyncExecutor:
#Override
#Bean(name="asyncExecutor")
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(5);
threadPoolTaskExecutor.setQueueCapacity(5);
threadPoolTaskExecutor.setMaxPoolSize(10);
threadPoolTaskExecutor.setThreadNamePrefix("async_thread_");
threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(true);
return threadPoolTaskExecutor;
}
which is injected into a class with a scheduled method:
#Autowired
#Qualifier("asyncExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Autowired
AsyncService asyncService;
#Autowired
private DaoService daoService;
#Scheduled(fixedRateString = "1000")
public void schedulerMetod() {
try {
while (threadPoolTaskExecutor.getActiveCount() < threadPoolTaskExecutor.getMaxPoolSize()) {
DbItem dbItem = daoService.retrieveNewItemFromDB();
if (dbItem != null){
asyncService.processNewItem(dbItem);
}
}
} catch (ObjectOptimisticLockingFailureException ole){
log.info(ole.getMessage());
} catch (Exception ex){
log.error(ex.getMessage());
}
}
#Service
public class AsyncServiceImpl implements AsyncService {
#Autowired
private TaskService taskService;
#Override
#Transactional
#Async("asyncExecutor")
public void processNewItem(DbItem dbItem) {
log.debug("Executing dbItem on the following asyncExecutor : " + Thread.currentThread().getName());
taskService.processNewItem(dbItem);
}
}
Using ThreadPoolTaskScheduler bean you will have twofold purpose as it implements both TaskExecutor (supports #Async) and TaskScheduler (supports #Scheduled) interfaces. You can configure like this:
#Bean
public ThreadPoolTaskScheduler taskScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
threadPoolTaskScheduler.setPoolSize(5);
threadPoolTaskScheduler.setThreadNamePrefix("async_thread_");
threadPoolTaskScheduler.initialize();
return threadPoolTaskScheduler;
}
If you give another executor name such as taskScheduler1 then you should provide executor name as an attribute in #Async("taskScheduler1").
By this implementation you already have answers for your questions 1 & 2.
About your 3rd question, if all threads in your thread pool are busy then your schedulers do nothing until a thread released then it executes the scheduling task. You may think to give the thread pool a bigger size.
Last but not least, only usage of #Scheduled is enough as it works asynchronously.
I am use spring boot 2.0.2, and use Async to execute tasks.
I register a TaskExecutor to control thread pool size, like below:
public class ThreadConfig {
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(2);
executor.setMaxPoolSize(3);
executor.setQueueCapacity(3);
executor.setThreadNamePrefix("anniversary");
executor.initialize();
System.out.println("******* name " + executor.getThreadNamePrefix());
System.out.println("********** core pool size " + executor.getCorePoolSize());
return executor;
}
}
But during running, there are more than 20 threads are created,
*********Thread[SimpleAsyncTaskExecutor-1,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-2,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-3,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-4,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-5,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-6,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-7,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-8,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-9,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-10,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-11,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-12,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-13,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
**********Thread[SimpleAsyncTaskExecutor-14,5,com.abc.team1.project1.anniversary.AnniversaryApplication]*********
I am reading Hazelcast documentation (http://docs.hazelcast.org/docs/latest-development/manual/html/Distributed_Computing/Executor_Service/Implementing_a_Runnable_Task.html)
To manage Executor in cloud environment, hazelcast retrieve Executor like ...
public class MasterMember {
public static void main( String[] args ) throws Exception {
HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
IExecutorService executor = hazelcastInstance.getExecutorService( "exec" );
for ( int k = 1; k <= 1000; k++ ) {
Thread.sleep( 1000 );
System.out.println( "Producing echo task: " + k );
executor.execute( new EchoTask( String.valueOf( k ) ) );
}
System.out.println( "EchoTaskMain finished!" );
}
}
However, I could not find the part which register ExecutorService named exec. I would like to know how to register Executor to HazelcastInstance.
FYI, My project is based on Spring boot, and initiate HazelcastInstance like below (in ApplicationContextAware implementation)
public static HazelcastInstance getHazelcastIntance(){
return context.getBean(HazelcastInstance.class) ;
}
summary. I would like to know how to register a Executor to HazelcastInstance.
Thanks.
===============================================
Edit.
I got a TaskExecutor bean in my application Context.
But I can not generate IExecutorService with Spring TaskExecutor bean.
IExecutorService executor = hazelcastInstance.getExecutorService("taskExecutor");
executor.execute(new ParallelDBSyncExecutor(executeId) {
#Override
public void action() throws Exception {
this.dbSyncService.executeWithIteration(this.executeId, start, total);
}
});
Hazelcast can detect Spring bean which is already registered?
I checked http://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/concurrent/ThreadPoolTaskExecutor.html
There is no getter for queue size, only queue capacity.
If I use jmx to monnitor ThreadPoolTaskExecutor, how can I monitor queue size level to make sure it is healthy?
executor.getThreadPoolExecutor().getQueue().size()
EDIT
#ManagedResource
public class MyTEMBean {
private final ThreadPoolTaskExecutor te;
public MyTEMBean(ThreadPoolTaskExecutor te) {
this.te = te;
}
#ManagedAttribute
public int getQueueSize() {
return this.te.getThreadPoolExecutor().getQueue().size();
}
}