I've a simple spring bean/component with one method and one int property (initialized to 0) as shown below and this method (Kafka consumer) is being called multiple times in a short span of time.
#Component
public class MyKafkaConsumer{
public int messageCount = 0;
#KafkaListener
public void consumeMessage(MyPayload payload){
messageCount++;
System.out.println("messageCount - "+ messageCount);
}
}
Now when I see the output of this method, it's coming as below and I'm surprised to see this output.
messageCount - 1
messageCount - 2
messageCount - 3
messageCount - 4
messageCount - 1
messageCount - 2
...
Here is my configuration
#Configuration
#ComponentScan(basePackages = {
"mypackages"})
#EnableAutoConfiguration(exclude = EmbeddedMongoAutoConfiguration.class)
#EnableDiscoveryClient
public class MyApplication {
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
System.out.println("MyApplication is ready........");
}
}
This behavior is not consistent but intermittent. Can someone help me understand why is it behaving like this? Am i missing something basic here?
I tried with synchronized block as shown below
#Component
public class MyKafkaConsumer{
public int messageCount = 0;
#KafkaListener
public void consumeMessage(MyPayload payload){
System.out.println("Thread - "+Thread.currentThread().getName()+" count "+messageCount);
synchronized (this){
messageCount++;
}
}
}
and here is the output and we can clearly see there is only one thread operating there and the count is expected to be 7 for but it's not
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 0
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 1
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 2
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 5
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 3
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 4
Thread - org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1 count 5
Related
How to switch back from blocking scheduler (blocking-pool) to previous scheduler (reactor-http-nio) using Spring Webflux + Netty + Reactor?
The code:
#RequiredArgsConstructor
#Service
#Slf4j
public class BookService {
private final IBookRepo bookRepo;
private final BlockingPoolConfig blockingPoolConfig;
public Mono<Optional<Book>> getBook(Long id) {
log.debug("getBook() - id: {}", id);
return asyncCallable(() -> {
log.trace("getBook() - invoking bookRepo.findById(id) ...");
return bookRepo.findById(id);
});
}
protected <S> Mono<S> asyncCallable(Callable<S> callable) {
return Mono.fromCallable(callable)
.subscribeOn(blockingPoolConfig.blockingScheduler());
}
}
#RestController
#RequiredArgsConstructor
#Slf4j
public class BookController {
private final BookService bookService;
#GetMapping("/book/{id}")
public Mono<Book> get(#PathVariable Long id) {
log.debug("get() - id: {}", id);
return bookService.getBook(id)
.publishOn(Schedulers.parallel()) //publishOn(... ?)
.map(optionalBook -> {
return optionalBook.map(book -> {
log.debug("get() result: {}", book);
return book;
}).orElseThrow(() -> {
log.debug("book with id: {} is not found.", id);
return new ResponseStatusException(HttpStatus.NOT_FOUND, "Book not found");
});
});
}
#Configuration
#Slf4j
public class BlockingPoolConfig {
#Value("${spring.datasource.maximumPoolSize:8}")
private int connectionPoolSize = 1;
#Scope("singleton")
#Bean
public Scheduler blockingScheduler() {
Scheduler scheduler = Schedulers.newBoundedElastic(connectionPoolSize, connectionPoolSize, "blocking-pool");
return scheduler;
}
}
Above i'm using publishOn(Schedulers.parallel()), but this one creates new thread pool (parallel). Instead of this I prefer to switch reactor-http-nio thread pool.
Log of actual result:
19:17:45.290 [reactor-http-nio-2 ] DEBUG t.a.p.controller.BookController - get() - id: 1
19:17:45.291 [reactor-http-nio-2 ] DEBUG t.a.p.service.BookService - getBook() - id: 1
19:17:45.316 [blocking-pool-1 ] TRACE t.a.p.service.BookService - getBook() - invoking bookRepo.findById(id) ...
19:17:45.427 [parallel-2 ] DEBUG t.a.p.controller.BookController - get() result: Book(id=1, title=Abc)
Log of expected result:
19:17:45.290 [reactor-http-nio-2 ] DEBUG t.a.p.controller.BookController - get() - id: 1
19:17:45.291 [reactor-http-nio-2 ] DEBUG t.a.p.service.BookService - getBook() - id: 1
19:17:45.316 [blocking-pool-1 ] TRACE t.a.p.service.BookService - getBook() - invoking bookRepo.findById(id) ...
19:17:45.427 [reactor-http-nio-2 ] DEBUG t.a.p.controller.BookController - get() result: Book(id=1, title=Abc)
This is currently not possible, because A) these HTTP threads are not controlled by a Reactor Scheduler, but by the underlying Netty event loop itself, and B) there's no generic way in Java to "return execution to an (arbitrary) thread" if that thread doesn't have an Executor/ExecutorService associated with it.
For reactor-netty, once you've switched out of the HTTP threads there should be little reason to want to switch back to the Netty threads anyway. It will be done naturally by reactor-netty once the response is sent.
Assuming blocking pool is something like Schedulers.boundedElastic(), you might indeed want to go to Schedulers.parallel() to limit the life of the blocking threads, and that's a perfectly fine solution.
I have next implementation of RecursiveAction, single purpose of this class - is to print from 0 to 9, but from different threads, if possible:
public class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
new MyRecursiveAction(num + 1).fork();
}
}
}
And I thought that invoking awaitQuiescence will make current thread to wait until all tasks (submitted and forked) will be completed:
public class Main {
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkJoinPool.execute(new MyRecursiveAction(0));
System.out.println(forkJoinPool.awaitQuiescence(5, TimeUnit.SECONDS) ? "tasks" : "time");
}
}
But I don't always get correct result, instead of printing 10 times, prints from 0 to 10 times.
But if I add helpQuiesce to my implementation of RecursiveAction:
public class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
new MyRecursiveAction(num + 1).fork();
}
RecursiveAction.helpQuiesce();//here
}
}
Everything works fine.
I want to know for what actually awaitQuiescence waiting?
You get an idea of what happens when you change the System.out.println(num); to System.out.println(num + " " + Thread.currentThread());
This may print something like:
0 Thread[ForkJoinPool-1-worker-3,5,main]
1 Thread[main,5,main]
tasks
2 Thread[ForkJoinPool.commonPool-worker-3,5,main]
When awaitQuiescence detects that there are pending tasks, it helps out by stealing one and executing it directly. Its documentation says:
If called by a ForkJoinTask operating in this pool, equivalent in effect to ForkJoinTask.helpQuiesce(). Otherwise, waits and/or attempts to assist performing tasks until this pool isQuiescent() or the indicated timeout elapses.
Emphasis added by me
This happens here, as we can see, a task prints “main” as its executing thread. Then, the behavior of fork() is specified as:
Arranges to asynchronously execute this task in the pool the current task is running in, if applicable, or using the ForkJoinPool.commonPool() if not inForkJoinPool().
Since the main thread is not a worker thread of a ForkJoinPool, the fork() will submit the new task to the commonPool(). From that point on, the fork() invoked from a common pool’s worker thread will submit the next task to the common pool too. But awaitQuiescence invoked on the custom pool doesn’t wait for the completion of the common pool’s tasks and the JVM terminates too early.
If you’re going to say that this is a flawed API design, I wouldn’t object.
The solution is not to use awaitQuiescence for anything but the common pool¹. Normally, a RecursiveAction that splits off sub tasks should wait for their completion. Then, you can wait for the root task’s completion to wait for the completion of all associated tasks.
The second half of this answer contains an example of such a RecursiveAction implementation.
¹ awaitQuiescence is useful when you don’t have hands on the actual futures, like with a parallel stream that submits to the common pool.
Everything works fine.
No it does not, you got lucky that it worked when you inserted:
RecursiveAction.helpQuiesce();
To explain this let's slightly change your example a bit:
static class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
new MyRecursiveAction(num + 1).fork();
}
}
}
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkJoinPool.execute(new MyRecursiveAction(0));
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(2));
}
If you run this, you will notice that you get the result you expect to get. And there are two main reasons for this. First, fork method will execute the task in the common pool as the other answer already explained. And second, is that threads in the common pool are daemon threads. JVM is not waiting for them to finish before exiting, it exists early. So if that is the case, you might ask why it works. It does because of this line:
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(2));
which makes the main thread (which is a non daemon thread) sleep for two seconds, giving enough time for the ForkJoinPool to execute your task.
Now let's change the code closer to your example:
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkJoinPool.execute(new MyRecursiveAction(0));
System.out.println(forkJoinPool.awaitQuiescence(5, TimeUnit.SECONDS) ? "tasks" : "time");
}
specifically, you use: forkJoinPool.awaitQuiescence(...), which is documented as:
Otherwise, waits and/or attempts to assist performing tasks...
It does not say that it will necessarily wait, it says it will "wait and/or attempt ...", in this case it is more or, than and. As such, it will attempt to help, but still it will not wait for all the tasks to finish. Is this weird or even stupid?
When you insert RecursiveAction.helpQuiesce(); you are eventually calling the same awaitQuiescence (with different arguments) under the hood - so essentially nothing changes; the fundamental problem is still there:
static ForkJoinPool forkJoinPool = new ForkJoinPool();
static AtomicInteger res = new AtomicInteger(0);
public static void main(String[] args) {
forkJoinPool.execute(new MyRecursiveAction(0));
System.out.println(forkJoinPool.awaitQuiescence(5, TimeUnit.SECONDS) ? "tasks" : "time");
System.out.println(res.get());
}
static class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10_000) {
res.incrementAndGet();
System.out.println(num + " thread : " + Thread.currentThread().getName());
new MyRecursiveAction(num + 1).fork();
}
RecursiveAction.helpQuiesce();
}
}
When I run this, it never printed 10000, showing that the insertions of that line changes nothing.
The usual default way to handle such things is to fork then join. And one more join in the caller, on the ForkJoinTask that you get back when calling submit. Something like:
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool(2);
ForkJoinTask<Void> task = forkJoinPool.submit(new MyRecursiveAction(0));
task.join();
}
static class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
MyRecursiveAction ac = new MyRecursiveAction(num + 1);
ac.fork();
ac.join();
}
}
}
I'm trying to use the idle between polls mentioned here to slow down the consumption rate, i also use the max.poll.interval.ms to double the idle between polls, but its always triggering partition rebalance, any idea what is the problem?
[Edit]
I have 5 hosts and i'm setting concurrency level to 1
[Edit 2]
I was setting the idle between polls to 5 min and max.poll.interval.ms to 10 min i also noticed this log "About to close the idle connection from 105 due to being idle for 540012 millis".
I decreased the idle between polls to 10 sec and the issue disappeared, any idea why?
private ConsumerFactory<String, GenericRecord> dlqConsumerFactory() {
Map<String, Object> configurationProperties = commonConfigs();
DlqConfiguration dlqConfiguration = kafkaProperties.getConsumer().getDlq();
final Integer idleBetweenPollInterval = dlqConfiguration.getIdleBetweenPollInterval()
.orElse(DLQ_POLL_INTERVAL);
final Integer maxPollInterval = idleBetweenPollInterval * 2; // two times the idleBetweenPoll, to prevent re-balancing
logger.info("Setting max poll interval to {} for DLQ", maxPollInterval);
overrideIfRequired(DQL_CONSUMER_CONFIGURATION, configurationProperties, ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, maxPollInterval);
dlqConfiguration.getMaxPollRecords().ifPresent(maxPollRecords ->
overrideIfRequired(DQL_CONSUMER_CONFIGURATION, configurationProperties, ConsumerConfig.MAX_POLL_RECORDS_CONFIG, maxPollRecords)
);
return new DefaultKafkaConsumerFactory<>(configurationProperties);
}
<time to process last polled records> + <idle between polls> must be less than max.poll.interval.ms.
EDIT
There is logic in the container to make sure we never exceed the max poll interval:
idleBetweenPolls = Math.min(idleBetweenPolls,
this.maxPollInterval - (System.currentTimeMillis() - this.lastPoll)
- 5000); // NOSONAR - less by five seconds to avoid race condition with rebalance
I can't reproduce the issue with this...
#SpringBootApplication
public class So63411124Application {
public static void main(String[] args) {
SpringApplication.run(So63411124Application.class, args);
}
#KafkaListener(id = "so63411124", topics = "so63411124")
public void listen(String in) {
System.out.println(in);
}
#Bean
public ApplicationRunner runner(ConcurrentKafkaListenerContainerFactory<?, ?> factory,
KafkaTemplate<String, String> template) {
factory.getContainerProperties().setIdleBetweenPolls(300000L);
return args -> {
while (true) {
template.send("so63411124", "foo");
Thread.sleep(295000);
}
};
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so63411124").partitions(1).replicas(1).build();
}
}
logging.level.org.springframework.kafka=debug
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.properties.max.poll.interval.ms=600000
If you can provide a small example like this that exhibits the behavior you describe, I will take a look to see what's wrong.
I am trying to understand how Subscriber and Publisher works in java 9.
Here I have created one subscriber here and using SubmissionPublisher for publishing item .
I am trying to publish 100 strings to subscriber. If I do not make the Client program to sleep (see commented code in MyReactiveApp), I do not see all the items are published.
why is it not waiting for all the strings processed here:
strs.stream().forEach(i -> publisher.submit(i)); // what happens here?
If I replace the above code with, I see all the strings are printed in console
strs.stream().forEach(System.out::println);
Client program that publishes using SubmissionPublisher.
import java.util.List;
import java.util.concurrent.SubmissionPublisher;
import java.util.function.Supplier;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class MyReactiveApp {
public static void main(String args[]) throws InterruptedException {
SubmissionPublisher<String> publisher = new SubmissionPublisher<>();
MySubscriber subs = new MySubscriber();
publisher.subscribe(subs);
List<String> strs = getStrs();
System.out.println("Publishing Items to Subscriber");
strs.stream().forEach(i -> publisher.submit(i));
/*while (strs.size() != subs.getCounter()) {
Thread.sleep(10);
}*/
//publisher.close();
System.out.println("Exiting the app");
}
private static List<String> getStrs(){
return Stream.generate(new Supplier<String>() {
int i =1;
#Override
public String get() {
return "name "+ (i++);
}
}).limit(100).collect(Collectors.toList());
}
}
Subscriber
import java.util.concurrent.Flow.Subscription;
public class MySubscriber implements java.util.concurrent.Flow.Subscriber<String>{
private Subscription subscription;
private int counter = 0;
#Override
public void onSubscribe(Subscription subscription) {
this.subscription = subscription;
subscription.request(100);
}
#Override
public void onNext(String item) {
System.out.println(this.getClass().getSimpleName()+" item "+item);
//subscription.request(1);
counter++;
}
#Override
public void onError(Throwable throwable) {
System.out.println(this.getClass().getName()+ " an error occured "+throwable);
}
#Override
public void onComplete() {
System.out.println("activity completed");
}
public int getCounter() {
return counter;
}
}
output:
Publishing Items to Subscriber
MySubscriber item name 1
MySubscriber item name 2
MySubscriber item name 3
MySubscriber item name 4
MySubscriber item name 5
Exiting the app
MySubscriber item name 6
MySubscriber item name 7
MySubscriber item name 8
MySubscriber item name 9
MySubscriber item name 10
MySubscriber item name 11
MySubscriber item name 12
SubmissionPublisher<String> publisher = new SubmissionPublisher<>();
Creates a new SubmissionPublisher using the ForkJoinPool.commonPool() for async delivery to subscribers
see: https://docs.oracle.com/javase/9/docs/api/java/util/concurrent/SubmissionPublisher.html#SubmissionPublisher--
So actually
strs.stream().forEach(i -> publisher.submit(i));
enqueues all submissions and delivers them asynchronously on another thread. But then the application is terminated. This is independent of the progress of the worker thread. This means that the application is terminated regardless of how many elements the worker thread has already delivered.
This can be different for each run. In the worst case, the application could be terminated before the first item is delivered.
Threads
If you want to verify that the main method of MyReactiveApp and the delivery in MySubscriber's onNext happen on different threads you can print out the names of the corresponding threads, e.g. in MyReactiveApp's main:
System.out.println(Thread.currentThread().getName())
will output main as thread name.
Whereas MySubscriber's onNext method will e.g. output something like ForkJoinPool.commonPool-worker-1.
User and Deamon Threads
Why does the application terminate although we still have a running thread?
There are two kind of threads in Java:
user threads
daemon threads
A Java program terminates when no longer any user threads are running, even when deamon threads are still running.
The main thread is a user thread. The SubmissionPublisher uses here worker threads from ForkJoinPool.commonPool(). These are daemon threads.
All worker threads are initialized with Thread.isDaemon() set true.
https://docs.oracle.com/javase/9/docs/api/java/util/concurrent/ForkJoinPool.html
I have the following ThreadPoolTaskExecutor thats gets created with the expected core/max pool size configurations.
#Slf4j
#Configuration
public class ThreadPoolConfig {
#Value("${corePoolSize}")
private Integer corePoolSize;
#Value("${queueCapacity}")
private Integer queueCapacity;
#Value("${maxPoolSize}")
private Integer maxPoolSize;
#Bean(name="myThreadPoolTaskExecutor")
public ThreadPoolTaskExecutor myThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setBeanName("myThreadPoolTaskExecutor");
executor.setCorePoolSize(corePoolSize);
executor.setQueueCapacity(queueCapacity);
executor.setMaxPoolSize(maxPoolSize);
executor.setThreadNamePrefix("my_thread_");
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.initialize();
log.debug("threadPoolTaskExecutor CorePoolSize is : " + executor.getCorePoolSize());
log.debug("threadPoolTaskExecutor MaxPoolSize is : " + executor.getMaxPoolSize());
return executor;
}
}
When my #scheduled method runs the max pool size is set to the DEFAULT value of 2147483647 and I don't understand why it's not using the configured ThreadPoolTaskExecutor above:
#EnableScheduling
public class SchedulingConfig {
}
#Component
public class Scheduler {
#Autowired
#Qualifier("myThreadPoolTaskExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Scheduled(fixedRateString = "${fixedRate}")
public void invokeScheduledThread() {
while (threadPoolTaskExecutor.getActiveCount() <= threadPoolTaskExecutor.getMaxPoolSize()) {
log.debug("Active Thread Pool count is : " + threadPoolTaskExecutor.getActiveCount() + ", Max Thread Pool count is : " + threadPoolTaskExecutor.getMaxPoolSize() + " on the scheduled Thread : " + Thread.currentThread().getName());
//call a service to retrieve some items to process
threadPoolTaskExecutor.execute(Some Object that implements runnable);
}
}
}
Output:
Active Thread Pool count is : 0, Max Thread Pool count is : 2147483647 on the scheduled Thread : task-scheduler-1
I put a break point into the initialise() method of org.springframework.scheduling.concurrent.ExecutorConfigurationSupport
and it looks like the method is getting invoked 3 times, twice with a ThreadName Prefix of "my_thread_"
which is expected and finally once for a Bean called "taskScheduler" with a ThreadName Prefix of "task-scheduler-".
Does anyone know why I can't use my own ThreadPoolTaskExecutor within the Scheduler class?
I wanted to use a default #Scheduler to run on a single thread every x number of seconds and create X number of Threads using my own ThreadPoolTaskExecutor.
Use ThreadPoolTaskScheduler instead of ThreadPoolTaskExecutor.
For example:
#Configuration
public class SpringSchedulerConfig {
private static final int THREAD_POOL_SIZE = 5;
#Bean
public ThreadPoolTaskScheduler getScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
//we want every Job in a separate Thread.
threadPoolTaskScheduler.setPoolSize(THREAD_POOL_SIZE);
return threadPoolTaskScheduler;
}
}