I am using Spring Kafka Consumer. I have set the concurrency to 10 and have 5 consumers created (for 5 topics). So there are 50 Spring Kafka Consumer threads.
What is the maximum number of threads I can have for Kafka Consumers? How can I increase the size of this thread pool? I have gone over the spring documentation but didn't find anything relavent.
There is no limit; by default, each container uses a SimpleAsyncTaskExecutor for the threads; there is no pool involved.
Related
I have around 10 different rabbit mq queues in 10 different virtual hosts to connect to. For each queue, a separate SimpleMessageListenerContainer bean is defined in my spring boot application and a separate Spring Integration flow is created using each specific SimpleMessageListenerContainer.
The concurrency for SimpleMessageListenerContainer is set to 1-3. Each of the SimpleMessageListenerContainer bean is using separate CachingConnectoryFacory beans. The Connection Factory mode is set as CHANNEL.
We also have another IntegrationFlow to publish messages to an outbound queue that is using a different connection factory. I am not setting any ThreadPool Task Executors in the ConnectionFactory, so it using the default one. While doing the Load test we are noticing that the multiple thread pool (prefixed with pool-) are getting crated and after a certain point application crashes may due to the high number of threads.
It looks like the default thread pool executor is having max value of Integer unbounded which may spinning up threads based on the demand. I tried setting custom Thread Pool task executors for each connection factory and I noticed that the threads are not growing like previously but from the java profiler it shows the SimpleMessageListenerContainer threads are getting BLOCKED frequently.
I want to know if there any best practices or to be followed while setting the custom thread pool task executors in the connection factory like a ratio between Lisneter threads and connection factory threads etc?
I have done some debugging; ...-1 gets renamed to, for example AMQP Connection 127.0.0.1:5672.
That thread is not from the pool, but it is created by the same thread factory.
Similarly, the scheduled executor (for heartbeats) uses the same thread factory, and gets ...-2.
Hence the pool starts at ...-3. So indeed, you have a fixed pool of 8 threads, an I/O thread, and a heartbeat thread for each factory.
With a large number of factories like that, you probably don't need so many threads; I would suggest a single pooled executor with sufficient threads to satisfy your workload; experimentation is probably the only way to determine the number, but I would guess it's something less than 88 (11x8).
How to configure spring boot server threads (for both IO and Worker) for undertow embedded server?
Thanks in advance.
As per the documentation:
server.undertow.threads.io //Number of I/O threads to create for the worker. The default is derived from the number of available processors.
server.undertow.threads.worker //Number of worker threads. The default is 8 times the number of I/O threads.
Versions
Spring Boot 1.5.x,
Spring Boot 2.4.x,
Apache Kafka 0.10.2
The Situation
We have two service instances hosted on different servers. Each instance initializes multiple Kafka consumers. All consumers are listening to the same topic and are part of the same consumer group.
We are not relying on Spring Boot/Spring Kafka to configure the ConcurrentKafkaListnerContainerFactory and its DefaultKafkaConsumerFactory. All the consumer configuration properties are set to the default Apache Kafka consumer property values except for max.poll.records, session.timeout.ms, and heartbeat.interval.ms. Acknowledgement mode is set to record.
We are using the #KafkaListener annotation and setting its containerFactory property with the bean name of the initialized ConcurrentKafkaListenerContainerFactory and setting it topics property.
The Problem
When a topic does not get any messages published to it for a day or two, all consumers are removed from the consumer group.
I can’t find any reason for this to happen. From my understanding of reading both the Apache Kafka and Spring Kafka documentation if poll is called within max.poll.interval.ms, the consumer is considered alive. And if heartbeats are continuously sent by the consumer within the session.timeout.ms, the consumer is considered alive. According to the documentation, poll is called continuously and heartbeats are sent at the interval set by heartbeat.interval.ms.
The Questions
Is there a setting or property Spring Boot/Spring Kafka is setting that causes a consumer that hasn’t consumed any records from an idle topic for a day or two to be removed from the consumer group?
If yes, can this be turned off and what are the downsides?
If no, is there a way to rejoin the consumer group without having to restart the service and what are the downsides?
That Kafka version is very, very old.
Older versions removed the consumer offsets after no activity for 24 hours, even if the consumer is still connected. In 2.0, this was increased to 7 days. With newer brokers (since 2.1), consumer offsets are only removed if the consumers are not actually connected for 7 days.
See https://kafka.apache.org/documentation/#upgrade_200_notable
You can increase the broker's offsets.retention.minutes with older brokers.
I am using spring, dsl, solace integration with spring boot. My application with one subscriber is able to poll approx 80 messages per second and 4 subscribers able to poll approx 125 messages per seconds. I want to process at least 500 messages per second. My JMS flow is stated below-
public #Bean IntegrationFlow defaultJmsFlow()
{
return IntegrationFlows.from(
//read JMS topic
Jms.messageDrivenChannelAdapter(this.connectionFactory).destination(this.config.getInputQueueName()).errorChannel(errorChannel()).configureListenerContainer(c ->
{
final DefaultMessageListenerContainer container = c.get();
container.setSessionTransacted(true);
container.setMaxConcurrentConsumers(10);
container.setConcurrentConsumers(4);
}).get())
.channel(messageProcessingChannel()).get();
}
After reading messages I'm sending those to DirectChannel. Any special configuration need to make to increase performance of my application so that at least 500 messages per seconds gets processed.
It is strongly recommended to cache consumers when integrating Spring JMS with Solace. This way, the connection stays persistent and messages will be delivered to the consumer quickly without the overhead of unbinding, disconnecting, and reconnecting. This can be set in the DefaultMessageListenerContainer with container.setCacheLevel(3).
If you are using concurrent consumers to read messages from the same Solace queue, ensure that the Solace queue is non-exclusive. Exclusive queues will deliver messages to one active consumer and non-exclusive queues will deliver messages round robin to all consumers.
I've done some tests with ActiveMQ and Spring JMS. I've configured DefaultMessageListenerContainer (DMLC) with WorkManagerTaskExecutor and CommonJ to manage the threads. I want to have control over the threads are running in the server.
I have made the decision of using DMLC based on a post written by Juergen Holler in Spring's forum: http://forum.spring.io/forum/other-spring-related/remoting/24208-what-s-the-best-practice-for-using-jms-in-spring?p=256420#post256420
In this post, he says "DMLC is the only listener container that does not impose the thread management onto the JMS provider, that is, does not use/block JMS provider threads." So I thought that all the threads would be managed by the server and there wouldn't be threads of ActiveMQ.
However, analysing the server's threads using JConsole, I've seen some threads of ActiveMQ that I didn't expect.
As you can see in the image, there are ActimeMQ threads (inactivity threads, transport threads, etc).
When I'm executing the tests, I see in the logs that the JMS messages are processed by CommonJ threads and not by ActiveMQ threads, so that's fine. However, I don't understand why are ActiveMQ threads are created if they are not used. Specially "ActiveMQ Transport" threads because for every queue I use I've got one thread "ActiveMQ Transport". Therefore if I'm consuming 50 queues, I've got 50 threads of "ActiveMQ Transport". Is it to keep the socket opened? Are these threads mandatory?
Details about the configuration:
ConnectionFactory used: org.apache.activemq.ActiveMQConnectionFactory
ActiveMQ version: 5.11.1
Transport protocol: TCP
Using DefaultMessageListenerContainer the client is in a loop invoking MessageConsumer.receive() method.
It seems a silly question and surely I've misunderstood some basic concepts.
Thanks to Rob Davies' explanation in this thread http://activemq.2283324.n4.nabble.com/ActimeMQ-Client-s-thread-management-td4705885.html, I've understood that all the interaction with ActiveMQ require a TCP thread per connection.
However, the amount of transport threads can be minimised replacing ActiveMQConnectionFactory with one of these ConnectionFactories:
PooledConnectionFactory (ActiveMQ)
SingleConnectionFactory (spring-jms)
CachingConnectionFactory (spring-jms)