I have a scenario where i am creating the predefined consumers for queues in DirectMessageListenerContainer bean as follows:
#Bean
public DirectMessageListenerContainer directMessageListenerContainer(ConnectionFactory conn, DynamicTenantListenerCreator listener){
DirectMessageListenerContainer factory = new DirectMessageListenerContainer();
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
factory.setConnectionFactory(conn);
factory.setConsumersPerQueue(5);
factory.setMessageListener(listener);
return factory;
}
As per my understanding, this will assign 5 consumers to each queue working on separate channel. for eg, if i add 5 queues to this bean so the total number of consumes will be 25 working on 5 different channels. My questions are:
What will be the impact of idle consumers on CPU in Cloud Foundry. In case some queues are having less messages, will the performance be impacted for the application as there will be some idle consumers?
Is there any way to define specific number of consumers of each queue separately?
Thanks in advance
Idle consumers will use minimal resources (just a small amount of memory, no cpu). No, you can't specify consumer counts for each queue, unless you have a separate container for each.
Related
I'm using Spring Boot with RabbitMq, and a question came up, is there a limited number of consumers I can create?
Where do I find this value?
Spring-Amqp has no limit for consumer number.
But usually it will be restricted by other things. For example, if you use SimpleMessageListner, one consumer corresponds to one thread. When your number of consumers is large, your app may not be able to create so many threads, resulting in OOM: unable to create new native thread.
// OOM in my computer
#RabbitListener(queues = "testq", concurrency = "10000-10000")
public void listen() {
}
If you use a CachingConnectionFactory(connections), set CacheMode to CONNECTION, maybe your rabbitmq server cannot carry a very large number of consumers (probably hit the maximum number of file descriptors.), and your app may not be able to connect to rabbitmq.
I'm using spring-kafka to consume messages from two Kafka topics, which sends same message format as below.
#KafkaListener(topics = {"topic_country1", "topic_country2"}, groupId = KafkaUtils.MESSAGE_GROUP)
public void onCustomerMessage(String message, Acknowledgment ack) throws Exception {
log.info("Message : {} is received", message);
ack.acknowledge();
}
Can KafkaListener allocate the number of consumer threads according to the number of topics that it listens by it's own and parallel process messages in two topics? Or it doesn't support parallel processing and messages have to wait in the topic till one message gets processed?
In case if the number of messages in the topic is higher, I need to autoscale my micro-service to start new instances (till the number of partitions). What are the parameters (CPU, memory) I can depend on to find out the number of messages in the topics is higher from KafkaListener point of view? (i.e In an API I can auto-scale the service by monitoring the HTTP latency)
You can set the concurrency property to run more threads; but each partition can only be processed by one thread. To increase concurrency you must increase the number of partitions in each topic. When listening to multiple topics in the same listener, if those topics only have one partition, you may not get the concurrency you desire unless you change the kafka consumer partition assignor.
See https://docs.spring.io/spring-kafka/docs/2.5.0.RELEASE/reference/html/#using-ConcurrentMessageListenerContainer
When listening to multiple topics, the default partition distribution may not be what you expect. For example, if you have three topics with five partitions each and you want to use concurrency=15, you see only five active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. This is because the default Kafka PartitionAssignor is the RangeAssignor (see its Javadoc). For this scenario, you may want to consider using the RoundRobinAssignor instead, which distributes the partitions across all of the consumers. Then, each consumer is assigned one topic or partition. ...
If you want to scale horizontal beyond the partition count and dynamically - consider using something like Parallel Consumer (PC). It can be used within a Spring context.
By using PC, you can processing all your keys in parallel, regardless of how long it takes to process, and you can be as concurrent as you wish - and this can scale dynamically.
PC directly solves for this, by sub partitioning the input partitions by key and processing each key in parallel.
It also tracks per record acknowledgement. Check out Parallel Consumer on GitHub (it's open source BTW, and I'm the author).
When using DirectMessageListenerContainer with consumersPerQueue property of 25, I noticed 25 rabbit channels get created per listener container's subscribed queue. The rabbit channel count quickly grows out of hand in our setup, as more queues are added to the listener container dynamically. We had to increase broker channel limit to accommodate the channel growth.
What is the relationship between channels and consumers in the DirectMessageListenerContainer. From my observations it appears to be 1 channel per consumer.
Does DirectMessageListenerContainer offer any channel pooling/recycling/rebalancing to keep channel growth under control. Specifically for queues that are mostly idle.
Does the simple SimpleMessageListenerContainer handle channels pooling differently, since it can dynamically resize the consumer count.
The DMLC uses a separate channel for each consumer.
No.
The SMLC uses one channel per concurrentConsumers; since 2.0, each channel is used for multiple consumers (when there is more than one queue listened to).
However dynamically adding or removing queues is much less efficient with the SMLC because the consumer(s) are canceled and re-created when changes are made.
I am playing with the SimpleMessageListenerContainer that is subscribed to 3 different queues. The SimpleMessageListenerContainer has been configured with a RetryOperationsInterceptor that has a exponential back off policy.
My SimpleMessageListenerContainer has been configured with :
container.addQueueNames("news.politics","news.science","news.tech");
container.setMaxConcurrentConsumers(10);
container.setAdviceChain(new Advice[]{retryInterceptor});
If a message from one of the 3 queue thats been consumed enters an erroneous state resulting in an exception, the consumer triggers the exponential back of policy as predicted. However, I am noticing that the consumer STOPS doing round robin, to process messages on the other 2 queues. I was thinking since the consumer is set with "MaxConcurrentConsumer" of 10, the consumer will start to spawn consumer threads and round robin the rest of the queues.
Is this the normal behavior of SimpleMessageListenerContainer?
Can this behavior be adjusted?
For my use-case, would it be recommended to have one SimpleMessageListenerContainer per queue to keep them segregated? Perhaps come up with a decorator CompositeSimpleMessageListener container, that has internally a map of SimpleMessageListenerContainers per queue?
The algorithm to increase consumers is limited; if the retry interceptor suspends the consumer thread, it will prevent new consumers from being started.
Either increase the concurrentConsumers (to at least 3) or switch to the DirectMessageListenerContainer.
See choosing a container.
This is the scenario:
I know that using latest API related to Spring kafka (like Spring-integration-kafka 2.10) we can do something like:
#KafkaListener(id = "id0", topicPartitions = { #TopicPartition(topic = "SpringKafkaTopic", partitions = { "0" }) })
#KafkaListener(id = "id1", topicPartitions = { #TopicPartition(topic = "SpringKafkaTopic", partitions = { "1" }) })
and with that read from different partitions related to the same kafka topic.
I'm wondering if we can do the same using, for example spring-integration-kafka 1.3.1
I didn't find any tip about how to do that (I'm interesting in the xml version).
In Kafka you can decide from which topics you want to read,
but we can't decide from which partitions we want to read, it's up to Kafka to decide that in order to avoid reading the same message more than once.
Consumers don't share partitions for reading purposes, by Kafka definition.
If you'll have more consumers than partitions some consumers will stay idle and won't consume from any partition. for example, if we'll have 5 consumers and 4 partitions, 1 consumer will stay idle and won't consume data from kafka broker.
The actual partition assignment is being done by a kafka broker (the group coordinator) and a leader consumer. we can't control that.
This definition helped me the most:
In Apache Kafka, the consumer group concept is a way of achieving two
things:
Having consumers as part of the same consumer group means providing the “competing consumers” pattern with whom the messages
from topic partitions are spread across the members of the group. Each
consumer receives messages from one or more partitions
(“automatically” assigned to it) and the same messages won’t be
received by the other consumers (assigned to different partitions). In
this way, we can scale the number of the consumers up to the number of
the partitions (having one consumer reading only one partition); in
this case, a new consumer joining the group will be in an idle state
without being assigned to any partition.
Having consumers as part of different consumer groups means providing the “publish/subscribe” pattern where the messages from
topic partitions are sent to all the consumers across the different
groups. It means that inside the same consumer group, we’ll have the
rules explained above, but across different groups, the consumers will
receive the same messages. It’s useful when the messages inside a
topic are of interest for different applications that will process
them in different ways. We want all the interested applications to
receive all the same messages from the topic.
From here Don't Use Apache Kafka Consumer Groups the Wrong Way!