Spring Kafka consumer: Is there a way to read from multiple partitions using Kafka 0.8? - spring

Question

This is the scenario:
I know that using latest API related to Spring kafka (like Spring-integration-kafka 2.10) we can do something like:
#KafkaListener(id = "id0", topicPartitions = { #TopicPartition(topic = "SpringKafkaTopic", partitions = { "0" }) })
#KafkaListener(id = "id1", topicPartitions = { #TopicPartition(topic = "SpringKafkaTopic", partitions = { "1" }) })
and with that read from different partitions related to the same kafka topic.
I'm wondering if we can do the same using, for example spring-integration-kafka 1.3.1
I didn't find any tip about how to do that (I'm interesting in the xml version).

Answer 1

In Kafka you can decide from which topics you want to read,
but we can't decide from which partitions we want to read, it's up to Kafka to decide that in order to avoid reading the same message more than once.
Consumers don't share partitions for reading purposes, by Kafka definition.
If you'll have more consumers than partitions some consumers will stay idle and won't consume from any partition. for example, if we'll have 5 consumers and 4 partitions, 1 consumer will stay idle and won't consume data from kafka broker.
The actual partition assignment is being done by a kafka broker (the group coordinator) and a leader consumer. we can't control that.

Answer 2

This definition helped me the most:
In Apache Kafka, the consumer group concept is a way of achieving two
things:
Having consumers as part of the same consumer group means providing the “competing consumers” pattern with whom the messages
from topic partitions are spread across the members of the group. Each
consumer receives messages from one or more partitions
(“automatically” assigned to it) and the same messages won’t be
received by the other consumers (assigned to different partitions). In
this way, we can scale the number of the consumers up to the number of
the partitions (having one consumer reading only one partition); in
this case, a new consumer joining the group will be in an idle state
without being assigned to any partition.
Having consumers as part of different consumer groups means providing the “publish/subscribe” pattern where the messages from
topic partitions are sent to all the consumers across the different
groups. It means that inside the same consumer group, we’ll have the
rules explained above, but across different groups, the consumers will
receive the same messages. It’s useful when the messages inside a
topic are of interest for different applications that will process
them in different ways. We want all the interested applications to
receive all the same messages from the topic.
From here Don't Use Apache Kafka Consumer Groups the Wrong Way!