I have a problem with kafka cluster installed on k8s. When consumer is connected to the given topic and there are no messages in that topic for about 60 seconds, the connection is aborted. The problem is only on the cluster (when i am using kafka on local computer with the same configuration than it works fine). Kafka is installed from bitnami helm chart. I tried to set different properties like session timeout, but it doesn't work. Thanks for help
Kafka consumer properties:
spring.kafka.bootstrap-servers=kafka:9092
spring.kafka.consumer.group-id=group-id
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.max-poll-records=1000
spring.kafka.properties.spring.json.trusted.packages=*
spring.kafka.consumer.properties.spring.json.use.type.headers=false
spring.kafka.consumer.properties.spring.json.value.default.type=java.lang.String
Related
We are facing a strange issue in only one of our environment (with same consumer app).
Basically, it is observed that suddenly a lag starts to build up with only one of the topics on kafka broker (it has multiple topics), with 10 consumer members under a single consumer group.
Even after multiple restarts, adding another pod of consumer application, changing defaults configuration properties (max poll records, session timeout) so far have NOT helped much.
Looking for any suggestions, advice on how to possibly debug the issue (we tried enabling apache logs, cloud watch etc, but so we only saw that regular/periodic rebalancing is happening, even for very low load of 7k messages waiting for processing).
Below are env details:
App - Spring boot app on version 2.7.2 Platform
AWS Kafka - MSK
Kafka Broker - 3 brokers (version 2.8.x)
Consumer Group - 1 with 15 members (partition 8, Topic 1)
Versions
Spring Boot 1.5.x,
Spring Boot 2.4.x,
Apache Kafka 0.10.2
The Situation
We have two service instances hosted on different servers. Each instance initializes multiple Kafka consumers. All consumers are listening to the same topic and are part of the same consumer group.
We are not relying on Spring Boot/Spring Kafka to configure the ConcurrentKafkaListnerContainerFactory and its DefaultKafkaConsumerFactory. All the consumer configuration properties are set to the default Apache Kafka consumer property values except for max.poll.records, session.timeout.ms, and heartbeat.interval.ms. Acknowledgement mode is set to record.
We are using the #KafkaListener annotation and setting its containerFactory property with the bean name of the initialized ConcurrentKafkaListenerContainerFactory and setting it topics property.
The Problem
When a topic does not get any messages published to it for a day or two, all consumers are removed from the consumer group.
I can’t find any reason for this to happen. From my understanding of reading both the Apache Kafka and Spring Kafka documentation if poll is called within max.poll.interval.ms, the consumer is considered alive. And if heartbeats are continuously sent by the consumer within the session.timeout.ms, the consumer is considered alive. According to the documentation, poll is called continuously and heartbeats are sent at the interval set by heartbeat.interval.ms.
The Questions
Is there a setting or property Spring Boot/Spring Kafka is setting that causes a consumer that hasn’t consumed any records from an idle topic for a day or two to be removed from the consumer group?
If yes, can this be turned off and what are the downsides?
If no, is there a way to rejoin the consumer group without having to restart the service and what are the downsides?
That Kafka version is very, very old.
Older versions removed the consumer offsets after no activity for 24 hours, even if the consumer is still connected. In 2.0, this was increased to 7 days. With newer brokers (since 2.1), consumer offsets are only removed if the consumers are not actually connected for 7 days.
See https://kafka.apache.org/documentation/#upgrade_200_notable
You can increase the broker's offsets.retention.minutes with older brokers.
We have a kafka cluster and have build the cluster with kafka from this maven repository https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10/0.10.0.0
We are using another kafka client library to send messages into this cluster. I would like to log the messages that are sent to this cluster and send them to AWS cloudwatch (my cluster in running on AWS EC2).
How can I achieve this?
Also, once a producer has sent the message, what is the entry point of code that will be invoked in this kafka library?
-Thanks!
I got my zookeeper and 3 kafka broker running locally.
I started one producer and one consumer. I can see consumer is consuming message.
I then started three consumers with same consumer group name (different ports since its a spring boot project). but what I found is that all the consumers are now consuming (receiving) messages. But I expect the message to be load-balanced in that only messages are not repeated across the consumers. I don't know what the problem is.
Here is my property file
spring.cloud.stream.bindings.input.destination=timerTopicLocal
spring.cloud.stream.kafka.binder.zkNodes=localhost
spring.cloud.stream.kafka.binder.brokers=localhost
spring.cloud.stream.bindings.input.group=timerGroup
Here the group is timerGroup.
consumer code : https://github.com/codecentric/edmp-sample-stream-sink
producer code : https://github.com/codecentric/edmp-sample-stream-source
Can you please update dependencies to Camden.RELEASE (and start using Kafka 0.9+) ? In Brixton.RELEASE, Kafka consumers were 0.8-based and required passing instanceIndex/instanceCount as properties in order to distribute partitions correctly.
In Camden.RELEASE we are using the Kafka 0.9+ consumer client, which does load-balancing in the way you are expecting (we also support static partition allocation via instanceIndex/instanceCount, but I suspect this is not what you want). I can enter into more details on how to configure this with Brixton, but I guess an upgrade should be a much easier path.
We have an application running in a GlassFish 3.1.2.2 cluster (two instances) that writes its results to "the_output_queue".
GlassFish sets up Message Queue as an embedded broker cluster, which in turn has also two message broker instances corresponding directly to the two GlassFish instances.
Now I would like to consume results from the_output_queue with an external JMS client (think Android app).
I assumed that a broker cluster can somehow be accessed transparently by a JMS client, but I cannot get this to work. I only succeed in connecting a JMS client to one individual broker.
If I have one JMS client running, connected to one broker I get only half of the messages. The physical queue (the_output_queue) defined in the GlassFish Administration Console exists in both brokers and the messages get evenly distributed thanks to load balancing.
This text from the Oracle manuals sounds to me like every message should be available in all broker instances of the cluster, i.e. if only a single JMS consumer is running it should receive all messages irrespective of the broker instance it is connected to.
"The home broker is responsible for routing and delivering the messages to all consumers of the destination, whether these consumers are local (connected to the home broker) or remote (connected to other brokers in the cluster)."
Have I misunderstood this completely?
Can a JMS client access a Oracle Message Queue broker cluster transparently?
How would the connection string look?
Is there some "global cluster target" (instead of an individual broker) to which the JMS client can connect? Where could I find the connection details for the cluster?
Is there something special in the GlassFish setup I have to verify? The settings currently are (default setup created by jelastic.com, looks good to me):
JMS Availability:
JMS Service Type: Embedded
JMS Cluster Type: Conventional
JMS Configuration Store Type: Master Broker
JMS Message Store Type: File
GMS is enabled
Answer to the main question: Yes, a JMS client can connect to any instance of a cluster and GlassFish will replicate the messages. I have tested it on my PC.
The problem in Jelastic is discussed in this posting.