I faced considerable latency in receiving topic messages and thought of writing this test program which uses spring kafka and apache kafka. I am using a test program to send data to a topic and use the following consumers to log latency of receiving the message in consumer. Difference between record timestamp and time actually received in consumer.
Spring Consumer
#Component
public class SpringKafkaConsumer {
#KafkaListener(topics = "websocket",
containerFactory = "dynamicKafkaListenerContainerFactory")
public void receiveDeliveryTopic(ConsumerRecord<String, String> record, #Headers MessageHeaders headers) {
Log.info("0", "Kafka latency in spring receiver: Topic: {} latency: {}", record.topic(),
System.currentTimeMillis() - record.timestamp());
}
Apache Consumer
#Component
public class ApacheKafkaConsumer {
#PostConstruct
public void start() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("websocket"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records)
Log.info("0", "Latency in apache kafka consumer: {}", System.currentTimeMillis() - record.timestamp());
}
}
}
Result
Spring Consumer
Kafka latency in spring receiver: Topic: websocket latency: 246
Kafka latency in spring receiver: Topic: websocket latency: 246
Kafka latency in spring receiver: Topic: websocket latency: 247
Kafka latency in spring receiver: Topic: websocket latency: 267
Kafka latency in spring receiver: Topic: websocket latency: 260
Kafka latency in spring receiver: Topic: websocket latency: 261
Kafka latency in spring receiver: Topic: websocket latency: 260
Kafka latency in spring receiver: Topic: websocket latency: 306
Kafka latency in spring receiver: Topic: websocket latency: 300
Kafka latency in spring receiver: Topic: websocket latency: 300
Kafka latency in spring receiver: Topic: websocket latency: 299
Kafka latency in spring receiver: Topic: websocket latency: 340
Kafka latency in spring receiver: Topic: websocket latency: 337
Kafka latency in spring receiver: Topic: websocket latency: 337
Kafka latency in spring receiver: Topic: websocket latency: 337
Kafka latency in spring receiver: Topic: websocket latency: 345
Kafka latency in spring receiver: Topic: websocket latency: 345
Kafka latency in spring receiver: Topic: websocket latency: 345
Kafka latency in spring receiver: Topic: websocket latency: 346
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 352
Kafka latency in spring receiver: Topic: websocket latency: 350
Kafka latency in spring receiver: Topic: websocket latency: 349
Kafka latency in spring receiver: Topic: websocket latency: 359
Kafka latency in spring receiver: Topic: websocket latency: 355
Kafka latency in spring receiver: Topic: websocket latency: 354
Kafka latency in spring receiver: Topic: websocket latency: 354
Kafka latency in spring receiver: Topic: websocket latency: 361
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 354
Kafka latency in spring receiver: Topic: websocket latency: 351
Kafka latency in spring receiver: Topic: websocket latency: 351
Kafka latency in spring receiver: Topic: websocket latency: 350
Kafka latency in spring receiver: Topic: websocket latency: 358
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 362
Kafka latency in spring receiver: Topic: websocket latency: 362
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 363
Kafka latency in spring receiver: Topic: websocket latency: 361
Kafka latency in spring receiver: Topic: websocket latency: 359
Kafka latency in spring receiver: Topic: websocket latency: 359
Kafka latency in spring receiver: Topic: websocket latency: 358
Kafka latency in spring receiver: Topic: websocket latency: 351
Kafka latency in spring receiver: Topic: websocket latency: 352
Kafka latency in spring receiver: Topic: websocket latency: 351
Kafka latency in spring receiver: Topic: websocket latency: 370
Kafka latency in spring receiver: Topic: websocket latency: 370
Kafka latency in spring receiver: Topic: websocket latency: 369
Kafka latency in spring receiver: Topic: websocket latency: 368
Kafka latency in spring receiver: Topic: websocket latency: 368
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 364
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 365
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 360
Kafka latency in spring receiver: Topic: websocket latency: 358
Kafka latency in spring receiver: Topic: websocket latency: 358
Kafka latency in spring receiver: Topic: websocket latency: 358
Kafka latency in spring receiver: Topic: websocket latency: 359
Kafka latency in spring receiver: Topic: websocket latency: 356
Kafka latency in spring receiver: Topic: websocket latency: 353
Kafka latency in spring receiver: Topic: websocket latency: 370
Kafka latency in spring receiver: Topic: websocket latency: 363
Kafka latency in spring receiver: Topic: websocket latency: 362
Kafka latency in spring receiver: Topic: websocket latency: 361
Kafka latency in spring receiver: Topic: websocket latency: 379
Kafka latency in spring receiver: Topic: websocket latency: 378
Kafka latency in spring receiver: Topic: websocket latency: 379
Kafka latency in spring receiver: Topic: websocket latency: 373
Kafka latency in spring receiver: Topic: websocket latency: 371
Kafka latency in spring receiver: Topic: websocket latency: 371
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 372
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 364
Kafka latency in spring receiver: Topic: websocket latency: 374
Kafka latency in spring receiver: Topic: websocket latency: 368
Kafka latency in spring receiver: Topic: websocket latency: 367
Kafka latency in spring receiver: Topic: websocket latency: 366
Kafka latency in spring receiver: Topic: websocket latency: 367
Kafka latency in spring receiver: Topic: websocket latency: 362
Kafka latency in spring receiver: Topic: websocket latency: 363
Kafka latency in spring receiver: Topic: websocket latency: 362
Kafka latency in spring receiver: Topic: websocket latency: 362
Kafka latency in spring receiver: Topic: websocket latency: 361
Kafka latency in spring receiver: Topic: websocket latency: 364
Kafka latency in spring receiver: Topic: websocket latency: 363
Apache Consumer
Latency in apache kafka consumer: 134
Latency in apache kafka consumer: 130
Latency in apache kafka consumer: 130
Latency in apache kafka consumer: 130
Latency in apache kafka consumer: 130
Latency in apache kafka consumer: 138
Latency in apache kafka consumer: 133
Latency in apache kafka consumer: 133
Latency in apache kafka consumer: 131
Latency in apache kafka consumer: 143
Latency in apache kafka consumer: 141
Latency in apache kafka consumer: 141
Latency in apache kafka consumer: 142
Latency in apache kafka consumer: 140
Latency in apache kafka consumer: 143
Latency in apache kafka consumer: 137
Latency in apache kafka consumer: 225
Latency in apache kafka consumer: 224
Latency in apache kafka consumer: 225
Latency in apache kafka consumer: 223
Latency in apache kafka consumer: 225
Latency in apache kafka consumer: 222
Latency in apache kafka consumer: 217
Latency in apache kafka consumer: 217
Latency in apache kafka consumer: 217
Latency in apache kafka consumer: 230
Latency in apache kafka consumer: 234
Latency in apache kafka consumer: 233
Latency in apache kafka consumer: 233
Latency in apache kafka consumer: 233
Latency in apache kafka consumer: 231
Latency in apache kafka consumer: 231
Latency in apache kafka consumer: 230
Latency in apache kafka consumer: 228
Latency in apache kafka consumer: 226
Latency in apache kafka consumer: 225
Latency in apache kafka consumer: 229
Latency in apache kafka consumer: 226
Latency in apache kafka consumer: 226
Latency in apache kafka consumer: 227
Latency in apache kafka consumer: 227
Latency in apache kafka consumer: 258
Latency in apache kafka consumer: 253
Latency in apache kafka consumer: 252
Latency in apache kafka consumer: 252
Latency in apache kafka consumer: 254
Latency in apache kafka consumer: 252
Latency in apache kafka consumer: 263
Latency in apache kafka consumer: 262
Latency in apache kafka consumer: 261
Latency in apache kafka consumer: 261
Latency in apache kafka consumer: 261
Latency in apache kafka consumer: 257
Latency in apache kafka consumer: 256
Latency in apache kafka consumer: 255
Latency in apache kafka consumer: 270
Latency in apache kafka consumer: 269
Latency in apache kafka consumer: 266
Latency in apache kafka consumer: 263
Latency in apache kafka consumer: 263
Latency in apache kafka consumer: 263
Latency in apache kafka consumer: 262
Latency in apache kafka consumer: 263
Latency in apache kafka consumer: 260
Latency in apache kafka consumer: 257
Latency in apache kafka consumer: 254
Latency in apache kafka consumer: 253
Latency in apache kafka consumer: 253
Latency in apache kafka consumer: 271
Latency in apache kafka consumer: 270
Latency in apache kafka consumer: 267
Latency in apache kafka consumer: 270
Latency in apache kafka consumer: 266
Latency in apache kafka consumer: 266
Latency in apache kafka consumer: 266
Latency in apache kafka consumer: 266
Latency in apache kafka consumer: 264
Latency in apache kafka consumer: 296
Latency in apache kafka consumer: 295
Latency in apache kafka consumer: 297
Latency in apache kafka consumer: 293
Latency in apache kafka consumer: 294
Latency in apache kafka consumer: 298
Latency in apache kafka consumer: 297
Latency in apache kafka consumer: 295
Latency in apache kafka consumer: 295
Latency in apache kafka consumer: 298
Latency in apache kafka consumer: 298
Latency in apache kafka consumer: 302
Latency in apache kafka consumer: 302
Latency in apache kafka consumer: 302
Latency in apache kafka consumer: 300
Latency in apache kafka consumer: 303
Latency in apache kafka consumer: 302
Latency in apache kafka consumer: 303
Latency in apache kafka consumer: 301
Latency in apache kafka consumer: 300
Latency in apache kafka consumer: 299
Latency in apache kafka consumer: 300
Latency in apache kafka consumer: 299
Why am I seeing this high latency in kafka consumer?
Related
I am using Spring Kafka Consumer. I have set the concurrency to 10 and have 5 consumers created (for 5 topics). So there are 50 Spring Kafka Consumer threads.
What is the maximum number of threads I can have for Kafka Consumers? How can I increase the size of this thread pool? I have gone over the spring documentation but didn't find anything relavent.
There is no limit; by default, each container uses a SimpleAsyncTaskExecutor for the threads; there is no pool involved.
I have a RabbitMQ message broker running in a server, to which I'm trying to configure a Producer and a Consumer using Spring Cloud Stream. My Producer is creating messages in a queue every second, while my Consumer reads them at the same rate. However, if I stop my Consumer and the Producer keeps pushing messages, when I restart my Consumer again it is unable to retrieve the messages created in that period of time it was down, only picking up the messages produced from the time that it was started. How can I make my Consumer consume existing messages in the queue when it starts?
Here are my Consumer properties:
cloud:
stream:
bindings:
input:
destination: spring-cloud-stream-demo
consumer:
auto-bind-dlq: true
republishToDlq: true
maxAttempts: 5
And my Producer properties:
cloud:
stream:
bindings:
output:
destination: spring-cloud-stream-demo
Appreciate any help!
You need to add a group to the consumer (input) binding; otherwise it will bind an anonymous, auto-delete, queue to the exchange.
With a group, a permanent, durable, queue is bound instead.
i have a Spring boot application and using Spring Kafka. we have create a consumer which is consuming messages from 4 topics. these topics doesnt have any partition. the issue i am facing here a rendom behavior that out of three topics, in any one topic offset stop and my consumer keep on consuming same messages from that topic again and again until we need to manually move the offset to latest.below is the configuration YAML configuration i have :
spring:
kafka:
consumer:
bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS}
group-id: group_id
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
kafka:
consumer:
allTopicList: user.topic,student.topic,class.topic,teachers.topic**
as it is a Spring boot application, default offset is set to latest.
what i am doing wrong here, please help me to understand.
What version are you using?
You should set
...consumer:
enable-auto-commit: false
The listener container will more reliably commit the offsets.
You should also consider
ack-mode: RECORD
and the container will commit the offset for each successfully processed record (default is BATCH).
is there an bridge which can send data from a kafka topic to amqp(rabbitmq) or mqtt ?
I just find only the from mqtt/amqp to kafka but not backwards...
BR,
I believe the Stimzi supplies a Kafka AMQP 1.0 bridge that could be used for such a task. The project source is located on Github here so I'd suggest starting there and reading the documentation.
I have a Spring Cloud Streaming transformer application using RabbitMQ. It is reading from a Rabbit queue, doing some transformation, and writing to a Rabbit exchange. I have my application deployed to PCF and am binding to a Rabbit service.
This works fine, but now I am needing a separate connection for consuming and producing the message. (I want to read from the Rabbit queue using one connection, and write to a Rabbit exchange using a different connection). How would I configure this? Is it possible to bind my applications to 2 different Rabbit services using 1 as the producer and 1 as the consumer?
Well, starting with version 1.3 Rabbit Binder indeed creates a separate ConnectionFactory for producers: https://docs.spring.io/spring-cloud-stream/docs/Ditmars.RELEASE/reference/htmlsingle/#_rabbitmq_binder
Starting with version 1.3, the RabbitMessageChannelBinder creates an internal ConnectionFactory copy for the non-transactional producers to avoid dead locks on consumers when shared, cached connections are blocked because of Memory Alarm on Broker.
So, maybe that is just enough for you as is after upgrading to Spring Cloud Stream Ditmars.
UPDATE
How would I go about configuring this internal ConnectionFactory copy with different connection properties?
No, that's different story. What you need is called multi-binder support: https://docs.spring.io/spring-cloud-stream/docs/Ditmars.RELEASE/reference/htmlsingle/#multiple-binders
You should declare several blocks for different connection factories:
spring.cloud.stream.bindings.input.binder=rabbit1
spring.cloud.stream.bindings.output.binder=rabbit2
...
spring:
cloud:
stream:
bindings:
input:
destination: foo
binder: rabbit1
output:
destination: bar
binder: rabbit2
binders:
rabbit1:
type: rabbit
environment:
spring:
rabbitmq:
host: <host1>
rabbit2:
type: rabbit
environment:
spring:
rabbitmq:
host: <host2>