Spring Kafka receive and forward another broker - spring-boot

Question

I'm using spring-kafka to receive and send messages.
What i'm gonna do is read some messages from X kafka broker and do some enhancements and send to another Y kafka broker
Here is my beans and configurations.
#Bean
public ProducerFactory<String, AuditLog> forwarderKafkaProducerFactory() {
Map<String, Object> configs = new HashMap<>();
configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9093");
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return new DefaultKafkaProducerFactory<>(configs);
}
#Bean
public KafkaTemplate<String, AuditLog> forwarderKafkaClient() {
return new KafkaTemplate<>(forwarderKafkaProducerFactory());
}
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, AuditLog>> kafkaListenerContainerFactoryV2() {
ConcurrentKafkaListenerContainerFactory<String, AuditLog> factory = new ConcurrentKafkaListenerContainerFactory<>();
Map<String, Object> configs = new HashMap<>();
configs.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
configs.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
configs.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
configs.put(ConsumerConfig.GROUP_ID_CONFIG, "receiver-sender");
configs.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
ConsumerFactory<String, AuditLog> consumerFactory = new DefaultKafkaConsumerFactory<>(
configs, new StringDeserializer(), new JsonDeserializer<>(AuditLog.class));
factory.setConsumerFactory(consumerFactory);
return factory;
}
as you can see one broker runs in 9092 the other one runs in 9093
Receive/Forward logic is like that.
#KafkaListener(topics = "audit_log", containerFactory = "kafkaListenerContainerFactoryV2")
public void listen(#Payload AuditLog payload) {
log.info("Audit log [{}]", payload);
if (!payload.isForwarded()) {
String key = "some-value";
String system = String.join("/", key, payload.getSystem());
payload.setSystem(system);
payload.setForwarded(true);
template.send("audit_log", key, payload);
}
}
Template which is above the code snippet configurations are correct. I can confirm by looking ((DefaultKafkaProducerFactory)template.getProducerFactory()).getConfigurationProperties()
In this configuration i can receive messages from 9092 but i can't send to 9093. Template always sending to 9092.
Thanks.

Answer 1

The problem is Kafka setup was incorrect.
I use wurstmeister/kafka on my current environment.
Docker compose file is like this.
zookeeper:
image: wurstmeister/zookeeper:3.4.6
kafka:
image: wurstmeister/kafka:2.12-2.5.0
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_ADVERTISED_HOST_NAME: "127.0.0.1"
depends_on:
- zookeeper
Then i just add second Kafka
zookeeper2:
image: wurstmeister/zookeeper:3.4.6
kafka2:
image: wurstmeister/kafka:2.12-2.5.0
ports:
- "9093:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper2:2181"
KAFKA_ADVERTISED_HOST_NAME: "127.0.0.1"
depends_on:
- zookeeper2
But registry owners warns :) I saw this warning now while writing this answer.
modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)
I tried bitnami zookeeper/kafka
version: "2"
services:
zookeeper1:
image: docker.io/bitnami/zookeeper:3.7
ports:
- "2001:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka1:
image: docker.io/bitnami/kafka:3
ports:
- "9201:9201"
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper1:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
- KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9201
- KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka1:9092,EXTERNAL://localhost:9201
- KAFKA_INTER_BROKER_LISTENER_NAME=CLIENT
depends_on:
- zookeeper1
zookeeper2:
image: docker.io/bitnami/zookeeper:3.7
ports:
- "2002:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka2:
image: docker.io/bitnami/kafka:3
ports:
- "9202:9202"
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper2:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
- KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9202
- KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka2:9092,EXTERNAL://localhost:9202
- KAFKA_INTER_BROKER_LISTENER_NAME=CLIENT
depends_on:
- zookeeper2
The code in question runs correctly with this setup.