Cannot access the Kafka Broker from inside the Kubernetes cluster - spring-boot

Question

I have a kafka broker and a spring boot application in my Kubernetes cluster. They are running on their own containers.
The spring boot application is a message producer. It needs to access the kafkabroker to send the messages. But it couldn't access the Kafka broker by providing the Kafka's servicename:port in the producers bootstrap.servers
Any help would be greatly appreciated.
Zookeper and KafkaBroker configuration in yaml:
---
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
namespace: mynamespace-k8s
spec:
type: NodePort
ports:
- name: zookeeper-port
port: 2181
nodePort: 30181
targetPort: 2181
selector:
app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: zookeeper
name: zookeeper
namespace: mynamespace-k8s
spec:
replicas: 1
selector:
matchLabels:
app: zookeeper
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- image: wurstmeister/zookeeper
imagePullPolicy: IfNotPresent
name: zookeeper
ports:
- containerPort: 2181
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-broker
name: kafka-service
namespace: mynamespace-k8s
spec:
ports:
- port: 9092
selector:
app: kafka-broker
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
namespace: mynamespace-k8s
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
hostname: kafka-broker
containers:
- env:
- name: KAFKA_PORT
value: "9092"
- name: KAFKA_ADVERTISED_PORT
value: "9092"
- name: KAFKA_ADVERTISED_HOST_NAME
value: kafka-service.mynamespace-k8s
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper-service.mynamespace-k8s:2181
- name: KAFKA_BROKER_ID
value: "1"
image: wurstmeister/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
My springboot application conf in yaml:
apiVersion: v1
kind: Service
metadata:
name: locationmanager-service
namespace: mynamespace-k8s
labels:
app: locationmanager
spec:
selector:
app: locationmanager
type: LoadBalancer
ports:
- protocol: TCP
port: 8080
targetPort: 8080
nodePort: 32588
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: locationmanager-deployment
namespace: mynamespace-k8s
labels:
app: locationmanager
spec:
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels:
app: locationmanager
template:
metadata:
labels:
app: locationmanager
spec:
containers:
- name: locationmanager
image: aef/locmanager:latest
ports:
- containerPort: 8081
resources:
limits:
memory: "1Gi"
cpu: "1000m"
requests:
memory: "256Mi"
cpu: "500m"
env:
- name: CONFIG_KAFKA_BOOTSTRAP_SERVERS
value: kafka-service.mynamespace-k8s:9092
Spring boot's bootstrap.server in application.properties:
spring.kafka.producer.bootstrap-servers= ${CONFIG_KAFKA_BOOTSTRAP_SERVERS}
When springboot application tries to create a topic, I receive the exception below:
2022-07-07 10:51:50,078 ERROR o.s.k.c.KafkaAdmin [main] Could not configure topics
org.springframework.kafka.KafkaException: Timed out waiting to get existing topics; nested exception is java.util.concurrent.TimeoutException
at org.springframework.kafka.core.KafkaAdmin.lambda$checkPartitions$5(KafkaAdmin.java:275) ~[spring-kafka-2.8.4.jar!/:2.8.4]
at java.util.HashMap.forEach(HashMap.java:1337) ~[?:?]
at org.springframework.kafka.core.KafkaAdmin.checkPartitions(KafkaAdmin.java:254) ~[spring-kafka-2.8.4.jar!/:2.8.4]
at org.springframework.kafka.core.KafkaAdmin.addOrModifyTopicsIfNeeded(KafkaAdmin.java:240) ~[spring-kafka-2.8.4.jar!/:2.8.4]
at org.springframework.kafka.core.KafkaAdmin.initialize(KafkaAdmin.java:178) ~[spring-kafka-2.8.4.jar!/:2.8.4]
at org.springframework.kafka.core.KafkaAdmin.afterSingletonsInstantiated(KafkaAdmin.java:145) ~[spring-kafka-2.8.4.jar!/:2.8.4]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:972) ~[spring-beans-5.3.18.jar!/:5.3.18]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.18.jar!/:5.3.18]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.18.jar!/:5.3.18]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.6.jar!/:2.6.6]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) ~[spring-boot-2.6.6.jar!/:2.6.6]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) ~[spring-boot-2.6.6.jar!/:2.6.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-2.6.6.jar!/:2.6.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312) ~[spring-boot-2.6.6.jar!/:2.6.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) ~[spring-boot-2.6.6.jar!/:2.6.6]
at com.trendyol.locationmanager.LocationManagerApplication.main(LocationManagerApplication.java:24) ~[classes!/:0.0.1-SNAPSHOT]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[locationmanager.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[locationmanager.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[locationmanager.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[locationmanager.jar:0.0.1-SNAPSHOT]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180) ~[kafka-clients-3.0.1.jar!/:?]
at org.springframework.kafka.core.KafkaAdmin.lambda$checkPartitions$5(KafkaAdmin.java:257) ~[spring-kafka-2.8.4.jar!/:2.8.4]
... 23 more

Answer 1

There are several issues with your configuration, which are leading to the services not working correctly:
The zookeeper-service is of type NodePort. Therefore, there is no need to specify the port parameter. The traffic will be received on nodePort: 30181 and forwarded to the targetPort: 2181.
The kafka-broker service is not specifying any targetPort parameter. The service is of ClusterIP type by default which requires this parameter. You are receiving traffic on port: 9092 but you're not forwarding it to any pods. You need to add targetPort: 9092 which is the value of your KAFKA_PORT environment variable. This will correctly forward the incoming traffic to the right kafka-service pods.
The springboot application locationmanager is of type Loadbalancer therefore, there is no need to specify the nodePort parameter. Remove it. Additionally, the service is receiving traffic on port: 8080 and forwards it to the pods on targetPort: 8080. This is incorrect, since your application deployment exposes containerPort: 8081 instead of 8080.
Fixing these configuration issues will fix your problem.