I am building a spring boot application which can produce some messages via Apache Kafka.
I could connect the spring boot application to kafka when running locally.
But when I containerized it, it fails to connect
2022-09-05 11:42:49.354 INFO 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Node -1 disconnected.
2022-09-05 11:42:49.355 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
docker-compse.yml file
version: "3"
services:
zookeeper:
image: zookeeper
network_mode: bridge
container_name: zookeeper
ports:
- "2181:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: wurstmeister/kafka
network_mode: bridge
container_name: kafka
ports:
- "9092:9092"
environment:
HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
below are the commands used to build spring boot app
1> mvn clean install -DskipTests
2> docker build -f Dockerfile -t url-feeder-service .
3> docker run --name url-feeder-service -p 8085:8085 --network bridge url-feeder-service
Related
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 10 months ago.
I am converting test project to docker
I have working containers: Mysql,Application,Kafka.
Im having error in zookeeper or maybe wrong set up in docker-compose.yaml
//docker-compose.yaml
version: '3.8'
networks:
product-net:
driver: bridge
services:
productmicroservice:
image: productmicroservice:latest
container_name: productmicroservice
depends_on:
- product-mysqldb
- kafka
restart: always
build:
context: ./
dockerfile: Dockerfile
ports:
- "9001:8091"
environment:
MYSQL_HOST: product-mysqldb
MYSQL_USER: root
MYSQL_PASSWORD: root
links:
- kafka:localhost
networks:
- product-net
product-mysqldb:
image: mysql:8.0.28
restart: unless-stopped
container_name: product-mysqldb
ports:
- "3307:3306"
cap_add:
- SYS_NICE
environment:
MYSQL_DATABASE: dbpoc
MYSQL_ROOT_PASSWORD: root
networks:
- product-net
zookeeper:
image: wurstmeister/zookeeper:latest
container_name: zookeeper
restart: on-failure
ports:
- "2181:2181"
networks:
- product-net
kafka:
image: wurstmeister/kafka:2.11-1.1.1
restart: unless-stopped
container_name: kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_MESSAGE_MAX_BYTES: 2000000
KAFKA_CREATE_TOPICS: "producttopic:1:1"
BROKER_ID: 1
ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- product-net
depends_on:
- zookeeper
//Dockerfile
FROM openjdk:8
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
//application.yaml
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://${MYSQL_HOST:localhost}:${MYSQL_PORT:3306}/dbpoc
username: root
password: root
kafka:
template:
default-topic: producttopic
producer:
bootstrap-servers:
- localhost:9092
key-serializer:
org.apache.kafka.common.serialization.StringSerializer
value-serializer:
org.springframework.kafka.support.serializer.JsonSerializer
jpa:
hibernate:
naming:
implicit-strategy: org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
physical-strategy: org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy
hibernate.ddl-auto: update
generate-ddl: "false"
show-sql: "false"
properties:
hibernate:
dialect: org.hibernate.dialect.MySQL5InnoDBDialect
mvc:
throw-exception-if-no-handler-found: "true"
web:
resources:
add-mappings: "false"
sql:
init:
mode: always
continue-on-error: "true"
server:
port: 8091
//Error show
kafka | creating topics: producttopic:1:1
zookeeper | 2022-05-03 12:21:55,223 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#215] - Accepted socket connection from /172.27.0.4:34150
zookeeper | 2022-05-03 12:21:55,225 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#949] - Client attempting to establish new session at /172.27.0.4:34150
zookeeper | 2022-05-03 12:21:55,229 [myid:] - INFO [SyncThread:0:ZooKeeperServer#694] - Established session 0x100008559dd0003 with negotiated timeout 30000 for client /172.27.0.4:34150
zookeeper | 2022-05-03 12:21:55,377 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor#487] - Processed session termination for sessionid: 0x100008559dd0003
zookeeper | 2022-05-03 12:21:55,381 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1056] - Closed socket connection for client /172.27.0.4:34150 which had sessionid 0x100008559dd0003
//after i try to send a data
2022-05-03 12:15:52.695 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
you have to set the ADVERTISE_LISTENERS properly for the internal and external docker network, something like:
KAFKA_ADVERTISED_LISTENERS: >-
LISTENER_DOCKER_INTERNAL://kafka:19092,
LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-localhost}:9092
you can find a working example here: https://github.com/stockgeeks/docker-compose/blob/master/one-to-run-them-all/docker-compose.yml
And an article with explanation here: https://dev.to/thegroo/one-to-run-them-all-1mg6
And some explanation on the advertise addresses in this article: https://dev.to/thegroo/running-kafka-on-kubernetes-for-local-development-with-storage-class-4oa9 in the ned the section "Connecting a Kafka client"
how can docker-compose run comands as no-root user? My docker-compose.yml just show below:
version: "3"
services:
sonarqube-h:
image: sonarqube:community
user: "${UID}:${GID}"
depends_on:
- db
environment:
SONAR_JDBC_URL: jdbc:postgresql://db:5432/sonar
SONAR_JDBC_USERNAME: sonar
SONAR_JDBC_PASSWORD: sonar
volumes:
- /home/win/docker-app/sonarqube/docker-compose-sonardata/data:/opt/sonarqube/data
- /home/win/docker-app/sonarqube/docker-compose-sonardata/extensions:/opt/sonarqube/extensions
- /home/win/docker-app/sonarqube/docker-compose-sonardata/logs:/opt/sonarqube/logs
- /home/win/docker-app/sonarqube/docker-compose-sonardata/conf:/opt/sonarqube/conf
ports:
- "9001:9000"
networks:
sonarqube-net:
ipv4_address: 172.19.0.8
db:
image: postgres:12
environment:
POSTGRES_USER: sonar
POSTGRES_PASSWORD: sonar
volumes:
- /home/win/docker-app/sonarqube/docker-compose-sonardata/postgresql:/var/lib/postgresql
- /home/win/docker-app/sonarqube/docker-compose-sonardata/postgresql/data:/var/lib/postgresql/data
networks:
sonarqube-net:
ipv4_address: 172.19.0.7
networks:
sonarqube-net:
driver: bridge
ipam:
config:
- subnet: 172.19.0.0/16
when I use env UID=1000 GID=1000 docker-compose up to build, It turns out to be logs belows, due to the es was run as root user:
sonarqube-h_1 | 2022.03.03 15:33:37 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:39789]
sonarqube-h_1 | 2022.03.03 15:33:38 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
sonarqube-h_1 | could not find java in ES_JAVA_HOME at /usr/lib/jvm/java-11-openjdk/bin/java
sonarqube-h_1 | 2022.03.03 15:33:38 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
sonarqube-h_1 | 2022.03.03 15:33:38 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
sonarqube-h_1 | 2022.03.03 15:33:38 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped
sonarqube-h_1 | 2022.03.03 15:33:38 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
sonarqube_sonarqube-h_1 exited with code 0
I have a problem with kafka when I work with http protocol kafka run with me without any problem
but after upgrading the protocol from http to https
after upgrade it all services return the same error
2021-10-28 23:16:26.367 WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Connection to node 1 (/206.189.178.228:9092) could not be established. Broker may not be available.
2021-10-28 23:18:02.623 WARN 1 --- [ntainer#1-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-2, groupId=groupId] Connection to node 1 (/206.189.178.228:9092) could not be established. Broker may not be available.
in producer and consumer i dont know what happens or if this error appeared from SSL
but i did not change any configuration about Kafka
this is my kafak.yml file
kafka.yml
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.5.0
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
networks:
- food_default
kafka:
image: confluentinc/cp-kafka:5.5.0
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: kafka
LISTENERS: PLAINTEXT://206.189.178.228:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://206.189.178.228:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
networks:
- food_default
networks:
food_default:
external: true
and this is service config this config existing in each other service
version: '2'
services:
food-order-service:
image: altshiftcreative/food-app-order-service:v1.5
environment:
# - _JAVA_OPTIONS=-Xmx512m -Xms256m
- SPRING_PROFILES_ACTIVE=prod,swagger
- MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED=false
- EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://admin:$${jhipster.registry.password}#jhipster-registry:8761/eureka
- SPRING_CLOUD_CONFIG_URI=http://admin:$${jhipster.registry.password}#jhipster-registry:8761/config
- SPRING_DATASOURCE_URL=jdbc:mysql://food-mysql:3306/food_db?useUnicode=true&characterEncoding=utf8&useSSL=false&allowPublicKeyRetrieval=true&useLegacyDatetimeCode=false&serverTimezone=UTC&createD>
- SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_OIDC_ISSUER_URI=https://shopbia.shop/auth/realms/jhipster
- SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_OIDC_CLIENT_ID=internal
- SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_OIDC_CLIENT_SECRET=internal
# - JHIPSTER_SLEEP=30 # gives time for other services to boot before the application
- KAFKA_BOOTSTRAPSERVERS=kafka:9092
ports:
- 8082:8082
networks:
- food_default
networks:
food_default:
external: true
Kafka isn't an HTTP protocol, so your problem has nothing to do with that
You've set KAFKA_BOOTSTRAPSERVERS=kafka:9092
i did not change any configuration about Kafka
I think you did. That IP address is not the default of anything. To revert to expected behavior within the same Docker network, you need to set
LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
Unclear where you're getting the IP address from, but if you want to expose the Docker container to an external network in that subnet, then you will need other settings
I am trying to send data from Kafka to Elasticsearch. I checked that my Kafka Broker is working because I can see the messages I produce to a topic is read by a Kafka Consumer. However, when I try to connect Kafka to Elasticsearch I get the following error.
Command:
connect-standalone etc/schema-registry/connect-avro-standalone.properties \
etc/kafka-connect-elasticsearch/quickstart-elasticsearch.properties
Error:
ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone)
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:83)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
... 2 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
My Docker Compose File:
version: '3'
services:
zookeeper:
container_name : zookeeper
image: zookeeper
ports:
- 2181:2181
- 2888:2888
- 3888:3888
kafka:
container_name : kafka
image: bitnami/kafka:1.0.0-r5
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_BROKER_ID: "42"
KAFKA_ADVERTISED_HOST_NAME: "kafka"
ALLOW_PLAINTEXT_LISTENER: "yes"
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
elasticsearch:
container_name : elasticsearch
image:
docker.elastic.co/elasticsearch/elasticsearch:7.8.0
environment:
- node.name=elasticsearch
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elasticsearch
- bootstrap.memory_lock=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data99:/usr/share/elasticsearch/data
ports:
- 9200:9200
kibana:
container_name : kibana
image: docker.elastic.co/kibana/kibana:7.8.0
# environment:
# - SERVER_NAME=Local kibana
# - SERVER_HOST=0.0.0.0
# - ELASTICSEARCH_URL=elasticsearch:9400
ports:
- "5601:5601"
depends_on:
- elasticsearch
kafka-connect:
container_name : kafka-connect
image: confluentinc/cp-kafka-connect:5.3.1
ports:
- 8083:8083
depends_on:
- zookeeper
- kafka
volumes:
- $PWD/connect-plugins:/connect-plugins
environment:
CONNECT_BOOTSTRAP_SERVERS: kafka:9092
CONNECT_REST_ADVERTISED_HOST_NAME: "localhost"
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: kafka-connect
CONNECT_CONFIG_STORAGE_TOPIC: docker-kafka-connect-configs
CONNECT_OFFSET_STORAGE_TOPIC: docker-kafka-connect-offsets
CONNECT_STATUS_STORAGE_TOPIC: docker-kafka-connect-status
CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_KEY_CONVERTER-SCHEMAS_ENABLE: "false"
CONNECT_VALUE_CONVERTER-SCHEMAS_ENABLE: "false"
CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
CONNECT_LOG4J_ROOT_LOGLEVEL: "ERROR"
CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_TOPICS: "test-elasticsearch-sink"
CONNECT_TYPE_NAME: "type.name=kafka-connect"
CONNECT_PLUGIN_PATH: '/usr/share/java' #'/usr/share/java'
# Interceptor config
CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-5.3.1.jar
CONNECT_KAFKA_HEAP_OPTS: "-Xms256m -Xmx512m"
volumes:
data99:
driver: local
I checked some other questions and answers but couldn't come up with a solution to this problem.
Thanks in advance!
The Connect container starts Connect Distributed Server already. You should use HTTP and JSON properties to configure the Elastic connector rather than exec into the container shell and issue connect-standalone commands which default to using a broker running in the container itself.
Similarly, the Elastic quickstart file expects Elasticsearch running within the Connect container, by default
I have this docker-compose.yml in which I run Zookeeper, Kafka, Kafka Connect, and KafDrop, the thing is, when I run locally I can connect from my Spring Boot application to consume some topic messages.
What I need is to run the same configuration on a Linux machine and be able to connect from the Spring Boot application the same way.
When run it remotely on the Linux machine everything seems to be running Ok, but when I try to connect from the Spring Boot application I receive some erros showing that somethin is wrong in the connection.
I will try to explain step by step and see if someone can give a "light" on that:
docker-compose.yml:
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
networks:
- broker-kafka
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
networks:
- broker-kafka
restart: unless-stopped
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS:
INTERNAL://kafka:29092,
EXTERNAL://localhost:9092
KAFKA_ADVERTISED_LISTENERS:
INTERNAL://kafka:29092,
EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
INTERNAL:PLAINTEXT,
EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG_RETENTION_HOURS: 12
connect:
image: cdc:latest
networks:
- broker-kafka
depends_on:
- zookeeper
- kafka
ports:
- 8083:8083
environment:
CONNECT_BOOTSTRAP_SERVERS: kafka:29092
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-1
CONNECT_CONFIG_STORAGE_TOPIC: connect-1-config
CONNECT_OFFSET_STORAGE_TOPIC: connect-1-offsets
CONNECT_STATUS_STORAGE_TOPIC: connect-1-status
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_OFFSET.STORAGE.REPLICATION.FACTOR: 1
CONNECT_CONFIG.STORAGE.REPLICATION.FACTOR: 1
CONNECT_OFFSET.STORAGE.PARTITIONS: 1
CONNECT_STATUS.STORAGE.REPLICATION.FACTOR: 1
CONNECT_STATUS.STORAGE.PARTITIONS: 1
CONNECT_REST_ADVERTISED_HOST_NAME: localhost
kafdrop:
image: obsidiandynamics/kafdrop:latest
networks:
- broker-kafka
depends_on:
- kafka
ports:
- 19000:9000
environment:
KAFKA_BROKERCONNECT: kafka:29092
networks:
broker-kafka:
driver: bridge
What I need is to expose to my network this IP machine to be accessed by my Spring Boot application.
Let´s suppose this Linux machine has the IP 10.12.54.99.
How can I make it Kafka be accessible by: 10.12.54.99:9090 ?
Here is my application.properties:
spring.kafka.bootstrap-servers=10.12.54.99:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-commit-interval=100
spring.kafka.consumer.max-poll-records=10
spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.group-id=connect-sql-server
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.ack-mode=manual-immediate
spring.kafka.listener.poll-timeout=3000
spring.kafka.listener.concurrency=3
spring.kafka.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.properties.spring.deserializer.value.delegate.class=org.apache.kafka.common.serialization.StringDeserializer
This is a only consumer-specif application (no producers are used here).
When I run the application:
2020-12-07 10:59:40.361 WARN 58716 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-connect-sql-server-1, groupId=connect-sql-server] Connection to node -1 (/10.12.54.99:9092) could not be established. Broker may not be available.
2020-12-07 10:59:40.362 WARN 58716 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-connect-sql-server-1, groupId=connect-sql-server] Bootstrap broker 10.12.54.99:9092 (id: -1 rack: null) disconnected
All the firewall ports are enabled in the Linux firewall machie.
Any enlightenment would be very much appreciated.
You need to bind your server's public ip in order to be able to access brokers remotely. However if you don't want to hardcode the ip, you can use .env file.
Do the following:
Create config.env file.
Add this line in config.env and add your host ip as below:
DOCKER_HOST_IP=111.111.11.111
Update your docker-compose:
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
networks:
- broker-kafka
ports:
- ${DOCKER_HOST_IP:-127.0.0.1}:2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:latest
networks:
- broker-kafka
restart: unless-stopped
depends_on:
- zookeeper
ports:
- ${DOCKER_HOST_IP:-127.0.0.1}:9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS:
INTERNAL://kafka:29092,
EXTERNAL://localhost:9092
KAFKA_ADVERTISED_LISTENERS:
INTERNAL://kafka:29092,
EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
INTERNAL:PLAINTEXT,
EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LOG_RETENTION_HOURS: 12
connect:
image: cdc:latest
networks:
- broker-kafka
depends_on:
- zookeeper
- kafka
ports:
- 8083:8083
environment:
CONNECT_BOOTSTRAP_SERVERS: kafka:29092
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-1
CONNECT_CONFIG_STORAGE_TOPIC: connect-1-config
CONNECT_OFFSET_STORAGE_TOPIC: connect-1-offsets
CONNECT_STATUS_STORAGE_TOPIC: connect-1-status
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_OFFSET.STORAGE.REPLICATION.FACTOR: 1
CONNECT_CONFIG.STORAGE.REPLICATION.FACTOR: 1
CONNECT_OFFSET.STORAGE.PARTITIONS: 1
CONNECT_STATUS.STORAGE.REPLICATION.FACTOR: 1
CONNECT_STATUS.STORAGE.PARTITIONS: 1
CONNECT_REST_ADVERTISED_HOST_NAME: localhost
kafdrop:
image: obsidiandynamics/kafdrop:latest
networks:
- broker-kafka
depends_on:
- kafka
ports:
- 19000:9000
environment:
KAFKA_BROKERCONNECT: kafka:29092
networks:
broker-kafka:
driver: bridge
It will bind to 127.0.0.1, if DOCKER_HOST_IP is not found.
Run the following command:
sudo docker-compose -f path-to-docker-compose.yml --env-file path-to-config.env up -d --force-recreate