So I just recently started on using Kafka and for some odd reason, although the kafka and zookeeper servers containers are running as intended on the documentation and by using the command docker ps. I keep getting this specific error
I have been trying to solve this issue for about a week now with different configs and yet it still keeps giving me the endless loop of an error.
I followed both of these tutorials:
https://rmoff.net/2018/08/02/kafka-listeners-explained/ and https://www.youtube.com/watch?v=-0vrqMMGQbc
docker-compose up runs perfectly fine and the broker starts but when I start the mvn spring-boot:run command that's where it fails.
I've also tried changing the address through my local machine but still nothing. Would appreciate some help
This is my docker-compose.yml
version: '3.2'
services:
zookeeper:
container_name: zookeeper
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
kafka:
container_name: kafka
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: INTERNAL://:19092,EXTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
depends_on:
- zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock
And this is my application.yml
# Server
server:
port: 8090
# Spring Kafka & H2
spring:
autoconfigure:
exclude: org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration
kafka:
bootstrap-servers: localhost:9092
h2:
console:
enabled: true
path: /h2
datasource:
url: jdbc:h2:mem:api;MODE=MySQL
username: api
password: root
driverClassName: org.h2.Driver
result:
topic.name: result
# Keycloak
keycloak:
auth-server-url: http://localhost:8080/auth
realm: demo
resource: demo-api
enabled: false
public-client: false
bearer-only : true
principal-attribute: preferred_username
allow-any-hostname: true
# credentials:
# secret:
# Logging
logging:
level:
org.springframework.web: DEBUG
org.hibernate: DEBUG
org: INFO
com.API.demo.Controllers: DEBUG
config: src/main/resources/logback-spring.xml
file:
path: logs/log
The Advertised Listener is not defined correctly
Instead of
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092
you need
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:19092,EXTERNAL://localhost:9092
Also note that if you want the broker to create topics automagically you'll want to set
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
Related
I am building up a jar file for my spring boot microservice, but upon maven install i am getting the error as stated below.
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:730) ~[kafka-clients-3.1.1.jar:na]
my docker-compose file for the application is
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
restart: always
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 2181:2181
networks:
- app-network
kafka:
image: confluentinc/cp-kafka:latest
container_name: kafka
restart: always
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
networks:
- app-network
mongodb:
image: mongo:latest
restart: always
container_name: mongodb
networks:
- app-network
ports:
- 27017:27017
matching-engine-helper:
image: matching-engine-helper:latest
restart: always
container_name: "matching-engine-helper"
networks:
- app-network
ports:
- 9192:9192
depends_on:
- mongodb
- zookeeper
- kafka
matching-engine-core:
image: matching-engine-core:latest
restart: always
container_name: "matching-engine-core"
networks:
- app-network
ports:
- 9191:9191
depends_on:
- matching-engine-helper
- zookeeper
- kafka
- mongodb
networks:
app-network:
driver: bridge
and the application.yml is
spring:
data:
mongodb:
database: matching-engine-mongodb
host: mongodb
port: 27017
kafka:
bootstrap-servers: kafka:9092
server:
port: 9192
let me know if i need some environment variable's configuration here. as its running fine on local environment on local host but not on docker containers as i am unable to make a jar out of it.
Assuming mvn install is running on your host, then your host doesn't know how to resolve kafka as a DNS name running in a container. It that's what you wanted, then you should rename your file to application-docker.yml and use Spring profiles to properly override any defaults when you actually do run your code in a container. In your case, you would need to have localhost:29092 for Kafka, and similarly use localhost:27017 for Mongo in your default application.yml. You can also use environment variables for those two properties rather than hard-code them.
Or, if mvn install is running in a Docker layer itself, then that is completely isolated from other Docker networks. You can pass --network flag to docker build, though.
Assuming you don't want to mvn install -DskipTests, ideally, your tests do not rely on external services to be running. If you want to run a Kafka unit test, which is failing to connect, then you should either mock that, or using Spring-Kafka's EmbeddedKafka for integration-tests.
Worth mentioning that Spring boot has a Maven plugin for building Docker images, and it doesn't need a JAR to do so.
I'm very new to everything about docker, spring framework... They run on localhost environment successfully, now I want to push them into docker, but always gave me error there, i got stuck on it about a week. Can someone help me figure it out PLEASE!!!
now I have eureka-server and cloud-config-server services, here is my code:
eureka-server application.yml
server:
port: 8761
spring:
application:
name: service-registration
eureka:
client:
register-with-eureka: false
fetch-registry: false
management:
security:
enabled: false
and Dockerfile:
FROM openjdk:11
COPY target/service-registration-0.0.1-SNAPSHOT.jar service-registration.jar
EXPOSE 8761
CMD ["java", "-jar", "service-registration.jar"]
Now I have cloud-config-server application.yml:
In this file, i tried to backup it on github.
From the start, I tried to change the hostname as localhost or any other hostname like service-registration, eureka-server etc... but not work, and "service-url" or "serviceUrl'....
server:
port: 9291
spring:
application:
name: cloud-config-server
profiles:
active: production
# cloud:
# config:
# server:
# git:
# uri: https://github.com/sshaunn/config-server
# clone-on-start: true
eureka:
client:
register-with-eureka: true
fetch-registry: true
serviceUrl:
defaultZone: http://service-registration:8761/eureka/
instance:
# prefer-ip-address: true
# ip-address: http://eureka-server:8761/eureka/å
hostname: service-registration
Dockerfile:
FROM openjdk:11
COPY target/cloud-config-server-0.0.1-SNAPSHOT.jar cloud-config-server.jar
EXPOSE 9291
ENTRYPOINT ["java", "-jar", "cloud-config-server.jar"]
and the docker-compose.yml file:
version: '3.7'
services:
service-registration:
image: service-registration
networks:
- eureka-server
ports:
- "8761:8761"
container_name: service-registration
hostname: service-registration
cloud-config-server:
build: cloud-config-server
networks:
- eureka-server
ports:
- "9291:9291"
depends_on:
- service-registration
container_name: cloud-config-server
hostname: service-registration
environment:
EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://service-registration:8761/eureka
networks:
eureka-server:
name: eureka-server
driver: bridge
# route:
# image: route
# ports:
# - "9191:9191"
# container_name: api-gateway
# environment:
# EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://service-registration:8761/eureka
The error logs here:
2022-05-29 07:22:41.713 WARN 1 --- [freshExecutor-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: I/O error on GET request for "http://localhost:8761/eureka/apps/": Connect to localhost:8761 [localhost/127.0.0.1] failed: Connection refused (Connection refused);
2022-05-29 07:06:11.565 WARN 1 --- [tbeatExecutor-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: I/O error on PUT request for "http://localhost:8761/eureka/apps/CONFIG-SERVER/service-registration:CONFIG-SERVER:9291": Connect to localhost:8761 [localhost/127.0.0.1] failed: Connection refused (Connection refused);
Ok, I dont know why but I set up a docker-compose.yml file like this and it works:
Note, I set up environment about "service-url" defaultZone in docker-compose.yml file.
version: '3.7'
services:
service-registration:
container_name: service-registration
build:
context: ./service-registration
dockerfile: Dockerfile
ports:
- "8761:8761"
cloud-config-server:
container_name: cloud-config-server
build:
context: ./cloud-config-server
dockerfile: Dockerfile
environment:
- eureka.client.service-url.defaultZone=http://service-registration:8761/eureka
depends_on:
- service-registration
ports:
- "9191:9191"
route:
container_name: route
build:
context: ./route
dockerfile: Dockerfile
environment:
- eureka.client.service-url.defaultZone=http://service-registration:8761/eureka
depends_on:
- service-registration
- employee
- income
ports:
- "9291:9291"
employee:
container_name: employee
build:
context: ./employee
dockerfile: Dockerfile
environment:
- eureka.client.service-url.defaultZone=http://service-registration:8761/eureka
depends_on:
- service-registration
ports:
- "9001:9001"
income:
container_name: income
build:
context: ./income
dockerfile: Dockerfile
environment:
- eureka.client.service-url.defaultZone=http://service-registration:8761/eureka
depends_on:
- service-registration
ports:
- "9002:9002"
You should use your host ip address to run in your local machine. For the docker it should not be localhost
Try with the below configuration in your application.properties file.
eureka.client.service-url.defaultZone=http://host.docker.internal:8761/eureka/
I am trying to dockerize my spring boot and mongo.
This is my application.yml
spring:
application:
name: service-app
data:
mongodb:
uri: mongodb://localhost:27017/user
I have found this to be recommended in other posts.
The error I get is:
Exception in monitor thread while connecting to server localhost:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~.
[mongodb-driver-core-4.1.1.jar:na]
version: '3'
services:
mongo:
image: library/mongo:latest
container_name: mongo
restart: always
ports:
- 27017:27017
network_mode: host
volumes:
- $HOME/mongo:/data/db
healthcheck:
test: "exit 0"
user-service:
build:
context: .
dockerfile: Dockerfile
image: user-service
depends_on:
- mongo
network_mode: "host"
hostname: localhost
restart: always
ports:
- 8082:8082
healthcheck:
test: "exit 0"
FROM openjdk:8-jdk-alpine
ADD ./target/demo-0.0.1-SNAPSHOT.jar /usr/src/user-service-0.0.1-SNAPSHOT.jar
WORKDIR usr/src
ENTRYPOINT ["java -Djdk.tls.client.protocols=TLSv1.2","-jar", "user-service-0.0.1-SNAPSHOT.jar"]
It seems to be a common problem for noobsters, I couldn't solve it with what was on the internet already.
The problem is in not building the url properly.
By the information I gave, I should have gotten a proper answer, instead I got -1.
Some people are true trolls with privileges.
uri: mongodb://mongo/user
I feel like it became a gamble on Stackoverflow, depends on the type of people crossing over your question.
I have a sample project which consists of a registry module that starts a Eureka server, a spring cloud gateway, and Keycloak.
The problem is by running Keycloak, registry using docker-compose and the gateway as a normal application, everything works, that is, when I want to see the eureka dashboard, I get redirected to Keycloak to authenticate and then redirected to the dashboard, but it is not the case when I execute the gateway together with others using docker-compose and it complains:
Caused by: java.lang.IllegalArgumentException: Unable to resolve Configuration with the provided Issuer of "http://127.0.0.1:8090/auth/realms/dev"
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://127.0.0.1:8090/auth/realms/dev/.well-known/openid-configuration": Connection refused
while I can get the configuration by pasting the link in the browser.
I expect it to be my misconfiguration with docker-compose and I would appreciate it if you can give me some ideas.
gateway application.yml
spring:
application:
name: gateway
cloud:
gateway:
routes:
- id: firstService
uri: lb://first-microservice
predicates:
- Path=/first/**
filters:
- TokenRelay=
- RemoveRequestHeader=Cookie
- id: secondService
uri: lb://second-microservice
predicates:
- Path=/second/**
# filters:
# - StripPrefix=1
filters:
- TokenRelay=
- RemoveRequestHeader=Cookie
- id: registry
uri: lb://registry
predicates:
- Path=/registry/**
filters:
- StripPrefix=1
- id: eureka
uri: lb://registry
predicates:
- Path=/eureka/**
autoconfigure:
# TODO: remove when fixed https://github.com/spring-projects/spring-security/issues/6314
exclude: org.springframework.boot.actuate.autoconfigure.security.reactive.ReactiveManagementWebSecurityAutoConfiguration
security:
oauth2:
client:
registration:
keycloak:
client-id: backend
client-secret: '2baa28ce-9607-44a3-a42c-a0bb2102a66d'
provider:
keycloak:
issuer-uri: ${ISSUER_URI:http://127.0.0.1:8090/auth/realms/dev}
user-name-attribute: preferred_username
server:
port: 8079
info:
app:
name: ${spring.application.name}
eureka:
client:
registerWithEureka: true
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}
healthcheck:
enabled: true
docker-compose.yml
version: '3'
volumes:
postgres_data:
driver: local # is already local by default
keycloak-data-volume:
driver: local # is already local by default
# external: true
services:
postgres:
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
networks:
- net
keycloak:
image: jboss/keycloak:11.0.0
environment:
DB_VENDOR: POSTGRES
DB_ADDR: postgres
DB_DATABASE: keycloak
DB_USER: keycloak
DB_SCHEMA: public
DB_PASSWORD: password
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: Pa55w0rd
volumes:
- keycloak-data-volume:/var/lib/keycloak/data
command: ["-Djboss.socket.binding.port-offset=10"]
expose:
- 8090
ports:
- 8090:8090
depends_on:
- postgres
networks:
- net
registry:
image: sample-cloud-registry:latest
container_name: registry
expose:
- 8761
networks:
- net
environment:
- EUREKA_SERVER=http://registry:8761/eureka/
gateway:
image: sample-cloud-gateway:latest
container_name: gateway
expose:
- 8079
ports:
- 127.0.0.1:8080:8079
networks:
- net
restart: always
depends_on:
- registry
- keycloak
environment:
- ISSUER_URI=http://127.0.0.1:8090/auth/realms/dev
- EUREKA_SERVER=http://registry:8761/eureka/
networks:
net:
I managed to solve my issue and here are my changes:
As I am using Eureka, I changed ISSUER_URI=http://127.0.0.1:8090/auth/realms/dev to use the hostname of keycloak container so the result is :
- ISSUER_URI=http://keycloak:8090/auth/realms/dev
You need to pay attention that the port number in the above line is the container port and not necessarily the host port.
And then, you need to add keycloak to your known hosts under etc to let login page be accessible via browser:
127.0.0.1 keycloak
i m trying to deploy a Spring Server on Docker. The Spring server is connecting to a RabbitMQ Server that runs in another app. I get an error while connecting to the rabbitmq. I added the host in my application.properties, didnt work. Than added to the app-container as environment variable and still doesnt work. I also rebuilded the jar and changed the images version.
version: '3'
services:
rabbitmq:
image: rabbitmq:management
ports:
- "5672:5672" #JMS Port
- "15672:15672" #Management Port - default user:pass = guest:guest
db:
image: mysql:5.7.22
environment:
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "hospital"
MYSQL_PASSWORD: "root"
ports:
- "3306:3306"
networks:
- mysql_bridge
restart: always
springboot-docker-compose-app-container:
image: app-image
build:
context: ./
dockerfile: Dockerfile
environment: # Pass environment variables to the service
SPRING_DATASOURCE_URL: jdbc:mysql://db:3306/hospital?useSSL=false&serverTimezone=UTC&useLegacyDatetimeCode=false&allowPublicKeyRetrieval=true
SPRING_DATASOURCE_USERNAME: root
SPRING_DATASOURCE_PASSWORD: root
SPRING_RABBITMQ_HOST: rabbitmq
depends_on:
- rabbitmq
- db
volumes:
- /data/VerzorgerSOAP
ports:
- "8080:8080"
networks:
- mysql_bridge
- rabbiy_mq
networks:
mysql_bridge:
rabbiy_mq:
Also this is my application.properties
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.generate-ddl=true
spring.datasource.url = jdbc:mysql://db:3306/hospital?useSSL=false&serverTimezone=UTC&useLegacyDatetimeCode=false&allowPublicKeyRetrieval=true
spring.jpa.hibernate.ddl-auto = update
spring.datasource.username = root
spring.datasource.password = root
spring.jackson.serialization.WRITE_DATES_AS_TIMESTAMPS = false
server.ssl.enabled=false
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
spring.jpa.properties.hibernate.globally_quoted_identifiers=true
spring.jpa.properties.hibernate.enable_lazy_load_no_trans=true
spring.rabbitmq.host= rabbitmq
This is the error i get
org.springframework.amqp.AmqpIOException: java.net.UnknownHostException: rabbitmq
The networks configuration was missing for rabbitmq in the dockercompose file:
rabbitmq:
image: rabbitmq:management
ports:
- "5672:5672" #JMS Port
- "15672:15672" #Management Port - default user:pass = guest:guest
networks:
- rabbiy_mq
Another solution suggested by #david-maze is to remove all networks