Spring Cloud Consul and Consul Clients dockerized - spring-boot

I have 2 applications, both written using spring boot. Both are running in different docker containers. I also have consul running in a different docker container. I have exposed port 8500 for consul using docker-compose.yml file. So, how do I specify to my spring boot applications where to register themselves, i.e, where is consul running. Do I give the address of the mapped port (port mapped to my local machine), or some other change?
The example I'm using right now: https://github.com/Java-Techie-jt/cloud-consul-service-discovery
Edit:
docker-compose.yml:
version: "2"
services:
consul:
container_name: consul
image: consul
expose:
- "8300"
- "8400"
- "8500"
restart: always
registrator:
container_name: registrator
image: gliderlabs/registrator:master
volumes:
- "/var/run/docker.sock:/tmp/docker.sock"
command: -internal consul://consul:8500
restart: always
depends_on:
- consul
web1:
image: deis/mock-http-server
container_name: web1
expose:
- "8080"
environment:
SERVICE_NAME: "web"
SERVICE_TAGS: "web"
restart: always
depends_on:
- registrator
web2:
image: deis/mock-http-server
container_name: web2
expose:
- "8080"
environment:
SERVICE_8080_NAME: "web"
SERVICE_8080_TAGS: "web"
restart: always
depends_on:
- registrator
haproxy:
build: ./haproxy
container_name: my-haproxy
image: anthcourtney/haproxy-consul
ports:
- 80
depends_on:
- web1
- web2
test:
container_name: test-client
build: ./test
depends_on:
- haproxy
networks:
default:

You can use registrator for your service registry.
Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online. Registrator supports pluggable service registries, which currently includes Consul, etcd and SkyDNS 2.
You can run registrator as a container.It will register each port of your application. Below is the sample compose file :-
version: '2'
services:
registrator:
image: "${REGISTRY}gliderlabs/registrator:latest"
command: [
"-ip=<docker-host-ip>",
"-retry-attempts", "100",
"-cleanup",
# "-internal",
"consul://vconsul:8500"
]
official documentation : https://gliderlabs.github.io/registrator/latest/

Related

No resolvable bootstrap urls given in bootstrap.servers on maven install

I am building up a jar file for my spring boot microservice, but upon maven install i am getting the error as stated below.
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:730) ~[kafka-clients-3.1.1.jar:na]
my docker-compose file for the application is
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
restart: always
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 2181:2181
networks:
- app-network
kafka:
image: confluentinc/cp-kafka:latest
container_name: kafka
restart: always
depends_on:
- zookeeper
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
networks:
- app-network
mongodb:
image: mongo:latest
restart: always
container_name: mongodb
networks:
- app-network
ports:
- 27017:27017
matching-engine-helper:
image: matching-engine-helper:latest
restart: always
container_name: "matching-engine-helper"
networks:
- app-network
ports:
- 9192:9192
depends_on:
- mongodb
- zookeeper
- kafka
matching-engine-core:
image: matching-engine-core:latest
restart: always
container_name: "matching-engine-core"
networks:
- app-network
ports:
- 9191:9191
depends_on:
- matching-engine-helper
- zookeeper
- kafka
- mongodb
networks:
app-network:
driver: bridge
and the application.yml is
spring:
data:
mongodb:
database: matching-engine-mongodb
host: mongodb
port: 27017
kafka:
bootstrap-servers: kafka:9092
server:
port: 9192
let me know if i need some environment variable's configuration here. as its running fine on local environment on local host but not on docker containers as i am unable to make a jar out of it.
Assuming mvn install is running on your host, then your host doesn't know how to resolve kafka as a DNS name running in a container. It that's what you wanted, then you should rename your file to application-docker.yml and use Spring profiles to properly override any defaults when you actually do run your code in a container. In your case, you would need to have localhost:29092 for Kafka, and similarly use localhost:27017 for Mongo in your default application.yml. You can also use environment variables for those two properties rather than hard-code them.
Or, if mvn install is running in a Docker layer itself, then that is completely isolated from other Docker networks. You can pass --network flag to docker build, though.
Assuming you don't want to mvn install -DskipTests, ideally, your tests do not rely on external services to be running. If you want to run a Kafka unit test, which is failing to connect, then you should either mock that, or using Spring-Kafka's EmbeddedKafka for integration-tests.
Worth mentioning that Spring boot has a Maven plugin for building Docker images, and it doesn't need a JAR to do so.

How to access an external server folder from a spring boot docker container?

I'm using docker-compose to run spring boot application inside a docker container. A spring boot uses embedded tomcat to run.
There is an external folder on a server (/opt/cp/uploads) with images which I would like to access from a spring boot docker container.
Within the docker-compose.yaml file there are following containers defined:
nginx
mysql
springboot-app
nginx and CloudFlare redirect the domain to the spring app on the port 8080.
I'd like to access images like this:
https://domainname.com/uploads/imageName.png
Using tomcat this was done in server.xml:
<Context docBase="/opt/uploads" path="/uploads"/>
but this option is obviously not available via application.properties with embedded tomcat.
How does one come about this?
My docker-compose.yaml:
version: '3.8'
services:
nginx:
container_name: some-nginx
hostname: nginx
image: nginx:1.19.2-alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d
restart: unless-stopped
network_mode: host
mysqldb:
image: mysql:8.0.20
hostname: mysqldb
container_name: cp-mysqldb
environment:
- MYSQL_ROOT_PASSWORD=pass1234
- MYSQL_DATABASE=db_name
- MYSQL_USER=root
- MYSQL_PASSWORD=pass1234
ports:
- "3306:3306"
volumes:
- cp-mysqldb-data:/opt/mysql
restart: unless-stopped
springboot-app:
image: openjdk:8
hostname: cp
container_name: cp-springboot
environment:
- SPRING_DATASOURCE_URL=jdbc:mysql://mysqldb:3306/db_name?autoReconnect=true&useSSL=false&useUnicode=yes&characterEncoding=UTF-8&characterSetResults=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&allowPublicKeyRetrieval=true&serverTimezone=UTC
ports:
- "8080:8080"
depends_on:
- mysqldb
volumes:
- cp-springboot-data:/opt/cp
- ./target/cp-springboot-0.0.1-SNAPSHOT.war:/ROOT.war
command: ["java", "-jar",
"-Dspring.profiles.active=prod",
"ROOT.war"]
restart: always
volumes:
cp-springboot-data: {
}
cp-mysqldb-data: {
}
Maybe better use Tomcat for docker image and not OpenJDK.
That way you can setup your Tomcat configuration for the external path
Take a look here: https://medium.com/#iamvickyav/deploying-spring-boot-war-in-tomcat-based-docker-2b689b206496
Actually, this option IS available via application.properties with embedded tomcat:
spring.mvc.static-path-pattern=/uploads/**
spring.resources.static-locations=file:/opt/uploads

Register Zipkin with Eureka server

I have an application that is divided into a few microservices (using Spring Eureka project).
All the services are registered to Eureka Server - so that the communication between the services can be realized through a "Gateway API" (Eureka Server)
The logs produced by the services are reported to a Zipkin server that runs as a separate service.
All works as expected but when I go to the Eureka dashboard I am not able to see my Zipkin service since it is not registered with Eureka.
Question: Is it possible to register Zipkin with Eureka Server?
Here is my docker-compose file:
version: '3'
services:
logs-aggregator:
container_name: logs-aggregator
image: maimas/sr-logs-aggregator-service:0.1.2-SNAPSHOT
ports:
- 9411:9411
database-service:
container_name: database-service
image: maimas/sr-database-service:0.1.2-SNAPSHOT
ports:
- 27017:27017
- 28017:28017
gateway-api:
depends_on:
- logs-aggregator
- database-service
container_name: gateway-api
image: maimas/sr-gateway-api-service:0.1.2-SNAPSHOT
ports:
- 8081:8081
environment:
EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://admin:12341234#gateway-api:8081/eureka
SPRING_ZIPKIN_BASEURL: http://logs-aggregator:9411/
user-service:
depends_on:
- gateway-api
container_name: user-service
image: maimas/sr-user-service:0.1.2-SNAPSHOT
ports:
- 8082:8082
environment:
EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://admin:12341234#gateway-api:8081/eureka
SPRING_ZIPKIN_BASEURL: http://logs-aggregator:9411/
SPRING_DATA_MONGODB_URI: mongodb://database-service/smartrent2
property-service:
container_name: property-service
image: maimas/sr-property-service:0.1.2-SNAPSHOT
ports:
- 8083:8083
environment:
EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://admin:12341234#gateway-api:8081/eureka
SPRING_ZIPKIN_BASEURL: http://logs-aggregator:9411/
SPRING_DATA_MONGODB_URI: mongodb://database-service/smartrent2
renter-service:
container_name: renter-service
image: maimas/sr-renter-service:0.1.2-SNAPSHOT
ports:
- 8084:8084
environment:
EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://admin:12341234#gateway-api:8081/eureka
SPRING_ZIPKIN_BASEURL: http://logs-aggregator:9411/
SPRING_DATA_MONGODB_URI: mongodb://database-service/smartrent2
I think you must have found your answer by now. But I am posting this for future reference.
Take a look at this Github issue, it basically explains everything and provides a few workarounds.

Deploying Dockerized SpringBoot Microservices with Eureka to AWS Fargate

I have a couple of dockerized microservices that register with Eureka & works on my local env with Docker Compose.
The same when deployed to AWS Fargate via its CLI (using the same Compose file), fails.
I also tried changing the reference to Eureka, in clients, from (which works with Docker Compose)
eureka.client.serviceUrl.defaultZone=http://eureka:8761/eureka
to
eureka.client.serviceUrl.defaultZone=http://127.0.0.1:8761/eureka
The Docker Compose file is shown below:
version: "3.0"
services:
eureka:
image: navraj28/eureka
ports:
- "8761:8761"
networks:
ecsfs:
aliases:
- eureka
facade-service:
image: navraj28/facade-service
ports:
- "8100:8100"
environment:
- eureka.host=eureka
- eureka.client.enabled='true'
- eureka.instance.preferIpAddress='true'
depends_on:
- eureka
- geri-service
logging:
driver: awslogs
options:
awslogs-group: ai-facade
awslogs-region: us-east-1
awslogs-stream-prefix: facade-service
networks:
ecsfs:
geri-service:
image: navraj28/geri-service
ports:
- "8000:8000"
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/credentials.json
- eureka.host=eureka
- eureka.client.enabled='true'
- eureka.instance.preferIpAddress='true'
depends_on:
- eureka
logging:
driver: awslogs
options:
awslogs-group: ai-facade
awslogs-region: us-east-1
awslogs-stream-prefix: geri-service
networks:
ecsfs:
networks:
ecsfs:
Anyone had success with this combo? And the error log messages just don't make sense:

Send mail from a container using a postfix container

I'm using an application hosted on a docker container.
This application executes bash scripts / instructions to send mails.
I made another container which executes Postfix as a SMTP Relay.
I want to send mails from my application container by using a bash script using my Postfix container as a relay.
I tried to connect with SSH from my application container to my Postfix container. But that doesn't seem to work.
How can i make it so a script executed in my application container can use my Postfix relay while not allowing anything outside of the docker network, or even better, to only allow some containers, to send mails from this relay.
EDIT 1 : Docker-compose files
Application docker compose :
version: "3.4"
volumes:
[...]
services:
application:
restart: always
build: ./application
depends_on:
- mariadb
container_name: application
volumes:
[...]
ports:
- "80:80"
- "443:443"
- "5669:5669"
deploy:
restart_policy:
window: 300s
links:
- mariadb
external_links:
- smtp-server
mariadb:
restart: always
image: mariadb
command: mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
container_name: application-mariadb
volumes:
[...]
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
deploy:
restart_policy:
window: 300s
Here's my docker compose for my SMTP server :
version: "3.4"
services:
postfix:
restart: always
build: ./postfix
container_name: smtp-server
deploy:
restart_policy:
window: 300s
{a quick response, because I "cicle" in my work ... and I'm taking 10 minutes of clear my mint, I hope it serves you}
Are you using "docker-compose" ?, could you give an example of your YML file? (a little more context)
[you can not connect to by ssh to a container unless you have "supervisor" installed,which I do not recommend at all.]
from what I see, you only need to make private networks; You could use this:
https://docs.docker.com/compose/networking/
to hide everything, I also recommend using a load balancer / Inverse Proxy like TRAEFIK (if they have access to port 80 or 443 in some clear way this ...)
so you only expose 1/2 port(s) (80 + 443 for example) and everything else is protected by your reverse proxy
Watch as I separate the networks as you need the different containers.
bash have access to db and smtp
db does not have access smtp neither nginx
nginx have access to bash
nginx have access to proxy network to expose 80 and 443
no other container is exposed to the outside more than nginx
--
version: "3"
services:
bash:
####### use hostname "smtp" as SMTP server
image: bash
depends_on:
- db
networks:
- smtp_internal_network
- internal_network
- data_network
volumes:
- ../html:/var/www/html
restart: always
db:
image: percona:5.7
# ports: # for debug connections and querys
# - 3306:3306
volumes:
- ../db_data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
networks:
- data_network
restart: always
smtp:
image: mwader/postfix-relay
environment:
- POSTFIX_myhostname=smtp.domain.tld
networks:
- smtp_internal_network
restart: always
nginx:
image: nginx
volumes:
- ../html:/var/www/html
networks:
- external_network
- internal_network
labels:
- "traefik.backend=nginx_${COMPOSE_PROJECT_NAME}"
- "traefik.port=80"
- "traefik.frontend.rule=Host:${FRONTEND_RULE}"
- "traefik.frontend.passHostHeader=true"
- "traefik.enable=true"
- "traefik.docker.network=traefik_proxy"
restart: always
depends_on:
- db
- bash
networks:
external_network:
external:
name: traefik_proxy
internal_network:
driver: bridge
smtp_internal_network:
driver: bridge
data_network:
driver: bridge
Edit:
version: "3"
volumes:
[...]
services:
####### use hostname "smtp" as SMTP server in your application
application:
restart: always
build: ./application
depends_on:
- mariadb
volumes:
[...]
ports:
- "80:80"
- "443:443"
- "5669:5669"
deploy:
restart_policy:
window: 300s
networks:
- smtp_external_network
- data_network
mariadb:
restart: always
image: mariadb
command: mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
networks:
- data_network
volumes:
[...]
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
deploy:
restart_policy:
window: 300s
networks:
smtp_external_network:
external:
name: [ReplaceForFolderParentNameOfSmtpYmlWithoutSquareBrackets]_smtp
data_network:
driver: bridge
--- (in your other file)
services:
smtp:
restart: always
build: ./postfix
networks:
- smtp
deploy:
restart_policy:
window: 300s
networks:
smpt:
driver: bridge

Resources