I am trying to run consul, fabio and my microservice called email-api in docker. When I run it, i see that consul registers all my services through registrator and I also added urlprefix- tag to my microservice. For some reason fabio can't pickup my microservice. Here is my docker compose to start with:
---
version: '3'
services:
consul-agent-1:
image: consul:latest
networks:
- consul-demo
command: "agent -retry-join consul-server-bootstrap -client 0.0.0.0"
consul-agent-2:
# Complete copy of 'consul-agent-1'
image: consul:latest
networks:
- consul-demo
command: "agent -retry-join consul-server-bootstrap -client 0.0.0.0"
consul-agent-3:
# Complete copy of 'consul-agent-1'
image: consul:latest
networks:
- consul-demo
command: "agent -retry-join consul-server-bootstrap -client 0.0.0.0"
consul-server-1:
# Uses only image and networks from 'consul-agent-1'
# while 'command' is overwritten
image: consul:latest
networks:
- consul-demo
command: "agent -server -retry-join consul-server-bootstrap -client 0.0.0.0"
consul-server-2:
# Complete copy of 'consul-server-1'
image: consul:latest
networks:
- consul-demo
command: "agent -server -retry-join consul-server-bootstrap -client 0.0.0.0"
consul-server-bootstrap:
# Uses only image and networks from 'consul-agent-1'
# while 'command' is overwritten and 'ports' are defined
image: consul:latest
networks:
- consul-demo
ports:
- "8400:8400"
- "8500:8500"
- "8600:8600"
- "8600:8600/udp"
command: "agent -server -bootstrap-expect 3 -ui -client 0.0.0.0"
fabio:
image: fabiolb/fabio
volumes:
- "/mnt/shared/fabio/fabio.properties:/etc/fabio/fabio.properties"
ports:
- "9999:9999"
- "9998:9998"
container_name: fabio
restart: unless-stopped
networks:
- consul-demo
dns:
- 192.168.16.5
email_api:
image: registry.local/emailapi:latest
container_name: sr-email-api
restart: unless-stopped
environment:
- ASPNETCORE_ENVIRONMENT=dev
#- SERVICE_80_CHECK_HTTP=/
- SERVICE_NAME=email_api
- SERVICE_CHECK_INTERVAL=10s
- SERVICE_CHECK_TIMEOUT=5s
- SERVICE_TAGS=urlprefix-/email-api
ports:
- "80"
networks:
- consul-demo
registrator:
image: gliderlabs/registrator:latest
volumes:
- "/var/run/docker.sock:/tmp/docker.sock"
#network_mode: host
container_name: consul-registrator1
restart: unless-stopped
command: 'consul://consul-server-bootstrap:8500'
networks:
- consul-demo
networks:
consul-demo:
email api is a .net core microservice.
Here is my fabio.properties
registry.consul.addr = 192.168.16.6:8500
I have confirmed that 192.168.16.6 is a one of the consul's server address.
When I query consul catalog services, I see the following output:
agent
consul
consul-8400
consul-8500
consul-8600
email_api-80
fabio
fabio-9998
fabio-9999
So I know for sure that my emailapi is certainly registered with consul service registry
When I visit http://<HOST_NAME>:9998/routes, I see nothing in fabio.
Am I missing something here?
Related
I have created two container laravel webapp (project1 and project2) with own nginx/php-fpm and linked them with traefik container. Each project has its own folder and docker-compose.yaml properly configured with traefik labels.
Thanks to traefik what I expect is that when I visit the project1.laravel.test I look at the contents of project1 and when I visit the project2.laravel.test I look at the content of project2.
The issue is that when I visit project1.laravel.test, alternately, the content of project1 is shown and other times the content of project2 is shown. If I shut down the container of the project2, project 1 works fine. It seems that traefik is configured as a load balancer but I don't understand where is the issue.
How to replicate my issue?
1. git clone https://github.com/gtoto007/traefik-laravel-docker
2. cd traefik-laravel-docker
3. docker-compose -f traefik/docker-compose.yaml up -d
4. docker-compose -f project1/docker-compose.yaml up -d
5. docker-compose -f project1/docker-compose.yaml up -d
in your host file
127.0.0.1 traefik.laravel.test
127.0.0.1 project1.laravel.test
127.0.0.1 project2.laravel.test
MY DOCKER-COMPOSE FILES
Below for simplicity I put the three docker-compose of project1, project2 and traefik:
./traefik/docker-compose.yaml
version: "3.3"
networks:
my-network:
external: true
services:
traefik:
image: "traefik:v2.9"
container_name: "traefik"
networks:
- my-network
command:
- "--api"
- "--providers.docker.exposedbydefault=false"
- "--api.insecure=true"
- "--accesslog.filepath=/data/access.log"
# entrypoints
- "--entrypoints.http.address=:80"
- "--entrypoints.https.address=:443"
- "--entrypoints.traefik.address=:8888"
ports:
- "80:80"
- "443:443"
- "8888:8888"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik.rule=Host(`traefik.laravel.test`)"
- "traefik.http.routers.traefik.entrypoints=traefik"
./project1/docker-compose.yaml
version: '3.8'
networks:
my-network:
external: true
services:
nginx:
build:
dockerfile: docker/nginx/Dockerfile
context: ./
image: my-nginx
volumes:
- ./docker/nginx/conf.d:/etc/nginx/conf.d/
- my-data_project1:/var/www
networks:
- my-network
depends_on:
- php-fpm
labels:
- "traefik.enable=true"
- "traefik.http.routers.project1.rule=Host(`project1.laravel.test`)"
- "traefik.docker.network=my-network"
php-fpm:
build:
dockerfile: docker/php/Dockerfile
context: ./
image: my-php-fpm
ports:
- "5173:5173"
volumes:
- ./docker/php/local.ini:/usr/local/etc/php/conf.d/local.ini
- my-data_project1:/var/www
networks:
- my-network
project1:
build:
dockerfile: docker/project1/Dockerfile
context: ./
image: project1:1.0
volumes:
- my-data_project1:/var/www
networks:
- my-network
volumes:
my-data_project1:
./project2/docker-compose.yaml
version: '3.8'
networks:
my-network:
external: true
services:
nginx:
build:
dockerfile: docker/nginx/Dockerfile
context: ./
image: my-nginx
volumes:
- ./docker/nginx/conf.d:/etc/nginx/conf.d/
- my-data_project2:/var/www
networks:
- my-network
depends_on:
- php-fpm
labels:
- "traefik.enable=true"
- "traefik.http.routers.project2.rule=Host(`project2.laravel.test`)"
- "traefik.docker.network=my-network"
php-fpm:
build:
dockerfile: docker/php/Dockerfile
context: ./
image: my-php-fpm
ports:
- "5174:5173"
volumes:
- ./docker/php/local.ini:/usr/local/etc/php/conf.d/local.ini
- my-data_project2:/var/www
networks:
- my-network
project2:
build:
dockerfile: docker/project2/Dockerfile
context: ./
image: project2:1.0
volumes:
- my-data_project2:/var/www
networks:
- my-network
# - ./:/var/www
volumes:
my-data_project2:
I resolved the issue.
The problem is caused by container name conflict because the projects share the same network and some containers have the same hostname of default.
For example, if you do a ping nginx sometimes you get the ip of the nginx service of project1 and other times you get the ip of the nginx service of project2 because both services use the same hostname.
I fixed it by overwriting the hostnames for each container with unique hostname by Hostname property of docker-compose and corrected the nginx configuration for each project.
You can watch my fixed in this commit :
https://github.com/gtoto007/traefik-laravel-docker/commit/707925465b979168448128b8b307660bde2b5aeb
I have a docker-compose file with Kafka, zookeeper, and spring boot application.
while I run the entire file everything works fine.
when I run it without my spring boot application in order to debug it via intellij It cannot connect to Kafka and doesn't work properly.
my docker-compose file:
version: "3.5" services: # Install Zookeeper. zookeeper:
container_name: zookeeper
image: debezium/zookeeper:1.2
networks:
- mynetwork
ports:
- 2181:2181
- 2888:2888
- 3888:3888 # Install Kafka. kafka:
container_name: kafka
image: debezium/kafka:1.2
depends_on:
- zookeeper
ports:
- 9092:9092
- 29092:29092
networks:
- mynetwork
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
- ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP= INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
- KAFKA_ADVERTISED_LISTENERS= INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
- KAFKA_LISTENERS= EXTERNAL_SAME_HOST://:29092,INTERNAL://:9092
- KAFKA_INTER_BROKER_LISTENER_NAME= PLAINTEXT # Install Postgres. postgres:
container_name: postgres
image: debezium/postgres:12
volumes:
- ./sql/init.sql:/docker-entrypoint-initdb.d/init.sql
ports:
- 5432:5432
networks:
- mynetwork
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres kafka-ui:
container_name: kafka-ui
image: provectuslabs/kafka-ui:0.2.1
ports:
- 8080:8080
networks:
- mynetwork
environment:
- KAFKA_CLUSTERS_0_NAME=local
- KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092 #Deploy a Consumer. consumer:
build:
context: .
container_name: pledge-consumer
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/postgres
ports:
- 8101:8080
networks:
- mynetwork
image: isber/ssm-pledgeservice:v1
depends_on:
- zookeeper
- kafka
- postgres
networks: mynetwork:
external: true
In the application I tried:
spring.kafka.bootstrap-servers=kafka:9092
which works when I run it via docker but not from intellij
I also tried when running with intellij:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.bootstrap-servers=localhost:29092
I found the problem, the image I used:
image: debezium/kafka:1.2
had a problem and it didn't read any of the parameters of the environment I added.
I upgraded to:
image: debezium/kafka:1.4
everything works.
I'm struggling to create a Docker Compose to create a Redis Cluster. I saw that there is a Redis Cluster image from Bitnami, I tried but it my Spring Boot App cannot connect to it due to the below error:
I tried another approach is to create 2 Redis instances master-slave and I can connect to it. Now I'm trying to create 6 Redis Instances and later create a Redis Cluster with 3 master and 3 slaves with the following command:
redis-cli --cluster create 127.0.0.1:6380 127.0.0.1:6381 \
127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384 127.0.0.1:6385 --cluster-replicas 1
But when I executed the command it said that
Could not connect to Redis at 127.0.0.1:6380: Connection refused
Below is my current Docker-compose.yaml:
version: '3.8'
services:
redis-node-0:
image: redis:latest
container_name: redis-0
ports:
- "6380:6379"
command: ["redis-server","--appendonly yes","--cluster-enabled yes","--cluster-node-timeout 5000"]
volumes:
- redis-cluster_data-0:/redis/data
redis-node-1:
image: redis:latest
container_name: redis-1
ports:
- "6381:6379"
command: ["redis-server","--appendonly yes","--cluster-enabled yes","--cluster-node-timeout 5000"]
volumes:
- redis-cluster_data-1:/redis/data
redis-node-2:
image: redis:latest
container_name: redis-2
ports:
- "6382:6379"
command: ["redis-server","--appendonly yes","--cluster-enabled yes","--cluster-node-timeout 5000"]
volumes:
- redis-cluster_data-2:/redis/data
redis-node-3:
image: redis:latest
container_name: redis-3
ports:
- "6383:6379"
command: ["redis-server","--appendonly yes","--cluster-enabled yes","--cluster-node-timeout 5000"]
volumes:
- redis-cluster_data-3:/redis/data
redis-node-4:
image: redis:latest
container_name: redis-4
ports:
- "6384:6379"
command: ["redis-server","--appendonly yes","--cluster-enabled yes","--cluster-node-timeout 5000"]
volumes:
- redis-cluster_data-4:/redis/data
redis-node-5:
image: redis:latest
container_name: redis-5
ports:
- "6385:6379"
command: ["redis-server","--appendonly yes","--cluster-enabled yes","--cluster-node-timeout 5000"]
volumes:
- redis-cluster_data-5:/redis/data
networks:
default:
name: overlay
volumes:
redis-cluster_data-0:
driver: local
redis-cluster_data-1:
driver: local
redis-cluster_data-2:
driver: local
redis-cluster_data-3:
driver: local
redis-cluster_data-4:
driver: local
redis-cluster_data-5:
driver: local
I'm totally new to Both Docker and Redis, I'm learning so any help would be really appreciated. Thanks in advance.
The way to do this is not obvious, because Redis Cluster doesn't easily work with Docker bridge networking. The simplest way to setup a single-node cluster is to cheat and bind it to 127.0.0.1:
version: '3.8'
services:
redis-single-node-cluster:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=yes'
- 'REDIS_CLUSTER_REPLICAS=0'
- 'REDIS_NODES=127.0.0.1 127.0.0.1 127.0.0.1'
- 'REDIS_CLUSTER_CREATOR=yes'
- 'REDIS_CLUSTER_DYNAMIC_IPS=no'
- 'REDIS_CLUSTER_ANNOUNCE_IP=127.0.0.1'
ports:
- '6379:6379'
You can then connect to it with this Spring Boot config:
spring:
redis:
cluster:
nodes: [localhost:6379]
ssl: false
If you want to setup multiple nodes, you'll have to create a custom network and assign static IPs. You'll probably also have to set network_mode: host.
I have a local docker-compose setup in which i am testing some HTTPS requirements. To setup the same, LetsEncrypt Boulder Docker image was used as a local CA. I have tested CertBot with the same and I am able to generate certificates.
In addition, traefik is being used as a reverse proxy system which tries to verify the HTTPS TXT, AAAA, etc on the DNS server.
I need to know if i can use Boulder as a local DNS server and if yes then if there is any documentation on the same.
Also I am currently using DNSMASQ as a local DNS server. is there any way i can update the TXT values and all in DNSMASQ on run time..?
Thanks in advance
Yes, you can. Check this docker-compose.yml file:
version: "3"
networks:
test:
driver: bridge
ipam:
driver: default
config:
- subnet: 10.77.77.0/24
services:
boulder:
# To minimize fetching this should be the same version used below
image: containous/boulder:containous-acmev2
environment:
FAKE_DNS: 10.77.77.1
PKCS11_PROXY_SOCKET: tcp://boulder-hsm:5657
restart: unless-stopped
extra_hosts:
- docker.com:10.77.77.66
- boulder:10.77.77.77
ports:
- 4000:4000 # ACME
- 4001:4001 # ACMEv2
- 4002:4002 # OCSP
- 4003:4003 # OCSP
- 4430:4430 # ACME via HTTPS
- 4431:4431 # ACMEv2 via HTTPS
- 8055:8055 # dns-test-srv updates
depends_on:
- bhsm
- bmysql
networks:
test:
ipv4_address: 10.77.77.77
aliases:
- sa2.boulder
- ca2.boulder
- ra2.boulder
- va2.boulder
- publisher2.boulder
bhsm:
# To minimize fetching this should be the same version used above
image: letsencrypt/boulder-tools:2018-03-07
hostname: boulder-hsm
environment:
PKCS11_DAEMON_SOCKET: tcp://0.0.0.0:5657
command: /usr/local/bin/pkcs11-daemon /usr/lib/softhsm/libsofthsm2.so
expose:
- 5657
networks:
test:
aliases:
- boulder-hsm
bmysql:
image: mariadb:10.1
hostname: boulder-mysql
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
command: mysqld --bind-address=0.0.0.0
logging:
driver: none
networks:
test:
aliases:
- boulder-mysql
proxy:
image: containous/traefik
depends_on:
- boulder
extra_hosts:
- traefik.boulder.com:10.77.77.77
networks:
test:
ipv4_address: 10.77.77.66
ports:
- "0.0.0.0:80:80"
- "5002:80"
- "0.0.0.0:443:443"
- "0.0.0.0:8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik.toml:/traefik.toml
- "./acme/:/acme/:rw"
consul:
image: consul
networks:
- test
command: agent -server -bootstrap -ui -client 0.0.0.0 -log-level debug
ports:
- "8400:8400"
- "0.0.0.0:8500:8500"
- "8600:53/udp"
expose:
- "8300"
- "8301"
- "8301/udp"
- "8302"
- "8302/udp"
whoami:
image: containous/whoami
networks:
- test
labels:
- traefik.enable=true
- traefik.port=80
- traefik.backend=whoami
- traefik.network=test
- traefik.frontend.rule=Host:whoami.docker.com
storeconfig:
image: containous/traefik
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik.toml:/traefik.toml
- "./acme/:/acme/:rw"
command: storeconfig --debug --configfile=/traefik.toml --logLevel="DEBUG"
networks:
- test
I'm using an application hosted on a docker container.
This application executes bash scripts / instructions to send mails.
I made another container which executes Postfix as a SMTP Relay.
I want to send mails from my application container by using a bash script using my Postfix container as a relay.
I tried to connect with SSH from my application container to my Postfix container. But that doesn't seem to work.
How can i make it so a script executed in my application container can use my Postfix relay while not allowing anything outside of the docker network, or even better, to only allow some containers, to send mails from this relay.
EDIT 1 : Docker-compose files
Application docker compose :
version: "3.4"
volumes:
[...]
services:
application:
restart: always
build: ./application
depends_on:
- mariadb
container_name: application
volumes:
[...]
ports:
- "80:80"
- "443:443"
- "5669:5669"
deploy:
restart_policy:
window: 300s
links:
- mariadb
external_links:
- smtp-server
mariadb:
restart: always
image: mariadb
command: mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
container_name: application-mariadb
volumes:
[...]
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
deploy:
restart_policy:
window: 300s
Here's my docker compose for my SMTP server :
version: "3.4"
services:
postfix:
restart: always
build: ./postfix
container_name: smtp-server
deploy:
restart_policy:
window: 300s
{a quick response, because I "cicle" in my work ... and I'm taking 10 minutes of clear my mint, I hope it serves you}
Are you using "docker-compose" ?, could you give an example of your YML file? (a little more context)
[you can not connect to by ssh to a container unless you have "supervisor" installed,which I do not recommend at all.]
from what I see, you only need to make private networks; You could use this:
https://docs.docker.com/compose/networking/
to hide everything, I also recommend using a load balancer / Inverse Proxy like TRAEFIK (if they have access to port 80 or 443 in some clear way this ...)
so you only expose 1/2 port(s) (80 + 443 for example) and everything else is protected by your reverse proxy
Watch as I separate the networks as you need the different containers.
bash have access to db and smtp
db does not have access smtp neither nginx
nginx have access to bash
nginx have access to proxy network to expose 80 and 443
no other container is exposed to the outside more than nginx
--
version: "3"
services:
bash:
####### use hostname "smtp" as SMTP server
image: bash
depends_on:
- db
networks:
- smtp_internal_network
- internal_network
- data_network
volumes:
- ../html:/var/www/html
restart: always
db:
image: percona:5.7
# ports: # for debug connections and querys
# - 3306:3306
volumes:
- ../db_data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
networks:
- data_network
restart: always
smtp:
image: mwader/postfix-relay
environment:
- POSTFIX_myhostname=smtp.domain.tld
networks:
- smtp_internal_network
restart: always
nginx:
image: nginx
volumes:
- ../html:/var/www/html
networks:
- external_network
- internal_network
labels:
- "traefik.backend=nginx_${COMPOSE_PROJECT_NAME}"
- "traefik.port=80"
- "traefik.frontend.rule=Host:${FRONTEND_RULE}"
- "traefik.frontend.passHostHeader=true"
- "traefik.enable=true"
- "traefik.docker.network=traefik_proxy"
restart: always
depends_on:
- db
- bash
networks:
external_network:
external:
name: traefik_proxy
internal_network:
driver: bridge
smtp_internal_network:
driver: bridge
data_network:
driver: bridge
Edit:
version: "3"
volumes:
[...]
services:
####### use hostname "smtp" as SMTP server in your application
application:
restart: always
build: ./application
depends_on:
- mariadb
volumes:
[...]
ports:
- "80:80"
- "443:443"
- "5669:5669"
deploy:
restart_policy:
window: 300s
networks:
- smtp_external_network
- data_network
mariadb:
restart: always
image: mariadb
command: mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
networks:
- data_network
volumes:
[...]
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
deploy:
restart_policy:
window: 300s
networks:
smtp_external_network:
external:
name: [ReplaceForFolderParentNameOfSmtpYmlWithoutSquareBrackets]_smtp
data_network:
driver: bridge
--- (in your other file)
services:
smtp:
restart: always
build: ./postfix
networks:
- smtp
deploy:
restart_policy:
window: 300s
networks:
smpt:
driver: bridge