Re-use YAML sections - yaml

I'm putting together a docker-compose file. I'd like to re-use sections that are repetitive. For example, each container re-uses the same deploy config. I tried making a template for it:
...
redis:
image: redis
ports:
- 6379:6379
deploy: deploy_template
volumes:
- /srv/redis/data:/data
deploy_template:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 10
window: 120s
However this didn't work. Is there any way to do this?

You could use the YAML anchor and alias facility for that, effectively:
version: '2'
dummy: &deploy_template
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 10
window: 120s
services:
redis:
image: redis
ports:
- 6379:6379
deploy: *deploy_template
volumes:
- /srv/redis/data:/data
will be parsed as if you had specified:
version: '2'
dummy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 10
window: 120s
services:
redis:
image: redis
ports:
- 6379:6379
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 10
window: 120s
volumes:
- /srv/redis/data:/data
You can have multipe *deploy_template values for a single &deploy_template anchor.
The problem however is that the dummy key, and its value, will trip docker-compose, and at least in version 2 there was no place to put this information.
I therefore preprocess my docker-compose file using ruamel.dcw (I am the author of that package), which allows for a user-data top-level key that will not appear in the output and where you can put such anchor information. Starting with:
version: '2'
user-data:
author: dthree <calvin#hobbes.org>
description: redis container
env-defaults:
NAME: redis # default values if not specified in the environment
PORT: 6379
dummy:
- &deploy_template
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 10
window: 120s
- &some_other_template:
x: null
services:
redis:
image: ${NAME}
ports:
- "${PORT}:${PORT}"
deploy: *deploy_template
volumes:
- /srv/${NAME}/data:/data
this will expand to:
version: '2'
services:
redis:
image: ${NAME}
ports:
- ${PORT}:${PORT}
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 10
window: 120s
volumes:
- /srv/${NAME}/data:/data
before being handed to docker-compose itself (using the -f option). Any variables in the env-defaults "section" that are not already set in the environment in which you execute the preprocessor, will be given their default value, making it easy to override them.
As an aside: you should be careful with:
- 6379:6379
because if the port number gets below 60, the old YAML parser that docker-compose uses, interprets that scalar as a sexagesimal. I tend to always quote such values, especially when using env. variables.

Related

Collecting Containers metrics and Hostsystem metrics with node-exporter?. How can I do this?

My hostmachine is a windows system and I'm running docker desktop. I've running prometheus / node-exporter / cadvisor and grafana in a container.
Currently I get only the metrics of the containers, not from the windows host system.
How is it possible to collect data from host system?
There is a simular question in Stackoverflow but this not work for me, probably it's for a linux host system.
https://stackoverflow.com/questions/66060894/how-to-resolve-prometheus-node-exporter-node-filesystem-device-error-within-do#:~:text=To%20emit%20host%20filesystem%20metrics%20from%20within%20a,so%20it%20knows%20where%20to%20find%20the%20filesystems.
Here is my compose-file:
version: '3'
services:
prometheus:
container_name: Monitoring-Prometheus
image: prometheus
networks:
- monitor-net
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus_db:/var/lib/prometheus
- ./prometheus_db:/prometheus
- ./prometheus_db:/etc/prometheus
- ./alert.rules:/etc/prometheus/alert.rules
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--web.route-prefix=/'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
ports:
- '1840:9090'
restart: unless-stopped
node-exporter:
container_name: Monitoring-Node-Exporter
image: node-exporter
ports:
- '1841:9100'
cadvisor:
container_name: Monitoring-Cadvisor
image: cadvisor
networks:
- monitor-net
ports:
- '1842:8080/tcp'
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
grafana:
container_name: Monitoring-Grafana
image: grafana:latest
networks:
- monitor-net
ports:
- "1843:3000"
volumes:
- ./grafana_db:/var/lib/grafana
depends_on:
- Monitoring-Prometheus
restart: always
reports:
image: skedler
container_name: Monitoring-Reports
privileged: true
cap_add:
- SYS_ADMIN
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:ro
- reportdata:/var/lib/skedler
- ./reporting.yml:/opt/skedler/config/reporting.yml
ports:
- '1844:3001'
networks:
monitor-net:
name: monitoring-network
driver: bridge
volumes:
reportdata:
name: reports-data
driver: local
here my prometheus.yml file:
global:
scrape_interval: 5s
external_labels:
monitor: 'Monitoring'
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['host.docker.internal:1840']
- job_name: 'node-exporter'
static_configs:
- targets: ['host.docker.internal:1841']
- job_name: 'cadvisor'
static_configs:
- targets: ['host.docker.internal:1842']
You'll need to run your exporters directly as Windows processes to get metrics from your host. Otherwise, containers are running in a Linux hypervisor, and that's what you'd be getting metrics from with host.docker.internal references.

Why docker sync files with map folder extremely slow? (Ubuntu)

On my local machine (Ubuntu 18.04, 8GB RAM, i5, HDD) I have two docker-compose files with my laravel project
docker-compose.yml
version: '3.7'
networks:
backend-network:
driver: bridge
frontend-network:
driver: bridge
services:
&app-service app: &app-service-template
container_name: k4fntr_app
build:
context: ./docker/php-fpm
args:
UID: ${UID?Use your user ID}
GID: ${GID?Use your group ID}
USER: ${USER?Use your user name}
user: "${UID}:${GID}"
hostname: *app-service
volumes:
- /etc/passwd/:/etc/passwd:ro
- /etc/group/:/etc/group:ro
- ./:/var/www/k4fntr
environment:
APP_ENV: "${APP_ENV}"
CONTAINER_ROLE: app
FPM_PORT: &php-fpm-port 9000
FPM_USER: "${UID:-1000}"
FPM_GROUP: "${GID:-1000}"
networks:
- backend-network
&queue-service queue:
<<: *app-service-template
container_name: k4fntr_queue
restart: always
hostname: *queue-service
depends_on:
- app
environment:
CONTAINER_ROLE: queue
&schedule-service schedule:
<<: *app-service-template
container_name: k4fntr_schedule
restart: always
hostname: *schedule-service
depends_on:
- app
environment:
CONTAINER_ROLE: scheduler
&sportlevel-listener sportlevel_listener:
<<: *app-service-template
container_name: k4fntr_sl_listener
restart: always
hostname: *sportlevel-listener
ports:
- "${SPORTLEVEL_LISTEN_PORT}:${SPORTLEVEL_LISTEN_PORT}"
depends_on:
- app
environment:
CONTAINER_ROLE: sl_listener
&php-fpm-service php-fpm:
<<: *app-service-template
container_name: k4fntr_php-fpm
user: 'root:root'
restart: always
hostname: *php-fpm-service
ports: [*php-fpm-port]
entrypoint: /fpm-entrypoint.sh
command: php-fpm --nodaemonize
networks:
- backend-network
- frontend-network
echo-server:
container_name: k4fntr_echo
image: oanhnn/laravel-echo-server
volumes:
- ./:/app
environment:
GENERATE_CONFIG: "false"
depends_on:
- app
ports:
- "6001:6001"
networks:
- backend-network
- frontend-network
redis:
container_name: k4fntr_redis
image: redis
restart: always
command: redis-server
volumes:
- ./docker/redis/config/redis.conf:/usr/local/etc/redis/redis.conf
- ./docker/redis/redis-data:/data:rw
ports:
- "16379:6379"
networks:
- backend-network
and docker-compose-dev.yml
version: '3.7'
volumes:
redis-data:
pg-data:
k4fntr_sync:
external: true
services:
&app-service app: &app-service-template
container_name: k4fntr_app
build:
context: ./docker/php-fpm
args:
UID: ${UID?Use your user ID}
GID: ${GID?Use your group ID}
USER: ${USER?Use your user name}
user: "${UID}:${GID}"
hostname: *app-service
volumes:
- /etc/passwd/:/etc/passwd:ro
- /etc/group/:/etc/group:ro
- k4fntr_sync:/var/www/k4fntr:nocopy
environment:
APP_ENV: "${APP_ENV}"
CONTAINER_ROLE: app
FPM_PORT: &php-fpm-port 9000
FPM_USER: "${UID:-1000}"
FPM_GROUP: "${GID:-1000}"
networks:
- backend-network
&php-fpm-service php-fpm:
<<: *app-service-template
container_name: k4fntr_php-fpm
user: 'root:root'
restart: always
hostname: *php-fpm-service
ports: [*php-fpm-port]
entrypoint: /fpm-entrypoint.sh
command: php-fpm --nodaemonize -d "opcache.enable=0" -d "display_startup_errors=On" -d "display_errors=On" -d "error_reporting=E_ALL"
networks:
- backend-network
- frontend-network
mail:
container_name: k4fntr_mail
image: mailhog/mailhog
ports:
- "1025:1025"
- "8025:8025"
networks:
- backend-network
nginx:
container_name: k4fntr_nginx
image: nginx
volumes:
- ./docker/nginx/config/default:/etc/nginx/conf.d
- k4fntr_sync:/var/www/k4fntr:nocopy
depends_on:
- *php-fpm-service
ports:
- "${NGINX_LISTEN_PORT}:80"
networks:
- frontend-network
database:
container_name: k4fntr_database
build: ./docker/postgres
restart: always
environment:
ENV: ${APP_ENV}
TESTING_DB: ${DB_DATABASE_TESTING}
POSTGRES_DB: ${DB_DATABASE}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_PASSWORD: ${DB_PASSWORD}
ports:
- "15432:5432"
volumes:
- ./docker/postgres/prod/:/prod
- ./docker/postgres/pg-data:/var/lib/postgresql/data:rw
networks:
- backend-network
The problem is the fact that when I change some files in my project I have to wait a lot of time. From 15 to 40 seconds. It is impossible for local development. How can I solve this problem?
I learned some information with similar problems with other OS such as Mac or Windows, but I can't found the same problems with Linux.
The problem was that I thought that second file (docker-compose-dev.yml) overrided first file. I mean php-fpm section. If you look at docker-compose-dev you can see that there is the command
command: php-fpm --nodaemonize -d "opcache.enable=0" -d "display_startup_errors=On" -d "display_errors=On" -d "error_reporting=E_ALL"
Actually I used first file (what is very strongely, because I used the command
docker-compose -f docker-compose-dev.yml -f docker-compose.yml up
) and my opcache was cached. This was the main reason why I had to wait so long

How to start CosmosDB emulator with docker-compose?

I've got a docker-compose project in Visual Studio which starts 3 services. One of them use cosmosdb.
I've followed the instructions on https://hub.docker.com/r/microsoft/azure-cosmosdb-emulator/ to start the emulator in a docker container and it worked.
But now I want to get it up and running through docker-compose file. Following is my current configuration.
version: '3.4'
services:
gateway:
environment:
- ASPNETCORE_ENVIRONMENT=Development
image: ${DOCKER_REGISTRY-}gateway
ports:
- "7000:80"
depends_on:
- servicea
- serviceb
build:
context: .\ApiGateways\IAGTO.Fenix.ApiGateway
dockerfile: Dockerfile
servicea:
environment:
- ASPNETCORE_ENVIRONMENT=Development
image: ${DOCKER_REGISTRY-}servicea
depends_on:
- email.db
build:
context: .\Services\ServiceA
dockerfile: Dockerfile
serviceb:
environment:
- ASPNETCORE_ENVIRONMENT=Development
image: ${DOCKER_REGISTRY-}serviceb
build:
context: .\Services\ServiceB
dockerfile: Dockerfile
email.db:
image: microsoft/azure-cosmosdb-emulator
container_name: cosmosdb-emulator
ports:
- "8081:8081"
I can see the container running when I run docker container list
But requests to https://localhost:8081/_explorer/index.html fails.
Any help on this much appreciated
I was in the same situation but the container was started with the following docker-compose.yml and it became accessible.
I can browse https://localhost:8081/_explorer/index.html
version: '3.7'
services:
cosmosdb:
container_name: cosmosdb
image: microsoft/azure-cosmosdb-emulator
tty: true
restart: always
ports:
- "8081:8081"
- "8900:8900"
- "8901:8901"
- "8979:8979"
- "10250:10250"
- "10251:10251"
- "10252:10252"
- "10253:10253"
- "10254:10254"
- "10255:10255"
- "10256:10256"
- "10350:10350"
volumes:
- vol_cosmos:C:\CosmosDB.Emulator\bind-mount
volumes:
vol_cosmos:
Probably I needed to set "tty" or "volumes".
Using the linux cosmos db image, I set it up like this:
version: '3.4'
services:
db:
container_name: cosmosdb
image: "mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator"
tty: true
restart: always
mem_limit: 2G
cpu_count: 2
environment:
- AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
- AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
ports:
- "8081:8081"
- "8900:8900"
- "8901:8901"
- "8979:8979"
- "10250:10250"
- "10251:10251"
- "10252:10252"
- "10253:10253"
- "10254:10254"
- "10255:10255"
- "10256:10256"
- "10350:10350"
volumes:
- vol_cosmos:/data/db
volumes:
vol_cosmos:
Part of the problem is that the emulator takes a while to start, and there is a timeout of 2 minutes before it's just stops waiting.
I'm trying to hack my way through it, but I haven't had much success.
For now the image only works stand alone (via docker run) and that's it.

Use a hostname for a custom service

Is it possible to use use a hostname for a custom service. Currently, I have the following:
Redis service: docker-compose.redis.yml
version: '3.6'
services:
redis:
container_name: ddev-${DDEV_SITENAME}-redis
image: redis:latest
restart: always
ports:
- 6379
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
com.ddev.app-url: $DDEV_URL
environment:
- VIRTUAL_HOST=$DDEV_HOSTNAME
- HTTP_EXPOSE=6379
volumes: []
web:
links:
- redis:$DDEV_HOSTNAME
Redis Commander Service: docker-compose.commander.yml
version: '3.6'
services:
redis:
container_name: ddev-${DDEV_SITENAME}-commander
image: rediscommander/redis-commander:latest
restart: always
ports:
- 8081
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
com.ddev.app-url: $DDEV_URL
environment:
- VIRTUAL_HOST=$DDEV_HOSTNAME
- HTTP_EXPOSE=8081
- REDIS_HOSTS=local:redis:6379
volumes: []
web:
links:
- commander:$DDEV_HOSTNAME
At the moment I can access the Redis Commander from the outside with <project-name>.ddev.local:8081/.
What I want to achieve, if possible is to access the Redis Commander from a custom hostname or subdomain like: comander.<project-name>.ddev.local or commander.local.
After a bit of research and a lot of help from Randy Fay, we were able to accomplish it. We had to run the following:
$ sudo ddev hostname commander.local 127.0.0.1
The Redis Commander Service file(docker-compose.commander.yml) had to be updated to:
version: '3.6'
services:
commander:
container_name: ddev-${DDEV_SITENAME}-commander
image: rediscommander/redis-commander:latest
restart: always
ports:
- 8081
labels:
com.ddev.site-name: ${DDEV_SITENAME}
com.ddev.approot: $DDEV_APPROOT
com.ddev.app-url: $DDEV_URL
environment:
- VIRTUAL_HOST=commander.local
- HTTP_EXPOSE=80
- REDIS_HOSTS=local:redis:6379
volumes: []
web:
links:
- commander:$DDEV_HOSTNAME
- commander:commander.local
for it to work.

Send mail from a container using a postfix container

I'm using an application hosted on a docker container.
This application executes bash scripts / instructions to send mails.
I made another container which executes Postfix as a SMTP Relay.
I want to send mails from my application container by using a bash script using my Postfix container as a relay.
I tried to connect with SSH from my application container to my Postfix container. But that doesn't seem to work.
How can i make it so a script executed in my application container can use my Postfix relay while not allowing anything outside of the docker network, or even better, to only allow some containers, to send mails from this relay.
EDIT 1 : Docker-compose files
Application docker compose :
version: "3.4"
volumes:
[...]
services:
application:
restart: always
build: ./application
depends_on:
- mariadb
container_name: application
volumes:
[...]
ports:
- "80:80"
- "443:443"
- "5669:5669"
deploy:
restart_policy:
window: 300s
links:
- mariadb
external_links:
- smtp-server
mariadb:
restart: always
image: mariadb
command: mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
container_name: application-mariadb
volumes:
[...]
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
deploy:
restart_policy:
window: 300s
Here's my docker compose for my SMTP server :
version: "3.4"
services:
postfix:
restart: always
build: ./postfix
container_name: smtp-server
deploy:
restart_policy:
window: 300s
{a quick response, because I "cicle" in my work ... and I'm taking 10 minutes of clear my mint, I hope it serves you}
Are you using "docker-compose" ?, could you give an example of your YML file? (a little more context)
[you can not connect to by ssh to a container unless you have "supervisor" installed,which I do not recommend at all.]
from what I see, you only need to make private networks; You could use this:
https://docs.docker.com/compose/networking/
to hide everything, I also recommend using a load balancer / Inverse Proxy like TRAEFIK (if they have access to port 80 or 443 in some clear way this ...)
so you only expose 1/2 port(s) (80 + 443 for example) and everything else is protected by your reverse proxy
Watch as I separate the networks as you need the different containers.
bash have access to db and smtp
db does not have access smtp neither nginx
nginx have access to bash
nginx have access to proxy network to expose 80 and 443
no other container is exposed to the outside more than nginx
--
version: "3"
services:
bash:
####### use hostname "smtp" as SMTP server
image: bash
depends_on:
- db
networks:
- smtp_internal_network
- internal_network
- data_network
volumes:
- ../html:/var/www/html
restart: always
db:
image: percona:5.7
# ports: # for debug connections and querys
# - 3306:3306
volumes:
- ../db_data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
networks:
- data_network
restart: always
smtp:
image: mwader/postfix-relay
environment:
- POSTFIX_myhostname=smtp.domain.tld
networks:
- smtp_internal_network
restart: always
nginx:
image: nginx
volumes:
- ../html:/var/www/html
networks:
- external_network
- internal_network
labels:
- "traefik.backend=nginx_${COMPOSE_PROJECT_NAME}"
- "traefik.port=80"
- "traefik.frontend.rule=Host:${FRONTEND_RULE}"
- "traefik.frontend.passHostHeader=true"
- "traefik.enable=true"
- "traefik.docker.network=traefik_proxy"
restart: always
depends_on:
- db
- bash
networks:
external_network:
external:
name: traefik_proxy
internal_network:
driver: bridge
smtp_internal_network:
driver: bridge
data_network:
driver: bridge
Edit:
version: "3"
volumes:
[...]
services:
####### use hostname "smtp" as SMTP server in your application
application:
restart: always
build: ./application
depends_on:
- mariadb
volumes:
[...]
ports:
- "80:80"
- "443:443"
- "5669:5669"
deploy:
restart_policy:
window: 300s
networks:
- smtp_external_network
- data_network
mariadb:
restart: always
image: mariadb
command: mysqld --sql-mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
networks:
- data_network
volumes:
[...]
environment:
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
deploy:
restart_policy:
window: 300s
networks:
smtp_external_network:
external:
name: [ReplaceForFolderParentNameOfSmtpYmlWithoutSquareBrackets]_smtp
data_network:
driver: bridge
--- (in your other file)
services:
smtp:
restart: always
build: ./postfix
networks:
- smtp
deploy:
restart_policy:
window: 300s
networks:
smpt:
driver: bridge

Resources