I wrote a script that creates a local development environment using a docker-compose.yml file.
When running the script, I want to use a yaml linter command to validate that the file is a valid yaml before upping the environment and to do that I'm using the command yamllint.
In this docker-compose.yml file, there is more than one service which "depeneds_on" another service but when I run yamllint, it returns the following error:
47:5 error duplication of key "depends_on" in mapping (key-duplicates)
Which is not a real error, but since the lint is part of the script run then I cannot count on its exit code as it counts this error as an error while in reality, it's not.
An example portion of the docker-compose.yml file:
microservice-one:
image: ms-one:feature-local_development_env
environment:
NODE_ENV: 'development'
NPM_TOKEN: 'SECRET'
ports:
- "3013:3000"
depends_on:
- redis-cluster
microservice-two:
image: ms-two:feature-local_development_env
environment:
NODE_ENV: 'development'
NPM_TOKEN: 'SECRET'
ports:
- "3014:3000"
depends_on:
- redis-cluster
networks:
default:
Is there any other command line yaml linter that you know which will not count more than one "depends_on" as an error?
I found my answer and thought I'll share it with whoever gets here.
So the solution is to override yamllint's default configuration by creating a specific yamllint configuration file.
In my case, the file looks like so:
extends: default
rules:
key-duplicates: disable
Then, I'm running the command like so:
yamllint -d config_file docker-compose.yml
More options can be found in yamllint's official documentation page,
If you need only syntax error and nothing else , below command can be used.
yamllint -d "{rules:{}}"
Related
Actually I have checked some questions like this
What I do not understand is; if I change my docker-compose.yml and add profile to it then should I leave the Dockerfile without profile ?
For example my docker-compose file:
backend:
container_name: backend
image: backend
build: ./backend
restart: always
deploy:
restart_policy:
condition: on-failure
max_attempts: 15
ports:
- '8080:8080'
environment:
- MYSQL_ROOT_PASSWORD=DbPass3008
- MYSQL_PASSWORD=DbPass3008
- MYSQL_USER=DbUser
- MYSQL_DATABASE=db
depends_on:
- mysql
And I will add:
environment:
- "SPRING_PROFILES_ACTIVE=test
As far as I understand I need to put 3 different compose file and run them with -f parameter for different environments like:
docker-compose -f docker-compose-local/test/prod up -d
But my question is that my Dockerfile is already specifying profile as:
FROM openjdk:17-oracle
ADD ./target/backend-0.0.1-SNAPSHOT.jar backend.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar", "-Dspring.profiles.active=TEST", "backend.jar"]
So how should I change this Dockerfile? Even if I create 3-4 different compose file, they are all using same Dockerfile. Should I create different Dockerfiles too (seems ridicilous) but what is the correct way ?
There's no need to add a java -Dspring.profiles.active=... command-line option; Spring will recognize the runtime SPRING_PROFILES_ACTIVE environment variable on its own. That means all of your environments can use the same image (which is generally a good practice).
Compose can also expand host environment variables in some contexts, so you may be able to use a single Compose file with environment-variable references
version: '3.8'
services:
backend:
environment:
- SPRING_PROFILES_ACTIVE=${ENVIRONMENT:-dev}
ENVIRONMENT=test docker-compose up -d
I tend to discourage putting environment-specific settings in a src/main/resources/*.yml file, since it means you need to recompile the application jar file whenever you deploy to a new environment. Another possibility is to set most Spring properties as environment variables, and then use multiple Compose files to include environment-specific settings. The one downside here is that you need multiple docker-compose -f options and you need to repeat them on every docker-compose invocation.
Example
Consider this example docker-compose file with custom .env file:
version: '3'
services:
service_example:
build:
dockerfile: Dockerfile
context: .
args:
AAA: ${AAA}
command: python3 src/service/run.py
env_file:
- custom_env.env
custom_env.env:
AAA=qqq
When I run docker-compose config I get the following output:
WARNING: The AAA variable is not set. Defaulting to a blank string.
services:
service_example:
build:
args:
AAA: '' <----------------------------- ??????
context: /Users/examples
dockerfile: Dockerfile
command: python3 src/service/run.py
environment:
AAA: qqq
version: '3'
Question
Why AAA is unset in build section?
What should I do to set it properly (to the value provided from custom file: AAA=qqq)?
I've also noticed that if I change the env file name to the default setting mv custom_env.env .env and remove env_file section from docker-compose.yml - everything will be just fine:
services:
service_example:
build:
args:
AAA: qqq
context: /Users/examples
dockerfile: Dockerfile
command: python3 src/service/run.py
version: '3'
Quick Answer
docker-compose --env-file custom_env.env config
Answers Explanation
Question-1: Why AAA is unset in build section?
Because the env file specified in env_file property custom_env.env is specific for the Container only, i.e. those variables are to be passed to container while running not during image build.
Question-2: What should I do to set it properly (to the value provided from custom file: AAA=qqq)?
To provide environment variables for build step in docker-compose file using custom env file, we need to specify the custom file path. Like
Syntax: docker-compose --env-file FILE_PATH config
Example: docker-compose --env-file custom_env.env config
Question-3: How .env works?
Because that is the default file for which docker-compose looks for.
Summary
So, In docker-compose for current scenario we can consider 2 stages for specifying env
Build Stage(Image)
Running Stage(Container)
For Build stage - we can use .env default file or we can use --env-file option to specify custom env file
For Running Stage - we can specify environment variables using environment: property or we can use env_file: property to specify a env file
References
https://docs.docker.com/compose/env-file/
https://docs.docker.com/compose/environment-variables/
https://docs.docker.com/compose/compose-file/#env_file
https://docs.docker.com/compose/compose-file/#environment
i don't know how to approach my problem because i don`t find similar cases to have an example.
I want to setup influx with 2 buckets to save telegraf data but only setups with init bucket.
These are the two influxdb services in my docker composer file:
influxdb:
image: influxdb:latest
volumes:
- ./influxdbv2:/root/.influxdbv2
environment:
# Use these same configurations parameters in your telegraf configuration, mytelegraf.conf.
- DOCKER_INFLUXDB_INIT_MODE=setup
- DOCKER_INFLUXDB_INIT_USERNAME=User
- DOCKER_INFLUXDB_INIT_PASSWORD=****
- DOCKER_INFLUXDB_INIT_ORG=org
- DOCKER_INFLUXDB_INIT_BUCKET=data
- DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=****
ports:
- "8086:8086"
influxdb_cli:
image: influxdb:latest
links:
- influxdb
volumes:
# Mount for influxdb data directory and configuration
- ./influxdbv2:/root/.influxdbv2
entrypoint: ["./entrypoint.sh"]
restart: on-failure:10
depends_on:
- influxdb
when inits runs influxdb setup correctly but doesn`t run the script and telegraf returns 404 when trying to write to buckets.
I ran into the same issue today and as far as I am aware you cannot currently initialize two buckets with the DOCKER_INFLUXDB_INIT_BUCKET environment variable.
So I created a shellscript called createSecondBucket.sh that I found in another answer for this question. It uses the influx cli to create a new bucket. The script looks like this:
#!/bin/sh
set -e
influx bucket create -n YOUR_BUCKET_NAME -o YOUR_ORG_NAME -r 0
Note that I had to change the line endings to unix (LF) to run the script without errors.
Inside my Dockerfile I added the following lines:
COPY ./createSecondBucket.sh /docker-entrypoint-initdb.d
RUN chmod +x /docker-entrypoint-initdb.d/createSecondBucket.sh
which have the effect that the script is executed after the container starts for the first time. I found this information on the MongoDB dockerhub page which you can find here under the "Initializing a fresh instance" headline.
I'm facing a strange problem (or better: two different, weird problems) trying to pass build-args to my Dockerfile through docker-compose up.
My files - initial setup
Dockerfile:
ARG NODE_VERSION
FROM node:${NODE_VERSION}
ARG NPM_REGISTRY_TOKEN
RUN echo "=====> token ${NPM_REGISTRY_TOKEN}"
... ... ...
docker-compose.yml:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
With this initial setup in place, I have the following behaviour (on Linux Mint 20, docker-compose version 1.26.2, build eefe0d31):
running docker build --build-arg NPM_REGISTRY_TOKEN=xyz123 produces in output =====> token xyz123: the NPM_REGISTRY_TOKEN arg flows to the Dockerfile
running docker-compose build --build-arg NPM_REGISTRY_TOKEN=xyz123 myservice produces in output =====> token xyz123: the NPM_REGISTRY_TOKEN arg flows to the Dockerfile
running NPM_REGISTRY_TOKEN=xyz123 docker-compose up myservice produces in output =====> token : the NPM_REGISTRY_TOKEN env arg should flow to the Dockerfile due to - NPM_REGISTRY_TOKEN (according to https://docs.docker.com/compose/compose-file/#args: You can omit the value when specifying a build argument, in which case its value at build time is the value in the environment where Compose is running) but it seems to not be available during build
My files - reloaded
Simply changing my docker-compose.yml file to
version: '3'
services:
myservice:
build:
context: ./myservice
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
dockerfile: ../Dockerfile
seems to solve the problem: switching args and dockerfile entries in yml file unlocks the capability to pass environment variables to Dockerfile as build-args through docker-compose up, too. Problem solved. Or not?
Changing OS, getting new problem
So, developers in my team use a bunch of different operating systems: Linux, Mac Os, and Windows, too.
Running the same commands on the same version (1.26.2) of docker-compose on Windows 10 Professional 1909 we're getting the same problem we faced initially, both using the initial version of the docker-compose.yml file and using the version that works on Linux.
We tried passing env var from command line, setting them in the command prompt, setting them as system variables through GUI... we tried launching docker-compose up for git-bash, too, but we're not able to get the variable value in Dockerfile.
I googled a bit aaround but I've not found any reference to known bugs or limitation of the Windows version of docker-compose.
Anyone have any idea what the problem might be? Thank you very much in advance!
So, finally, after some try-and-fail on different OSs and with different configurations, I ended up with an explanation of my problem - and therefore with a viable workaround, which allowed me to reach a satisfactory configuration for my docker-compose-yml file.
Short answer: it wasn't a matter of OSs nor env var passing nor order of context / dockerfile sections - it was a matter of clash between different services in my compose file.
More in detail: my docker-compose.yml file contained an additional service, too, whose job was to initialize the database the application was pointing to:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
depends_on:
- persistence
- db_initializer
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- npm run start:dev'
persistence:
# Setting up the DBMS here
db_initializer:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
depends_on:
- persistence
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- npm run db:migrate'
So, the problem was that I was configuring two services based on the same, self-build image, launching it with different commands (npm run db:migrate for the db_initializer service, npm run start:dev for the application service). Apparently compose took the configuration provided for the first initialized service (db_initializer, because myservice was dependant on it) and used that configuration for both services, ignoring the (different) args section I was providing for the second container: so I was able to solve (this time really!) the problem simply merging services declaration, including all args I needed:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
depends_on:
- persistence
- db_initializer
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- run db:migrate && npm run start:dev'
persistence:
# Setting up the DBMS here
So, after a bunch of months without collecting answers, I think it's time to share my experience, hoping it can help someone encountering this weird behaviour.
I'm using docker-compose to manage a multi container application. 1 of those containers needs access to the contents of a directory on the host.
This seems simple according to the various sources of documentation on docker and docker-compose but I'm struggling to get it working.
event_processor:
environment:
- COMPOSE_CONVERT_WINDOWS_PATHS=1
build: ./Docker/event_processor
ports:
- "15672:15672"
entrypoint: python -u /src/event_processor/event_processor.py
networks:
- app_network
volumes:
- C/path/to/interesting/directory:/interesting_directory"
Running this I get the error message:
ERROR: Named volume
"C/path/to/interesting/directory:/interesting_directory:rw" is used in
service "event_processor" but no declaration was found in the
volumes section.
I understand from the docs that a top level declaration is only necessary if data is to be shared between containers
which isn't the case here.
The docs for docker-compose I linked above have an example which seems to do exactly what I need:
version: "3.2"
services:
web:
image: nginx:alpine
ports:
- "80:80"
volumes:
- type: volume
source: mydata
target: /data
volume:
nocopy: true
- type: bind
source: ./static
target: /opt/app/static
networks:
webnet:
volumes:
mydata:
However when I try, I get errors about the syntax:
ERROR: The Compose file '.\docker-compose.yaml' is invalid because:
services.audio_event_processor.volumes contains an invalid type, it
should be a string
So I tried to play along:
volumes:
- type: "bind"
source: "C/path/to/interesting/directory"
target: "/interesting_directory"
ERROR: The Compose file '.\docker-compose.yaml' is invalid because:
services.audio_event_processor.volumes contains an invalid type, it should be a string
So again the same error.
I tried the following too:
volumes:
- type=bind, source=C/path/to/interesting/directory,destination=/interesting_directory
No error, but attaching to the running container, I see the following two folders;
type=bind, source=C
So it seems that I am able to create a number of volumes with 1 string (though the forward slashes are cutting the string in this case) but I am not mapping it to the host directory.
I've read the docs but I think I'm missing something.
Can someone post an example of mounting a a windows directory from a host to a linux container so that the existing contents of the windows dir is available from the container?
OK so there were multiple issues here:
1.
I had
version: '3'
at the top of my docker-compose.yml. The long syntax described here wasn't implemented until 3.4 so I stopped receiving the bizarre syntax error when I updated this to:
version: '3.6'
2.
I use my my docker account on 2 windows PCs. Following a hint from another stackoverflow post, I reset Docker to the factory settings. I had to give docker the computer username and password with the notice that this was necessary to access the contents of the local filesystem - at this point I remembered doing this on another PC so I'm not sure whether the credentials were correct on this on. With the correct credentials for the current PC, I was able to bind-mount the volume with the expected results as follows:
version: '3.6'
event_processor:
environment:
- COMPOSE_CONVERT_WINDOWS_PATHS=1
build: ./Docker/event_processor
ports:
- "15672:15672"
entrypoint: python -u /src/event_processor/event_processor.py
networks:
- app_network
volumes:
- type: bind
source: c:/path/to/interesting/directory
target: /interesting_directory
Now it works as expected. I'm not sure if it was the factory reset or the updated credentials that fixed it. I'll find out tomorrow when I use another PC and update.