I am trying to set an environment variable in my docker-compose file and it is not able to be accessed when running the container.
version: "2"
services:
newman_runner:
build:
context: https://github.com/postmanlabs/newman/tree/develop/docker/images/alpine
args:
NEWMAN_VERSION: 4.6.0
image: postman/newman:alpine
environment:
- FOO=BOO
volumes:
- ./postman:/etc/newman
entrypoint: echo ${FOO}
# command:
# run /etc/newman/Google_Civic_Information_API.postman_collection.json
# --env-var API_KEY=${API_KEY}
FOO comes back as Empty and I get the following WARNING after running docker-compose.exe up
WARNING: The FOO variable is not set. Defaulting to a blank string.
I've tried to set FOO as both an array and a dictionary with similar effects.
FOO: BOO vs - FOO=BOO
Any ideas?
I'm on windows at the moment if that makes a difference.
What ended up working was this:
version: "2"
services:
newman_runner:
build:
context: https://github.com/postmanlabs/newman/tree/develop/docker/images/alpine
args:
NEWMAN_VERSION: 4.6.0
image: postman/newman:alpine
environment:
- FOO
From there I can export FOO before running docker-compose
export FOO=BOO
docker-compose up
Related
Example
Consider this example docker-compose file with custom .env file:
version: '3'
services:
service_example:
build:
dockerfile: Dockerfile
context: .
args:
AAA: ${AAA}
command: python3 src/service/run.py
env_file:
- custom_env.env
custom_env.env:
AAA=qqq
When I run docker-compose config I get the following output:
WARNING: The AAA variable is not set. Defaulting to a blank string.
services:
service_example:
build:
args:
AAA: '' <----------------------------- ??????
context: /Users/examples
dockerfile: Dockerfile
command: python3 src/service/run.py
environment:
AAA: qqq
version: '3'
Question
Why AAA is unset in build section?
What should I do to set it properly (to the value provided from custom file: AAA=qqq)?
I've also noticed that if I change the env file name to the default setting mv custom_env.env .env and remove env_file section from docker-compose.yml - everything will be just fine:
services:
service_example:
build:
args:
AAA: qqq
context: /Users/examples
dockerfile: Dockerfile
command: python3 src/service/run.py
version: '3'
Quick Answer
docker-compose --env-file custom_env.env config
Answers Explanation
Question-1: Why AAA is unset in build section?
Because the env file specified in env_file property custom_env.env is specific for the Container only, i.e. those variables are to be passed to container while running not during image build.
Question-2: What should I do to set it properly (to the value provided from custom file: AAA=qqq)?
To provide environment variables for build step in docker-compose file using custom env file, we need to specify the custom file path. Like
Syntax: docker-compose --env-file FILE_PATH config
Example: docker-compose --env-file custom_env.env config
Question-3: How .env works?
Because that is the default file for which docker-compose looks for.
Summary
So, In docker-compose for current scenario we can consider 2 stages for specifying env
Build Stage(Image)
Running Stage(Container)
For Build stage - we can use .env default file or we can use --env-file option to specify custom env file
For Running Stage - we can specify environment variables using environment: property or we can use env_file: property to specify a env file
References
https://docs.docker.com/compose/env-file/
https://docs.docker.com/compose/environment-variables/
https://docs.docker.com/compose/compose-file/#env_file
https://docs.docker.com/compose/compose-file/#environment
I'm dockerizing a bunch of windows apps in Windows Containers.
All my apps require the same mappings, here's a short snippet of my config:
version: '3.9'
services:
shell0:
build:
target: myimage
context: .
image: 'salimfadhley/myimage:latest'
entrypoint: c:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe
working_dir: "c:\\"
volumes:
- type: "bind"
source: "x:"
target: "x:"
volumes: # THIS BIT DOESN'T WORK!
xdrive:
source: "x:"
"xdrive" is a network drive share used by all of my applications. Every single process needs access do "xdrve", that's why I'm bind-mounting this into each service.
I'm doing this by repeating the configuration for every single service in this Docker Compose file. There's going to be quite a few of them. It's going to make my docker-compose file very repetitive.
Is there a way to define the "xdrive" just once, for example in the global "volumes" section? I'd like to be able to do something like this per-service:
...service
volumes:
- xdrive: "x:"
Can it be done? What is the syntax to define a bind-mount globally?
You can solve this with YAML syntax:
version: "3.5"
services:
one:
image: busybox
command: ls /foo
volumes:
- &volume-foo
type: bind
source: .
target: /foo
two:
image: busybox
command: ls /foo
volumes:
- *volume-foo
&volume-foo is an anchor, *volume-foo is an alias. An alias repeats what's been declared after the corresponding anchor, in this case a single object of the array. After parsing it will look like this:
version: "3.5"
services:
one:
image: busybox
command: "ls /foo"
volumes:
-
source: "."
target: /foo
type: bind
two:
image: busybox
command: "ls /foo"
volumes:
-
source: "."
target: /foo
type: bind
I'm facing a strange problem (or better: two different, weird problems) trying to pass build-args to my Dockerfile through docker-compose up.
My files - initial setup
Dockerfile:
ARG NODE_VERSION
FROM node:${NODE_VERSION}
ARG NPM_REGISTRY_TOKEN
RUN echo "=====> token ${NPM_REGISTRY_TOKEN}"
... ... ...
docker-compose.yml:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
With this initial setup in place, I have the following behaviour (on Linux Mint 20, docker-compose version 1.26.2, build eefe0d31):
running docker build --build-arg NPM_REGISTRY_TOKEN=xyz123 produces in output =====> token xyz123: the NPM_REGISTRY_TOKEN arg flows to the Dockerfile
running docker-compose build --build-arg NPM_REGISTRY_TOKEN=xyz123 myservice produces in output =====> token xyz123: the NPM_REGISTRY_TOKEN arg flows to the Dockerfile
running NPM_REGISTRY_TOKEN=xyz123 docker-compose up myservice produces in output =====> token : the NPM_REGISTRY_TOKEN env arg should flow to the Dockerfile due to - NPM_REGISTRY_TOKEN (according to https://docs.docker.com/compose/compose-file/#args: You can omit the value when specifying a build argument, in which case its value at build time is the value in the environment where Compose is running) but it seems to not be available during build
My files - reloaded
Simply changing my docker-compose.yml file to
version: '3'
services:
myservice:
build:
context: ./myservice
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
dockerfile: ../Dockerfile
seems to solve the problem: switching args and dockerfile entries in yml file unlocks the capability to pass environment variables to Dockerfile as build-args through docker-compose up, too. Problem solved. Or not?
Changing OS, getting new problem
So, developers in my team use a bunch of different operating systems: Linux, Mac Os, and Windows, too.
Running the same commands on the same version (1.26.2) of docker-compose on Windows 10 Professional 1909 we're getting the same problem we faced initially, both using the initial version of the docker-compose.yml file and using the version that works on Linux.
We tried passing env var from command line, setting them in the command prompt, setting them as system variables through GUI... we tried launching docker-compose up for git-bash, too, but we're not able to get the variable value in Dockerfile.
I googled a bit aaround but I've not found any reference to known bugs or limitation of the Windows version of docker-compose.
Anyone have any idea what the problem might be? Thank you very much in advance!
So, finally, after some try-and-fail on different OSs and with different configurations, I ended up with an explanation of my problem - and therefore with a viable workaround, which allowed me to reach a satisfactory configuration for my docker-compose-yml file.
Short answer: it wasn't a matter of OSs nor env var passing nor order of context / dockerfile sections - it was a matter of clash between different services in my compose file.
More in detail: my docker-compose.yml file contained an additional service, too, whose job was to initialize the database the application was pointing to:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
depends_on:
- persistence
- db_initializer
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- npm run start:dev'
persistence:
# Setting up the DBMS here
db_initializer:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
depends_on:
- persistence
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- npm run db:migrate'
So, the problem was that I was configuring two services based on the same, self-build image, launching it with different commands (npm run db:migrate for the db_initializer service, npm run start:dev for the application service). Apparently compose took the configuration provided for the first initialized service (db_initializer, because myservice was dependant on it) and used that configuration for both services, ignoring the (different) args section I was providing for the second container: so I was able to solve (this time really!) the problem simply merging services declaration, including all args I needed:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
depends_on:
- persistence
- db_initializer
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- run db:migrate && npm run start:dev'
persistence:
# Setting up the DBMS here
So, after a bunch of months without collecting answers, I think it's time to share my experience, hoping it can help someone encountering this weird behaviour.
I define a variable in an .env file:
LOCAL_IP=127.0.0.1
When using this in the following docker-compose.yml I can see that the variable is expanded to 127.0.0.1 by running docker-compose config.
docker-compose.yml
services:
my-service:
environment:
# This gets expanded to http://127.0.0.1/services
SERVICE_ENDPOINT: http://${LOCAL_IP}/services
Now I'd like to reference the value of LOCAL_IP in another variable in the same file like:
LOCAL_IP=127.0.0.1
SOME_OTHER_VAR=$LOCAL_IP
Updated docker-compose.yml
services:
my-service:
environment:
SERVICE_ENDPOINT: http://${SOME_OTHER_VAR}/services
Running docker-compose config then gives me:
services:
my-service:
environment:
# I'd have expected http://127.0.0.1/services
SERVICE_ENDPOINT: http://$$LOCAL_IP:8090/services
I'm new to Docker so bear with me for any wrong term.
I have Docker Tools installed on Windows 7 and I'm trying to run a Docker compose file of a proprietary existing project stored in a git repository and that has probably been only run on Linux.
These are the commands I ran:
docker-machine start
docker-machine env
#FOR /f "tokens=*" %i IN ('docker-machine env') DO #%i
this was output by step (2)
docker-compose -f <docker-file.yml> up
Most of the Docker work has gone fine (image download, extraction, etc).
It is failing at container start, where some containers run fine - I recognize a working MongoDB instance since its log doesn't report any error - but other containers exit pretty soon with an error code, i.e.:
frontend_1 exited with code 127
Scrolling up a bit the console, I can see lines like:
No such file or directoryr/bin/env: bash
I have no idea where to go from here. I tried launching composer from a CygWin terminal, but got the same result.
Docker Compose file
version: "2"
services:
frontend:
command: "yarn start"
image: company/application/frontend:1
build:
context: frontend
dockerfile: docker/Dockerfile
environment:
<env entries>
ports:
- "3000:3000"
volumes:
- ./frontend:/opt/app
backend:
restart: "no"
# source ~/.bashrc is needed to add the ssh private key, used by git
command: bash -c "source ~/.bashrc && yarn run dev"
image: company/application/backend:1
build:
context: backend
dockerfile: docker/Dockerfile
environment:
<env entries>
ports:
- "4000:4000"
volumes:
- ./backend:/opt/app
- ./:/opt:rw
- ./.ssh/company_utils:/tmp/company_utils
depends_on:
- db
generator-backend:
restart: "no"
# source ~/.bashrc is needed to add the ssh private key, used by git
command: bash -c "source ~/.bashrc && npm run dev"
image: company/generator/backend:1
build:
context: generator-backend
dockerfile: docker/Dockerfile
environment:
<env entries>
ports:
- "5000:5000"
volumes:
- ./generator-backend:/opt/app
- ./:/opt:rw
- ./.ssh/company_utils:/tmp/company_utils
depends_on:
- db
db:
image: mongo:3.4
volumes:
- mongo:/data/db
ports:
- "27017:27017"
volumes:
mongo:
It turned out it was a matter of file line endings, caused by git clone, as pointed out by #mklement0 in his answer to env: bash\r: No such file or directory question.
Disabling core.autocrlf then recloning the repo solved it.