I'm developing a Node application using Docker and docker-compose, which all have awesome ways to import multiple environment variables from a .env file.
Just wondering if there's a shortcut for updating a Heroku deployment's env vars instead of typing them out.
I've noticed there's no --env-file option on the official docs: https://devcenter.heroku.com/articles/config-vars
Using some xargs magic
cat .env.prod | xargs heroku config:set
# Output:
# API_PORT: 3000
# API_KEY_1: ab-cd
# DB_PORT: 6379
# REDISCLOUD_URL: xxx
Happily sets multiple args from a local .env file
Related
When running Github actions on a self hosted runner machine, how do I access existing custom environment variables that have been set on the machine, in my Github action .yaml script?
I have set those variables and restarted the runner virtual machine several times, but they are not accessible using the $VAR syntax in my script.
If you want to set a variable only for one run, you can add an export command when you configure the self-hosted runner on the Github repository, before running the ./run.sh command:
Example (linux) with a TEST variable:
# Create the runner and start the configuration experience
$ ./config.sh --url https://github.com/owner/repo --token ABCDEFG123456
# Add new variable
$ export TEST="MY_VALUE"
# Last step, run it!
$ ./run.sh
That way, you will be able to access the variable by using $TEST, and it will also appear when running env:
job:
runs-on: self-hosted
steps:
- run: env
- run: echo $VAR
If you want to set a variable permanently, you can add a file to the etc/profile.d/<filename>.sh, as suggested by #frennky above, but you will also have to update the shell for it be aware of the new env variables, each time, before running the ./run.sh command:
Example (linux) with a HTTP_PROXY variable:
# Create the runner and start the configuration experience
$ ./config.sh --url https://github.com/owner/repo --token ABCDEFG123456
# Create new profile http_proxy.sh file
$ sudo touch /etc/profile.d/http_proxy.sh
# Update the http_proxy.sh file
$ sudo vi /etc/profile.d/http_proxy.sh
# Add manually new line in the http_proxy.sh file
$ export HTTP_PROXY=http://my.proxy:8080
# Save the changes (:wq)
# Update the shell
$ bash
# Last step, run it!
$ ./run.sh
That way, you will also be able to access the variable by using $HTTP_PROXY, and it will also appear when running env, the same way as above.
job:
runs-on: self-hosted
steps:
- run: env
- run: echo $HTTP_PROXY
- run: |
cd $HOME
pwd
cd ../..
cat etc/profile.d/http_proxy.sh
The etc/profile.d/<filename>.sh will persist, but remember that you will have to update the shell each time you want to start the runner, before executing ./run.sh command. At least that is how it worked with the EC2 instance I used for this test.
Reference
Inside the application directory of the runner, there is a .env file, where you can put all variables for jobs running on this runner instance.
For example
LANG=en_US.UTF-8
TEST_VAR=Test!
Every time .env changes, restart the runner (assuming running as service)
sudo ./svc.sh stop
sudo ./svc.sh start
Test by printing the variable
I would like to use some environment variables in a bash script that contains :
#!/usr/bin/env bash
docker-compose exec apache bash -c "\
printenv | grep BLACKFIRE\
&& blackfire-agent --register --server-id=$BLACKFIRE_SERVER_ID --server-token=$BLACKFIRE_SERVER_TOKEN \
&& /etc/init.d/blackfire-agent restart"
echo "Blackfire agent configured !"
I pass the variable from my .env :
BLACKFIRE_SERVER_ID=xxxxx
BLACKFIRE_SERVER_TOKEN=xxxxx
BLACKFIRE_CLIENT_ID=xxxxx
BLACKFIRE_CLIENT_TOKEN=xxxxx
using docker-compose.yml
environment:
- DAMART_ENV=dev
- BLACKFIRE_SERVER_ID=${BLACKFIRE_SERVER_ID}
- BLACKFIRE_SERVER_TOKEN=${BLACKFIRE_SERVER_TOKEN}
- BLACKFIRE_CLIENT_ID=${BLACKFIRE_CLIENT_ID}
- BLACKFIRE_CLIENT_TOKEN=${BLACKFIRE_CLIENT_TOKEN}
The apache container has the environment variables (here is the result of printenv)
If I change the variables by their values it works but I don't want to use them directly in this script.
How should I call the variables for them to work.
The code repository usually has a function to store secret keys and environment tokens. For instance, github secrets can be kept in https://github.com/user/repo/settings/secrets/actions for your actions or https://github.com/user/repo/settings/secrets/codespaces/new for the repo.
The problem comes from the use of " (double quotes) that made the string to be interpreted not by the container but by the machine running the docker exec. This machine did not have the env variables as they are set in the container.
Replace with ' (simple quotes) make the script to use the environment variables set in the container.
docker-compose exec apache bash -c '\
blackfire-agent --register --server-id=$BLACKFIRE_SERVER_ID --server-token=$BLACKFIRE_SERVER_TOKEN \
&& /etc/init.d/blackfire-agent restart'
I want to set an environment variable in my Dockerfile.
I've got a .env file that looks like this:
FOO=bar.
Inside my Dockerfile, I've got a command that parses the contents of that file and assigns it to FOO.
RUN 'export FOO=$(echo "$(cut -d'=' -f2 <<< $(grep FOO .env))")'
The problem I'm running into is that the script above doesn't return what I need it to. In fact, it doesn't return anything.
When I run docker-compose up --build, it fails with this error.
The command '/bin/sh -c 'export FOO=$(echo "$(cut -d'=' -f2 <<< $(grep FOO .env))")'' returned a non-zero code: 127
I know that the command /bin/sh -c 'echo "$(cut -d'=' -f2 <<< $(grep FOO .env))"' will generate the correct output, but I can't figure out how to assign that output to an environment variable.
Any suggestions on what I'm doing wrong?
Environment Variables
If you want to set a number of environment variables into your docker image (to be used within the containers) you can simply use env_file configuration option in your docker-compose.yml file. With that option, all the entries in the .env file will be set as the environment variables in image and hence into containers.
More Info about env_file
Build ARGS
If your requirement is to use some variables only within your Dockerfile then you specify them as below
ARG FOO
ARG FOO1
ARG FOO2
etc...
And you have to specify these arguments under the build key in your docker-compose.yml
build:
context: .
args:
FOO: BAR
FOO1: BAR1
FOO2: BAR2
More info about args
Accessing .env values within the docker-compose.yml file
If you are looking into passing some values into your docker-compose file from the .env then you can simply put your .env file same location as the docker-compose.yml file and you can set the configuration values as below;
ports:
- "${HOST_PORT}:80"
So, as an example you can set the host port for the service by setting it in your .env file
Please check this
First, the error you're seeing. I suspect there's a "not found" error message not included in the question. If that's the case, then the first issue is that you tried to run an executable that is the full string since you enclosed it in quotes. Rather than trying to run the shell command "export", it is trying to find a binary that is the full string with spaces in it and all. So to work past that error, you'd need to unquote your RUN string:
RUN export FOO=$(echo "$(cut -d'=' -f2 <<< $(grep FOO .env))")
However, that won't solve your underlying problem. The result of a RUN command is that docker saves the changes to the filesystem as a new layer to the image. Only changes to the filesystem are saved. The shell command you are running changes the shell state, but then that shell exits, the run command returns, and the state of that shell, including environment variables, is gone.
To solve this for your application, there are two options I can think of:
Option A: inject build args into your build for all the .env values, and write a script that calls build with the proper --build-arg flag for each variable. Inside the Dockerfile, you'll have two lines for each variable:
ARG FOO1=default value1
ARG FOO2=default value2
ENV FOO1=${FOO1} \
FOO2=${FOO2}
Option B: inject your .env file and process it with an entrypoint in your container. This entrypoint could run your export command before kicking off the actual application. You'll also need to do this for each RUN command during the build where you need these variables. One shorthand I use for pulling in the file contents to environment variables is:
set -a && . .env && set +a
I'd like to print all environment variables set on my Heroku server. How can I do that with command line only?
Ok, I found the way:
heroku config
The heroku run command runs a one-off process inside a Heroku dyno. The unix command that prints environment variables is printenv (manual page). Thus
heroku run -a app-name printenv
is the command you are looking for.
step 1 : list your apps
heroku apps
Copy the name of your app
step 2 : view config variables of this app
heroku config -a acme-web
Append --json to get the output as JSON.
heroku config -a acme-web --json
Append -s to get the output in shell format, to paste directly to a .env file, for example.
heroku config -a your-app -s
Is it possible to get a unique key per slug / release from a running dyno? I was following this article to setup RAILS_CACHE_ID (for expiring etags after deployments) but found that the dynos no longer ship with GIT configured (which causes this error):
fatal: Not a git repository (or any parent up to mount point /app)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
I also considered setting it in a config/initializers to the current Time but obviously that wouldn't work across multiple dynos. Any ideas?
There's a new (Nov 2015) labs feature which does just what you need "Dyno metadata" https://devcenter.heroku.com/changelog-items/768
heroku labs:enable runtime-dyno-metadata -a <app name>
Then on heroku:
~ $ env
HEROKU_APP_ID: 9daa2797-e49b-4624-932f-ec3f9688e3da
HEROKU_APP_NAME: example-app
HEROKU_DYNO_ID: 1vac4117-c29f-4312-521e-ba4d8638c1ac
HEROKU_RELEASE_VERSION: v42
HEROKU_SLUG_COMMIT: 2c3a0b24069af49b3de35b8e8c26765c1dba9ff0
HEROKU_SLUG_DESCRIPTION: Deploy 2c3a0b2
One solution is to use a git pre-push hook to set a heroku config value. Since it's done before the push and slug compilation, the config variable will be available as an ENV var to your rails app.
.git/hooks/pre-push:
#!/bin/sh
remote="$1"
url="$2"
while read local_ref local_sha remote_ref remote_sha
do
if [[ $url =~ heroku ]] ; then
app=$(git remote show -n $remote | sed -n -E -e 's/[[:space:]]+(Push[[:space:]]+URL)(\/|:).+(:|\/)(.*)\.git$/\4/gp')
echo Setting RAILS_CACHE_ID to $local_sha on app $app
heroku config:set RAILS_CACHE_ID=$local_sha --app $app
fi
done
exit 0
The pre-push.sample file has some documentation about the parameters that the hook is called with. I'm using verbose output of git remote to determine which app to set the config value on. The '-E' option for sed is for Mac OS X -- if you are using GNU sed replace that with '-r'.
Another solution is to use the heroku-api via a profile.d script to get the unique release id. This example uses curl to get the latest release id using the RANGE header. It's not the commit reference but it will be unique to every release, including rollbacks and config changes. You'll want to set the API_KEY and the APP_NAME as heroku config variables.
.profile.d/release.sh
# get release id and set as RAILS_CACHE_ID
# Heroku config variables that need to be set
# API_KEY: heroku api key (get from dashboard or `heroku auth:token`
# APP_NAME: set this to your app_name (this could be hardcoded in the profile.d script but
# would make it harder to manage apps with multiple environments
res=$(curl -s -H "Accept: application/vnd.heroku+json; version=3"\
-H "Authorization: Bearer $API_KEY"\
-H "Range: version ..; order=desc, max=1"\
-X GET https://api.heroku.com/apps/$APP_NAME/releases)
release_id=$(ruby -rjson -e "j = JSON.parse('$res'); puts j[0]['id']")
export RAILS_CACHE_ID=$release_id
In the rails app, ENV['RAILS_CACHE_ID'] should now be set to the most recent release id. You could also use this same strategy in a rails initializer.