I'm using a Makefile to run various docker-compose commands and I'm trying to capture the output of a script run on my local machine and pass that value to a Docker image.
start-service:
VERSION=$(shell aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$(VERSION) \
connect make run-service
When I run this I can see the variable being assigned but it still errors. Why is the value not getting passed into the -e argument:
VERSION=1.2.3-build342 && \
docker-compose -f ./compose/docker-compose.yml run --rm \
-e VERSION?=$(VERSION) \
connect make run-connect
/bin/sh: VERSION: command not found
You're mixing several different Bourne shell and Make syntaxes here. The Make $$(VERSION) translates to shell $(VERSION), which is command-substitution syntax; GNU Make $(shell ...) generally expands at the wrong time and isn't what you want here.
If you were writing this as an ordinary shell command it would look like
# Set VERSION using $(...) substitution syntax
# Refer to just plain $VERSION
VERSION=$(aws s3 ls s3://redact/downloads/1.2.3/) && ... \
-e VERSION=$VERSION ... \
So when you use this in a Make context, if none of the variables are Make variables (they get set and used in the same command), just double the $ to $$ not escape them.
start-service:
VERSION=$$(aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$VERSION \
connect make run-service
Related
I have a variable that has a command that I want to run.
It has a bunch of double-quotes. when I echo it, it looks beautiful.
I can copy-paste it and run it just fine.
I tried simply $cmd, but it doesn't work. I get an error as if the command is malformed.
I then tried running it via eval "$cmd" or similarly, bash -c "$cmd", which works, but I don't get any output until the command is done running.
Example with bash -c "$cmd":
This runs the command, BUT I don't get any output until the command is done running, which sucks and I'm trying to fix that:
cmd="docker run -v \"$PROJECT_DIR\":\"$PROJECT_DIR\" \
-v \"$PPI_ROOT_DIR/utilities/build_deploy/terraform/modules/\":/ppi_modules \
--workdir \"$PROJECT_DIR/terraform\" \
--env TF_VAR_aws_account_id=$AWS_ACCOUNT_ID \
--env TF_VAR_environment=${ENVIRONMENT} \
--env TF_VAR_region=${AWS_DEFAULT_REGION:-us-west-2} \
${OPTIONAL_AWS_ENV_VARS} \
${CUSTOM_TF_VARS} \
${TERRAFORM_BASE_IMAGE} \
init --plugin-dir=/.terraform/providers \
-reconfigure \
-backend-config=\"bucket=${AWS_ACCOUNT_ID}-tf-remote-state\" \
-backend-config=\"key=${ENVIRONMENT}/${PROJECT_NAME}\" \
-backend-config=\"region=us-west-2\" \
-backend-config=\"dynamodb_table=terraform-locks\" \
-backend=true"
# command output looks good. I can copy and paste it and run it my terminal too.
echo $cmd
# Running the command via bash works,
# but I don't get the output until the command is done running,
# which is what I'm trying to fix:
bash -c "$cmd"
Here is an example using bash array.
It prints it to screen perfectly, but just like running it like $cmd, it throws an error as if the command is malformed:
cmd=(docker run -v \"$PROJECT_DIR\":\"$PROJECT_DIR\" \
-v \"$PPI_ROOT_DIR/utilities/build_deploy/terraform/modules/\":/ppi_modules \
--workdir \"$PROJECT_DIR/terraform\" \
--env TF_VAR_aws_account_id=$AWS_ACCOUNT_ID \
--env TF_VAR_environment=${ENVIRONMENT} \
--env TF_VAR_region=${AWS_DEFAULT_REGION:-us-west-2} \
${OPTIONAL_AWS_ENV_VARS} \
${CUSTOM_TF_VARS} \
${TERRAFORM_BASE_IMAGE} \
init --plugin-dir=/.terraform/providers \
-reconfigure \
-backend-config=\"bucket=${AWS_ACCOUNT_ID}-tf-remote-state\" \
-backend-config=\"key=${ENVIRONMENT}/${PROJECT_NAME}\" \
-backend-config=\"region=us-west-2\" \
-backend-config=\"dynamodb_table=terraform-locks\" \
-backend=true)
echo "${cmd[#]}"
"${cmd[#]}"
How can I execute a bash variable that has double-quotes, but run it so I get the output in realtime, just as if I executed via $cmd (which doesn't work)
Similar to these questions, but my question is to run it AND get the output in realtime:
Execute command containing quotes from shell variable
Bash inserting quotes into string before execution
bash script execute command with double quotes, single quotes and spaces
In your array version, double quotes escaped by a backslash become part of the arguments, which is not intended.
So removing backslashes should fix the issue.
I am trying to run the following command within a script and keep getting the following error
docker: invalid reference format.
I also tried running the cmd without quotes around the env variable names and also without $ and { , same result.
sudo docker run --name "${container_name}-${number}" --cpus="${cpus_per_container}" \
--cpuset-cpus="${cpu_1}","${cpu_2}" \
--restart always -e VIRTUAL_HOST="${virtual_host}" \
-e NODE_ENV="${environment}" -d "${container_registry}:${tag_name}"
Anything that should be changed in the above?
I have a docker image that I want to run locally and to make my life easier I am using make file to pass AWS environment variable.
aws_access_key_id := $(shell aws configure get aws_access_key_id)
aws_secret_access_key := $(shell aws configure get aws_secret_access_key)
aws_region := $(shell aws configure get region)
docker-run:
docker run -e AWS_ACCESS_KEY_ID="$(aws_access_key_id)" -e AWS_SECRET_ACCESS_KEY="$(aws_secret_access_key)" -e AWS_DEFAULT_REGION="$(aws_region)" --rm mydocker-image
And I need to find a way to do something like this in my terminal
make docker-run -d my_db -s dev -t my_table -u my_user -i URI://redshift
make docker-run --pre-actions "delete from dev.my_table where first_name = 'John'" -s dev -t my_table
make docker-run -s3 s3://temp-parquet/avro/ -s dev -t my_table -u myuser -i URI://redshift
These are the arguments that my docker (python application with argparse) will accept.
You can't do that, directly. The command line arguments to make are parsed by make, and must be valid make program command line arguments. Makefiles are not shell scripts and make is not a general interpreter: there's no facility for passing arbitrary options to it.
You can do this by putting them into a variable, like this:
make docker-run DOCKER_ARGS="-d my_db -s dev -t my_table -u my_user -i URI://redshift"
make docker-run DOCKER_ARGS="-d my_db -s dev -t my_table"
then use $(DOCKER_ARGS) in your makefile. But that's the only way.
If you want to do argument parsing yourself, you probably don't want a Makefile! You should probably write a Bash script instead.
Example:
#!/usr/bin/env bash
set -euo pipefail
aws_access_key_id="$(aws configure get aws_access_key_id)"
aws_secret_access_key="$(aws configure get aws_secret_access_key)"
aws_region="$(aws configure get region)"
docker run -e AWS_ACCESS_KEY_ID="$aws_access_key_id" -e AWS_SECRET_ACCESS_KEY="$aws_secret_access_key" -e AWS_DEFAULT_REGION="$aws_region" --rm mydocker-imagedocker "$#"
Note the $# at the end, which passes the arguments from Bash to the docker command.
You might want to try someting like:
$ cat Makefile
all:
#echo make docker-run -d my_db -s dev -t my_table $${MYUSER+-u "$(MYUSER)"} $${URI+-i "URI://$(URI)"}
$ make
make docker-run -d my_db -s dev -t my_table
$ make MYUSER=myuser URI=redshift
make docker-run -d my_db -s dev -t my_table -u myuser -i URI://redshift
I have access to a computing cluster (LSF) and the basic way to send stuff to the compute nodes is by doing:
bsub -I <command>
I had this in a file:
bsub -I ../configure --prefix="..." \
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install
However I just noticed that actually only the first command (configure) was running on the cluster, the remaining two were running locally. What's the best way to group the whole command and pass it to bsub?
Assuming the bsub you are referring to is the one documented here, you have two options:
Surround the entire command to be executed with single quotes (assuming you don't use a single quote anywhere in the command):
bsub -I '../configure --prefix="..."\
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install'
Feed the command to bsub's standard input, using a HERE document to avoid quoting issues:
bsub -I <<END
../configure --prefix="..." \
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install
END
Or, very similar to the second one, put the command into a file and provide the file as input.
bsub -I sh -c '../configure --prefix="..." \
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install'
The following works fine in docker:
docker run -i -t -rm -e a="hello world" b=world ubuntu /bin/bash
What it does is passes env var a with value "hello world" and env var b with value "world" into the docker container.
Thing is, I need to generate that from script.
It is super easy to get this working for env vars without space:
ENV_VARS='-e a=helloworld b=world'
docker run -i -t -rm $ENV_VARS ubuntu /bin/bash
However, once there is a space in the env var I am hosed:
ENV_VARS='-e a="hello world" b=world'
docker run -i -t -rm $ENV_VARS ubuntu /bin/bash
Unable to find image 'world"' (tag: latest) locally
2014/01/15 16:28:40 Invalid repository name (world"), only [a-z0-9-_.] are allowed
How can I get the above example to work? I also tried arrays but can not get them to work.
Bash arrays are designed to solve exactly this sort of problem
First step is to declare the array:
docker_env=(-e "a=hello world" "b=world")
Which lets you programmatically populate more environment variables, for example:
docker_env+=("c=foo bar")
Finally run it:
docker run -i -t -rm "${docker_env[#]}" ubuntu /bin/bash
How about instead:
a='hello world'
b='some more'
docker run -i -t -rm -e a -e b ...
Does this do what you need in an eaiser way?
eval docker run -i -t -rm "$ENV_VARS" ubuntu /bin/bash
I solved problem of the space containing variables passed to docker environment using option --env-file. In the env file the line separated variable definitions are expected. Following example illustrates the pattern.
DOCKER_ENV=$(mktemp run-docker-env.XXXXXXX)
echo Using env file: $DOCKER_ENV
echo VAR1=$VARZZZ > $DOCKER_ENV
echo VAR2=value >> $DOCKER_ENV
echo VAR_WITH_SPACE=variable value with space >> $DOCKER_ENV
docker run --env-file=${DOCKER_ENV} image:latest
rm $DOCKER_ENV