I have a docker image that I want to run locally and to make my life easier I am using make file to pass AWS environment variable.
aws_access_key_id := $(shell aws configure get aws_access_key_id)
aws_secret_access_key := $(shell aws configure get aws_secret_access_key)
aws_region := $(shell aws configure get region)
docker-run:
docker run -e AWS_ACCESS_KEY_ID="$(aws_access_key_id)" -e AWS_SECRET_ACCESS_KEY="$(aws_secret_access_key)" -e AWS_DEFAULT_REGION="$(aws_region)" --rm mydocker-image
And I need to find a way to do something like this in my terminal
make docker-run -d my_db -s dev -t my_table -u my_user -i URI://redshift
make docker-run --pre-actions "delete from dev.my_table where first_name = 'John'" -s dev -t my_table
make docker-run -s3 s3://temp-parquet/avro/ -s dev -t my_table -u myuser -i URI://redshift
These are the arguments that my docker (python application with argparse) will accept.
You can't do that, directly. The command line arguments to make are parsed by make, and must be valid make program command line arguments. Makefiles are not shell scripts and make is not a general interpreter: there's no facility for passing arbitrary options to it.
You can do this by putting them into a variable, like this:
make docker-run DOCKER_ARGS="-d my_db -s dev -t my_table -u my_user -i URI://redshift"
make docker-run DOCKER_ARGS="-d my_db -s dev -t my_table"
then use $(DOCKER_ARGS) in your makefile. But that's the only way.
If you want to do argument parsing yourself, you probably don't want a Makefile! You should probably write a Bash script instead.
Example:
#!/usr/bin/env bash
set -euo pipefail
aws_access_key_id="$(aws configure get aws_access_key_id)"
aws_secret_access_key="$(aws configure get aws_secret_access_key)"
aws_region="$(aws configure get region)"
docker run -e AWS_ACCESS_KEY_ID="$aws_access_key_id" -e AWS_SECRET_ACCESS_KEY="$aws_secret_access_key" -e AWS_DEFAULT_REGION="$aws_region" --rm mydocker-imagedocker "$#"
Note the $# at the end, which passes the arguments from Bash to the docker command.
You might want to try someting like:
$ cat Makefile
all:
#echo make docker-run -d my_db -s dev -t my_table $${MYUSER+-u "$(MYUSER)"} $${URI+-i "URI://$(URI)"}
$ make
make docker-run -d my_db -s dev -t my_table
$ make MYUSER=myuser URI=redshift
make docker-run -d my_db -s dev -t my_table -u myuser -i URI://redshift
Related
I'm using a Makefile to run various docker-compose commands and I'm trying to capture the output of a script run on my local machine and pass that value to a Docker image.
start-service:
VERSION=$(shell aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$(VERSION) \
connect make run-service
When I run this I can see the variable being assigned but it still errors. Why is the value not getting passed into the -e argument:
VERSION=1.2.3-build342 && \
docker-compose -f ./compose/docker-compose.yml run --rm \
-e VERSION?=$(VERSION) \
connect make run-connect
/bin/sh: VERSION: command not found
You're mixing several different Bourne shell and Make syntaxes here. The Make $$(VERSION) translates to shell $(VERSION), which is command-substitution syntax; GNU Make $(shell ...) generally expands at the wrong time and isn't what you want here.
If you were writing this as an ordinary shell command it would look like
# Set VERSION using $(...) substitution syntax
# Refer to just plain $VERSION
VERSION=$(aws s3 ls s3://redact/downloads/1.2.3/) && ... \
-e VERSION=$VERSION ... \
So when you use this in a Make context, if none of the variables are Make variables (they get set and used in the same command), just double the $ to $$ not escape them.
start-service:
VERSION=$$(aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$VERSION \
connect make run-service
I have a bash file should bring the postgres docker container online and then run a .sql file to create the databases. But it's throwing the error.
psql: error: provision-db.sql: No such file or directory
I have checked the path and the file exists at the same level of this bash script. Following is the content of my bash file.
#!/usr/bin/env bash
docker-compose up -d db
# Ensure the Postgres server is online and usable
until docker exec -i boohoo.postgres pg_isready --host="${POSTGRES_HOST}" --username="${POSTGRES_USER}"
do
echo "."
sleep 1
done
docker exec -i boohoo.postgres psql -h "${POSTGRES_HOST}" -U "${POSTGRES_USER}" -a -q -f provision-db.sql
And this is the provision-db.sql file.
DROP DATABASE "boo-hoo";
CREATE DATABASE "boo-hoo";
GRANT ALL PRIVILEGES ON DATABASE "boo-hoo" TO postgres;
This is the part of docker-compose.yml
version: '3.3'
services:
db:
container_name: boohoo.postgres
hostname: postgres.boohoo
image: postgres
ports:
- "15432:5432"
environment:
POSTGRES_USER: "postgres"
POSTGRES_PASSWORD: "postgres"
The short version
This works
cat provision-db.sql | docker exec -i boohoo.postgres bash -c 'psql -U ${POSTGRES_USER} -w -a -q -f -'
The long version
multiple things here
1) why does following command not find the provision-db.sql?
docker exec -i boohoo.postgres psql -h "${POSTGRES_HOST}" -U "${POSTGRES_USER}" -a -q -f provision-db.sql
because the provision-db.sql is on your host and not in your container. Therefore, when you execute the psql command inside the container it can not find the file
2) Why didn't my first solution work?
cat provision-db.sql | docker exec -i boohoo.postgres psql -h "${POSTGRES_HOST}" -U "${POSTGRES_USER}" -a -q -f - should do the trick asuming provision-db.sql
That is due to the fact, that the variables ${POSTGRES_USER} and ${POSTGRES_PASSWORD} get evaluated on your host machine and I guess they are not set there. In addition, I forgot to specify the -w flag to avoid the password prompt
3) Why does that work?
cat provision-db.sql | docker exec -i boohoo.postgres bash -c 'psql -U ${POSTGRES_USER} -w -a -q -f -'
Well, let's go through it step by step.
First, we print the content of provision-db.sql, which resides on the host machine to stdout and pipe it to the next command via |.
docker-exec executes a command in the container specified (boohoo.postgres). By specifying the -i flag we allow the stdin from your host to go to stdin in the container <- that's important.
In the container, we execute bash -c which is just a wrapper to avoid evaluating the bash variables on the host. We want the variables from the container and by putting it into single quotes we can do that.
docker-exec boohoo.postgres bash -c "echo $POSTGRES_USER"
evaluates the host env variable named POSTGRES_USER.
docker-exec boohoo.postgres bash -c "echo $POSTGRES_USER"
evaluates the container env variable named POSTGRES_USER.
Next we just have to get our postgres command in order.
psql -U ${POSTGRES_USER} -w -a -q -f -
-U specifies the user
-w does not ask for password
-q do it quietly
-f - process whatever you get from stdin
-f is an option for psql and not for docker exec, and psql is running inside the container, so it can only access the file if it is inside the container as well.
In a directory I have a config file with my db variables.
This file (db/database.ini) looks like this:
[PostgreSQL]
host=localhost
database=...
user=postgres
password=...
I have another file (db/create_stmts.sql) where I have all my raw create table statements, and i am trying to experiment the use of a Makefile to have a command like this:
make create-db from_file=db/create_stmts.sql
In order not to repeat myself, I thought of tailing the variables of db/database.ini to a file which I would then source, creating shell variables to pass to psql in the make file.
Here's my plan:
make-db:
# from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=$(PWD)/file.sh
tail -n4 db/database.ini > file.sh && . $(file_path)
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U $(user) -d $(database) -q -f $(from_file) && rm file.sh
Which I run by: make create-db from_file=db/create_stmts.sql
Which gives me this message - from which i kindof understand that the sourcing just did not work.
#from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=/home/gabriele/Desktop/TIUK/companies-house/file.sh
tail -n4 db/database.ini > file.sh && .
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U -d -q -f db/schema_tables.sql && rm file.sh
psql: FATAL: Peer authentication failed for user "-d"
Makefile:3: recipe for target 'create-db' failed
make: *** [create-db] Error 2
Any help?
Another solution, perhaps simpler to understand:
make-db:
file_path=$$PWD/file.sh; \
tail -n4 db/database.ini > file.sh && . $$file_path; \
psql -U $$user -d $$database -q -f $$from_file && rm file.sh
Note using ; and \ to convince make to run all commands in a single shell, and using $$ to escape the $ and use shell variable references.
The error is in the text, namely
psql -U -d -q -f db/schema_tables.sql && rm file.sh
This happens because the variables $(user) and $(database) aren't set. Every line within a target is executed in a sub shell. There is now way to use source like you would in a regular script.
You could create a file named database.mk in which you define these variables and use include database.mk at the top of your makefile to include them:
Makefile
CONFILE ?= database
include $(CONFILE).mk
test:
#echo $(user)
#echo $(database)
database.mk
user := user
database := data
If you want to parse the ini file you could do that as such
CONFILE := db/database.ini
make-db: _setup_con
echo $(user) $(database)
# your target
_setup_con:
$(eval user=$(shell grep "user=" $(CONFILE) | grep -Eo "[^=]*$$"))
$(eval database=$(shell grep "database=" $(CONFILE) | grep -Eo "[^=]*$$"))
# and so forward
I would make it more Make-way by using feature of automatic Makefile generation. Given that a configuration file is a simple properties file, its syntax is easily parseable by Make, it's sufficient to just get the lines with variables, i.e.:
include database.mk
database.mk: db/database.ini
grep -E '^\w+=\w+$$' $< > $#
.PHONY: create-db
create-db: $(from_file)
psql -U $(user) -d $(database) -q -f $<
Some additional notes:
create-db should be made .PHONY to avoid situation when nothing is done due to somebody creating (accidentally or not) a file named create-db,
by making create-db depending on from_file one can get a clean and readable error from make that a file does not exist instead of possibly cryptic error later.
I want to ssh into a host and start a container and run some commands. So the code will be like this:
ssh $host 'screen -L -d -m bash -c "docker run "\
"--network=host -v ~/data:/data myimage:${TAG_NAME}"\
" /bin/bash -c \" some command.... \""'
The question is simple, since I was using single quote, I can't read the ${TAG_NAME}. Is there any way to write this kind of nested quotes and also pass the variable?
You can stop and start your single quotes to include the environment variable, like so:
echo 'foo'"$HOME"'foo'
For your example, the way to include an env var (from your local system) in the command that runs on $host would be:
ssh $host 'screen -L -d -m bash -c "docker run'\
' --network=host -v ~/data:/data myimage:'"$TAG_NAME"\
' /bin/bash -c \" some command.... \""'
I am trying to run a nested ssh -t -t but it won't provide me the environment variables when working with cat and echo.
#!/bin/bash
pass="password\n"
bla="cat <(echo -e '$pass') - | sudo -S su -"
ssh -t -t -t -t jumpserver "ssh -t -t -t -t server \"$bla\" "
I get an output without any variables taken into consideration. (e.g. PS1 does not get shown but commands work fine) The problem is related to cat <(echo -e '$pass') - but this was the way to keep echo alive after providing the password for sudo.
How can i achieve this and get environment variables to get a proper output?
Thanks.
The -tt is enough. Using more -t does not add any more effect and just makes an impression that you have no idea what are you doing.
What is the point of cat <(echo -e) construction? Writing just echo would result in the same, isn't it?
Why to use sudo su? sudo already does all you need, isn't it?
So how can it look in some fashionable manner?
pass="password\n"
bla="echo '$pass' | sudo -Si"
ssh -tt jumpserver "ssh -tt server \"$bla\""
And does it work? Try to debug the commands with -vvv switches to the ssh. It will show you what is actually executed and passed to each other shell.