I have a rule within a Makefile which tries to render some text with env variables, I always get this variable empty whenever I pass it to the $(shell ) command
the following is the Makefile I am trying to pass the env variables with
and I am using sqoop to fetch data from sql server running a query with defined date in where statement
SHELL := /bin/bash
ifndef DATE
override DATE=$(shell date --date="1 day ago" +%Y%m%d)
endif
test:
sqoop --target-dir /tmp/output/ --fields-terminated-by '\t' --split-by id --query '$(shell cat query.sql | env DATE=$(DATE) python -m mypackage.render)'
I run commands like make test DATE=20190728
when I pass the env argument like | env DATE=20190728 python -m mypackage.render then I am able to see the query with where convert(varchar,created_at,112) = '20190728' but when I use env var I always get an empty string with the env variable like where convert(varchar,created_at,112) = ''
The problem is the make $(shell ...) invocation. You probably want command substitution (a shell feature). And kill a useless cat.
test:
$(MAKE) do_test VAR=test
do_test:
VAR=$(VAR) python -m mypackage.render < file.txt
Note that it is not recommended to set SHELL in a Makefile.
Related
I am passing an argument to a Makefile routine, and am based on that determining which argument to pass to a subsequent shell command.
target:
if [ "$(tensorflowgpu)" = "false" ]; then \
conda env create -f temp-environment.yaml \
else \
conda env create -f environment.yaml \
fi
I'm wondering if there is a way to plug the respectively apt .yaml file name directly into the conda create command without using the if else fi syntax. A ternary operator which does not assign its result to a variable seems to be unavailable in shell scripts.
If that really isn't feasible at all, I'm wondering if there's some other way to make this piece of code a little more maintainable and readable, apart from refactoring the conda create command to a function and passing the yaml file to that, maybe.
Disclaimer: I'm not too proficient with shell scripting.
You could do it like this:
target:
[ "$(tensorflowgpu)" = "false" ] && prefix=temp- || prefix= ; \
conda env create -f $${prefix}environment.yaml
I don't know if you like that any better.
Or, since you're using make variables here not shell variables, if you're willing to rely on GNU make, you could use constructed variable names:
YAMLFILE_false = temp-environment.yaml
YAMLFILE_base = environment.yaml
YAMLFILE = $(firstword $(YAMLFILE_$(tensorflowgpu)) $(YAMLFILE_base))
target:
conda env create -f $(YAMLFILE)
Or you could use GNU make's if function; for example:
target:
conda env create -f $(if $(filter false,$(tensorflowgpu)),temp-)environment.yaml
or you could put it in a variable to make it a little cleaner:
YAMLFILE = $(if $(filter false,$(tensorflowgpu)),temp-)environment.yaml
target:
conda env create -f $(YAMLFILE)
I am trying to pass a shell variable from one makefile command to another, but so far have not been successful.
target1:
curl ... ${myvar} ## using myvar from target2
target2:
export myvar=$(shell curl .....);
echo $myvar
In the above case, I am not even getting the output on echo $myvar. Is there something I'm doing wrong?
In a Makefile, every line of a target is run in a separate shell. Additionally, a command can only change the environment for itself and its children. So when you have:
target2:
export myvar=$(shell curl .....);
echo $myvar
And you run make target2, the following happens:
Make starts a shell that runs export myvar=...some value...
The shell exits.
Make runs another shell that runs echo $myvar
That shell exits.
First, there's a syntax problem here: when you write $myvar, this
will be interpreted by make as a request for the value $m followed
by the text yvar. To access shell variables, you need to escape the
$, like this:
echo $$myvar
But that won't solve this problem, because as we see from the above
sequence, the export command happens in a shell process which
immediately exits, so it's effectively invisible to anything else.
This target would work the way you expect if you were to write:
target2:
export myvar=$(shell curl .....); \
echo $$myvar
Here, because we're using the \ to escape the end-of-line, this is
all one long "virtual" line and executes in a single shell process, so
the echo statement will see the variable value set in the previous
statement.
But nothing will make an environment variable set in a shell process
in one target visible in another target, because there's no way to get
these to execute in the same process.
If you need to set variables in your Makefile that are visible
across all targets, set make variables:
myvar = $(shell curl ...)
target1:
curl ... $(myvar)
A workaround, as you have discovered, is to re-execute make as a
child process from within the process that set the environment
variable as in:
target2:
export myvar=$(shell curl .....); \
echo $$myvar; \
$(MAKE) myvar=$$myvar
But often this sort of recursive call to make results in a more
complicated Makefile.
Found the answer. Posting if anyone comes across this. I needed to use $$ instead of $
target1:
curl ... ${myvar} ## using myvar from target2
target2:
export myvar=$(shell curl .....);
echo $$myvar
$(MAKE) myvar=$${myvar} target1
I'm trying to run some commands only if a shell command output is empty, like this:
setup-database:
database=$(shell docker-compose exec mariadb mysql -e "select schema_name from information_schema.schemata where schema_name = 'dbname'" --silent --silent)
ifeq ($(strip $(database)),)
docker-compose exec mariadb mysql -e "create database dbname"
# ...
endif
but it doesn't work. It executes the commands inside the if, regardless the first command's output.
The problem is that you're mixing Make commands and shell commands.
setup-database:
database=$(shell docker-compose some things)
ifeq ($(strip $(database)),)
docker-compose some other things
# ...
endif
That ifeq ... endif is a Make conditional. Make will evaluate it before running any rule; the variable database is empty to begin with, so Make always includes that block of rules in the recipe. (And in fact the make variable database remains empty. The database that gets assigned a value is a shell variable.
Since you want to test the variable when the rule is executed, it should be a shell variable, tested with a shell conditional. On the command line, this would be:
database=`docker-compose some things`
if [ -z $database ]
then
docker-compose some other things
...
fi
(I don't think [ -z STRING ] cares about whitespace, so we don't need $(strip ...).)
Since every command in a recipe runs in a separate subshell, the whole thing must be on one line or the value of the variable will be lost:
database=`docker-compose some things`; if [ -z $database ] ; then docker-compose some other things; ...; fi
And when we put this in a makefile, we use $(shell ...) instead of backticks, and escape the $ (and optionally use some backslashes to make the rule more readable):
setup-database:
database=$(shell docker-compose some things);\
if [ -z $$database ] ; then \
docker-compose some other things; \
... \
fi
Make does not provide direct option to extract values of specific operation into variables.
Two options to consider: - using a make symbol, tracking the result a file.
create make symbol
# Note: nothing get executed.
db_check=$(shell docker-compose exec mariadb mysql -e "select schema_name from information_schema.schemata where schema_name = 'dbname'" --silent --silent)
In setup-database:
# Create shell variable. Useful only on the same line.
database="${db_check}" ; [ "$database" ] || docker-compose exec mariadb mysql -e "create database dbname"
# ...
I'd like to assign the output of a command to a Makefile variable, but the constraint is that I need to preserve the new line characters.
The value is a private key, hence the following code will result in a malformed key:
SHELL=/bin/bash
APP_NAME?=foo
ifdef CI
export DEPS_PRIVATE_KEY=$(shell echo $(CI_BASE64_PRIVATE_KEY) | base64 -d)
endif
build:
#docker build --no-cache --build-arg DEPS_PRIVATE_KEY -t $(APP_NAME) .
.PHONY: build
I realize that I can set the value in bash and then call make build like so:
DEPS_PRIVATE_KEY="$(echo $CI_BASE64_PRIVATE_KEY | base64 -d)" make build
but I was wondering if it's possible to encapsulate this logic in the Makefile.
Any help much appreciated.
Using a make variable as an intermediate for a shell environment variable -- especially when the values are multi-line strings -- is probably uselessly complex. The simplest is thus probably to add the environment variable definition to your recipe:
$ cat Makefile
build:
#export TXT="$$(echo "$$(B64)" | base64 -d)"; \
printenv TXT
$ make B64="$(printf 'a\nb\n' | base64)"
export TXT="$(echo "YQpiCg==" | base64 -d)"; \
printenv TXT
a
b
I'm catching the following error from my GNUmakefile, and I'm trying to understand why /bin/sh is being used:
$ make diff
/bin/sh: 541: Bad file descriptor
rm -f cryptopp563.diff
The recipe is:
.PHONY: diff
diff:
-rm -f cryptopp$(LIB_VER).diff
$(shell svn diff -r 541> cryptopp$(LIB_VER).diff)
I understand where the 541 is coming from. But I don't understand why my shell is being changed from /bin/bash to /bin/sh.
$ printenv | grep -i shell
SHELL=/bin/bash
$ cat GNUmakefile | grep SHELL
SHELL ?= /bin/bash
Where or why is the shell being changed?
The ?= operator only sets the variable if it has no value. If you need bash always, use = or := instead.
Make's SHELL is not the same as your shell's SHELL.
It's documented in Recipes > Execution > Choosing the Shell:
Unlike most variables, the variable SHELL is never set from the
environment. This is because the SHELL environment variable is used
to specify your personal choice of shell program for interactive use.
It would be very bad for personal choices like this to affect the
functioning of makefiles. See Variables from the Environment.