In my Makefile deploy target I create environment variables and I want to reuse those in the following lines:
SHELL=/bin/sh
deploy:
export $(shell sh this-script-generate-key-values.sh | xargs)
echo ${VAR1} #there is no variable here
echo ${VAR2} #there is no variable here
Where:
this-script-generate-key-values.sh generates this output:
VAR1="somevalue"
VAR2="somevalue"
Why the variables are not set in subsequent lines? How can I make it work?
Notes:
This line works: sh this-script-generate-key-values.sh | xargs
The shell must be /bin/sh (no bash)
All lines in a Makefile recipe run in a separate shell. You need to run the lines in a single shell. Also you need to escape the dollar sign ($) so that variable substitution is not done by make but by the shell.
SHELL=/bin/sh
deploy:
export $$(this-script-generate-key-values.sh | xargs) ;\
echo $${VAR1} ;\
echo $${VAR2}
Just to expand on my comment -- you could output to a file, and use the file to generate your output as so:
vars.txt:
this-script-generate-key-values.sh > $#
deploy : vars.txt
echo VAR1=$$(sed -n 's|VAR1=\(.*\)|\1|p' vars.txt)
echo VAR2=$$(sed -n 's|VAR2=\(.*\)|\1|p' vars.txt)
note: you may have to generate dependencies for vars.txt or declare it .PHONY, otherwise, this will not run on every invocation of make.
If the .ONESHELL special target appears anywhere in the makefile then all recipe lines for each target will be provided to a single invocation of the shell. Newlines between recipe lines will be preserved.
.ONESHELL:
deploy:
export $$(this-script-generate-key-values.sh)
echo $${VAR1}
echo $${VAR2}
Related
I am trying to pass a shell variable from one makefile command to another, but so far have not been successful.
target1:
curl ... ${myvar} ## using myvar from target2
target2:
export myvar=$(shell curl .....);
echo $myvar
In the above case, I am not even getting the output on echo $myvar. Is there something I'm doing wrong?
In a Makefile, every line of a target is run in a separate shell. Additionally, a command can only change the environment for itself and its children. So when you have:
target2:
export myvar=$(shell curl .....);
echo $myvar
And you run make target2, the following happens:
Make starts a shell that runs export myvar=...some value...
The shell exits.
Make runs another shell that runs echo $myvar
That shell exits.
First, there's a syntax problem here: when you write $myvar, this
will be interpreted by make as a request for the value $m followed
by the text yvar. To access shell variables, you need to escape the
$, like this:
echo $$myvar
But that won't solve this problem, because as we see from the above
sequence, the export command happens in a shell process which
immediately exits, so it's effectively invisible to anything else.
This target would work the way you expect if you were to write:
target2:
export myvar=$(shell curl .....); \
echo $$myvar
Here, because we're using the \ to escape the end-of-line, this is
all one long "virtual" line and executes in a single shell process, so
the echo statement will see the variable value set in the previous
statement.
But nothing will make an environment variable set in a shell process
in one target visible in another target, because there's no way to get
these to execute in the same process.
If you need to set variables in your Makefile that are visible
across all targets, set make variables:
myvar = $(shell curl ...)
target1:
curl ... $(myvar)
A workaround, as you have discovered, is to re-execute make as a
child process from within the process that set the environment
variable as in:
target2:
export myvar=$(shell curl .....); \
echo $$myvar; \
$(MAKE) myvar=$$myvar
But often this sort of recursive call to make results in a more
complicated Makefile.
Found the answer. Posting if anyone comes across this. I needed to use $$ instead of $
target1:
curl ... ${myvar} ## using myvar from target2
target2:
export myvar=$(shell curl .....);
echo $$myvar
$(MAKE) myvar=$${myvar} target1
I want to write a Makefile to install go program and other assets, so I want a installdir. I want to check if GOBIN, GOPATH is set, if not, want user to enter a installdir.
I wrote the Makefile as following, but the makefile variable installdir is empty. echo output nothing.
installdir:=$(shell echo $(GOPATH) | cut -d':' -f1)
all: *.go
#GO111MODULE=on GOPATH=$(GOPATH) go build -o trpc main.go
install:
ifeq ($(installdir),)
installdir=$(shell echo $(GOBIN) | cut -d':' -f1)
endif
ifeq ($(installdir),)
installdir=$(shell bash -c 'read -s -p "Please input installdir: " tmpdir; echo $$tmpdir')
endif
echo $(installdir)
Please help!
This code:
install:
ifeq ($(installdir),)
installdir=$(shell echo $(GOBIN) | cut -d':' -f1)
endif
ifeq ($(installdir),)
installdir=$(shell bash -c 'read -s -p "Please input installdir: " tmpdir; echo $$tmpdir')
endif
echo $(installdir)
simply cannot work and represents a fundamental misunderstanding of how make operates.
Make works in two distinct stages: first, it reads and parses all the makefiles, included makefiles, etc. Second, it determines which targets are out of date and runs the recipes to update those targets. Recipes will start a shell and pass the recipe text to that shell. When the shell exits make determines whether it worked or not by looking at the exit code. All make variables and functions in the entire recipe are expanded first, the the shell is invoked on the results. Further, every logical line in the recipe is started in a different shell.
So in your makefile, the ifeq options (which are makefile constructs) are parsed during the first stage, as the makefile is read in. The recipe lines are not run until the second stage, so changes to the installdir variable in a recipe cannot impact the ifeq lines. Further, changes to installdir in a recipe cannot even be seen by make because they happen in a shell, then the shell exits and those changes are lost.
You'll have to write this entire thing in shell syntax and put all of it into a recipe, something like this:
install:
installdir='$(installdir)'; \
[ -n "$$installdir" ] || installdir=$$(echo $(GOBIN) | cut -d':' -f1); \
[ -n "$$installdir" ] || read -s -p "Please input installdir: " installdir; \
echo $$installdir
(untested). You have to use shell constructs, not make constructs. You should virtually never use the $(shell ...) make function inside a recipe: a recipe is already running in a shell. And you have to use backslash/newline pairs to ensure make considers the entire recipe one logical line, else variables set on one line will not be set on the next line.
Finally, I should point out that this (reading input during make) is just generally a bad idea. For example, if you run make install with the -j option, only one recipe can have control of stdin and make will choose more-or-less randomly which it is.
Generally instead you want to have the user pass the value on the command line, with something like:
$ make installdir=my/dir
so your check in the makefile should instead be something like this:
install:
installdir='$(installdir)'; \
[ -n "$$installdir" ] || installdir=$$(echo $(GOBIN) | cut -d':' -f1); \
[ -n "$$installdir" ] || { echo "Please add installdir=... on the command line"; exit 1; }; \
echo $$installdir
I compile my project with:
debug=yes make -j4 or debug=no make -j4
The debug variable changes some compiler flags in the Makefile
Instead of typing this repeatedly in the shell, I wrote this script (lets call it daemon):
#!/bin/bash
inotifywait -q -m -e close_write `ls *.c* *.h*` |
while read; do
make -j4
done
so I just do ./daemon which automatically builds whenever a file is written to.
However, I would like to be able to pass the debug=no make -j4 to the ./daemon script like this:
./daemon debug=no make -j4
So I modified the script:
#!/bin/bash
if [ $# -lt 1 ]; then
echo "Usage `basename $0` [COMMAND]"
exit 1;
fi
inotifywait -q -m -e close_write `ls *.c* *.h*` |
while read; do
"$#"
done
This works with ./daemon make -j4 but when I say daemon debug=no make -j4 I get the following error:
./daemon: line 9: debug=no: command not found
How can I make it so debug=no is recognized as a variable and not a command in the daemon script?
Thanks
The expansion of "$#" is parsed after any pre-command assignments are recognized. All you need to do is ensure that debug=... is in the environment of the command that runs make, which is your daemon script.
debug=no ./daemon make -j4
Variable expansions will only ever become arguments (including the zeroth argument: the command name).
They will never become:
Redirections, so you can't var='> file'; cmd $var
Shell keywords or operators, so you can't var='&'; mydaemon $var
Assignments, including prefix assignments, so you can't var='debug=yes'; $var make as you discovered
Command expansions, loops, process substitutions, &&/||, escape sequences, or anything else.
If you want to do this though, you're in luck: there's a standard POSIX tool that will turn leading key=value pairs into environment variables and run the program you want.
It's called env. Here's an example:
run() {
env "$#"
}
run debug=yes make -j 4
Though TBH I'd use chepner's solution
You always need to put the (env) variable settings at the beginning of the command, i.e. before "daemon".
What is the best way to use a .env in a Makefile, i.e. loading that file and exporting all variables for subshells in make?
It would be great if the proposed solution would work with make only, e.g. not using any third party tools. Also .env files support multiline variables like:
FOO="this\nis\na\nmultiline\nvar"
this is why this solution is probably not adequate.
Make does not offer any way to read a content of the file to some variable. So, I consider it impossible to achieve the result without using external tools. However, if I am wrong, I'd be glad to learn some new trick.
So, let's assume there are two files, .env, being a technically correct shell file:
FOO=bar
BAR="notfoo" # comment
#comment
MULTILINE="This\nis\nSparta!"
# comment
and script.sh:
#!/bin/bash
echo FOO=${FOO}
echo BAR=${BAR}
echo -e ${MULTILINE}
One solution is to include the .env file, then make sure variables are exported:
include .env
$(eval export $(shell sed -ne 's/ *#.*$$//; /./ s/=.*$$// p' .env))
all:
./script.sh
Because of different treatment of quotes by shell and make, you will see the quotes in output.
You can avoid that by reprocessing the variables by make:
include .env
VARS:=$(shell sed -ne 's/ *\#.*$$//; /./ s/=.*$$// p' .env )
$(foreach v,$(VARS),$(eval $(shell echo export $(v)="$($(v))")))
all:
./script.sh
but then the multiline variable will become a one-liner.
Finally, you can generate a temporary file to be processed by bash and source it before any command is run:
SHELL=bash
all: .env-export
. .env-export && ./script.sh
.env-export: .env
sed -ne '/^export / {p;d}; /.*=/ s/^/export / p' .env > .env-export
Oh, new lines got messed in this case in multiline variable. You need to additionally quote them.
Finally, you can add export to .env using above sed command, and do:
SHELL=bash
%: .env-export
. .env-export && make -f secondary "$#"
Found this and it worked great:
at top of makefile
ifneq (,$(wildcard ./.env))
include .env
export
endif
Then you have make variables for all your env, for example MY_VAR use as $(MY_VAR)
You can load specific .env file for each target by creating a function and target to use it with other targets when necessary. Here as an sample:
define setup_env
$(eval ENV_FILE := $(1).env)
#echo " - setup env $(ENV_FILE)"
$(eval include $(1).env)
$(eval export)
endef
devEnv:
$(call setup_env, dev)
prodEnv:
$(call setup_env, prod)
clean:
rm -rf bin/
build: clean
GOOS=linux GOARCH=amd64 go build -o bin/ ./cmd/...
dev: build devEnv
cd cmd/api && ./../../bin/api
migrate-dev-db: devEnv
sh +x database/migration/migrate.sh dev
migrate-prod-db: prodEnv
sh +x database/migration/migrate.sh
deploy: prodEnv
sh +x script/deployment/production/ec2-deploy.sh
Is there a better way to source a script, which sets env vars, from within a makefile?
FLAG ?= 0
ifeq ($(FLAG),0)
export FLAG=1
/bin/myshell -c '<source scripts here> ; $(MAKE) $#'
else
...targets...
endif
Makefile default shell is /bin/sh which does not implement source.
Changing shell to /bin/bash makes it possible:
# Makefile
SHELL := /bin/bash
rule:
source env.sh && YourCommand
To answer the question as asked: you can't.
The basic issue is that a child process can not alter the parent's environment. The shell gets around this by not forking a new process when source'ing, but just running those commands in the current incarnation of the shell. That works fine, but make is not /bin/sh (or whatever shell your script is for) and does not understand that language (aside from the bits they have in common).
Chris Dodd and Foo Bah have addressed one possible workaround, so I'll suggest another (assuming you are running GNU make): post-process the shell script into make compatible text and include the result:
shell-variable-setter.make: shell-varaible-setter.sh
postprocess.py #^
# ...
else
include shell-variable-setter.make
endif
messy details left as an exercise.
If your goal is to merely set environment variables for Make, why not keep it in Makefile syntax and use the include command?
include other_makefile
If you have to invoke the shell script, capture the result in a shell command:
JUST_DO_IT=$(shell source_script)
the shell command should run before the targets. However this won't set the environment variables.
If you want to set environment variables in the build, write a separate shell script that sources your environment variables and calls make. Then, in the makefile, have the targets call the new shell script.
For example, if your original makefile has target a, then you want to do something like this:
# mysetenv.sh
#!/bin/bash
. <script to source>
export FLAG=1
make "$#"
# Makefile
ifeq($(FLAG),0)
export FLAG=1
a:
./mysetenv.sh a
else
a:
.. do it
endif
Using GNU Make 3.81 I can source a shell script from make using:
rule:
<tab>source source_script.sh && build_files.sh
build_files.sh "gets" the environment variables exported by source_script.sh.
Note that using:
rule:
<tab>source source_script.sh
<tab>build_files.sh
will not work. Each line is ran in its own subshell.
This works for me. Substitute env.sh with the name of the file you want to source. It works by sourcing the file in bash and outputting the modified environment, after formatting it, to a file called makeenv which is then sourced by the makefile.
IGNORE := $(shell bash -c "source env.sh; env | sed 's/=/:=/' | sed 's/^/export /' > makeenv")
include makeenv
Some constructs are the same in the shell and in GNU Make.
var=1234
text="Some text"
You can alter your shell script to source the defines. They must all be simple name=value types.
Ie,
[script.sh]
. ./vars.sh
[Makefile]
include vars.sh
Then the shell script and the Makefile can share the same 'source' of information. I found this question because I was looking for a manifest of common syntax that can be used in Gnu Make and shell scripts (I don't care which shell).
Edit: Shells and make understand ${var}. This means you can concatenate, etc,
var="One string"
var=${var} "Second string"
I really like Foo Bah's answer where make calls the script, and the script calls back to make. To expand on that answer I did this:
# Makefile
.DEFAULT_GOAL := all
ifndef SOME_DIR
%:
<tab>. ./setenv.sh $(MAKE) $#
else
all:
<tab>...
clean:
<tab>...
endif
--
# setenv.sh
export SOME_DIR=$PWD/path/to/some/dir
if [ -n "$1" ]; then
# The first argument is set, call back into make.
$1 $2
fi
This has the added advantage of using $(MAKE) in case anyone is using a unique make program, and will also handle any rule specified on the command line, without having to duplicate the name of each rule in the case when SOME_DIR is not defined.
If you want to get the variables into the environment, so that they are passed to child processes, then you can use bash's set -a and set +a. The former means, "When I set a variable, set the corresponding environment variable too." So this works for me:
check:
bash -c "set -a && source .env.test && set +a && cargo test"
That will pass everything in .env.test on to cargo test as environment variables.
Note that this will let you pass an environment on to sub-commands, but it won't let you set Makefile variables (which are different things anyway). If you need the latter, you should try one of the other suggestions here.
My solution to this: (assuming you're have bash, the syntax for $# is different for tcsh for instance)
Have a script sourceThenExec.sh, as such:
#!/bin/bash
source whatever.sh
$#
Then, in your makefile, preface your targets with bash sourceThenExec.sh, for instance:
ExampleTarget:
bash sourceThenExec.sh gcc ExampleTarget.C
You can of course put something like STE=bash sourceThenExec.sh at the top of your makefile and shorten this:
ExampleTarget:
$(STE) gcc ExampleTarget.C
All of this works because sourceThenExec.sh opens a subshell, but then the commands are run in the same subshell.
The downside of this method is that the file gets sourced for each target, which may be undesirable.
Depending on your version of Make and enclosing shell, you can implement a nice solution via eval, cat, and chaining calls with &&:
ENVFILE=envfile
source-via-eval:
#echo "FOO: $${FOO}"
#echo "FOO=AMAZING!" > $(ENVFILE)
#eval `cat $(ENVFILE)` && echo "FOO: $${FOO}"
And a quick test:
> make source-via-eval
FOO:
FOO: AMAZING!
An elegant solution found here:
ifneq (,$(wildcard ./.env))
include .env
export
endif
If you need only a few known variables exporting in makefile can be an option, here is an example of what I am using.
$ grep ID /etc/os-release
ID=ubuntu
ID_LIKE=debian
$ cat Makefile
default: help rule/setup/lsb
source?=.
help:
-${MAKE} --version | head -n1
rule/setup/%:
echo ID=${#F}
rule/setup/lsb: /etc/os-release
${source} $< && export ID && ${MAKE} rule/setup/$${ID}
$ make
make --version | head -n1
GNU Make 3.81
. /etc/os-release && export ID && make rule/setup/${ID}
make[1]: Entering directory `/tmp'
echo ID=ubuntu
ID=ubuntu
--
http://rzr.online.fr/q/gnumake
Assuming GNU make, can be done using a submake. Assuming that the shell script that exports the variables is include.sh in the current directory, move your Makefile to realmake.mk. Create a new Makefile:
all:
#. ./include.sh; \
$(MAKE) -f realmake.mk $(MAKECMDGOALS)
$(MAKECMDGOALS):
+#. ./include.sh; \
$(MAKE) -f realmake.mk $(MAKECMDGOALS)
Pay attention to the ./ preceding include.sh.
Another possible way would be to create a sh script, for example run.sh, source the required scripts and call make inside the script.
#!/bin/sh
source script1
source script2 and so on
make
target: output_source
bash ShellScript_name.sh
try this it will work, the script is inside the current directory.