test :
touch test
echo $(realpath test)
echo $(shell realpath test)
rm test
When I run make, $(realpath test) returns the empty string, whereas $(shell realpath test) returns the exptected result. Why is this? I've tried using .ONESHELL but it doesn't make a difference.
First, shell realpath and the GNU make realpath function are different. Shell realpath will return a path even if the file doesn't exist:
/home/me$ rm -f blahblah
/home/me$ realpath blahblah
/home/me/blahblah
But, GNU make's realpath will return the empty string if the file doesn't exist.
So, why does the file not exist? Because make will expand all lines of the recipe before running any line of the recipe.
That means that the make functions like $(realpath ...) and $(shell ...) are expanded first, before the first line of the recipe (touch test) is run... and so at the time they are expanded the test file doesn't exist.
In general you never want to use make's $(shell ...) function in a recipe and you can't use make constructs to "interact" with operations that happen inside your recipe. You should use shell functions for that:
test :
touch test;
echo $$(realpath test)
rm test
Related
I'm new to makefile. I'm trying to perform some shell operation inside a makefile under a target. I made a new_target without modifying the working code. The code looks like this:
all: new_target existing_target
new_target:
TEST_FILES:=$(wildcard $(HOME)/Test/*.cpp)
for f in $(TEST_FILES); do \
$(shell ls) $$f; \
done
Error:
TEST_FILES:=/docker_home/myhome/Test/b.cpp /docker_home/myhome/Test/file.cpp /docker_home/myhome/Test/a.cpp
/bin/sh: 1: TEST_FILES:=/docker_home/myhome/Test/b.cpp: not found
Makefile:6: recipe for target 'new_target' failed
make: *** [new_target] Error 127
The idea is to perform a shell operation(similar to ls) on all the .cpp files in a particular directory
This ...
TEST_FILES:=$(wildcard $(HOME)/Test/*.cpp)
... is (GNU) make syntax that assigns a value to a make variable. Your recipe instructs the shell to execute it as if it were a shell command. Obviously, that doesn't work.
Additionally, $(shell ls) doesn't do what you intend. It will run the ls command without arguments in make's working directory, at the time the makefile is parsed, and insert the results into the command to be run. If you want to run a shell command in your recipe then just put the command in the recipe.
The easiest solution would probably be to move that line outside the recipe (and dedent it):
TEST_FILES:=$(wildcard $(HOME)/Test/*.cpp)
new_target:
for f in $(TEST_FILES); do \
ls $$f; \
done
Note that the $(wildcard) function will be evaluated and the results assigned to TEST_FILES at the time that the makefile is parsed, not when the new_target target is built, but that appears unlikely to be an issue in this case.
Of course, unless you need TEST_FILES for something else, too, a much cleaner way would be to merge it together and get rid of wildcard:
new_target:
for f in $(HOME)/Test/*.cpp; do \
ls $$f; \
done
Or, best of all for this particular case:
new_target:
ls $(HOME)/Test/*.cpp
You need to run it in below way as TEST_FILES is a make variable and you should not mix make and shell:
TEST_FILES:=$(wildcard $(HOME)/Test/*.cpp)
new_target:
for f in $(TEST_FILES); do \
ls $$f; \
done
Note :
When it is time to execute recipes to update a target by make , they are executed by invoking a new sub-shell for each line of the recipe, unless the .ONESHELL special target is in effect. So you dont require a $(shell) explicitly.
I have a directory with test inputs and outputs. I wanted make to automatically test my program against this directory after build, for convenience. Thus I needed to somehow force the test target of Makefile to depend on the entire testing directory (it's called good, because it contains valid inputs and outputs for the program)
I read this question and the accepted answer and the comments about deleted files under this answer: Makefile rule that depends on all files under a directory (including within subdirectories) And, incorporating advice from this answer & comments, I came out with this:
my#comp:~/wtfdir$ cat Makefile
test : test.sh $(shell find good)
./test.sh
my#comp:~/wtfdir$
For the sake of MCVE, test.sh is very rudimentary:
my#comp:~/wtfdir$ cat test.sh
echo "blah"
my#comp:~/wtfdir$
However, I noticed, this behaves in a rather unexpected way:
my#comp:~/wtfdir$ ls good
test1 test1.out
my#comp:~/wtfdir$ make
./test.sh
blah
my#comp:~/wtfdir$ touch good/test1
my#comp:~/wtfdir$ make
cp good/test1 good/test1.out
./test.sh
blah
my#comp:~/wtfdir$
Why (expletive redacted) does modifying test1 cause make to overwrite test1.out with test1??? I'm not a big fan of data losses, you know.
What's going on here?
Your Make appears to be GNU Make. Here's why this happens. Your recipe:
test : test.sh $(shell find good)
./test.sh
adds to the prerequisites of test every file and directory that is listed
by find good in the current directory, which happen to be:
good
good/test1
good/test1.out
So to make target test, Make begins by determining if any of the specified
or built-in recipes require it to rebuild any of the prerequsities:
test.sh good good/test1 good/test1.out
Among its built-in recipes it finds:
%.out: %
# recipe to execute (built-in):
#rm -f $#
cp $< $#
as you can verify by running:
$ make --print-data-base | grep -A4 '%.out'
The rule for this recipe is matched by:
good/test1.out: good/test1
and by doing:
$ touch good/test1
you have made good/test1.out out of date with respect to good/test1.
So make executes the recipe:
#rm -f good/test1.out
cp good/test1 good/test1.out
the visible output of which is what you observed:
cp good/test1 good/test1.out
Then it proceeds with the recipe for test:
./test.sh
blah
There is always a risk of such booby-traps if you write a makefile that blindly
generates at runtime some set of preqrequisites or targets you don't know beforehand.
You could avoid this one in particular by explicitly deleting the offending
implicit pattern rule in your makefile by writing:
%.out: %
with no recipe. And you can avoid all possible booby-traps of this sort by disabling all
built-in recipes, with:
$ make --no-builtin-rules ...
but that will require you to write for yourself any builtin-recipes that your
makefile relies on.
The best solution for you is probably to amend your makefile as follows:
PREREQS := $(shell find good)
test : test.sh $(PREREQS)
./test.sh
$(PREREQS): ;
Then the last line explicitly specifies an empty recipe
for each of the $(PREREQS), and Make will not consult any pattern rules for targets
that have explicit recipes.
You should additionally make test a phony target:
.PHONY: test
for the avoidance of the booby-trap where something creates a file called test in the build directory.
I'd like to use $(error ...) to abort my make process if certain preconditions aren't met. The fails_to_work target should abort when failing test -d /foobar.
BAD.mk
all: this_works fails_to_work
this_works:
#echo echo works...
#test -d ~ || echo ~ is not a directory
#test -d /foobar || echo /foobar is not a directory
fails_to_work:
#echo error does not work...
#test -d ~ || $(error ~ is not a directory)
#test -d /foobar || $(error /foobar is not a directory)
$ make -f BAD.mk
echo works...
/foobar is not a directory
BAD.mk:9: *** ~ is not a directory. Stop.
As you can see, not even "error does not work..." is echoed to the screen. The recipe for fails_to_work fails before it gets started. How do I solve this? One of my use cases is#test -d $(MY_ENV_VAR), but I don't think that differs from the hard-coded paths given in the example.
UPDATE (version information)
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for x86_64-pc-linux-gnu
You're trying to get the shell stuff in a recipe to conditionally invoke makefile stuff, which doesn't work, as you've found.
I can think of two options:
Simply remove the $(error) stuff. If test fails, then it will return a non-zero exit status, and the Make process will terminate at that point.
Take the test out of the rule, and use a Make conditional (which in turn invokes shell functionality), e.g.:
ifeq ($(shell test -d /foobar; echo $$?),1)
$(error Not a directory)
endif
Shell commands for a make recipe are effectively stored as a single recursively expanded variable. At the point make decides to run the recipe, it expands the variable, and then runs each line in its own shell invocation. Any $(error ...) that gets expanded will cause make to abort even before invoking the first command.
Note though that the untaken branch of a $(if ...) or $(or ...) &c. will not be expanded. Thus, you could do
.PHONY: rule-with-assert
rule-with-assert:
$(if $(realpath ${should-be-file}/),$(error Assertion failure: ${should-be-file} is a folder!))
⋮
Note that trailing / in the realpath.
Of course macros help to tidy this up a lot.
assert-is-file = $(if $(realpath $1/),$(error Assertion failure: [$1] is a folder!))
.PHONY: rule-with-assert
rule-with-assert:
$(call assert-is-file,${should-be-file})
⋮
It's worth noting again that it doesn't matter where you put the $(call assert-is-file,…) in the recipe.
Any $(error)will be generated as the recipe is expanded,
before any shell commands are run.
Why don't you just use exit 1 shell command instead of $(error ...)? Is there any reason to use the latter?
try_this:
#test -d /foobar || { echo /foobar is not a directory; exit 1; }
or_this:
#if [ ! -d /foobar ]; then echo /foobar is not a directory; exit 1; fi
Both of these will abort the make process unless -k flag is specified.
-k
--keep-going
Continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other prerequisites of these targets can be processed all the same.
Is there a better way to source a script, which sets env vars, from within a makefile?
FLAG ?= 0
ifeq ($(FLAG),0)
export FLAG=1
/bin/myshell -c '<source scripts here> ; $(MAKE) $#'
else
...targets...
endif
Makefile default shell is /bin/sh which does not implement source.
Changing shell to /bin/bash makes it possible:
# Makefile
SHELL := /bin/bash
rule:
source env.sh && YourCommand
To answer the question as asked: you can't.
The basic issue is that a child process can not alter the parent's environment. The shell gets around this by not forking a new process when source'ing, but just running those commands in the current incarnation of the shell. That works fine, but make is not /bin/sh (or whatever shell your script is for) and does not understand that language (aside from the bits they have in common).
Chris Dodd and Foo Bah have addressed one possible workaround, so I'll suggest another (assuming you are running GNU make): post-process the shell script into make compatible text and include the result:
shell-variable-setter.make: shell-varaible-setter.sh
postprocess.py #^
# ...
else
include shell-variable-setter.make
endif
messy details left as an exercise.
If your goal is to merely set environment variables for Make, why not keep it in Makefile syntax and use the include command?
include other_makefile
If you have to invoke the shell script, capture the result in a shell command:
JUST_DO_IT=$(shell source_script)
the shell command should run before the targets. However this won't set the environment variables.
If you want to set environment variables in the build, write a separate shell script that sources your environment variables and calls make. Then, in the makefile, have the targets call the new shell script.
For example, if your original makefile has target a, then you want to do something like this:
# mysetenv.sh
#!/bin/bash
. <script to source>
export FLAG=1
make "$#"
# Makefile
ifeq($(FLAG),0)
export FLAG=1
a:
./mysetenv.sh a
else
a:
.. do it
endif
Using GNU Make 3.81 I can source a shell script from make using:
rule:
<tab>source source_script.sh && build_files.sh
build_files.sh "gets" the environment variables exported by source_script.sh.
Note that using:
rule:
<tab>source source_script.sh
<tab>build_files.sh
will not work. Each line is ran in its own subshell.
This works for me. Substitute env.sh with the name of the file you want to source. It works by sourcing the file in bash and outputting the modified environment, after formatting it, to a file called makeenv which is then sourced by the makefile.
IGNORE := $(shell bash -c "source env.sh; env | sed 's/=/:=/' | sed 's/^/export /' > makeenv")
include makeenv
Some constructs are the same in the shell and in GNU Make.
var=1234
text="Some text"
You can alter your shell script to source the defines. They must all be simple name=value types.
Ie,
[script.sh]
. ./vars.sh
[Makefile]
include vars.sh
Then the shell script and the Makefile can share the same 'source' of information. I found this question because I was looking for a manifest of common syntax that can be used in Gnu Make and shell scripts (I don't care which shell).
Edit: Shells and make understand ${var}. This means you can concatenate, etc,
var="One string"
var=${var} "Second string"
I really like Foo Bah's answer where make calls the script, and the script calls back to make. To expand on that answer I did this:
# Makefile
.DEFAULT_GOAL := all
ifndef SOME_DIR
%:
<tab>. ./setenv.sh $(MAKE) $#
else
all:
<tab>...
clean:
<tab>...
endif
--
# setenv.sh
export SOME_DIR=$PWD/path/to/some/dir
if [ -n "$1" ]; then
# The first argument is set, call back into make.
$1 $2
fi
This has the added advantage of using $(MAKE) in case anyone is using a unique make program, and will also handle any rule specified on the command line, without having to duplicate the name of each rule in the case when SOME_DIR is not defined.
If you want to get the variables into the environment, so that they are passed to child processes, then you can use bash's set -a and set +a. The former means, "When I set a variable, set the corresponding environment variable too." So this works for me:
check:
bash -c "set -a && source .env.test && set +a && cargo test"
That will pass everything in .env.test on to cargo test as environment variables.
Note that this will let you pass an environment on to sub-commands, but it won't let you set Makefile variables (which are different things anyway). If you need the latter, you should try one of the other suggestions here.
My solution to this: (assuming you're have bash, the syntax for $# is different for tcsh for instance)
Have a script sourceThenExec.sh, as such:
#!/bin/bash
source whatever.sh
$#
Then, in your makefile, preface your targets with bash sourceThenExec.sh, for instance:
ExampleTarget:
bash sourceThenExec.sh gcc ExampleTarget.C
You can of course put something like STE=bash sourceThenExec.sh at the top of your makefile and shorten this:
ExampleTarget:
$(STE) gcc ExampleTarget.C
All of this works because sourceThenExec.sh opens a subshell, but then the commands are run in the same subshell.
The downside of this method is that the file gets sourced for each target, which may be undesirable.
Depending on your version of Make and enclosing shell, you can implement a nice solution via eval, cat, and chaining calls with &&:
ENVFILE=envfile
source-via-eval:
#echo "FOO: $${FOO}"
#echo "FOO=AMAZING!" > $(ENVFILE)
#eval `cat $(ENVFILE)` && echo "FOO: $${FOO}"
And a quick test:
> make source-via-eval
FOO:
FOO: AMAZING!
An elegant solution found here:
ifneq (,$(wildcard ./.env))
include .env
export
endif
If you need only a few known variables exporting in makefile can be an option, here is an example of what I am using.
$ grep ID /etc/os-release
ID=ubuntu
ID_LIKE=debian
$ cat Makefile
default: help rule/setup/lsb
source?=.
help:
-${MAKE} --version | head -n1
rule/setup/%:
echo ID=${#F}
rule/setup/lsb: /etc/os-release
${source} $< && export ID && ${MAKE} rule/setup/$${ID}
$ make
make --version | head -n1
GNU Make 3.81
. /etc/os-release && export ID && make rule/setup/${ID}
make[1]: Entering directory `/tmp'
echo ID=ubuntu
ID=ubuntu
--
http://rzr.online.fr/q/gnumake
Assuming GNU make, can be done using a submake. Assuming that the shell script that exports the variables is include.sh in the current directory, move your Makefile to realmake.mk. Create a new Makefile:
all:
#. ./include.sh; \
$(MAKE) -f realmake.mk $(MAKECMDGOALS)
$(MAKECMDGOALS):
+#. ./include.sh; \
$(MAKE) -f realmake.mk $(MAKECMDGOALS)
Pay attention to the ./ preceding include.sh.
Another possible way would be to create a sh script, for example run.sh, source the required scripts and call make inside the script.
#!/bin/sh
source script1
source script2 and so on
make
target: output_source
bash ShellScript_name.sh
try this it will work, the script is inside the current directory.
In my GNUmakefile, I would like to have a rule that uses a temporary directory. For example:
out.tar: TMP := $(shell mktemp -d)
echo hi $(TMP)/hi.txt
tar -C $(TMP) cf $# .
rm -rf $(TMP)
As written, the above rule creates the temporary directory at the time that the rule is parsed. This means that, even I don't make out.tar all the time, many temporary directories get created. I would like to avoid my /tmp being littered with unused temporary directories.
Is there a way to cause the variable to only be defined when the rule is fired, as opposed to whenever it is defined?
My main thought is to dump the mktemp and tar into a shell script but that seems somewhat unsightly.
In your example, the TMP variable is set (and the temporary directory created) whenever the rules for out.tar are evaluated. In order to create the directory only when out.tar is actually fired, you need to move the directory creation down into the steps:
out.tar :
$(eval TMP := $(shell mktemp -d))
#echo hi $(TMP)/hi.txt
tar -C $(TMP) cf $# .
rm -rf $(TMP)
The eval function evaluates a string as if it had been typed into the makefile manually. In this case, it sets the TMP variable to the result of the shell function call.
edit (in response to comments):
To create a unique variable, you could do the following:
out.tar :
$(eval $#_TMP := $(shell mktemp -d))
#echo hi $($#_TMP)/hi.txt
tar -C $($#_TMP) cf $# .
rm -rf $($#_TMP)
This would prepend the name of the target (out.tar, in this case) to the variable, producing a variable with the name out.tar_TMP. Hopefully, that is enough to prevent conflicts.
A relatively easy way of doing this is to write the entire sequence as a shell script.
out.tar:
set -e ;\
TMP=$$(mktemp -d) ;\
echo hi $$TMP/hi.txt ;\
tar -C $$TMP cf $# . ;\
rm -rf $$TMP ;\
I have consolidated some related tips here: Multi-line bash commands in makefile
Another possibility is to use separate lines to set up Make variables when a rule fires.
For example, here is a makefile with two rules. If a rule fires, it creates a temp dir and sets TMP to the temp dir name.
PHONY = ruleA ruleB display
all: ruleA
ruleA: TMP = $(shell mktemp -d testruleA_XXXX)
ruleA: display
ruleB: TMP = $(shell mktemp -d testruleB_XXXX)
ruleB: display
display:
echo ${TMP}
Running the code produces the expected result:
$ ls
Makefile
$ make ruleB
echo testruleB_Y4Ow
testruleB_Y4Ow
$ ls
Makefile testruleB_Y4Ow
I dislike "Don't" answers, but... don't.
make's variables are global and are supposed to be evaluated during makefile's "parsing" stage, not during execution stage.
In this case, as long as the variable local to a single target, follow #nobar's answer and make it a shell variable.
Target-specific variables, too, are considered harmful by other make implementations: kati, Mozilla pymake. Because of them, a target can be built differently depending on if it's built standalone, or as a dependency of a parent target with a target-specific variable. And you won't know which way it was, because you don't know what is already built.