I have the following Makefile, with a rule which checks for dependencies:
#!/usr/bin/make -f
dependencies:
$(info START-info)
#echo "START-echo"
$(call assert-command-present,fastqc)
$(call assert-command-present,cutadapt)
$(call assert-command-present,bowtie2)
$(call assert-command-present,samtools)
$(call assert-command-present,bedtools)
$(call assert-command-present,fetchChromSizes)
$(call assert-command-present,bedGraphToBigWig)
$(info END-info)
#echo "END-echo"
pipeline: dependencies
assert-command-present = $(info Checking for $1) && $(if $(shell which $1),$(info OK.),$(error ERROR: could not find '$1'. Exiting))
The user-defined function assert-command-present checks for a command in the path, and returns an error if it is not found. When I run this Makefile, the echo and info commands are not returned in the order I expect:
START-info
Checking for fastqc
OK.
Checking for cutadapt
OK.
Checking for bowtie2
OK.
Checking for samtools
OK.
Checking for bedtools
OK.
Checking for fetchChromSizes
OK.
Checking for bedGraphToBigWig
OK.
END-info
START-echo
END-echo
The START-echo and START-info commands should run before any assert-command-presents functions run, but the echo command runs after the function calls.
Eugeniu Rosca is correct. More generally, "make" built-in functions are evaluated first, then the entire command sequence is run.
One way to see this is to use the GNU make debugger remake. You can stop at the target "dependencies", and write out the commands that would be run in a shell.
For example:
$ remake -X -f /tmp/Makefile dependencies
GNU Make 4.1+dbg0.91
...
Updating goal targets....
-> (/tmp/Makefile:3)
dependencies:
remake<0> write
START-info
END-info
File "/tmp/dependencies.sh" written.
remake<1>
Look at file /tmp/dependencies.sh and you will see all of the Make functions removed or expanded with whatever value they returned which in my case was empty lines.
Related
I'm trying to adapt the Example from the Official Documentation of GNU make to my Use Case:
GNU make - Example of a Conditional
libs_for_gcc = -lgnu
normal_libs =
ifeq ($(CC),gcc)
libs=$(libs_for_gcc)
else
libs=$(normal_libs)
endif
foo: $(objects)
$(CC) -o foo $(objects) $(libs)
So, I created this Prototype:
libs_for_gcc="gcc libs"
normal_libs="normal libs"
libs=
all: do_nothing
#echo "all: done."
do_nothing:
#echo "do_nothing: done."
example:
#echo "example: go ..."
#echo "example - cc: '$(CC)'"
#echo "libs_for_gcc: $(libs_for_gcc)"
#echo "normal_libs: $(normal_libs)"
#echo "libs: $(libs)"
ifeq ($(CC),gcc)
libs=$(libs_for_gcc) && echo "libs: '$$libs'"
#echo "example - libs: '$(libs)'"
else
libs=$(normal_libs) && echo "libs: '$$libs'"
#echo example - libs: $(libs)
endif
#echo "example - libs: '$(libs)'"
#echo "example: done."
test: libs += " -Ddebug"
test: example foo
bar:
#echo "bar: go ..."
ifeq (${libs}, "")
#echo "bar - libs: empty"
#echo "assigning libs"
libs=$(libs_for_gcc)
else
#echo "bar - libs: not empty"
#echo "bar - libs: '${libs}'"
endif
#echo "bar - libs: '${libs}'"
#echo "bar: done."
foo:
#echo "foo: go ..."
ifneq ("$(libs)", "")
#echo "foo - libs: not empty"
#echo "foo - libs: '$(libs)'"
else
#echo "foo - libs: empty"
endif
#echo "foo: done."
Now when I run the Default Target with $ make
it just produces:
$ make
example: go ...
example - cc: 'cc'
libs_for_gcc: gcc libs
normal_libs: normal libs
libs:
libs="normal libs"
example - libs:
example - libs: ''
example: done.
I see that the value of libs was not changed as intended.
When I run the make bar Target it produces:
$ make bar
bar: go ...
bar - libs: not empty
bar - libs: ''
bar - libs: ''
bar: done.
Here libs is not empty but it has nothing inside.
And when I run the make foo target it produces:
$ make foo
foo: go ...
foo - libs: empty
foo: done.
Here libs is understood as empty
As I see that libs is not changed correctly I tried to change the syntax to:
example:
# [...]
ifeq ($(CC),gcc)
libs := $(libs_for_gcc)
#echo "example - libs: '$(libs)'"
else
libs := $(normal_libs)
#echo example - libs: $(libs)
endif
But then I get the GNU make Error:
$ make
example: go ...
example - cc: 'cc'
libs_for_gcc: gcc libs
normal_libs: normal libs
libs:
libs := "normal libs"
/bin/sh: 1: libs: not found
Makefile:7: recipe for target 'example' failed
make: *** [example] Error 127
I couldn't find any documentation about this behaviour so I appreciate any advise.
Edit:
Added all and test Targets.
Background:
The GNU make Command is an important part of the toolchain for packaging and deploying software and thus important in the daily work of System Administrators and DevOps Engineers.
The Debian and RPM Packaging uses GNU make to package software.
Makefile Driven Packaging
It runs the ./configure && make && make install command sequence.
The Travis CI Workflow uses GNU make for running testsuites.
C Language Automated Testing
It runs the ./configure && make && make test sequence.
All the completely different Use Case are managed by the same Makefile.
Now for my concrete Use Case I'm working on setting up the Travis CI Workflow Sequence to enable Automated Testing for my static linked Source Code Library.
So, contrary to the Packaging Sequence the Automated Testing Sequence requires Debug Features and advanced Output Evaluation to produce a meaningful Test.
I want the Test to check the Error Report and also the Memory Usage Report to alert me of any hidden errors.
Using the Advice about setting the Variable at the Target Declaration Line I was able to change libs for test, example and foo targets.
I also saw the important hint about the Bash Variable libs which is only valid on the same line:
$ make
do_nothing: done.
all: done.
$ make test
example: go ...
example - cc: 'cc'
libs_for_gcc: gcc libs
normal_libs: normal libs
libs: -Ddebug
libs="normal libs" && echo "libs: '$libs'" ;
libs: 'normal libs'
example - libs: -Ddebug
example - libs: ' -Ddebug'
example: done.
foo: go ...
foo - libs: empty
foo - libs: ' -Ddebug'
foo: done.
The Recipe libs=$(libs_for_gcc) && echo "libs: '$$libs'" shows that a new Bash Variable libs was created and it did not affect the GNU make Variable.
Still the Conditional ifneq ($(libs),) cannot detect that libs was already set for the test target.
The difference between your example and the one in the GNU make manual is that in the GNU make manual they are setting a make variable named libs (they are assigning the variable outside of any recipe).
In your usage you are assigning a shell variable named libs (you are assigning it inside the recipe, indented with a TAB).
That's why you get an error when you try to use :=, because that's a make assignment and is not a valid shell assignment.
Then in your echo you print the make variable $(libs) which has not been set at all. Further, every logical line of the recipe is run in inside it's own shell. So you are running the equivalent of:
/bin/sh -c 'libs="gcc libs"'
/bin/sh -c 'echo '
so even if you DID change your echo command to print the shell variable (via $$libs) it would be empty because the previous shell, where it was set, has exited.
You want to use the same syntax as in the example: take the assignment of the variable OUT of the recipe and set the make variable instead:
libs_for_gcc = gcc libs
normal_libs = normal libs
ifeq ($(CC),gcc)
libs = $(libs_for_gcc)
else
libs = $(normal_libs)
endif
example:
#echo "example: go ..."
#echo "example - cc: '$(CC)'"
#echo "libs_for_gcc: $(libs_for_gcc)"
#echo "normal_libs: $(normal_libs)"
#echo "libs: $(libs)"
#echo "example - libs: '$(libs)'"
#echo "example: done."
Finally I achieved to get my Makefile make test Target working.
make test Test Suite with GNU make
The key was to understand that there are 3 different contexts where a GNU make Variable can be assigned.
And that GNU make often needs helper functions because of some strange issues with Conditionals
These issues are mostly undocumented and only can be solved by Try and Error and searching on StackOverflow.com
1. The first, easiest and best documented context to assign a Variable is in Global Context as seen in the official documentation and the first answer.
GNU make - Example of a Conditional
ifeq ($(CC),gcc)
libs=$(libs_for_gcc)
else
libs=$(normal_libs)
endif
The Global Variable ${libs} has global visibility.
2. The next context which is not that obvious but still documented in the Official Documentation is in the Target Definition Line
GNU make - Target-specific Variable Values
test: libs += " -Ddebug"
test: example foo
Here ${libs} is valid for the test Target and all dependent Targets that it calls.
But the only Condition is the Target itself.
3. To assign dynamically a Variable in the context of a Target is to use the $(eval ) Function in combination with the $(shell ) Function as seen in this StackOverflow.com Post:
Assign a Variable as Result of a Command
test: examples_debug
# [...]
$(eval error := $(shell cat ./demo_error.log))
ifeq ($(strip $(error)),)
#$(ECHO) "NO Error: '${error}'"
else
#$(ECHO) "Error detected: '${error}'"
$(error "Demo failed with Error [Code: '${error_code}']")
#exit 1
endif
Here ${error} is read from a File. Additionally the $(strip ) Function is required to be able to check if it is empty, which is some of the undocumented issues that GNU make has and are wierd to Bash Developers.
4. Another method that works but does not use a Makefile Variable and is somewhat bulky is evaluating the Variable entirely in Bash in 1 single Line which can be found at: Check Variable with Bash Command Recipe and was also hinted by the previous Answer.
Which would look like:
test: test_debug
# [...]
leak=`cat ./test_heap.log | grep -i "unfreed memory blocks" | awk '{print $$1}'` ; if [ $$leak -gt 1 ]; then echo "Memory Leaks: $$leak" && exit 1; else echo "Memory Leaks: NONE"; fi ;
Here $$leak is a Bash Variable and only valid within the same Line.
An approach that is somewhat similar to the GitHub Workflow Command Logic. (Actually is was directly ported from the GitHub Workflow for the same Project)
As about Evaluation of Conditionals there are many undocumented issues in GNU make that require Bash Workarounds to achieve the goal.
As documented at:
Preprocessing Numerical Values for the Makefile Evaluation
There are issues with comparing Numbers and it is impossible to compare against 0 which is extremely important Exit Code for Command Line Applications.
So the Workaround looked somewhat like:
test: test_debug
# Run the Testsuite Application
# [...]
$(eval error_code := $(shell export HEAPTRC="log=test_heap.log" ; echo -n "" >./test_heap.log ; ${wrkdir}/tests_hash-lists.dbg.run 2>./tests_error.log 1>./tests_exec.log ; echo "$$?"))
#$(ECHO) Application Tests: Execution finished with [${error_code}]
#Application Tests Execution Report
# [...]
$(eval is_error := $(shell if [ "${error_code}" = "0" ]; then echo false ; else echo true ; fi ;))
ifeq (${is_error}, true)
#$(ECHO) Exit Code non cero: '${error_code}'
#$(ECHO) "Tests failed with Error [Code: '${error_code}']"
#exit ${error_code}
endif
In this use case ${error_code} is tested with a Bash Conditional to populate the Helper Variable ${is_error} with true or false which then can be checked in GNU make.
Discussion:
The test Target cannot just exit on error.
For troubleshooting a failed Automated Test it is crucial to see the Exit Code, the Error Message and the Heap Report.
I have a directory with test inputs and outputs. I wanted make to automatically test my program against this directory after build, for convenience. Thus I needed to somehow force the test target of Makefile to depend on the entire testing directory (it's called good, because it contains valid inputs and outputs for the program)
I read this question and the accepted answer and the comments about deleted files under this answer: Makefile rule that depends on all files under a directory (including within subdirectories) And, incorporating advice from this answer & comments, I came out with this:
my#comp:~/wtfdir$ cat Makefile
test : test.sh $(shell find good)
./test.sh
my#comp:~/wtfdir$
For the sake of MCVE, test.sh is very rudimentary:
my#comp:~/wtfdir$ cat test.sh
echo "blah"
my#comp:~/wtfdir$
However, I noticed, this behaves in a rather unexpected way:
my#comp:~/wtfdir$ ls good
test1 test1.out
my#comp:~/wtfdir$ make
./test.sh
blah
my#comp:~/wtfdir$ touch good/test1
my#comp:~/wtfdir$ make
cp good/test1 good/test1.out
./test.sh
blah
my#comp:~/wtfdir$
Why (expletive redacted) does modifying test1 cause make to overwrite test1.out with test1??? I'm not a big fan of data losses, you know.
What's going on here?
Your Make appears to be GNU Make. Here's why this happens. Your recipe:
test : test.sh $(shell find good)
./test.sh
adds to the prerequisites of test every file and directory that is listed
by find good in the current directory, which happen to be:
good
good/test1
good/test1.out
So to make target test, Make begins by determining if any of the specified
or built-in recipes require it to rebuild any of the prerequsities:
test.sh good good/test1 good/test1.out
Among its built-in recipes it finds:
%.out: %
# recipe to execute (built-in):
#rm -f $#
cp $< $#
as you can verify by running:
$ make --print-data-base | grep -A4 '%.out'
The rule for this recipe is matched by:
good/test1.out: good/test1
and by doing:
$ touch good/test1
you have made good/test1.out out of date with respect to good/test1.
So make executes the recipe:
#rm -f good/test1.out
cp good/test1 good/test1.out
the visible output of which is what you observed:
cp good/test1 good/test1.out
Then it proceeds with the recipe for test:
./test.sh
blah
There is always a risk of such booby-traps if you write a makefile that blindly
generates at runtime some set of preqrequisites or targets you don't know beforehand.
You could avoid this one in particular by explicitly deleting the offending
implicit pattern rule in your makefile by writing:
%.out: %
with no recipe. And you can avoid all possible booby-traps of this sort by disabling all
built-in recipes, with:
$ make --no-builtin-rules ...
but that will require you to write for yourself any builtin-recipes that your
makefile relies on.
The best solution for you is probably to amend your makefile as follows:
PREREQS := $(shell find good)
test : test.sh $(PREREQS)
./test.sh
$(PREREQS): ;
Then the last line explicitly specifies an empty recipe
for each of the $(PREREQS), and Make will not consult any pattern rules for targets
that have explicit recipes.
You should additionally make test a phony target:
.PHONY: test
for the avoidance of the booby-trap where something creates a file called test in the build directory.
I am trying to build a generic task that will execute other task. What I need it to do is to loop against directories and use each dir name executing other task for it.
This is what I have:
# GENERIC TASKS
all-%:
for BIN in `ls cmd`; do
#$(MAKE) --no-print-directory BIN=$(BIN) $*
done
But I get this error, could anyone explain to me how can I make it work
bash
➜ make all-build
for BIN in `ls cmd`; do
/bin/sh: -c: line 1: syntax error: unexpected end of file
make: *** [all-build] Error 2
UPDATE
this is how the complete flow of my makefile looks like:
all-%:
for BIN in `ls cmd`; do \
#$(MAKE) --no-print-directory BIN=$BIN $*; \
done
build-%:
#$(MAKE) --no-print-directory BIN=$* build
build:
docker build --no-cache --build-arg BIN=$(BIN) -t $(BIN) .
Each line of a make-recipe is executed in a distinct invocation of the shell.
Your recipe fails with a shell-syntax error because this line:
for BIN in `ls cmd`; do
is not a valid shell command. Nor is the third line:
done
To have all three lines executed in a single shell you must join them
into a single shell command with make's line-continuation character \:
# GENERIC TASKS
all-%:
for BIN in `ls cmd`; do \
#$(MAKE) --no-print-directory BIN=$$BIN $*; \
done
Note also BIN=$$BIN, not $(BIN). BIN is a shell variable here, not a make variable: $$ escapes $-expansion by make, to preserve the shell-expansion $BIN.
Using ls to drive the loop in Make is an antipattern even in shell script (you want for f in cmd/* if I'm guessing correctly) but doubly so in a Makefile. A proper design would be to let make know what the dependencies are, and take it from there.
all-%: %: $(patsubst cmd/%,%,$(wildcard cmd/*))
$(MAKE) --no-print-directory -$(MAKEFLAGS) BIN=$< $*
I'm writing my first complex Makefile for a highly-modularized project.
I have various sub-directories, each one has its own Makefile which supports at least the all and the clean targets.
These sub-Makefiles work just fine, however I have a problem with the main Makefile, that should call all the sub-Makefiles automatically from the list contained in the variable COMPONENTS.
I tried with the following Makefile:
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY = all clean FORCE $(OUTFILE) $(COMPONENTS)
all: $(OUTFILE)
$(OUTFILE): $(COMPONENTS)
./make_image
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $component; \
done
FORCE:
clean:
for component in $(COMPONENTS); do \
make -C $component clean; \
done
This results in the following error message:
for component in bootloader; do \
make -C omponent; \
done
make: *** omponent: No such file or directory. Stop.
make: *** [bootloader] Error 2
as if the $component expression was only parsed as $c. I don't understand why that happens and how to fix it.
Just double the dollar sign:
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $$component; \
done
The trouble is that with your makefile, Make expands $component before executing the rule. And since $c has no value (there is no such variable), it expands to nothing, leaving "omponent", which it passes to she shell, which complains that there's no such directory. (If you had written $(component), Make would have expanded it to nothing, since Make knows of no such variable, and then the shell would have complained that you were not specifying a directory at all.)
With the double dollar sign, Make expands $$component to $component, which it then passes to the shell, which interprets it as the loop variable, and everything proceeds as planned.
You really should have played around with a simple loop in a command, before attempting to do actual work with one.
Several issues.
.PHONY should be written as a dependency, not a macro definition
Don't write shell loops, use make syntax instead
When you call make recursively, you must do it via the ${MAKE} macro invocation
Leading to
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY: all
all: ${OUTFILE}
.PHONY: ${OUTFILE}
${OUTFILE}: ${COMPONENTS}
./make_image
.PHONY: ${COMPONENTS}
${COMPONENTS}:
${MAKE} -C $#
The advantage of this formulation is that it is parallel make friendly.
Always a test of a good Makefile.
Here make -j5 all will cause make to keep 5 commands running at once,
across all invocations of make.
Nice if you have 4 CPUs.
What about clean?
(Personally I hate clean targets—it's a sign of dodgy dependencies,
and of unhygienic mixing of source and target folders.)
Just add -clean (say) to each of the component names,
and repeat the pattern above.
CLEANS := $(addsuxffix -clean,${COMPONENTS})
.PHONY: clean
clean: ${CLEANS} ; #echo Clean succesful
.PHONY: ${CLEANS}
${CLEANS}: %-clean:
${MAKE} -C $* clean
These two sections can tidied up and combined into one if you feel so inclined.
Tip
Always run make with --warn (or --warn-undefined-variables to give it its full name) to catch inadvertent expansion of $c in things like $component.
I'd like to use $(error ...) to abort my make process if certain preconditions aren't met. The fails_to_work target should abort when failing test -d /foobar.
BAD.mk
all: this_works fails_to_work
this_works:
#echo echo works...
#test -d ~ || echo ~ is not a directory
#test -d /foobar || echo /foobar is not a directory
fails_to_work:
#echo error does not work...
#test -d ~ || $(error ~ is not a directory)
#test -d /foobar || $(error /foobar is not a directory)
$ make -f BAD.mk
echo works...
/foobar is not a directory
BAD.mk:9: *** ~ is not a directory. Stop.
As you can see, not even "error does not work..." is echoed to the screen. The recipe for fails_to_work fails before it gets started. How do I solve this? One of my use cases is#test -d $(MY_ENV_VAR), but I don't think that differs from the hard-coded paths given in the example.
UPDATE (version information)
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for x86_64-pc-linux-gnu
You're trying to get the shell stuff in a recipe to conditionally invoke makefile stuff, which doesn't work, as you've found.
I can think of two options:
Simply remove the $(error) stuff. If test fails, then it will return a non-zero exit status, and the Make process will terminate at that point.
Take the test out of the rule, and use a Make conditional (which in turn invokes shell functionality), e.g.:
ifeq ($(shell test -d /foobar; echo $$?),1)
$(error Not a directory)
endif
Shell commands for a make recipe are effectively stored as a single recursively expanded variable. At the point make decides to run the recipe, it expands the variable, and then runs each line in its own shell invocation. Any $(error ...) that gets expanded will cause make to abort even before invoking the first command.
Note though that the untaken branch of a $(if ...) or $(or ...) &c. will not be expanded. Thus, you could do
.PHONY: rule-with-assert
rule-with-assert:
$(if $(realpath ${should-be-file}/),$(error Assertion failure: ${should-be-file} is a folder!))
⋮
Note that trailing / in the realpath.
Of course macros help to tidy this up a lot.
assert-is-file = $(if $(realpath $1/),$(error Assertion failure: [$1] is a folder!))
.PHONY: rule-with-assert
rule-with-assert:
$(call assert-is-file,${should-be-file})
⋮
It's worth noting again that it doesn't matter where you put the $(call assert-is-file,…) in the recipe.
Any $(error)will be generated as the recipe is expanded,
before any shell commands are run.
Why don't you just use exit 1 shell command instead of $(error ...)? Is there any reason to use the latter?
try_this:
#test -d /foobar || { echo /foobar is not a directory; exit 1; }
or_this:
#if [ ! -d /foobar ]; then echo /foobar is not a directory; exit 1; fi
Both of these will abort the make process unless -k flag is specified.
-k
--keep-going
Continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other prerequisites of these targets can be processed all the same.