I'm trying to adapt the Example from the Official Documentation of GNU make to my Use Case:
GNU make - Example of a Conditional
libs_for_gcc = -lgnu
normal_libs =
ifeq ($(CC),gcc)
libs=$(libs_for_gcc)
else
libs=$(normal_libs)
endif
foo: $(objects)
$(CC) -o foo $(objects) $(libs)
So, I created this Prototype:
libs_for_gcc="gcc libs"
normal_libs="normal libs"
libs=
all: do_nothing
#echo "all: done."
do_nothing:
#echo "do_nothing: done."
example:
#echo "example: go ..."
#echo "example - cc: '$(CC)'"
#echo "libs_for_gcc: $(libs_for_gcc)"
#echo "normal_libs: $(normal_libs)"
#echo "libs: $(libs)"
ifeq ($(CC),gcc)
libs=$(libs_for_gcc) && echo "libs: '$$libs'"
#echo "example - libs: '$(libs)'"
else
libs=$(normal_libs) && echo "libs: '$$libs'"
#echo example - libs: $(libs)
endif
#echo "example - libs: '$(libs)'"
#echo "example: done."
test: libs += " -Ddebug"
test: example foo
bar:
#echo "bar: go ..."
ifeq (${libs}, "")
#echo "bar - libs: empty"
#echo "assigning libs"
libs=$(libs_for_gcc)
else
#echo "bar - libs: not empty"
#echo "bar - libs: '${libs}'"
endif
#echo "bar - libs: '${libs}'"
#echo "bar: done."
foo:
#echo "foo: go ..."
ifneq ("$(libs)", "")
#echo "foo - libs: not empty"
#echo "foo - libs: '$(libs)'"
else
#echo "foo - libs: empty"
endif
#echo "foo: done."
Now when I run the Default Target with $ make
it just produces:
$ make
example: go ...
example - cc: 'cc'
libs_for_gcc: gcc libs
normal_libs: normal libs
libs:
libs="normal libs"
example - libs:
example - libs: ''
example: done.
I see that the value of libs was not changed as intended.
When I run the make bar Target it produces:
$ make bar
bar: go ...
bar - libs: not empty
bar - libs: ''
bar - libs: ''
bar: done.
Here libs is not empty but it has nothing inside.
And when I run the make foo target it produces:
$ make foo
foo: go ...
foo - libs: empty
foo: done.
Here libs is understood as empty
As I see that libs is not changed correctly I tried to change the syntax to:
example:
# [...]
ifeq ($(CC),gcc)
libs := $(libs_for_gcc)
#echo "example - libs: '$(libs)'"
else
libs := $(normal_libs)
#echo example - libs: $(libs)
endif
But then I get the GNU make Error:
$ make
example: go ...
example - cc: 'cc'
libs_for_gcc: gcc libs
normal_libs: normal libs
libs:
libs := "normal libs"
/bin/sh: 1: libs: not found
Makefile:7: recipe for target 'example' failed
make: *** [example] Error 127
I couldn't find any documentation about this behaviour so I appreciate any advise.
Edit:
Added all and test Targets.
Background:
The GNU make Command is an important part of the toolchain for packaging and deploying software and thus important in the daily work of System Administrators and DevOps Engineers.
The Debian and RPM Packaging uses GNU make to package software.
Makefile Driven Packaging
It runs the ./configure && make && make install command sequence.
The Travis CI Workflow uses GNU make for running testsuites.
C Language Automated Testing
It runs the ./configure && make && make test sequence.
All the completely different Use Case are managed by the same Makefile.
Now for my concrete Use Case I'm working on setting up the Travis CI Workflow Sequence to enable Automated Testing for my static linked Source Code Library.
So, contrary to the Packaging Sequence the Automated Testing Sequence requires Debug Features and advanced Output Evaluation to produce a meaningful Test.
I want the Test to check the Error Report and also the Memory Usage Report to alert me of any hidden errors.
Using the Advice about setting the Variable at the Target Declaration Line I was able to change libs for test, example and foo targets.
I also saw the important hint about the Bash Variable libs which is only valid on the same line:
$ make
do_nothing: done.
all: done.
$ make test
example: go ...
example - cc: 'cc'
libs_for_gcc: gcc libs
normal_libs: normal libs
libs: -Ddebug
libs="normal libs" && echo "libs: '$libs'" ;
libs: 'normal libs'
example - libs: -Ddebug
example - libs: ' -Ddebug'
example: done.
foo: go ...
foo - libs: empty
foo - libs: ' -Ddebug'
foo: done.
The Recipe libs=$(libs_for_gcc) && echo "libs: '$$libs'" shows that a new Bash Variable libs was created and it did not affect the GNU make Variable.
Still the Conditional ifneq ($(libs),) cannot detect that libs was already set for the test target.
The difference between your example and the one in the GNU make manual is that in the GNU make manual they are setting a make variable named libs (they are assigning the variable outside of any recipe).
In your usage you are assigning a shell variable named libs (you are assigning it inside the recipe, indented with a TAB).
That's why you get an error when you try to use :=, because that's a make assignment and is not a valid shell assignment.
Then in your echo you print the make variable $(libs) which has not been set at all. Further, every logical line of the recipe is run in inside it's own shell. So you are running the equivalent of:
/bin/sh -c 'libs="gcc libs"'
/bin/sh -c 'echo '
so even if you DID change your echo command to print the shell variable (via $$libs) it would be empty because the previous shell, where it was set, has exited.
You want to use the same syntax as in the example: take the assignment of the variable OUT of the recipe and set the make variable instead:
libs_for_gcc = gcc libs
normal_libs = normal libs
ifeq ($(CC),gcc)
libs = $(libs_for_gcc)
else
libs = $(normal_libs)
endif
example:
#echo "example: go ..."
#echo "example - cc: '$(CC)'"
#echo "libs_for_gcc: $(libs_for_gcc)"
#echo "normal_libs: $(normal_libs)"
#echo "libs: $(libs)"
#echo "example - libs: '$(libs)'"
#echo "example: done."
Finally I achieved to get my Makefile make test Target working.
make test Test Suite with GNU make
The key was to understand that there are 3 different contexts where a GNU make Variable can be assigned.
And that GNU make often needs helper functions because of some strange issues with Conditionals
These issues are mostly undocumented and only can be solved by Try and Error and searching on StackOverflow.com
1. The first, easiest and best documented context to assign a Variable is in Global Context as seen in the official documentation and the first answer.
GNU make - Example of a Conditional
ifeq ($(CC),gcc)
libs=$(libs_for_gcc)
else
libs=$(normal_libs)
endif
The Global Variable ${libs} has global visibility.
2. The next context which is not that obvious but still documented in the Official Documentation is in the Target Definition Line
GNU make - Target-specific Variable Values
test: libs += " -Ddebug"
test: example foo
Here ${libs} is valid for the test Target and all dependent Targets that it calls.
But the only Condition is the Target itself.
3. To assign dynamically a Variable in the context of a Target is to use the $(eval ) Function in combination with the $(shell ) Function as seen in this StackOverflow.com Post:
Assign a Variable as Result of a Command
test: examples_debug
# [...]
$(eval error := $(shell cat ./demo_error.log))
ifeq ($(strip $(error)),)
#$(ECHO) "NO Error: '${error}'"
else
#$(ECHO) "Error detected: '${error}'"
$(error "Demo failed with Error [Code: '${error_code}']")
#exit 1
endif
Here ${error} is read from a File. Additionally the $(strip ) Function is required to be able to check if it is empty, which is some of the undocumented issues that GNU make has and are wierd to Bash Developers.
4. Another method that works but does not use a Makefile Variable and is somewhat bulky is evaluating the Variable entirely in Bash in 1 single Line which can be found at: Check Variable with Bash Command Recipe and was also hinted by the previous Answer.
Which would look like:
test: test_debug
# [...]
leak=`cat ./test_heap.log | grep -i "unfreed memory blocks" | awk '{print $$1}'` ; if [ $$leak -gt 1 ]; then echo "Memory Leaks: $$leak" && exit 1; else echo "Memory Leaks: NONE"; fi ;
Here $$leak is a Bash Variable and only valid within the same Line.
An approach that is somewhat similar to the GitHub Workflow Command Logic. (Actually is was directly ported from the GitHub Workflow for the same Project)
As about Evaluation of Conditionals there are many undocumented issues in GNU make that require Bash Workarounds to achieve the goal.
As documented at:
Preprocessing Numerical Values for the Makefile Evaluation
There are issues with comparing Numbers and it is impossible to compare against 0 which is extremely important Exit Code for Command Line Applications.
So the Workaround looked somewhat like:
test: test_debug
# Run the Testsuite Application
# [...]
$(eval error_code := $(shell export HEAPTRC="log=test_heap.log" ; echo -n "" >./test_heap.log ; ${wrkdir}/tests_hash-lists.dbg.run 2>./tests_error.log 1>./tests_exec.log ; echo "$$?"))
#$(ECHO) Application Tests: Execution finished with [${error_code}]
#Application Tests Execution Report
# [...]
$(eval is_error := $(shell if [ "${error_code}" = "0" ]; then echo false ; else echo true ; fi ;))
ifeq (${is_error}, true)
#$(ECHO) Exit Code non cero: '${error_code}'
#$(ECHO) "Tests failed with Error [Code: '${error_code}']"
#exit ${error_code}
endif
In this use case ${error_code} is tested with a Bash Conditional to populate the Helper Variable ${is_error} with true or false which then can be checked in GNU make.
Discussion:
The test Target cannot just exit on error.
For troubleshooting a failed Automated Test it is crucial to see the Exit Code, the Error Message and the Heap Report.
Related
I have a Makefile as follows (excerpt):
# be a POSIX guy!
SHELL = /bin/dash
# avoid accursed tabs
.RECIPEPREFIX +=
PROJECT = my_project
# before-commit stuff
CHANGED_FILES = $(shell git ls-files --modified)
files ?= $(CHANGED_FILES)
lint:
pyflakes $(files)
lint-all:
pyflakes $(PROJECT)
STAGING_DB_PORT = 5437
staging-db-start:
ssh -fNL 0.0.0.0:$(STAGING_DB_PORT):localhost:$(STAGING_DB_PORT) staging-db
ss -tlpn sport eq :$(STAGING_DB_PORT)
staging-db-stop:
ssh -O check staging-db
ssh -O stop staging-db
ss -tlpn sport eq :$(STAGING_DB_PORT)
staging-db-check:
ss -tlpn sport eq :$(STAGING_DB_PORT)
ssh -O check staging-db
.PHONY: lint, lint-all, staging-db-start, staging-db-stop, staging-db-check
When I run target, say staging-db-check it works just fine. Although, when I run target lint, it fails with error:
Makefile:2:9: invalid syntax
SHELL = /bin/dash
^
For me, it is very strange. I read docs, they say that you always must set SHELL variable, so I decided to do so. But I can not figure out where there is an error?
I have GNU make, version 4.2.1.
GNU Make never generates diagnostics of form of:
Makefile:2:9: invalid syntax
SHELL = /bin/dash
^
But pyflakes does, which is the program run by your lint target's recipe:
lint:
pyflakes $(files)
As you know, pyflakes lints Python source files. Your $(files), as assigned
by:
# before-commit stuff
CHANGED_FILES = $(shell git ls-files --modified)
files ?= $(CHANGED_FILES)
expands to a list of files that includes Makefile. Your Makefile is not
a Python source file and the first line in Makefile that is not syntactically
valid Python is:
SHELL = /bin/dash
Here's a shorter makefile:
Makefile
# be a POSIX guy!
SHELL = /bin/dash
.PHONY: all
all:
echo "Hello World"
with which to reproduce your error:
$ pyflakes Makefile
Makefile:2:9: invalid syntax
SHELL = /bin/dash
^
Later
Is there a way to exclude non-python files from $files variable?
Yes. Assuming that Python files are files with the extension .py, change:
CHANGED_FILES = $(shell git ls-files --modified)
to:
CHANGED_FILES = $(filter %.py,$(shell git ls-files --modified))
See functions:
$(filter pattern...,text)
$(filter-out pattern...,text)
in 8.2 Functions for String Substitution and Analysis
in the GNU Make manual
And if you do that, maybe better change CHANGED_FILES to CHANGED_PYTHON_FILES.
Lets assume, i want to call
make somepath/abc.pot
which depends on somepath/somefiles.c
My target I've created so far looks like
%.pot: $(dir $#)*.c
#echo "it works"
ifeq (,$(wildcard $#))
# pot-file does not exist, do something
else
# pot-file already exists, do something else
endif
but does not work as the Automatic Variables
like $# are not available in the prerequisites.
If found, that i can enable second expansion
.SECONDEXPANSION:
%.pot: $$(dir $$#)*.c
#echo "it works"
ifeq (,$(wildcard $#))
# pot-file does not exist, do something
else
# pot-file already exists, do something else
endif
which allows me to use $# in the prerequisites but breaks my ifeq statement which then always results in the first branch. If I change the ifeq to
ifeq (,$$(wildcard $$#))
it's working again but I really don't get why.
Now there a two questions:
A) Is there another way but to enable second expansion to have the path of the target in my prerequisites?
B) Why does the ifeq (,$(wildcard $#)) statement always result in the first branch if second expansion is enabled?
Don't use ifeq in the recipe at all. Just use normal shell functionality. It works better.
.SECONDEXPANSION:
%.pot: $$(dir $$#)*.c
#echo "it works"
if [ ! -f $# ]; then \
: pot-file does not exist, do something; \
else \
: pot-file already exists, do something else; \
fi
That said using wildcard in prerequisite lists is generally a bad idea because the time that they are globbed is not reliable and can cause odd behaviors. See Pitfalls of Using Wildcards for one example of a problem.
If you need to write different recipe contents based on some external factor (like OS) then you need to detect that at make parse time and have two copies of your recipes/makefile that you switch between correctly. You can do that inline but you can't do that per-recipe inline.
Your original attempts (using ifeq in a recipe) do not work. They don't do what you think they do. They may appear to work but they aren't working the way you expect.
Consider this makefile:
all: c
a:
#touch a
c: a
.SECONDEXPANSION:
c d:
ifeq (,$(wildcard a))
#echo "a doesn't exist (make)"
else
#echo 'a does exist (make)'
endif
#if [ ! -f a ]; then \
echo "a doesn't exist (sh)"; \
else \
echo 'a does exist (sh)'; \
fi
ifeq (,$$(wildcard a))
#echo "a doesn't exist (make se)"
else
#echo 'a does exist (make se)'
endif
In an empty directory you would expect make to output (assuming ifeq works the way you want it to):
a does exist (make)
a does exist (sh)
a does exist (make se)
Right? But it doesn't. You get:
a doesn't exist (make)
a does exist (sh)
a does exist (make se)
Ok, you think, that's just things not working without secondary expansion. But the secondary expansion version is working correctly. But it isn't.
If you run make d in an empty directory (note the d target doesn't list a as a prerequisite so it won't create it) you would expect the following output:
a doesn't exist (make)
a doesn' exist (sh)
a doesn' exist (make se)
Right? But what you actually get is:
a doesn't exist (make)
a doesn't exist (sh)
a does exist (make se)
So it appears that the secondary expansion version isn't working either.
A look at the make database explains why not.
Run make -qprR | awk '/c: a/,/^$/; /d:/,/^$/' in an empty directory and you see:
c: a
# Implicit rule search has not been done.
# File does not exist.
# File has been updated.
# Needs to be updated (-q is set).
# variable set hash-table stats:
# Load=0/32=0%, Rehash=0, Collisions=0/2=0%
# commands to execute (from `Makefile', line 12):
#echo "a doesn't exist (make)"
#if [ ! -f a ]; then \
echo "a doesn't exist (sh)"; \
else \
echo 'a does exist (sh)'; \
fi
#echo 'a does exist (make se)'
d:
# Implicit rule search has not been done.
# Modification time never checked.
# File has not been updated.
# commands to execute (from `Makefile', line 12):
#echo "a doesn't exist (make)"
#if [ ! -f a ]; then \
echo "a doesn't exist (sh)"; \
else \
echo 'a does exist (sh)'; \
fi
#echo 'a does exist (make se)'
Which, as you can see, doesn't contain the ifeq lines but just the "correct" branch of the ifeq logic.
And that's the problem, the ifeq conditionals are evaluated at make parse time which is well before any recipes run (and thus before any files can be created, etc.).
I have the following Makefile, with a rule which checks for dependencies:
#!/usr/bin/make -f
dependencies:
$(info START-info)
#echo "START-echo"
$(call assert-command-present,fastqc)
$(call assert-command-present,cutadapt)
$(call assert-command-present,bowtie2)
$(call assert-command-present,samtools)
$(call assert-command-present,bedtools)
$(call assert-command-present,fetchChromSizes)
$(call assert-command-present,bedGraphToBigWig)
$(info END-info)
#echo "END-echo"
pipeline: dependencies
assert-command-present = $(info Checking for $1) && $(if $(shell which $1),$(info OK.),$(error ERROR: could not find '$1'. Exiting))
The user-defined function assert-command-present checks for a command in the path, and returns an error if it is not found. When I run this Makefile, the echo and info commands are not returned in the order I expect:
START-info
Checking for fastqc
OK.
Checking for cutadapt
OK.
Checking for bowtie2
OK.
Checking for samtools
OK.
Checking for bedtools
OK.
Checking for fetchChromSizes
OK.
Checking for bedGraphToBigWig
OK.
END-info
START-echo
END-echo
The START-echo and START-info commands should run before any assert-command-presents functions run, but the echo command runs after the function calls.
Eugeniu Rosca is correct. More generally, "make" built-in functions are evaluated first, then the entire command sequence is run.
One way to see this is to use the GNU make debugger remake. You can stop at the target "dependencies", and write out the commands that would be run in a shell.
For example:
$ remake -X -f /tmp/Makefile dependencies
GNU Make 4.1+dbg0.91
...
Updating goal targets....
-> (/tmp/Makefile:3)
dependencies:
remake<0> write
START-info
END-info
File "/tmp/dependencies.sh" written.
remake<1>
Look at file /tmp/dependencies.sh and you will see all of the Make functions removed or expanded with whatever value they returned which in my case was empty lines.
I want a different version of the clean target to run based on whether make dev or make prod are run on a makefile.
I'm not compiling anything per se, just want to conditionally call a particular target or set of targets based on a variable, for example:
ifeq ($(BUILD_ENV),"development")
clean: -clean
else
clean: -clean-info
endif
#---------------------------------
dev: BUILD_ENV = development
dev: dev-setup which-env
#---------------------------------
prod: BUILD_ENV = production
prod: prod-setup which-env
#---------------------------------
which-env: clean
#echo -e "$(GREEN)$(BUILD_ENV)!$(CLEAR)"
-clean: -clean-info -clean-logs | silent
#echo -e "$(GREEN)</CLEAN>$(CLEAR)"
-clean-info:
#echo -e "$(GREEN)<CLEAN>...$(CLEAR)"
-clean-logs:
#echo -e " $(GREY)Removing log and status files $(CLEAR)";
#if [ -d .stat ]; then rm -rf .stat; fi
#rm -f *.log || true
Is there a way to do this with Makefiles? I havent found anything yet that illustrates this use-case.
I'm not trying to specifically clean anything or build anything this is just an example of me trying to conditionally call a set of targets. The actual targets could be anything else.
It's not at all clear that what you're asking for is really what you want, but here goes:
all:
ifeq ($(BUILD_ENV),development)
all: clean-dev
else
all: clean-other
endif
clean-dev:
#echo running $#, doing something
clean-other:
#echo running $#, doing something else
If you run make BUILD_ENV=development, you'll get something; if you run make or make BUILD_ENV=production you'll get something else.
My open source project distributes a Makefile. "make" by itself works fine as long as the user has Boost and OpenSSL installed. If not, he gets a compilation error.
I would like to show the user an error message with instructions on how to fix rather than have him discern the issue from the compiler output.
I've put together a little script to embed inside a Makefile that will do a quick and dirty compilation to validate if a prerequisite header file exists before allowing the core code to build. It shows an error message and aborts the compile if the code won't compile. It seems to work good.
# BOOST_INCLUDE := -I/home/jselbie/boost_1_51_0
all: myapp
testforboost.o:
#echo "Testing for the presence of Boost header files..."
#rm -f testforboost.o
#echo "#include <boost/shared_ptr.hpp> " | $(CXX) $(BOOST_INCLUDE) -x c++ -c - -o testforboost.o 2>testerr; true
#rm -f testerr
#if [ -e testforboost.o ];\
then \
echo "Validated Boost header files are available";\
else \
echo "* ********************************************";\
echo "* Error: Boost header files are not avaialble";\
echo "* Consult the README file on how to fix";\
echo "* ********************************************";\
exit 1;\
fi
myapp: testforboost.o
$(CXX) $(BOOST_INCLUDE) myapp.cpp -o myapp
Is my script a good way to do this? I'm under the assumption that it's portable beyond Linux (Solaris, BSD, MacOS). Or are there other standard practices for doing this? I know that Autotools can do similar things, but I'm not too excited about learning all of Autotools and re-writing my Makefiles.
In principle it's possible like that. But since you're only preprocessing, and given that you can use any command as a condition, it can be simplified to:
.PHONY: testforboost
testforboost:
#echo "Testing for the presence of Boost header files..."
#if echo "#include <boost/shared_ptr.hpp> " | $(CXX) -x c++ -E - >/dev/null 2>&1;\
then \
echo "Validated Boost header files are available";\
else \
echo "* ********************************************";\
echo "* Error: Boost header files are not avaialble";\
echo "* Consult the README file on how to fix";\
echo "* ********************************************";\
exit 1;\
fi
OTOH, since you have the boost include path in a variable, why not just look for the file directly? That would need some string manipulation. Probably hard in make, but with makepp it would be $(map $(BOOST_INCLUDE),s/^-I//)