Must load modules before making target - makefile

I need to ensure that certain modules are loaded for the build process to proceed. I need one Makefile that can make one or several programs (submodel).
What I have currently:
all : submodel_01 submodel_02 submodel_03
check-modules: load_modules.sh
#if [ -z "${MODULES_LOADED}" ]; then \
echo "Please run"; \
echo "source $<"; \
echo "first."; \
exit 1; \
fi
.PHONY: check-modules
load_modules.sh:
#echo "export MODULES_LOADED=yes" > $#
#echo "module load intel-compiler" >> $#
#echo "module load openmpi" >> $#
submodel_01: check-modules
<command to make submodel_01>
submodel_02: check-modules
<command to make submodel_02>
submodel_03: check-modules
<command to make submodel_03>
This kinda works, but it's not very elegant. I'd much rather have make load the modules itself, then run a child make process where the modules are loaded.
I want to ensure though that the original intended target is preserved. So if I run just make, then it should make all three submodels, but if I specify a specific target, it should only make that one.
Thanks

Disclaimer: I'm not MPI expert, and I was not able to test.
I believe that for the module load command to work, it has to be executed by the parent process (before running Makefile), or in the same shell that will need the environment.
Given that each separate action in running in a separate shell, I think that the following will NOT work - the setting will not pass to the 2nd command. (Suggest you give it a try, just in case.)
submodel_01:
source load_modules.sh
<command that depends on modules>
However, this should work
submodel_01:
source load_modules.sh ; <commands that depends on modules.sh>
You can even include the 'guard' in load_modules.sh to prevent repeated sourcing by changing load_modules.sh to:
if [ ! "$MODULES_LOADED" ] ; then
module load intel-compiler"
module load openmpi
export MODULES_LOADED=yes"
fi

Related

Makefile: exit on conditional

I want to check that an environment variable is set before executing some code in a Makefile. If it's not set I want to throw an error with a simple error message:
run:
[ -z "$(MY_APP)" ] && echo "MY_APP must be set" && exit 1
echo "MY_APP is set. Yay!"
echo "Let's continue on with the command..."
When MY_APP is not set I get the following error, which is desired:
[ -z "" ] && echo "MY_APP must be set" && exit 1
MY_APP must be set
make: *** [run] Error 1
However, when MY_APP is set I get the following error:
[ -z "EXAMPLE_NAME" ] && echo "MY_APP must be set" && exit 1
make: *** [run] Error 1
Any idea what I'm doing wrong? And is there a better way to do this?
Recall that the && condition require that all conditions must be TRUE to pass. Since the first condition fail, the whole command will return a status of 1 (-> false), effectively stopping the make
You can use the following, so that the test will fail only when MY_APP is missing.
Note that I'm using false instead of exit 1. Also better to use "${MY_APP}", which make it easier to copy/paste from Make to shell prompt/script.
run:
{ [ -z "$(MY_APP)" ] && echo "MY_APP must be set" && false } || true
...
# Or just if-Then-Else
if [ -z "${MY_APP}" ] ; then echo "MY_APP must be set" ; false ; fi
...
You can test environment variables with Makefile conditional syntax, like this:
sometarget:
ifndef MY_APP
#echo "MY_APP environment variable missing"
exit 1
endif
somecommand to_run_if_my_app_is_set
Note that ifndef/ifdef operate on the name of the variable, not the variable itself.
It seems that you are trying to use a Makefile to run commands which are not building targets (the target name run is a giveaway). You already got bitten by one of Makefile and shells caveats. Makefile caveat: exit status is inspected after each line and if not zero abort immediately. Shell caveat: the test command ([) returns a non zero exit status so the entire line returns non zero.
The rule of thumb is: a recipe of a rule should create a filename named like the target of the rule.
Here is a rule (to clarify the terms):
target:
recipe command lines
should create file named target
There are some exceptions to this rule of thumb. Most notably make clean and make install. Both typically do not create files named clean or install. One can argue that make run maybe also be an exception to this rule of thumb.
If your run is as simple as a typical clean then I might agree about making an exception. But usually commands are run with command line arguments. Before long you will want make run accept arguments. And making make accept custom command line arguments is not fun at all.
You tried to manipulate the behaviour using environment variables which is somewhat less problematic than command line arguments. But still problematic enough to make you trip over a caveat.
My suggestion for a fix:
Put complex recipes in a shell script. There you have all the power and flexibility of a shell script without the awkwardness of makefiles. For example as explained here: Basic if else statement in Makefile
In case of a typical run target write a wrapper shell script around the makefile which lets the makefile rebuild the target and then run the target. For exampe as explained here: Passing arguments to "make run"
You can conditionally exit the Makefile using error control function, at least in the GNU version.
This snippet is a helpful condition to put into the head of the Makefile. It exits with a message of help, if make was not called from within the directory of the Makefile.
MAKEFILE_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
ifneq (${MAKEFILE_DIR}, $(shell pwd))
INVALID_LOCATION:=`make` must be called from within ${MAKEFILE_DIR} (or with option -C ${MAKEFILE_DIR})
$(error ERROR: $(INVALID_LOCATION))
endif
See: https://www.gnu.org/software/make/manual/html_node/Make-Control-Functions.html
Useful in case your paths are relative to the Makefile and you don't want them to prefix with a base.

How to change the return value of a `make` command

I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.

deferred include in make

I want to include another makefile after running a particular command because the first command generates that makefile. I tried this
debug2:
cd bench/${BENCH}; verilator --cc top.v
include ${BENCH_DIR}/Vtop_classes.mk
In this case make doesn't recognize the include command and gives an error "make: include: Command not found"
if I try to include without tab then it includes before the previous command is executed and so generates error of no such file
debug2:
cd bench/${BENCH}; verilator --cc top.v
include ${BENCH_DIR}/Vtop_classes.mk
Add - before : -include ${BENCH_DIR}/Vtop_classes.mk
INCLUDED-FILES=MakeA MakeB
-include $(INCLUDED-FILES)
# alternatively use sinclude
generate-makefile:
touch MakeA
touch MakeB
clean: generate-makefile
if [ -a MakeA ]; then make clean-A; else echo "No file: MakeA. Run make again."; fi;
if [ -a MakeB ]; then make clean-B; else echo "No file: MakeB. Run make again."; fi;
GNU make will not report errors because of -include. When make runs the deferred rules and the included files do not exist the shell prints a message "Run make again". The deferred rules in the example call make a second time, so no error is reported.
Because -include does not report errors, make can run the generation rule on the first run. The dependencies are resolved for the second run.

explicity chain dependency in Makefile

What's the pattern to follow when specialized Makefiles in a directory depends on the main one in a parent dir?
i have:
/
/Makefile
/src
/src/Makefile
/tests
/tests/Makefile
in /Makefile i have:
TESTING_COMMAND=something
dotest1:
make -C tests/ $#
in /tests/makefile i have
dotest1:
$(TESTING_COMMAND) $?
if i run:
me#host:/ $ Make dotest1
it works. but if i execute from the tests dir:
me#host:/tests/ $ Make dotest1
it will try to execute the test file in the shell, because $(TESTING_COMMAND) is empty, so it's first argument became the command passed to the shell.
I don't necessarily need that to work if executed in the /tests/ or /src/ dir, but need a way to gracefully fail.
Trying to send everything through the command line (or environment) seems like a bad idea to me. That's what inclusion was invented for. Put your common values into a separate file, something like config.mk, then in all your makefiles just use:
include config.mk
to get them included.
Your design scares me, but this will do the trick in the main Makefile:
TESTING_COMMAND=something
dotest1:
make -C tests/ $# TESTING_COMMAND=$(TESTING_COMMAND)
If you want tests/Makefile to fail well, you have a couple of options. If only that one target depends on TESTING_COMMAND, you can have it print a warning and do nothing:
ifdef TESTING_COMMAND
dotest1:
$(TESTING_COMMAND) $?
else
dotest1:
#echo warning: TESTING_COMMAND not defined
endif
Or if the whole Makefile depends on it, you can have Make print a warning or abort:
ifndef TESTING_COMMAND
$(warning TESTING_COMMAND is undefined, but Make will try to us it anyway)
$(error TESTING_COMMAND is undefined, Make will now abort)
endif
You can also have it abort the sub-make (the one that runs tests/Makefile) but still continue running the Make process that invoked it, but that's kind of a pain.

start other makefile within makefile

Since i am not so experienced with the building process / makefiles on linux i ran in follow problem:
the Setup:
i have an makefile A, which needs some enviroment variables set before running, this is done by running . ./set_A_vars.sh (set_A_vars.sh contains many export lines) before running make -f A
now i need to make project A within a makefile B.
i tried the following setup for makefile B:
all: debug release
A_debug:
. ./set_A_vars.sh && make -f A DEBUG=1
A_release:
. ./set_A_vars.sh && make -f A DEBUG=0
debug: A_debug some_B_stuff_debug
release: A_release some_B_stuff_debug
however i get lots of errors, which sound like the enviroment variables in set_A_vars.sh have not been set for make -f A ... in B.
How can i call makefile A from makefile B with the enviroment variables in set_A_vars.sh set in makefile B ??
Any help appreciated.
Your makefile looks good with these provisos:
When you call make from a makefile, please use the macro invocation ${MAKE} rather than plain make. (This ensures parallel make works, and also means it still works even if your make has another name (GNUmake say).)
If your targets do not correspond to actual files, then mark them with .PHONY (see below).
Does some_B_stuff_debug require A to be built first? Then you must tell make this.
some_B_stuff_debug: A_debug
some_B_stuff_debug: A_release
This is clearly wrong. One way is to enforce the ordering via the shell.
Try something like this:
.PHONY: debug
debug:
. ./set_A_vars.sh && ${MAKE} -f A DEBUG=1
${MAKE} some_B_stuff_debug
.PHONY: release
release:
. ./set_A_vars.sh && ${MAKE} -f A DEBUG=0
${MAKE} some_B_stuff_debug
.PHONY: some_B_stuff_debug
∶
Your makefiles should work. I suggest you try the following:
Try running set_A_vars.sh from the command line.
Verify that the variables you wanted set are set.
make -f MakefileA, to verify that MakefileA really does work nicely with these variables set.
Try a rule in MakefileB that will test one of the variables, say FOO:
test_var:
#echo FOO is $(FOO)
This should work if you have just run set_vars.sh. If it doesn't, then there are a couple of things that could be wrong...
Now clear the variables (including FOO) and try this rule in MakefileB:
set_vars_and_test_them:
./set_A_vars.sh && echo FOO is $(FOO)
Now put it together:
A_debug:
./set_A_vars.sh && make -f MakefileA DEBUG=1
(I recommend against calling a makefile "A".)

Resources