I have a Makefile that is organizing several steps of analysis that have to be run in a particular order. The analysis takes quite a while (a day or two) and I'd like to receive some email notifications when make completes. Is there a good way to have make automatically send an email at the end of the process so I can be alerted when it completes, especially when there is a failure with one of the steps?
I'm currently doing something like this:
# Makefile
all: results1.dat results2.dat results3.dat
python send_email_when_done.py
results1.dat: long_running_program1.py
python $< > $# # this takes ~12 hours
results2.dat: long_running_program1.py results1.dat
python $^ > $# # this takes ~2 hours
results2.dat: long_running_program1.py results2.dat
python $^ > $# # this takes ~30 hours
where the send_email_when_done.py script sends email notifications when the process has completed. But this only works if the entire process has run from start to finish without any errors. Any suggestions for a good way to do this?
+1 for suggestions that can accomplish this within the Makefile. I'm already running make in a separate session using setsid make > make.out 2>&1.
How about simply make the email sending as a separated target:
report: long_running_program1.py
python $< > $# # this takes ~12 hours
sendmail:
python send_email_when_done.py
And you call it:
make report sendmail
or
make report ; make sendmail
Let me make it more general so that it can send email for 100's of other such make. Use alias and add it in bashrc, here sagarsakre.blogspot.in is how i tried to use the alias command makes (make and send report). The added advantage would be u would get all the verbose printed regardless of worrying about rule and dependency. You can use it for all builds without really changing the makefile.
note: i used mailx, You can read this to setup mailx on ur machine
Since makepp allows you access to the underlying interpreter, you can do a little Perl programming to add an END-Handler anywhere in your makefile:
perl { END { system "python send_email_when_done.py" }}
Or more prettily layouted with an alternate direct subject-only mail
perl {
END {
system "mail -s 'Build finished with rc=$?' me#my.home <&-";
}
}
There is much more to makepp. Besides doing almost all that GNU make can, there are lots more useful things.
One way of obtaining that, would be to instruct make to ignore errors from the long-running programs. This can either be achieved by running make -i or by prepending each of the commands where failure should be ignored with a dash and just running plain make:
# Makefile
all: results1.dat results2.dat results3.dat
python send_email_when_done.py
results1.dat: long_running_program1.py
-python $< > $# # this takes ~12 hours
results2.dat: long_running_program1.py results1.dat
-python $^ > $# # this takes ~2 hours
results2.dat: long_running_program1.py results2.dat
-python $^ > $# # this takes ~30 hours
See the GNU make manual for more information on ignoring errors.
Related
I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.
I'm trying to create a makefile which downloads some pre-requisite files to a path.
But the foreach documentation is sadly lacking in detail and examples.
I want something like:
image_files = a b
image_versions = 701.2 802.1
image_path = images
images = $(foreach ...) ??
I'd like that to result in an expansion to:
images/701.2/a
images/701.2/b
images/802.1/a
images/802.1/b
And have a phony target to download them from a URL like:
mytarget: $(images)
wget somepath $<
How do I do that?
Ok I have gotten a little further with this. But I'm still a little perplexed as to how I can get this to work.
tag = my-registry:8443/boot-server-data
versions = 557.0.0 607.0.0
images_a = $(foreach ver, $(versions), images/$(ver)/coreos_production_pxe_image.cpio.gz)
images_b = $(foreach ver, $(versions), images/$(ver)/coreos_production_pxe.vmlinuz)
all: build
.PHONY: build $(images_a) $(images_b)
build:
./make-profiles
docker build -t $(tag) .
docker push $(tag)
$(images_a):
wget http://stable.release.core-os.net/amd64-usr/$(foreach version... but depends on each image)/coreos_production
How do you do this?
In fact I only want it to download the images if they aren't there. But for some reason it downloads it every time. It's literally been years since I used Make. I normally use another build tool, but that build tool needs to be modified to make it do what I want here. So I thought I'd just whip this up in the meantime. It's prooving to be a little harder than expected.
You are pretty close, but the problem does not lie with foreach. Let's have a look at just the bit that does the downloading. When make reads the makefile it ends up with something like (after shortening the names a bit for clarity):
images/1/file.cpio.gz images/2/file.cpio.gz:
<recipe>
If, for some reason, make decides to rebuild images/1/file.cpio.gz say, at this point it will expand the recipe, and pass each line of that expansion to a separate shell.
Your job is to write a recipe that does not care whether the target is images/1/file.cpio.gz or images/2/file.cpio.gz. That's another way of saying the recipe should use macros like $# (it will expand to the target).
A sketch:
${images_a}:
wget -O $# http://stable.release.core-os.net/amd64-usr/$#
You may have to munge $# so that wget gets the right url. Just one example:
${images_a}:
wget -O $# http://stable.release.core-os.net/$(dirname $#)/deeper/$(notdir $#)
One complaint about your original makefile: the dependencies are wrong. build needs the downloads to have completed before it runs.
.PHONY: build
build: $(images_a) $(images_b)
...
The images are not phony (just ensure you don't lie to make abut their filenames) either.
The massive advantage of writing your makefile in this way is that it's parallel safe (and that's the whole point of make). When -j is in force, both wgets can proceed at the same time, halving the download time.
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.
I have a compile job where linking is taking a lot of IO work. We have around a dozen of cores so we run make -j13, but when it comes to linking the 6 targets, I'd like those to be done in a round robin way. I thought about making one depend on the next but I think this would break the individual targets. Any ideas how to solve this small issue?
make itself doesn't provide a mechanism to request "N of these, but no more than M of those at a time".
You might try using the sem command from the GNU parallel package in the recipe of your linker rules. Its documentation has an example of ensuring only one instance of a tool runs at once. In your example, you would allow make to start up to 13 sems at a time, but only one of those at a time will run the linker, while the others block.
The downside is that you could get into a situation where 5 of your make's 13 job slots are tied up with instances of sem that are all waiting for a linker process to finish. Depending on the structure of your build, that might mean some wasted CPU time. Still beats 6 linkers thrashing the disk at once, though :-)
You should specify that your six targets cannot be built in parallel. Add a line like this to your makefile:
.NOTPARALLEL: target1 target2 target3 target4 target5 target6
For more information look here https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html.
I've stumbled upon a hacky solution:
For each recipe it runs, Make does two things: it expands variables/functions in the recipe, and then runs the shell commands.
Since the first step can read/write the global variables, it seems to be done synchronously.
So if you run all your shell commands during the first step (using $(shell )), no other recipe will be able to start while they're running.
E.g. consider this makefile:
all: a b
a:
sleep 1
b:
sleep 1
time make -j2 reports 1 second.
But if you rewrite it to this:
# A string of all single-letter Make flags, without spaces.
override single_letter_makeflags = $(filter-out -%,$(firstword $(MAKEFLAGS)))
ifneq ($(findstring n,$(single_letter_makeflags)),)
# See below.
override safe_shell = $(info Would run shell command: $1)
else ifeq ($(filter --trace,$(MAKEFLAGS)),)
# Same as `$(shell ...)`, but triggers a error on failure.
override safe_shell = $(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(.SHELLSTATUS)))
else
# Same functions but with logging.
override safe_shell = $(info Shell command: $1)$(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(>
endif
# Same as `safe_shell`, but discards the output and expands to nothing.
override safe_shell_exec = $(call,$(call safe_shell,$1))
all: a b
a:
$(call safe_shell_exec,sleep 1)
#true
b:
$(call safe_shell_exec,sleep 1)
#true
time make -j2 now reports 2 seconds.
Here, #true does nothing, and suppresses Nothing to be done for ?? output.
There are some problems with this approach though. One is that all output is discarded unless redirected to file or stderr...
It won't break individual targets.
You can create any number of (:) rules for a target, as long as only one of them has an actual recipe for building it. This appears to be a good use case for that.