I'm starting to use GNU Make as my frontend build tool, and most things work great. Only thing annoying is that compilation doesn't seem to stop when one of the steps reaches an error. The relevant portion of the Makefile:
js_files=$(filter-out $(ignore_js),$(wildcard \
js/ll/*.js js/ll/**/*.js))
ignore_js=js/ll/dist% js/ll/%.min.js
%.min.js: %.js
#echo ">>> Uglifying $?"
#$(BABELJS) $(BABELJSFLAGS) $? | $(UGLIFYJS) --source-map $(UGLIFYJSFLAGS) > $#
min_js_files=$(js_files:%.js=%.min.js)
main.js: $(min_js_files)
#echo ">>> Concatenating JavaScript"
mkdir -p $(DIST_DIR)
cat $^ > $(DIST_DIR)$#
prod: main.js clean
The output I get from running make prod looks something like this:
>>> Uglifying js/ll/DateEx.js
SyntaxError: js/ll/DateEx.js: Invalid number (22:36)
20 | day = today.getDate();
21 | }
> 22 | return new Date(year, month, day, 01, 0, 0);
| ^
23 | }
24 |
25 | function newDateS(s)
>>> Uglifying js/ll/Anonymization.js
>>> Uglifying js/ll/DummyStorage.js
(...)
I have the impression that this happens because the steps run in parallel, but I know nothing about Make to back that up. How can I have compilation stop when one of the steps returns non-zero?
What you are asking for is the well-established default behavior of Make. Something in your build chain is not properly setting a nonzero exit code on failure, or you are masking it out.
In particular, the exit code from a shell pipeline is always the exit code from the final command in the pipeline. In other words, any error from BABELJS in your recipe will be lost.
Maybe refactor to not use a pipe, perhaps something like this:
%.min.js: %.js.tmp
$(UGLIFYJS) --source-map $(UGLIFYJSFLAGS) <$< >$#
%.js.tmp: %.js
$(BABELJS) $(BABELJSFLAGS) $< >$#
.PHONY: clean
clean:
rm *.js.tmp
The use of a temporary file is a bit of a wart, and the choice whether to use a separate recipe for the first step is kind of arbitrary. I prefer this style because it better adheres to the spirit of Make (explicitly declare dependencies, let Make keep track) but if you want fine-grained control over the flow, you could go the other way.
I also removed the # prefix from the BABELJS rule. This is a bit of a private crusade of mine -- littering your Makefile with these makes debugging hard, and the proper solution if you want peace and quiet is to use make -s.
Related
I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.
DEPRECATED_CHECK := $(shell grep "test454" tex/*.tex)
ifneq ($(DEPRECATED_CHECK), )
$(warning \test454 is deprecated. Use \test2 instead)
endif
When I run this I get:
../common/Makefile.include:133: \test454 is deprecated. Use \test2 instead
That's fine, but I'd quite like to have only:
\test454 is deprecated. Use \test2 instead
Is this possible? Some sort of awk function? I think I need something with:
#echo \text454 is deprecated ...
But I don't know how to get this working with the basic purpose of my MWE, as it keeps complaining about missing separators.
Many thanks
You could use $(info ...) instead of $(warning ...). info doesn't prepend the file and line number.
just an aside -- I usually try to do those sort of checks as part of a sanity rule, and make everything depend on that rule instead of doing it at the top level. It gives you more flexibility that way. For example, if you didn't want to run the check when building clean, it becomes simple, or if you wanted to fail the build if a check failed, it becomes simple as well.
EDIT (adding more detail on aside)
Instead of doing an ifneq at the top level of make, you could add a target as so:
sanity_check:
# ! grep -q "test454" tex/*.txt || echo "test454 is depricated"
.PHONY: sanity check
The add dependencies of your main targets to sanity check:
all maintarg1 maintarg2: sanity_check
This way the sanity check will be run before any of your main targets, and will output as desired. This is in my opinion, a cleaner way of doing the test. This way the test is only run if you are building any of your targets, and will not be run, if for example you are making clean, or if your makefile was included by a parent makefile, or in a bunch of other corner cases that might pop up in the future.
Just a quick note on the recipe syntax: the # is a make directive that tells make not to echo the command as it's run. The ! is bash syntax to inverse the return of grep (so ! grep returns false if the text is found, thereby causing the || part of the statement to be evaluated.). The .PHONY: sanity_check tells make to run the rule, even if a file called sanity_check already exists
I have a Makefile that is organizing several steps of analysis that have to be run in a particular order. The analysis takes quite a while (a day or two) and I'd like to receive some email notifications when make completes. Is there a good way to have make automatically send an email at the end of the process so I can be alerted when it completes, especially when there is a failure with one of the steps?
I'm currently doing something like this:
# Makefile
all: results1.dat results2.dat results3.dat
python send_email_when_done.py
results1.dat: long_running_program1.py
python $< > $# # this takes ~12 hours
results2.dat: long_running_program1.py results1.dat
python $^ > $# # this takes ~2 hours
results2.dat: long_running_program1.py results2.dat
python $^ > $# # this takes ~30 hours
where the send_email_when_done.py script sends email notifications when the process has completed. But this only works if the entire process has run from start to finish without any errors. Any suggestions for a good way to do this?
+1 for suggestions that can accomplish this within the Makefile. I'm already running make in a separate session using setsid make > make.out 2>&1.
How about simply make the email sending as a separated target:
report: long_running_program1.py
python $< > $# # this takes ~12 hours
sendmail:
python send_email_when_done.py
And you call it:
make report sendmail
or
make report ; make sendmail
Let me make it more general so that it can send email for 100's of other such make. Use alias and add it in bashrc, here sagarsakre.blogspot.in is how i tried to use the alias command makes (make and send report). The added advantage would be u would get all the verbose printed regardless of worrying about rule and dependency. You can use it for all builds without really changing the makefile.
note: i used mailx, You can read this to setup mailx on ur machine
Since makepp allows you access to the underlying interpreter, you can do a little Perl programming to add an END-Handler anywhere in your makefile:
perl { END { system "python send_email_when_done.py" }}
Or more prettily layouted with an alternate direct subject-only mail
perl {
END {
system "mail -s 'Build finished with rc=$?' me#my.home <&-";
}
}
There is much more to makepp. Besides doing almost all that GNU make can, there are lots more useful things.
One way of obtaining that, would be to instruct make to ignore errors from the long-running programs. This can either be achieved by running make -i or by prepending each of the commands where failure should be ignored with a dash and just running plain make:
# Makefile
all: results1.dat results2.dat results3.dat
python send_email_when_done.py
results1.dat: long_running_program1.py
-python $< > $# # this takes ~12 hours
results2.dat: long_running_program1.py results1.dat
-python $^ > $# # this takes ~2 hours
results2.dat: long_running_program1.py results2.dat
-python $^ > $# # this takes ~30 hours
See the GNU make manual for more information on ignoring errors.
What's the pattern to follow when specialized Makefiles in a directory depends on the main one in a parent dir?
i have:
/
/Makefile
/src
/src/Makefile
/tests
/tests/Makefile
in /Makefile i have:
TESTING_COMMAND=something
dotest1:
make -C tests/ $#
in /tests/makefile i have
dotest1:
$(TESTING_COMMAND) $?
if i run:
me#host:/ $ Make dotest1
it works. but if i execute from the tests dir:
me#host:/tests/ $ Make dotest1
it will try to execute the test file in the shell, because $(TESTING_COMMAND) is empty, so it's first argument became the command passed to the shell.
I don't necessarily need that to work if executed in the /tests/ or /src/ dir, but need a way to gracefully fail.
Trying to send everything through the command line (or environment) seems like a bad idea to me. That's what inclusion was invented for. Put your common values into a separate file, something like config.mk, then in all your makefiles just use:
include config.mk
to get them included.
Your design scares me, but this will do the trick in the main Makefile:
TESTING_COMMAND=something
dotest1:
make -C tests/ $# TESTING_COMMAND=$(TESTING_COMMAND)
If you want tests/Makefile to fail well, you have a couple of options. If only that one target depends on TESTING_COMMAND, you can have it print a warning and do nothing:
ifdef TESTING_COMMAND
dotest1:
$(TESTING_COMMAND) $?
else
dotest1:
#echo warning: TESTING_COMMAND not defined
endif
Or if the whole Makefile depends on it, you can have Make print a warning or abort:
ifndef TESTING_COMMAND
$(warning TESTING_COMMAND is undefined, but Make will try to us it anyway)
$(error TESTING_COMMAND is undefined, Make will now abort)
endif
You can also have it abort the sub-make (the one that runs tests/Makefile) but still continue running the Make process that invoked it, but that's kind of a pain.
I am trying to build some simulation software using makefile provided by them after I have made some changes to the libraries. But when I run make clean, it stops midway and I get the following error
rm: invalid option -- 'l'
Try `rm --help' for more information.
make: *** [neat] Error 1
I checked the man page for rm and there is no -l option, but I don't know why this command is being executed with -l option. Is there anyway to ignore this, or find out which specific file is causing the problem?
EDIT:
I have figured out the source of the error, but dont know how to edit it to make it work properly. Below is a snippet from an included Makefile with the faulty line:
UDP_INTERFACE_SRCS = \
$(UDP_INTERFACE_DIR)/interfaceudp_app.cpp \
$(UDP_INTERFACE_DIR)/interfaceudp.cpp \
$(UDP_INTERFACE_DIR)/external_interface_udp.cpp \
$(UDP_INTERFACE_DIR)/packet_send.cpp \
$(UDP_INTERFACE_DIR)/addr.cpp \
$(UDP_INTERFACE_DIR)/packet_capture.cpp -lpcap \
$(UDP_INTERFACE_DIR)/queue.cpp
In particular, the line: $(UDP_INTERFACE_DIR)/packet_capture.cpp -lpcap \
is causing the error. What does the "-lpcap" added after "packet_capture.cpp" do? Now if I try to remove it, "make" gives an error saying:
./interfaces/extinterface/src/packet_capture.o: In function pcap_sniff_packets(void*)': /home/qualnet/4.5/main/../interfaces/extinterface/src/packet_capture.cpp:63: undefined reference to pcap_setdirection' make: *** [../bin/qualnet] Error 1
I checked the line number 63 in packet_capture.cpp in an effort to understand what -lpcap means. But I have no idea what that code does.
(This is clearly an iterative process, and the comments are getting long, so I'd better start an answer.)
You say: "When I put echo SIM_JOBS: $(SIM_OBJS) in the rule, i get the following when i run make clean: rm -f ../bin/qualnet ../bin/radio_range"
This doesn't make sense. You should get something like
SIM_JOBS: ../bin/qualnet ../bin/radio_range
rm -f ../bin/qualnet ../bin/radio_range
or
SIM_JOBS: something/else
rm -f ../bin/qualnet ../bin/radio_range
or at least
SIM_JOBS:
rm -f ../bin/qualnet ../bin/radio_range
This suggests that you are looking at the wrong rule: the rule which produces rm -f ../bin/qualnet ../bin/radio_range is not the rule in which you put the echo ... command. If it is the rule and you were just being imprecise in the report, put this right above the clean rule:
SIM_JOBS=../bin/qualnet ../bin/radio_range
and tell us what happens.
EDIT:
Sorry, I wrote `SIM_JOBS` when I meant `SIM_OBJS`.
The `echo` command was outside the rule, where it cannot work.
It looks as if the problem. Could you edit your question to show the line you mentioned to #thiton, and a few previous lines? It looks as if the flag "-lpcap" is getting into a variable, where it doesn't belong.
EDIT:
The "-lpcap" is a kludge. My guess is that it's an option intended for the linker. Suppose you want to link a bunch of object files into an executable, and you want to search a certain library called "pcap":
gcc foo.o -lpcap bar.o baz.o
The order is very important; when the linker is searching for something, you want it to search foo.o first, then pcap, then bar, then baz. It's a question of precedence. But you want to store those filenames in a nice tidy variable, and how will you insert the lpcap at the right place? You could do it a good way, or use a lazy hack like this:
OBJECTS = foo.o -lpcap bar.o baz.o
gcc $(OBJECTS)
And if you're deducing the OBJECTS from the SOURCES, you have to put the hack in earlier:
SOURCES = foo.cc -lpcap bar.cc baz.cc
OBJECTS = $(SOURCES:.cc=.o)
Whoever wrote these makefiles saved half an hour with this kludge, and it's taking you days to fix it. If you can confirm that this is what's happening, the easiest way is probably to split the list in two:
SOURCES_LEFT = foo.cc
SOURCES_RIGHT = bar.cc baz.cc
OBJECTS_LEFT = $(SOURCES_LEFT:.cc=.o)
OBJECTS_RIGHT = $(SOURCES_RIGHT:.cc=.o)
gcc $(OBJECTS_LEFT) -lcap $(OBJECTS_RIGHT)
Like #AndrejPanjkov noted in the comments, the standard tool to find out what exactly is going on in Makefiles is the -n (--dry-run) switch of make. It prints all commands as they would be run by the shell, even those normally silenced (e.g. via #).
Have a long, hard look in this output for any pattern like ' -l' (piping the output to less and using / helps). If that doesn't help, try running all commands output by -n by hand in make's shell (normally /bin/sh) or try removing all the silencers (# characters at a rule's start) from the Makefile and running make clean, checking the last line of output.