RAII-like action in makefile rule - makefile

I'm trying to find a good way to add some RAII-like actions into a makefile I'm maintaining. Currently, I have something similar to this:
out: in
lockfile in.lock
echo in // Some action which can fail
rm -f in.lock
This code works fine when using multiple jobs, as it is mainly meant sanity instead of performance. At least, if my action is not failing. So if I like to add a fallback to this. So in short, it'll look something like:
out: in
lockfile in.lock
(echo in) || (rm -f in.lock; false)
rm -f in.lock
Yet again looking good, though I don't like having to write twice rm -f in.lock, nor does the (echo in) looks elegant if the actual content is several lines of bash-script.
This would look similar to:
out: in
lockfile in.lock
trap "rm -f in.lock" EXIT; \
(echo in)
However, this would make the actual rules look more complex if you have different rules which are really separate.
out: in
lockfile in.lock
trap "rm -f in.lock" EXIT; \
$(SHOW_DEPENDENCY_ON_DEBUG) && \
(echo in)
Where SHOW_DEPENDENCY_ON_DEBUG can be defined as echo $# <=== $^ in certain circumstances and # in others. So I'm not that sure if I can nicely chain all commands. Therefore I hope any of you know of some tricks that I've missed.
In short, I like to transform:
out: in
lockfile in.lock
echo in // Some action(s) which can fail
rm -f in.lock
In a way that always executes rm -f in.lock, without having to chain bash-commands or duplicating the action(s) that have to be executed to finalize the actions in the rules.

For the problem of ensuring the your lockfile (or any file that make makes) is deleted come what may,
make has a stock solution: make it an .INTERMEDIATE target.
Then, if make creates the file, it will auto-delete it at the end, come what may, e.g.
Makefile
.PHONY: all clean
all: out
in:
touch $# # Whatever
.INTERMEDIATE: in.lock
%.lock: %
touch $# # Whatever
out: in.lock
if [ "`shuf -i 0-1 -n 1`" = "0" ]; then echo Fail; false ; else echo Succeed; touch $#; fi
rm -f $<
clean:
rm -f in out
Here the command:
if [ "`shuf -i 0-1 -n 1`" = "0" ]; then echo Fail; false ; else echo Succeed; touch $#; fi
will fail or succeed on a pseudo-random coin-toss.
Some runs:
$ make
touch in # Whatever
touch in.lock # Whatever
if [ "`shuf -i 0-1 -n 1`" = "0" ]; then echo Fail; false ; else echo Succeed; touch out; fi
Succeed
rm -f in.lock
$ make clean
rm -f in out
$ make
touch in # Whatever
touch in.lock # Whatever
if [ "`shuf -i 0-1 -n 1`" = "0" ]; then echo Fail; false ; else echo Succeed; touch out; fi
Fail
Makefile:14: recipe for target 'out' failed
make: *** [out] Error 1
rm in.lock
But don't push this feature so far as removing the
rm -f $<
from the recipe. make will delete the intermediates at exit, which is fine if the recipe fails.
But if the recipe succeeds you presumably want the lockfile deleted right away rather than when
make finishes, which might be arbitrarily later.
Later
Any chance the .INTERMEDIATE can refer to wildcard, like %.lock?
No. You'd have to mean:
.INTERMEDIATE: %.lock
and there is no "wildcard" there. With no % in lefthand side,
it's not a pattern-rule and % in the righthand side only just means %.
But you don't need this. You must know the names of the prerequisites you
want to lock or at least be able to compute them with makefunctions.
Otherwise you can't write the makefile at all. So say they are ina inb inc.
Then you make all the locks intermediate like:
inputs := ina inb inc
locks := $(patsubst %,%.lock,$(inputs))
.INTERMEDIATE: $(locks)

Related

Make with files that must be edited manually

I'm writing some Makefile. There is a file auto that can be processed in automatic way (build by make). There is recipe for it. So, this file needs to be edited manually and saved as manual. This manual is used for further automatic processing. How can I achieve the next:
If auto is not exists --- build it --- done by the recipe for it.
If manual is older than auto --- print a message and exit make.
If manual is newer --- build further --- done by the recipe for it.
How can the second step be done in some native way?
EDIT
The process description was not clear enough because I had no understanding how it should be. That was my mistake. Here is the process I assume.
There is source.
The auto can be built automatically from source.
The manual can be made manually from auto.
The result can be built automatically from manual.
Going 1->2 and 3->4 is what is make for.
The manual depends on source. But make itself can't build manual but only auto to help the user. So, if the manual is up-to-date (newer than source), then just build result from it.
But if manual is have to be rebuilt (it is older than source), rebuild auto (if needed), print a message and stop.
So, I think that Makefile should looks like:
SOURCE = source
result : manual ; cp manual result
manual : auto $(SOURCE) ; #echo "CREATE $#"
auto : $(SOURCE) ; cp source auto
But making result I have to stop after manual and let user to create it and run make again. The manual will be newer than source and auto and will not have to be rebuilt. But how can I do this?
The easiest is probably to describe your dependencies as they really are, even if you cannot really build manual with a make recipe. Then, for manual, instead of writing a building recipe, just print your message and exit with a non-zero status. If manualdoes not exist or is out of date, make will print the message and stop before trying to (re)build result:
# Makefile
result: manual
cp -f "$<" "$#"
manual: auto
#echo "CREATE $#"
#exit 1
auto: source
cp -f "$<" "$#"
Demo:
$ rm -f auto manual result source
$ touch source
$ make
cp -f "source" "auto"
CREATE manual
make: *** [Makefile:6: manual] Error 1
$ touch manual
$ make
cp -f "manual" "result"
$ make
make: 'result' is up to date.
$ touch source
$ make
cp -f "source" "auto"
CREATE manual
make: *** [Makefile:6: manual] Error 1
If you don't like the error situation you can defer the stop to the result recipe:
# Makefile
result: manual
[ -f "$#" ] && cp -f "$<" "$#" || true
manual: auto
#echo "CREATE $#"
auto: source
cp -f "$<" "$#"
Demo:
$ rm -f auto manual result source
$ touch source
$ make
cp -f "source" "auto"
CREATE manual
[ -f "manual" ] && cp -f "manual" "result" || true
$ touch manual
$ make
[ -f "manual" ] && cp -f "manual" "result" || true
$ make
make: 'result' is up to date.
$ touch source
$ make
cp -f "source" "auto"
CREATE manual
$ make
CREATE manual
$ touch manual
$ make
[ -f "manual" ] && cp -f "manual" "result" || true

How to separate different parts with exit control in Makefile

I am writing a complicated Makefile which have multiple parts like:
step1:
if [[ ${input} == "delete" ]]; then \
echo "this is a test to delete files"; \
else \
echo "error stop"; \
fi
step2:
rm -f *.txt
test:
make step1
make step2
So if I enter "make test input=delete", it will delete all the .txt files. And if I enter "make test input=none", it would not do anything. I know the simplest way is to combine step1 and step2 as:
test:
if [[ ${input} == "delete" ]]; then \
echo "this is a test to delete files"; \
rm -f *.txt; \
else \
echo "error stop"; \
fi
But my Makefile is so complicated that I have to separate into several parts. Does Makefile support similar features? If yes, what should I look for? Thanks.
You should always use $(MAKE) and never use make directly when invoking a sub-make.
If you do that, then all command-line overrides you provide will be correctly passed to the sub-makes.
BTW, you should not use the bash syntax [[ x == y ]]. If you run this makefile on a system where the default shell is limited to POSIX standard syntax this won't work. You should use POSIX syntax: [ x = y ].
As you use GNU Make you can use its conditionals, e.g.
ifeq:
test:
ifeq ($(input),delete)
echo "this is a test to delete files"
rm -f *.txt
else
echo "error stop"
endif
Or, if you want to control the prerequisites of a target:
step1:
echo "this is a test to delete files"
step2:
rm -f *.txt
# test always depends on step1
test: step1
# test also depends on step2, if input = delete
ifeq ($(input),delete)
test: step2
endif

Running mkdir -p in parallel under Mac OS X, quits before directory is created

I have a Makefile with multiple jobs.
Each job does a mkdir -p $(OBJDIR), to ensure the object directory exists before running gcc.
However, the mkdir -p sometimes does not ensure the directory exists before returning. I assume there is some sort of race condition going on. And then the compiler fails, as it can't write to the output file.
Is there a platform agnostic way of doing the mkdir -p so that this race condition doesn't occur?
This failure seems to only happen under OS X.
Question got sent to Stack Overflow, but really a sysadmin question.
# script named 'test'
if [ "$1" == "run" ] ; then
rm -rf XXX
NOW=`date +%s`
echo Now: $NOW
THEN=`echo "$NOW + 10" | bc`
RUNAT=`date -r $THEN +"%m%d%H%M.%S"`
echo $RUNAT
for i in {1..75}; do
./test $THEN &
sleep 0.05
done
wait
exit 0
fi
THEN=$1
# make sure things are cached
NOW=`python -c'import time; print repr(time.time())'`
NOW=`python -c'import time; print repr(time.time())'`
AMOUNT=`echo "$THEN - $NOW" | bc`
echo sleeping $AMOUNT
sleep $AMOUNT
mkdir -p XXX/YYY/ZZZ
touch XXX/YYY/ZZZ/hi
I wrote the above bash script, but it doesn't produce the predicted error. I will put the touch in the makefile, perhaps it will elucidate what is going on.
The makefile's CC statement looks like so:
$(OBJPATH)/%.o : %.cpp
mkdir -p $(dir $#)
#sleep 0.1
$(CPP) $(INCPATH) $(_FLAGS) $(CPPFLAGS) $< -o $#
The #sleep statement is there to possibly wait for the race condition to fix itself, but perhaps there is no race condition after all.

Make: .DELETE_ON_ERROR for directory targets

GNU Make includes a special target called .DELETE_ON_ERROR. If this is included in your Makefile, Make will delete any target whose build sequence completes with a non-zero return status. This is helpful so that in subsequent invocations Make does not assume that the target has been properly built.
Here's a dummy example.
.DELETE_ON_ERROR:
out.dat: in.dat
touch out.dat
false
Because false gives a non-zero return value, the build is considered failed and Make deletes the out.dat target. This is the advertised and expected behavior. However, this behavior does not seem to be preserved when the target is a directory. Consider another dummy example.
.DELETE_ON_ERROR:
outdir/: in.dat
mkdir outdir/
false
In this case, the build fails again but Make does not remove the outdir directory. Is there any way I can instruct Make to do this?
As noted in the comments, it is hard to use timestamps on directory. Few options:
proxy target (%.dir)
Atomic update using temporary folder.
Using proxy target, Makefile can be modified to incude a '%.done' target, which will embed the cleanup logic.
.PHONY: %.dir
outdir.dir:
$(MAKE) outdir ; if [ $? -ne 0 ] ; then echo CLEANUP $# ; rm -rf dir ; false ; fi
outdir: ... # as before
And use the outdir.dir as a dependency. Not elegant, but will get the work done. May be possible to to convert into a rule (disclaimer: I did not test this approach).
.PHONY %.dir
%.dir:
$(MAKE) $* ; if [ $? -ne 0 ] ; then echo CLEANUP $* ; rmd -rf $* ; false ; fi
Another variation is to change the outdir to add a "done" indicator file (if completed successfully), and use the proxy target to validate
%.dir:
$(MAKE) $* ; if [ ! -f $*.done ] ; then rm -rf $* ; false ; fi
outdir:
... commands, any can fail.
touch $*.done
As last resort (or first option, depending on your situation), consider, 'atomic' build for outdir - creating a temporary folder, and renaming it to outdir on success
outdir:
rm -rf $#.new $#
mkdir $#.new
# Command to create outdir.new here
mv $#.new $#

How to ignore mv error?

I'm making a Makefile that moves an output file (foo.o) to a different directory (baz).
The output file moves as desired to the directory. However since make won't recompile the output file if I type make again, mv gets an error when it tries to move the non-existent empty file to the directory baz.
So this is what I have defined in my rule make all after all compilation:
-test -e "foo.o" || mv -f foo.o ../baz
Unfortunately, I'm still getting errors.
Errors in Recipes (from TFM)
To ignore errors in a recipe line, write a - at the beginning of the
line's text (after the initial tab).
So the target would be something like:
moveit:
-mv foo.o ../baz
I notice nobody has actually answered the original question itself yet, specifically how to ignore errors (all the answers are currently concerned with only calling the command if it won't cause an error).
To actually ignore errors, you can simply do:
mv -f foo.o ../baz 2>/dev/null; true
This will redirect stderr output to null, and follow the command with true (which always returns 0, causing make to believe the command succeeded regardless of what actually happened), allowing program flow to continue.
+#[ -d $(dir $#) ] || mkdir -p $(dir $#)
is what I use to silently create a folder if it does not exist. For your problem something like this should work
-#[ -e "foo.o" ] && mv -f foo.o ../baz
-test -e "foo.o" || if [ -f foo.o ]; then mv -f foo.o ../baz; fi;
That should work
Something like
test -e "foo.o" && mv -f foo.o ../baz
should work: the operator should be && instead of ||.
You can experiment with this by trying these commands:
test -e testfile && echo "going to move the file"
test -e testfile || echo "going to move the file"
I faced the same problem and as I am generating files, they always have different time. Workaround is set the same time to the files: touch -d '1 June 2018 11:02' file. In that case, gzip generates the same output and same md5sum. In my scenario, I don't need the time for the files.

Resources