I have a Makefile as below, and I am concerned with the dependency order for up-clean:
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean up
.PHONY: down
down:
docker-compose down
.PHONY: down-clean
down-clean:
docker-compose down -v
Obviously it is important that in up-clean: down-clean up, down-clean must be executed before up. For ordinary make targets, the solution would be to add an entry, up: down-clean, but as these are PHONY targets, that would make up functionally equivalent to up-clean, removing volumes every time. Obviously, this is unacceptable.
In practice, GNU make respects the order of dependencies, but it does not guarantee it, and so is not entirely trustworthy, or with -j not trustworthy at all. What methods, if any, exist to ensure the order of execution of dependencies in this situation without changing the result of building other targets?
EDIT: Attempting to use order-only prerequisites does not appear to work, possibly because of an interaction with .PHONY. Adding
up: | down-clean
Causes the execution log to be:
$ make up
docker-compose down
<...>
docker-compose down -v
Removing volume <...>
Which is what is supposed to happen for normal prerequisites, not order-only ones.
The simplest answer is to use recursive invocations of make:
up-clean:
$(MAKE) down-clean
$(MAKE) up
Another alternative would be to model up-clean on up instead of making the latter a prerequisite for the former:
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean
docker-compose up -d
If you want to make that a little DRYer, you could factor out the docker-compose command to a variable:
UP_COMMAND = docker-compose up -d
.PHONY: up
up: down
$(UP_COMMAND)
.PHONY: up-clean
up-clean: down-clean
$(UP_COMMAND)
Starting with GNU make 4.4 you can explicitly serialize your prerequisites with the .WAIT pseudo-target, e.g.,
.PHONY: up-clean
up-clean: down-clean .WAIT up
Before 4.4, the GNU make documentation was avoiding making any commitments about the order or execution to enable parallel execution. However, any POSIX-compliant implementation of make (emphasis mine)
... shall treat all prerequisites as targets themselves and recursively ensure that they are up-to-date, processing them in the order in which they appear in the rule. The make utility shall use the modification times of files to determine whether the corresponding targets are out-of-date.
Of course, in the parallel mode, GNU make can't be fully compliant with this requirement.
But, the 4.4 release also adds the --shuffle option, and from the research made by the implementor of this option, it is evident that the only source of non-determinism is the -j option and the parallel mode of execution, which is also witnessed by the contents of the patch and the tests, which were reviewed by other members of the GNU make project.
Therefore, in version prior to 4.4, we can safely assume that the prerequisites are executed strictly in the order in which they are specified, from left to right, as long as make is executed in non-parallel mode. In versions before 4.4, we can disable parallelism with the .NOTPARALLEL pseudo target, just add it to your file, e.g.,
.NOTPARALLEL: # ensures that all deps are executed strictly in order
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean up
.PHONY: down
down:
docker-compose down
.PHONY: down-clean
down-clean:
docker-compose down -v
Notice also, that --shuffle respects the presence of the .NOTPARALLEL target, which corroborates our hypothesis the order could different from the syntactic order only because of the parallel execution.
Related
For a project spread over several sub-directories we use GNU make for the builds. Developers can use the -j <number> flag to parallelize the build tasks, and choose a number that suits the hardware of their machines.
However, the Makefiles of a third-party library that we use are not safe to parallelize - they apparently rely on implicit order of targets instead of explicit dependency rules between all dependent targets.
Since I have no desire to fix third-party Makefiles, we currently invoke their Makefiles with an explicit -j 1 parameter to restrict the number of jobs to 1 for building that library. The rule looks like this:
third_party_lib:
$(MAKE) -j 1 -C $#
This works as desired, however, make emits a warning for this:
make[1]: warning: -jN forced in submake: disabling jobserver mode.
which leads me to ask here if there is a better way to restrict the number of parallel jobs in one sub-make.
You can add the .NOTPARALLEL: special target to the makefiles which should not be parallelized.
If you don't want to modify those makefiles you can use the --eval option on the command line (note --eval was added in GNU make 3.82):
third_party_lib:
$(MAKE) --eval .NOTPARALLEL: -C $#
The GNU make manual says:
A phony target should not be a prerequisite of a real target file; if it is, its recipe will be run every time make goes to update that file.
What if that's what I want?
For example, what if I have a phony target called lint that lints app/scripts/main.js, and I want it to run every time make goes to update (transpile & minify) dist/scripts/main.js?
I just removed the lint target and put its recipe as the first command of the recipe for the dist/scripts/main.js target.
dist/scripts/main.js : app/scripts/main.js
./node_modules/.bin/eslint $< && \
mkdir -p dist/scripts && \
./node_modules/.bin/babel $< | \
./node_modules/.bin/uglifyjs - --screw-ie8 -o $# -m -c
Updated answer: the current GNU make manual does not specify the cited "rule" from the question but instead explicit says to make use of this in the entry about forced targets, which includes a note:
As you can see, using FORCE this way has the same results as using .PHONY clean.
Using .PHONY is more explicit and more efficient. However, other versions of make do not support .PHONY; thus FORCE appears in many makefiles
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I know this has been asked before, but please bear with me. run_test is the name of the file I would like to make. I prefer invoking as 'make run_test'. I want the prereq's makefile to be called each time I call make run_test. But I don't want the call to the prereq's make to force run_test to be rebuilt.
run_test: build_prereq prereq_dir/prereq_ts
rebuild_test.sh
.PHONY: build_prereq
build_prereq:
make -C prereq_dir
prereq_dir/prereq_ts is updated if make -C prereq (prereq_dir/makefile) detects that prereq's dependencies are old and prereq needs to be rebuilt.
The problem with this is that rebuilt_test.sh is always called, and I assume that is because one of run_test's dependencies is executed (as it is PHONY).
I have resolved this issue by changing run_test to:
run_test:: build_prereq
run_test:: prereq_dir/prereq_ts
rebuild_test.sh
.PHONY: build_prereq
build_prereq:
make -C prereq_dir
Is this a good solution? This first runs build_prereq, and then it runs rebuild_test.sh if prereq_dir/prereq_ts is updated (which would happen in build_prereq's make call if needed).
Is there a more proper way of doing this. I do not want to create additional targets as that doesn't allow a direct call of make ie 'make run_test'.
Make supports exactly what you want.
.PHONY: build_prereq
build_prereq:
${MAKE} -C prereq_dir
prereq_dir/prereq_ts: build_prereq ;
run_test: prereq_dir/prereq_ts
rebuild_test.sh
So, you ask make to build run_test.
To ensure prereq_dir/prereq_ts is up to date it first runs the
recipe for build_prereq.
Next it runs the recipe for prereq_dir/prereq_ts (which does nothing—see that ;).
Now it checks the time-stamp of prereq_dir/prereq_ts to decide whether run_test is out of date.
Perfect.
If you're willing to be non-portable, GNU make supports order-only prerequisites (see the GNU make manual) which do exactly what you want:
run_test: | build_prereq
build_prereq:
$(MAKE) -C prereq_dir
(note always use $(MAKE) when invoking sub-makes, never make)
I am using GNU make, where I have a top level makefile, which invokes
another makefile, for different types of builds, like:
LIST_OF_TYPES: 32 64 ...
tgt-name: deps
$(foreach i,$(LIST_OF_TYPES), \
$(MAKE) -f $(MY_MAKEFILE) ARCH=$i mylib;)
when running with higher j factor like -j100 etc, one of the build fails, but
the return value is still 0 so, I cannot make out if the build really did work!
Is there anything wrong with the way I'm using the foreach construct?
or its just the higher j with foreach which is causing problems?
The top-level makefile you have listed is safe under -j (though badly sub-optimal). After the expansion of the $(foreach...), make effectively sees:
tgt-name: deps
$(MAKE) -f $(MY_MAKEFILE) ARCH=32 mylib; $(MAKE) -f $(MY_MAKEFILE) ARCH=64 mylib; ...
When one of these sub-makes fails (due to mis-handling of -j), the failure is not reported to the top level make. You need to use something like:
tgt-name: deps
$(MAKE) -f $(MY_MAKEFILE) ARCH=32 mylib && $(MAKE) -f $(MY_MAKEFILE) ARCH=64 mylib && ... && :
The && tells bash to exit immediately with an error if the previous command fails. (The : at the end is the bash builtin that does nothing but issue a successful exit—it will simplify your writing of the $(foreach ...).)
EDIT:
The proper way to this of course is to use make dependencies, not serial processing in the shell. You want make to see something like:
tgt-name: # default target
.PHONY: target-32
target-32: deps
$(MAKE) -f ${MY_MAKEFILE} arch=32 mylib
.PHONY: target-64
target-64: deps
$(MAKE) -f ${MY_MAKEFILE} arch=64 mylib
# etc. etc.
tgt-name: target-32 target-64
#echo $# Success
This is -j safe. Under -j make will make all of the target-% at the same time. Nice. (Though in this case it seems that your $MY_MAKEFILE is not -j safe (naughty!).) A few macros to replace the boiler plate:
LIST_OF_TYPES := 32 64 ...
LIST_OF_TARGETS := $(add-prefix,target-,${LIST_OF_TYPES})
tgt-name: # default target
.PHONY: ${LIST_OF_TARGETS}
${LIST_OF_TARGETS}: target-%: deps # Static Pattern Rule will (conveniently) set $*
$(MAKE) -f ${MY_MAKEFILE} arch=$* mylib
tgt-name: ${LIST_OF_TARGETS}
#echo $# Success
P.S. I suspect that you should be marking tgt-name as .PHONY
I've never seen this kind of use of foreach, but it seems to work for you. Usually I use a bash for loop
tgt-name: deps
for i in $(LIST_OF_TYPES); do $(MAKE) -f $(MY_MAKEFILE) ARCH=$$i mylib; done
But this is not the problem, since in either case the makes are run sequentially, AFAICS.
Since you build a library, there's the possible clash of two objects being inserted into an archive simultaneously. When this happens the archive might become corrupted.
As with every parallel execution, be it make jobs or threads, you must protect the shared resources (the archive in your case). You must add the objects at the end of the library build or protect the insertion with some lock (e.g. man lockfile or similar).
There might be other problems, of course. Look out for the simultaneous access to shared resources (object files, archives, ...) or incomplete defined dependencies.
Update:
foreach seems not to be a problem. Set LIST_OF_TYPES to a single type (e.g. 32 only) and then do a make -j100 mylib. If the problem is with the building of a single archive, it will fail with only one type as well.
You can also test with make ARCH=32 -j100 mylib. This should show the problem too.