In this project, a top level rule looks like this:
all:
for l in $(LIBDIRS); do $(MAKE) -C $$l all; done
If I run make -jx, the files in each directory are compiled in parallel, but because of this bash loop, make doesn't move onto files in the next library until everything is done in the current one. What's the best way to change the loop so that it call truly be done in parallel? I only use gnumake, so gnu extensions are fine.
* Edit * as an aside, I was trying something like this
ALL_DEPENDENCY = $(foreach l, $(LIBDIRS), $(l).PHONY_LIB_RULE)
.PHONY: $(ALL_DEPENDENCY)
%.PHONY_LIB_RULE:
$(MAKE) --no-print-directory -C $(patsubst %.PHONY_LIB_RULE,%,$#) all
all: $(ALL_DEPENDENCY)
but these rules aren't tripped.
You can check this answer for an example of how to do this more correctly: makefile: foreach "make -C" call
Related
I have
$(foreach ___project___, $(UNIT_TEST_STUBS),$(MAKE) -C ../../$(___project___) $(UT_CMD) || exit 1;)
I want make to be parallel INSIDE each submake but I don't want the submakes to be executed in parallel.
How do I do this?
As #Michael Livshin points out, you already seem to have answered your own question.
Might be neater to rely on make rather than shell syntax to tie a load of commands together though.
Basically you want make to see something like this:
.PHONY: all
all:
${MAKE} -C ../../foo/ -j9 ut-cmd
${MAKE} -C ../../bar/ -j9 ut-cmd
${MAKE} -C ../../bum/ -j9 ut-cmd
Auto generation from a list is fairly straight-forward.
submakes := foo bar bum
define generate-submake
${MAKE} -C ../../$1/ ut-cmd
endef
.PHONY: all
all: ; $(foreach _,${submakes},$(call generate-submake,$1))
Note the blank line in the definition of generate-submake. It's important.
Run this with make -j9.
If you really want everything in this makefile to run serially (but the sub-makes to be parallel), then just introduce a .NOTPARALLEL target.
.NOTPARALLEL:
I have the following directory tree
moving_files/
Makefile
source/
a
b
c
target/
With my Makefile I want to cp every file in source/ to target/.
The catch: I want to be able to move other files to source/ afterwards without having to edit the Makefile. For this purpose I have written this:
FILES = $(filter-out Makefile, $(wildcard source/*) )
all: $(subst source,target,$(FILES))
$(subst source,target,$(FILES)): $(FILES)
cat $< >| $#
And it works fine.
However, when I execute touch source/d afterwards and make once again, in addition to d, a, b and c get cated, as well. What do I have to do in order to change this behavior.
$(subst source,target,$(FILES)): $(FILES)
expands to
target/a target/b target/c: source/a source/b source/c
which means that each single target depends on all files in source, probably not what you intended. Either a static or an implicit rule can fix this, static rules are generally better as they are more specific:
$(subst source,target,$(FILES)): target/%: source/%
In my project, I have a set of programs that are build from sources:
SRC_FILES = $(wildcard $(SRC_DIR)/*.cpp)
TARGETS = $(patsubst $(SRC_DIR)/%.cpp,$(BIN_DIR)/%,$(SRC_FILES))
My build target is simple, and works fine:
all: $(TARGETS)
#echo "- Done target $#"
Now, I want a run target so that all these programs are run from the shell on request. Say, if I have 3 files, I want make to run automatically:
>$ ./test1
>$ ./test2
>$ ./test3
Or
>$ ./test1 && ./test2 && ./test3
I tried this:
run: $(TARGETS)
$(addsuffix && ,$(TARGETS))
That generates the following command:
./test1&& ./test2&&
but it fails, due to the trailing &&
(Of course, I want these to be generated automatically as there can be 3... or 30.)
Edit: actually, the && separator is not required, so something like this:
>$ ./test1; ./test2; ./test3;
will be fine too.
Have some .PHONY line near start of Makefile with
.PHONY: all run
You might have
run: $(TARGETS)
$(addsuffix && ,$(TARGETS)) true
but it is a dirty trick.
Maybe you want to produce the output of test2 into test2.out then you might have
TESTSCRIPTS= $(wildcard test*[0-9])
run: $(patsubst %, %.out, $(TESTSCRIPTS))
test%.out: test%
# here some command to run the test%
As alternatives to Basile Starynkevitch's entirely correct answer here there are (at least) two other options as well.
You can avoid the need to run an unnecessary command (builtin though it might be) to end the list by manually pulling off the first entry (this may in fact be more costly then the shell builtin though).
run: $(TARGETS)
$< $(addprefix &&,$(wordlist 2,$(words $^),$^))
A better option I think, assuming that connecting the commands with && isn't a necessity would be to use $(foreach) to generate the command to be run.
run: $(TARGETS)
$(foreach t,$^,$t;)
The trailing ; in that is crucial as the output from $(foreach) is a single line and you need ; to terminate each shell command (or it is seen as one long command with arguments).
I need to work on a system that uses automake tools and makes recursively.
'make -n' only traces the top level of make.
Is there a way to cause make to execute a make -n whenever he encounters a make command?
Use $(MAKE) to call your submakefiles, instead of using make. That should work. Check out How the MAKE variable works in the manual. Here's a quick example:
Makefile:
all:
#$(MAKE) -f Makefile2
Makefile2:
all:
#echo Makefile2
Command line:
$ make
Makefile2
$ make -n
make -f Makefile2
echo Makefile2
$
Does your recursive makefile look like this:
foo:
make -C src1
make -C src2
Or like this:
foo:
${MAKE} -C src1
${MAKE} -C src2
I think you need to use the second style if you want flags passed to child make processes. Could be your problem.
Setting the environment variable "MAKEFLAGS" to "n" may do what you need.
There are some more advanced tricks for tracing make commands here:
http://www.cmcrossroads.com/ask-mr-make/6535-tracing-rule-execution-in-gnu-make
The simplest of these tricks comes down to adding SHELL="sh -x" to your make command (running without "-n" in that case).
I am looking for an elegant way for the parallelization of jobs in GNU make. Here is a sample of what I did so far. Make processes the directories dir-1, dir-2 and dir-3 in a serial fashion which is logical but not my intention:
SUBDIRS=dir-1 dir-2 dir-3
default: all
all:
#for dir in $(SUBDIRS); do (cd $$dir; $(MAKE)); done
.PHONY: clean
clean:
#for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) clean); done
Is there a way to support parallel processing of these directories using the "-j" option without specifying specific targets for each directory?
SUBDIRS = a b c
default: all
$(SUBDIRS)::
$(MAKE) -C $# $(MAKECMDGOALS)
all clean : $(SUBDIRS)
This probably will not answer your question directly, but besides what the other answers suggest, I would recommend to look into non-recursive make techniques. That is truly an elegant way to parallelize build, although, depending on what are the existing Makefiles, can require significant effort. Non-recursive make has advantages not limited to easy parallelization: it sees a complete dependency graph and so it does not need to build too little or too much, meaning faster (sometimes much faster) build times.
Some resources:
Discussion on SO
The classic text about trouble with recursive make
Design of non-recursive makefile
Implementation
Another implementation
Benchmarks (quite outdated though)
I found that when using the :: (double-colon rule), I could not find a way to force order if there were any dependencies among app, lib, and doc. After reading much of the GNU Make manual, I came up with the following rules to enforce the dependencies at the same time as having a generic recursive make. Notice that the .PHONY rule is there to force make to enter the directory even though the directory already exists.
SUBDIRS = app lib doc
default: all
app: lib
.PHONY: $(SUBDIRS)
$(SUBDIRS):
$(MAKE) -C $# $(MAKECMDGOALS)
all clean install: $(SUBDIRS)
Are dir-1, dir-2 and dir-3 interdependent or independent?
I have a similar structure but dependence between the subdirectories so with that I preferred to just use parallel builds within each of the subdirectories. You'd get that via
## default to four parallel runs
MAKEFLAGS += -j 4
all:
#for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) ); done
But another trick is to read up on SUBDIRS in the make manual -- you do not need the for loop as make can unroll this for you. Try something like
## default to four parallel runs
MAKEFLAGS += -j 4
SUBDIRS = dir-1 dir-2 dir-3
$(SUBDIRS): #whatever your depends here
$(MAKE) -C $#