Consider a makefile
all : a c
(cmd3)
a : b1 b2 b3 b4
(cmd2)
b% :
(cmd) $*
Consider that the dependencies b% can be parallelized and run on grid engine.
What would be the simplest way to qsub these dependencies with minimal changes to the makefile so that cmd2 is executed only after all the dependencies are satisfied ?
Use something like this in your makefile:
b%:
qsub -sync yes (cmd) $*
And then use the -j option to make to run them in parallel. For example:
make -j all
Related
Thanks for all your time and response -
Currently, we are using the nested build, multiple Makefiles, and individual subdirectories having their own Makefile, all are connected with a top-level Makefile. We are running
make xxxxx_yyyy_defconfig
make
this builds and creates an output file which is xxxxx.elf file. --- Till here everything works fine.
Now we are having multiple def-configs(around 50), I want to build all configurations using one "make all" command. is that possible?
This is not a simple case where we can put all "all: prog01 prog02 prog03" as every program needs to have a different configuration. Configuration can be achieved by using "make xxxxx_yyyy_defconfig". The output of "make config" is the .config file, which is used during the "make" command.
Based on .config file many variables are exported which is used at the subdirectory level.
So How can I build multiple configurations using a single "make all" command?
Environment - Ubuntu, Cross compile for ARM, output file xxxx.elf.
With the use of script and make file I am able to solve, But I have to solve only using Makefile.
in Makefile add one PHONY target
all:
./build_all.sh #shell script calling.
Created one shell script like this
#! /usr/bin/bash
echo "Make All"
for entry in `ls conf`; do
make $entry
wait
make
if [ $? -eq 0 ] ; then
for xxxfile in `ls xxx*_*` ; do
xxxdir=$(echo $xxxfile | cut -b yy-zz)
mkdir -p $xxxdir
mv $xxxfile $xxxdir/
done
else
break
fi
done
If you want to build several configurations you must do this out of tree in separate build directories (make O=/tmp/builds/foo foo_defconfig; make -C /tmp/builds/foo) to avoid conflicts. A shell script could do this as well as a Makefile but if you insist on using a Makefile you could try the following that assumes your source tree is in /src/kernel and you want to build configuration foo in /tmp/builds/foo; adapt to your needs:
$ pwd
/tmp/builds
$ cat Makefile
CONFIGS := uuuu_vvvv xxxx_yyyy ...
BUILD := /tmp/build
KERNEL := /src/kernel
.PHONY: $(CONFIGS) all
all: $(CONFIGS)
$(CONFIGS):
rm -rf $#
mkdir -p $(BUILD)/$#
$(MAKE) -C $(KERNEL) O=$(BUILD)/$# $#_defconfig
$(MAKE) -C $(BUILD)/$#
$ make
I'd like to use make to process a large number of inputs to outputs using a script (python, say.) The problem is that the script takes an incredibly short amount of time to run per input, but the initialization takes a while (python engine + library initialization.) So, a naive makefile that just has an input->output rule ends up being dominated by this initialization time. Parallelism doesn't help with that.
The python script can accept multiple inputs and outputs, as so:
python my_process -i in1 -o out1 -i in2 -o out2 ...
and this is the recommended way to use the script.
How can I make a Makefile rule that best uses my_process, by sending in out of date input-output pairs in batches? Something like parallel but aware of which outputs are out of date.
I would prefer to avoid recursive make, if at all possible.
I don't completely grasp your problem: do you really want make to operate in batches or do you want a kind of perpetual make process checking the file system on the fly and feeding to the Python process whenever it finds necessary? If the latter, this is quite the opposite of a batch mode and rather a pipeline.
For the batch mode there is a work-around which needs a dummy file recording the last runnning time. In this case we are abusing make for because the makefile is in this part a one-trick pony which is unintuitive and against the good rules:
SOURCES := $(wildcard in*)
lastrun : $(SOURCES)
python my_process $(foreach src,$?,-i $(src) -o $(patsubst in%,out%,$(src)))
touch lastrun
PS: please note that this solution has a substantial flaw in that it doesn't detect the update of in-files when they happen during the run of the makefile. All in all it is more advisable to simply collect the filenames of the in-files which were updated by the update process itself and avoid make althogether.
This is what I ended up going with, a makefile with one layer of recursion.
I tried using $? both with grouped and ungrouped targets, but couldn't get the exact behavior needed. If one of the output targets was deleted, the rule would be re-run but $? wouldn't necessarily have some input files but not the correct corresponding input file, very strange.
Makefile:
all:
INDIR=in
OUTDIR=out
INFILES=$(wildcard in/*)
OUTFILES=$(patsubst in/%, out/%, $(INFILES))
ifdef FIRST_PASS
#Discover which input-output pairs are out of date
$(shell mkdir -p $(OUTDIR); echo -n > $(OUTDIR)/.needs_rebuild)
$(OUTFILES) : out/% : in/%
#echo $# $^ >> $(OUTDIR)/.needs_rebuild
all: $(OUTFILES)
#echo -n
else
#Recurse to run FIRST_PASS, builds .needs_rebuild:
$(shell $(MAKE) -f $(CURDIR)/$(firstword $(MAKEFILE_LIST)) FIRST_PASS=1)
#Convert .needs_rebuild into batches, creates all_batches phony target for convenience
$(shell cat $(OUTDIR)/.needs_rebuild | ./make_batches.sh 32 > $(OUTDIR)/.batches)
-include $(OUTDIR)/.batches
batch%:
#In this rule, $^ is all inputs needing rebuild.
#The corresponding utputs can be computed using a patsubst:
targets="$(patsubst in/%, out/%, $^)"; touch $$targets
clean:
rm -rf $(OUTDIR)
all: all_batches
endif
make_batches.sh:
#!/bin/bash
set -beEu -o pipefail
batch_size=$1
function _make_batches {
batch_num=$1
shift 1
#echo ".PHONY: batch$batch_num"
echo "all_batches: batch$batch_num"
while (( $# >= 1 )); do
read out in <<< $1
shift 1
echo "batch$batch_num: $in"
echo "$out: batch$batch_num"
done
}
export -f _make_batches
echo ".PHONY: all_batches"
parallel -N$batch_size -- _make_batches {#} {} \;
Unfortunately, the makefile is a one trick pony and there's quite a bit of boilerplate to pull this recipe off.
Say I have a simple Makefile.
a1: some_script b
some_command $< b
a2: some_other_script b
some_command $< b
b: c
touch $#
c:
touch $#
Where b is some database-like file that is required to make a1 and a2. However, every time b is accessed (even if not altered) the modification date changes. Therefore, anytime the rule for a2 is executed, Make thinks that a1 needs to be remade because the database b was used (even if c hasn't changed and b remains the same). I only want to update a1 and a2 if c is newer (and therefore b is in need of actual re-compiling).
I could simply have a1 and a2 depend on c directly, but that misrepresents the true workflow.
I do not want to remove b, so having it as an intermediate file won't work.
I've also tried including b as an order-only dependency, but a1 and a2 will never be re-made unless forced to.
Notes: The Makefile is meant to automate executing scripts and keep track of dependencies for a research project (rather than a software project). Perhaps Make is not the right tool for this. The database-like files are GeoPackages.
If you can't rely on the timestamp of b to be accurate, then you need to not use it in your makefile. You can do something like this:
a1: some_script .buildc
some_command $< b
a2: some_other_script .buildc
some_command $< b
.buildc: c
command to update b
touch $#
c:
touch $#
This will run command to update b only if c is newer than .buildc which is set each time this command is invoked, not when b is used.
You could maybe just prevent useless changes of the timestamp of b:
a1: some_script b
touch -r b .b.a1
some_command $< b
touch -r .b.a1 b && rm .b.a1
a2: some_other_script b
touch -r b .b.a2
some_command $< b
touch -r .b.a2 b && rm .b.a2
But be careful: if you run make in parallel mode (make -j), a1 and a2 recipes could be run in parallel with potential race conditions. It is thus probably better to serialize them with .NOTPARALLEL: or by using flock in the recipes.
I think the "order-only prerequisite" might do the trick:
a1: some_script | b
some_command $< b
a2: some_other_script | b
some_command $< b
b: c
touch $#
c:
touch $#
I can invoke (GNU) make with the -j option, enabling multiple jobs to be run in parallel:
make -j
Is it possible from within the Makefile to determine whether or not make was invoked with this option? I have tried looking for some automatic variable storing all options given, but I could not find it.
I ask because I want the build process to be slightly different depending on whether or not -j was used.
I am not sure if this is a standard practice (or a good advise), but here it is:
makefile:
something:
#echo MAKEFLAGS : $(MAKEFLAGS)
Running:
/home/user> make something
MAKEFLAGS :
/home/user> make -j something
MAKEFLAGS : -j
/home/user> make -j 10 something
MAKEFLAGS : -j10 --jobserver-auth=3,4
/home/user> make -j20 something
MAKEFLAGS : -j20 --jobserver-auth=3,4
Look in the MAKEFLAGS GNU make variable.
echo 'all:;: $(MAKEFLAGS)' | make -f-
:
echo 'all:;: $(MAKEFLAGS)' | make -f- -j
: -j
Criteria: Makefile is a GNU Make Makefile - I'm not interested in makepp, qmake, cmake, etc. They're all nice (especially cmake), but this is for work and at work we use GNU Make. The optimal solution is a pure Makefile solution rather than a shell script that parses make for you.
I also don't want to do a 'continue on failure' solution - if it's broken, it's broken and needs to be fixed.
The situation is this, I've got a makefile that builds several directories in parallel - if one of them fails, of course the whole build fails, but not until all the running makes run to completion (or failure). This means that the reason why make actually failed is buried somewhere arbitrarily far from the end of make's output.
Here's an example of what I've got:
all: $(SUBDIRS)
SUBDIRS = \
apple \
orange \
banana \
pineapple \
lemon \
watermelon \
grapefruit
$(SUBDIRS):
cd $# && $(MAKE) $(MFLAGS) 2>&1 | sed -e "s/^/$(notdir $(#)): /g"
If I run 'make -j 5' and 'orange' happens to fail - I'd like to see a table like this at the end
of the make process
apple - passed
orange - FAILED
banana - passed
pineapple - passed
lemon - passed
I've considered having an && echo "passed" >.result || echo "FAILED" >.result, but make still needs some sort of TRAP or __onexit() cleanup command to print at them on exit.
Any Makefile ninjas out there have a pure-makefile solution for this?
un-edit - my solution wasn't actually working the way I had hoped.. STYMIED!
When you want make to abort at the first failure, end immediately and kill all in-flight jobs instead of waiting for them to finish, you need to patch GNU Make like this
http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html
Then you need to set a trap for every shell that make invokes (as well as set -o pipefail if you use a pipe), as described in this post http://lists.gnu.org/archive/html/help-make/2009-02/msg00011.html
In a nutshell:
target1:
trap 'kill $$(jobs -p)'; command && something || something-else
target2:
trap 'kill $$(jobs -p)'; set -o pipefail; command | sed '...'
The only way I see is self-execution with a sub-make:
all : subdirs
subdirs :
$(MAKE) -f $(lastword $(MAKEFILE_LIST)) subdirs-recursive || cat log
subdirs-recursive: $(SUBDIRS)