How to make GNU make run in batches? - makefile

Question

I'd like to use make to process a large number of inputs to outputs using a script (python, say.) The problem is that the script takes an incredibly short amount of time to run per input, but the initialization takes a while (python engine + library initialization.) So, a naive makefile that just has an input->output rule ends up being dominated by this initialization time. Parallelism doesn't help with that.
The python script can accept multiple inputs and outputs, as so:
python my_process -i in1 -o out1 -i in2 -o out2 ...
and this is the recommended way to use the script.
How can I make a Makefile rule that best uses my_process, by sending in out of date input-output pairs in batches? Something like parallel but aware of which outputs are out of date.
I would prefer to avoid recursive make, if at all possible.

Answer 1

I don't completely grasp your problem: do you really want make to operate in batches or do you want a kind of perpetual make process checking the file system on the fly and feeding to the Python process whenever it finds necessary? If the latter, this is quite the opposite of a batch mode and rather a pipeline.
For the batch mode there is a work-around which needs a dummy file recording the last runnning time. In this case we are abusing make for because the makefile is in this part a one-trick pony which is unintuitive and against the good rules:
SOURCES := $(wildcard in*)
lastrun : $(SOURCES)
python my_process $(foreach src,$?,-i $(src) -o $(patsubst in%,out%,$(src)))
touch lastrun
PS: please note that this solution has a substantial flaw in that it doesn't detect the update of in-files when they happen during the run of the makefile. All in all it is more advisable to simply collect the filenames of the in-files which were updated by the update process itself and avoid make althogether.

Answer 2

This is what I ended up going with, a makefile with one layer of recursion.
I tried using $? both with grouped and ungrouped targets, but couldn't get the exact behavior needed. If one of the output targets was deleted, the rule would be re-run but $? wouldn't necessarily have some input files but not the correct corresponding input file, very strange.
Makefile:
all:
INDIR=in
OUTDIR=out
INFILES=$(wildcard in/*)
OUTFILES=$(patsubst in/%, out/%, $(INFILES))
ifdef FIRST_PASS
#Discover which input-output pairs are out of date
$(shell mkdir -p $(OUTDIR); echo -n > $(OUTDIR)/.needs_rebuild)
$(OUTFILES) : out/% : in/%
#echo $# $^ >> $(OUTDIR)/.needs_rebuild
all: $(OUTFILES)
#echo -n
else
#Recurse to run FIRST_PASS, builds .needs_rebuild:
$(shell $(MAKE) -f $(CURDIR)/$(firstword $(MAKEFILE_LIST)) FIRST_PASS=1)
#Convert .needs_rebuild into batches, creates all_batches phony target for convenience
$(shell cat $(OUTDIR)/.needs_rebuild | ./make_batches.sh 32 > $(OUTDIR)/.batches)
-include $(OUTDIR)/.batches
batch%:
#In this rule, $^ is all inputs needing rebuild.
#The corresponding utputs can be computed using a patsubst:
targets="$(patsubst in/%, out/%, $^)"; touch $$targets
clean:
rm -rf $(OUTDIR)
all: all_batches
endif
make_batches.sh:
#!/bin/bash
set -beEu -o pipefail
batch_size=$1
function _make_batches {
batch_num=$1
shift 1
#echo ".PHONY: batch$batch_num"
echo "all_batches: batch$batch_num"
while (( $# >= 1 )); do
read out in <<< $1
shift 1
echo "batch$batch_num: $in"
echo "$out: batch$batch_num"
done
}
export -f _make_batches
echo ".PHONY: all_batches"
parallel -N$batch_size -- _make_batches {#} {} \;
Unfortunately, the makefile is a one trick pony and there's quite a bit of boilerplate to pull this recipe off.