Parallel execution on multiple directories - makefile

Question

I would like to create multiple directories with some files with one makefile.
I've a directory structure like this:
conf_a/conf.json
conf_b/conf.json
main.py
Makefile
requirements.txt
And would like to type make conf_a and having a new directory like this:
build/conf_a/conf.json
build/conf_a/main.py
build/conf_a/requirements.txt
conf_a/conf.json
conf_b/conf.json
main.py
Makefile
requirements.txt
Or something like make conf_b and having a new directory like this:
build/conf_b/conf.json
build/conf_b/main.py
build/conf_b/requirements.txt
conf_a/conf.json
conf_b/conf.json
main.py
Makefile
requirements.txt
So I've made a Makefile like this:
# Disable built-in rules and variables
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables
.ONESHELL:
.SHELLFLAGS: -ec
.SILENT:
BUILD_DIR := $(CURDIR)/build
CONF_FILE := conf.json
FILES_TO_COPY := requirements.txt main.py
FUNCTION_DIRS := $(shell ls */$(CONF_FILE) | xargs -n 1 -I {} dirname {})
HIDDEN_FUNCTION_DIRS := $(shell ls .*/$(CONF_FILE) 2> /dev/null | xargs -n 1 -I {} dirname {})
clean:
rm -rf $(BUILD_DIR)
all: clean $(FUNCTION_DIRS) deploy
$(FUNCTION_DIRS) $(HIDDEN_FUNCTION_DIRS):
tmp=$#
FUNCTION_DIR=$${tmp%/}
export FUNCTION=$${FUNCTION_DIR#.}
mkdir -p $(BUILD_DIR)/$$FUNCTION
cp -f $(FILES_TO_COPY) $$FUNCTION_DIR/$(CONF_FILE) $(BUILD_DIR)/$$FUNCTION/
test:
for FUNCTION in $(shell ls $(BUILD_DIR))
do
echo "Testing $$FUNCTION"
done
deploy:
for FUNCTION in $(shell ls $(BUILD_DIR))
do
echo "Deploying $$FUNCTION"
done
Well, it works...
So if I want to test a conf I do: make conf_a test.
If I want to deploy: make conf_b deploy
It work quiet well but test or deploy target are sequential (because the for loop) and they could have been parallel.
My problem is that I'se too much configuration directories and because the deploy is slow parallel could have been a lot better.
But I do not know how to structure a Makefile to make it this way.
Any idea ?
Truth be told, the task deploy deploy a GCP cloud function, and the test just run the function locally

Answer 1

Generally, the easiest way to structure a makefile to facilitate parallel operations is to define separate targets that can be processed in parallel. Then you can use make's -j option to request that it take care of the parallelization across (up to) a specific maximum number of parallel tasks.
For example:
deploy: deploy_a deploy_b
deploy_a: conf_a
echo deploying conf_a
deploy_b: conf_b
echo deploying conf_b
Then you can make -j2 deploy and (probably) the deploy_a and deploy_b rules will be processed in parallel. But note that that might not help much. Even though you have separate processes for the two deployments, if you're deploying both to the same local disk then they won't be able to both use write to different files on the disk at the same time. As a result, you'll probably not see a significantly better time to completion, and it might even be worse.
Note, too, that the above example eschews dynamically determining the available component directories. Such dynamism is atypical of makefiles, and IMO it rarely provides a net benefit. Nevertheless, GNU make (on which specific implementation you are already relying) does offer mechanisms by which you could generate the needed per-directory deployment rules dynamically.