Parallel make generates code multiple times - makefile

Question

let's assume an easy project consisting of two static sources and three dynamic ones, which are generated.
The makefile looks like this:
SRC_DIR = src
OUTPUT = output
TARGET = $(OUTPUT)/lib.so
GEN_SRCS = dynamic_1.c dynamic_2.c dynamic_3.c
SRCS = static_1.c static_2.c $(GEN_SRCS)
OBJS = $(SRCS:%.c=$(OUTPUT)/%.o)
all: $(TARGET)
$(TARGET): $(OBJS)
gcc -shared -o $# $^
$(OUTPUT)/%.o: $(SRC_DIR)/$*%.c | $(OUTPUT)
#echo "Creating object $# from $^"
gcc -c -o $# $<
$(SRC_DIR)/dynamic_%.c: definition_for_generator.xml
#echo "---------------------------------------"
#echo "Generating dynamic Cs"
#echo "---------------------------------------"
./generate.sh $<
$(OUTPUT):
mkdir -p $(OUTPUT)
.PHONY: clean
clean:
rm -rf $(OUTPUT) $(SRC_DIR)/dynamic_1.c $(SRC_DIR)/dynamic_2.c $(SRC_DIR)/dynamic_3.c
While building using one core only, it builds just fine:
mkdir -p output
Creating object output/static_1.o from src/static_1.c
gcc -c -o output/static_1.o src/static_1.c
Creating object output/static_2.o from src/static_2.c
gcc -c -o output/static_2.o src/static_2.c
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
Creating object output/dynamic_1.o from src/dynamic_1.c
gcc -c -o output/dynamic_1.o src/dynamic_1.c
Creating object output/dynamic_2.o from src/dynamic_2.c
gcc -c -o output/dynamic_2.o src/dynamic_2.c
Creating object output/dynamic_3.o from src/dynamic_3.c
gcc -c -o output/dynamic_3.o src/dynamic_3.c
gcc -shared -o output/lib.so output/static_1.o output/static_2.o output/dynamic_1.o output/dynamic_2.o output/dynamic_3.o
rm src/dynamic_1.c src/dynamic_2.c src/dynamic_3.c
However, with parallel building (-j 2), the dynamic_* files are generated twice:
mkdir -p output
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
Creating object output/static_1.o from src/static_1.c
gcc -c -o output/static_1.o src/static_1.c
Creating object output/static_2.o from src/static_2.c
gcc -c -o output/static_2.o src/static_2.c
Creating object output/dynamic_1.o from src/dynamic_1.c
gcc -c -o output/dynamic_1.o src/dynamic_1.c
Creating object output/dynamic_2.o from src/dynamic_2.c
gcc -c -o output/dynamic_2.o src/dynamic_2.c
Creating object output/dynamic_3.o from src/dynamic_3.c
gcc -c -o output/dynamic_3.o src/dynamic_3.c
gcc -shared -o output/lib.so output/static_1.o output/static_2.o output/dynamic_1.o output/dynamic_2.o output/dynamic_3.o
rm src/dynamic_1.c src/dynamic_2.c src/dynamic_3.c
How should I change the makefile to generate the files only one?
A clue can be here, but I haven't found a way to use it:
multiple targets from one recipe and parallel execution
The project to download is here: http://stuff.pitris.info/make.tar.gz
Thanks!

Answer 1

That linked answer is the answer. Your rule:
$(SRC_DIR)/dynamic_%.c: definition_for_generator.xml
#echo "---------------------------------------"
#echo "Generating dynamic Cs"
#echo "---------------------------------------"
./generate.sh $<
Tells make that a single $(SRC_DIR)/dynamic_%.c file is created by each invocation of generate.sh. However running that script actually generates multiple files (but make doesn't know that).
So when you run make normally make operates sequentially and when it sees the first dynamic_*.c file it runs the script to generate that single file (but all the files are generated). When make then goes to determine if the next dynamic_*.c file needs to be generated it sees that the file already exists and skips the rule for generating it.
When you run make -j make spawns parallel processes and those just so happen to each hit a dynamic_*.c file at the same time and so they all run the the script (even though only one of them needs to).
That's why you see it running N times. Change your target order/etc. and you may see fewer generate.sh runs as the parallelism works out differently (you might even manage to race it to only run once) but that's all coincidental.
The right solution, as in that linked answer, is to teach make that all the files are created by running the script by listing multiple patterns for that rule. Or, alternatively, modify generate.sh to only generate a single file at a time (and always run it N times).
The simplest (though somewhat ugly) way to do the multiple patterns thing would likely be to replace
GEN_SRCS = dynamic_1.c dynamic_2.c dynamic_3.c
with
GEN_SRCS = %ynamic_1.c %ynamic_2.c %ynamic_3.c
or something like that and then replace
$(SRC_DIR)/dynamic_%.c: definition_for_generator.xml
with
$(GEN_SRCS): definition_for_generator.xml

Answer 2

There is another issue with your code: dependency on $(OUTPUT) directory. It never works as expected.
The below makefile works as expected, please try:
SRC_DIR = src
OUTPUT = output
TARGET = $(OUTPUT)/lib.so
GEN_SRCS = dynamic_1.c dynamic_2.c dynamic_3.c
SRCS = static_1.c static_2.c $(GEN_SRCS)
ifeq ($(filter clean,$(MAKECMDGOALS)),)
-include $(OUTPUT)/.timestamp
endif
OBJS = $(SRCS:%.c=$(OUTPUT)/%.o)
mkdir_# = $(if $(wildcard $(#D)),,mkdir -p $(#D))
all: $(TARGET)
$(TARGET): $(OBJS)
gcc -shared -o $# $^
$(OUTPUT)/%.o: $(SRC_DIR)/$*%.c
$(mkdir_#)
#echo "Creating object $# from $^"
gcc -c -o $# $<
$(OUTPUT)/.timestamp: definition_for_generator.xml
$(mkdir_#)
#echo "---------------------------------------"
#echo "Generating dynamic Cs"
#echo "---------------------------------------"
./generate.sh $<
touch $#
.PHONY: clean
clean:
rm -rf $(OUTPUT) $(SRC_DIR)/dynamic_1.c $(SRC_DIR)/dynamic_2.c $(SRC_DIR)/dynamic_3.c
The idea is to inject dependency on generated empty makefile - $(OUTPUT)/.timestamp . This is to force GNU Make reread you Makefile. The rule to generate $(OUTPUT)/.timestamp runs your source generator. Thus when GNU Make starts processing your Makefile the second time, generated sources are in place already. Thus you don't need to define any dependencies for them, i.e. they are like your regular source files.
Hint: writing generated sources into source tree is not a good idea preventing use of this source tree by multiple builds at the same time. This is the most obvious issue with this approach.
Also look at mkdir_# function: it will efficiently create directory for the target if it doesn't exist first trying to use GNU Make builtin to check if the directory exists before spawning mkdir process. Makes a difference when used with Cygwin as process creation in this environment is very time consuming operation compared to the first class UNIX systems.
Output:
$ make -j
mkdir -p output
---------------------------------------
Generating dynamic Cs
---------------------------------------
./generate.sh definition_for_generator.xml
touch output/.timestamp
Creating object output/static_1.o from src/static_1.c
gcc -c -o output/static_1.o src/static_1.c
Creating object output/static_2.o from src/static_2.c
gcc -c -o output/static_2.o src/static_2.c
Creating object output/dynamic_1.o from src/dynamic_1.c
Creating object output/dynamic_2.o from src/dynamic_2.c
gcc -c -o output/dynamic_1.o src/dynamic_1.c
gcc -c -o output/dynamic_2.o src/dynamic_2.c
Creating object output/dynamic_3.o from src/dynamic_3.c
gcc -c -o output/dynamic_3.o src/dynamic_3.c
gcc -shared -o output/lib.so output/static_1.o output/static_2.o output/dynamic_1.o output/dynamic_2.o output/dynamic_3.o