GNU Make how to make a static pattern rule for files that are not in the same directory? - bash

Question

I want to use make and create a static pattern rule that has the target in a output directory, and the prerequisite files are in the preceeding directory, and it has to work recursively.
I have a minimal example here:
.
├── anotherdir
│   ├── output
│   │   ├── source3.md
│   │   └── source4.md
│   ├── source3.json
│   └── source4.json
├── output
│   ├── source1.md
│   └── source2.md
├── source1.json
└── source2.json
I want to generate the output directories if they do not exist, and I want to generate *.md files from the *.json using make if they do not exist, or *.json is updated.
So far, I have the following Makefile:
SOURCE_FILES := $(shell find ./ -name "*.json")
OUTPUT_FILES := $(join $(addsuffix output/,$(dir $(SOURCE_FILES))), $(addsuffix .md,$(basename $(notdir $(SOURCE_FILES)))))
.PHONY: all
all: $(OUTPUT_FILES)
$(OUTPUT_FILES): %.md: %.json
mkdir -p $(dir $#)
# Command to create MD file from json file into the output directory here
The actual command to create the MD file from the json file doesn't matter here, because I have a script that I will call that will do this for me. The problem here, is that when I try to even run this at all, I get the following output:
> make all
make: *** No rule to make target 'anotherdir/output/source4.json', needed by 'anotherdir/output/source4.md'. Stop.
Obviously, source4.json is not in anotherdir/output, but rather, it's in the preceeding directory, which is just anotherdir. I don't know how to make it so that the pattern $(OUTPUT_FILES): %.md: %.json will match it properly.
Or is a static pattern rule not good here? I'm not sure what to do to fit my scenario.
EDIT: I tried to do something like this:
$(OUTPUT_FILES): %.md: $(join $(subst output,,$(dir %)), $(addsuffix .json,$(basename $(notdir %))))
and this doesn't work, I still get:
> make all
make: *** No rule to make target 'anotherdir/output/source4.json', needed by 'anotherdir/output/source4.md'. Stop.
Edit 2: to clarify, i start with the following files
.
├── anotherdir
│ ├── source3.json
│ └── source4.json
├── source1.json
└── source2.json
And then when i run make, i want it to generate the output folders like this
.
├── anotherdir
│ ├── output
│ │ ├── source3.md
│ │ └── source4.md
│ ├── source3.json
│ └── source4.json
├── output
│ ├── source1.md
│ └── source2.md
├── source1.json
└── source2.json
I want to use some kind of smart makefile syntax to pick up these files names without me hard coding it in myself. Hence, i looked at the documentation and saw that static pattern rules might be the solution that i want, except that i can't get the right prerequisite pattern down.

Answer 1

I would do it this way:
First, find the source files just as you did (with a small change to prevent the unsightly double-slash):
SOURCE_FILES := $(shell find . -name "*.json")
A pattern file would be nice, if we could use two wildcards at once, but Make can't quite do that. So I recommend using a template:
define template
TDIR := $(dir $(1))output
TARG := $$(TDIR)/$(notdir $(basename $(1))).md
$$(TARG): $(1)
mkdir -p $$#
#echo building $$# from $$<
# Command to create MD file from json file into the output directory here
endef
$(foreach SOURCE,$(SOURCE_FILES),$(eval $(call template,$(SOURCE))))
If this works, all that's left is to construct a list of output files, and a default rule that has all of them as prerequisites:
define template
TDIR := $(dir $(1))output
TARG := $$(TDIR)/$(notdir $(basename $(1))).md
OUTPUT_FILES += $$(TARG)
$$(TARG): $(1)
mkdir -p $$#
#echo building $$# from $$<
# Command to create MD file from json file into the output directory here
endef
all:
$(foreach SOURCE,$(SOURCE_FILES),$(eval $(call template,$(SOURCE))))
all: $(OUTPUT_FILES)
It isn't pretty, but it seems to work.

Answer 2

If it had not been proposed already in another answer I would have suggested foreach-eval-call. For completeness here are different solutions for GNU make (they may work also with other versions of make but I did not check):
Creating the output directories beforehand
If the output directories exist already you can refer to ../%.json in your pattern rule:
SOURCE_FILES := $(shell find . -name "*.json")
OUTPUT_FILES := $(join $(dir $(SOURCE_FILES)),\
$(patsubst %.json,output/%.md,$(notdir $(SOURCE_FILES))))
$(shell mkdir -p $(dir $(OUTPUT_FILES)))
.PHONY: all
all: $(OUTPUT_FILES)
%.md: ../%.json
: json2md $< -o $#
This may look strange but if you read carefully the Pattern match section of the GNU make manual you should quickly understand. The only constraint for this to work is that the output directories exist before make searches pattern rules that match the targets. If one does not exist make will complain that there is no eligible rule to build the target. This is the reason for the:
$(shell mkdir -p $(dir $(OUTPUT_FILES)))
at the beginning of the Makefile. Demonstration:
$ make
: json2md output/../source2.json -o output/source2.md
: json2md output/../source1.json -o output/source1.md
: json2md anotherdir/output/../source4.json -o anotherdir/output/source4.md
: json2md anotherdir/output/../source3.json -o anotherdir/output/source3.md
Using the secondary expansion
Secondary expansion gives you the possibility to use automatic variables in the list of prerequisites. The $$ are needed to escape the first expansion by make.
SOURCE_FILES := $(shell find . -name "*.json")
OUTPUT_FILES := $(join $(dir $(SOURCE_FILES)),\
$(patsubst %.json,output/%.md,$(notdir $(SOURCE_FILES))))
.PHONY: all
all: $(OUTPUT_FILES)
$(sort $(dir $(OUTPUT_FILES))):
mkdir -p $#
.SECONDEXPANSION:
$(OUTPUT_FILES): $$(patsubst %output,%,$$(#D))$$(basename $$(#F)).json | $$(dir $$#)
: json2md $< -o $#
Demonstration:
$ make
mkdir -p output/
mkdir -p anotherdir/output/
: json2md source2.json -o output/source2.md
: json2md source1.json -o output/source1.md
: json2md anotherdir/source4.json -o anotherdir/output/source4.md
: json2md anotherdir/source3.json -o anotherdir/output/source3.md
Note: instead of creating the output directories in the json-to-md rule (which has the drawback of creating them several times), I added them as order-only prerequisites and added a specific rule to create them.
Note: the sort function also removes duplicates.
Using recursive make
Here we invoke make (with always the same Makefile) recursively in each sub-directory (except output, of course). Each invocation handles only the local json files, which makes the paths of prerequisites and targets much simpler.
MF := $(realpath $(lastword $(MAKEFILE_LIST)))
SUB_DIRS := $(filter-out . ./output,$(shell find . -maxdepth 1 -type d))
SOURCE_FILES := $(filter-out $(SUB_DIRS),$(wildcard *.json))
OUTPUT_FILES := $(patsubst %.json,output/%.md,$(SOURCE_FILES))
.PHONY: $(SUB_DIRS) all
all: $(SUB_DIRS) $(OUTPUT_FILES)
$(OUTPUT_FILES): output/%.md: %.json | output
: json2md $< -o $#
output:
mkdir -p $#
$(SUB_DIRS):
$(MAKE) -C $# -f $(MF)
Demonstration:
$ make
make -C anotherdir -f /home/doe/json2md/Makefile
make[1]: Entering directory '/home/doe/json2md/anotherdir'
mkdir -p output
: json2md source4.json -o output/source4.md
: json2md source3.json -o output/source3.md
make[1]: Leaving directory '/home/doe/json2md/anotherdir'
mkdir -p output
: json2md source2.json -o output/source2.md
: json2md source1.json -o output/source1.md