I have a project structure that looks something like this:
.
└── src
├── Module1
│ ├── source1.cc
│ ├── source2.cc
│ └── source3.cc
├── Module2
│ ├── source1.cc
│ ├── source2.cc
│ └── source3.cc
└── Module3
├── source1.cc
├── source2.cc
└── source3.cc
I have an implicit rule that will create an object file for each source file, and maintain the directory structure (e.g. src/Module1/source2.cc would compile to obj/Module1/source2.o).
However, I would now like to have an implicit rule to create archive files for each module. For example, each object file compiled from src/Module2 would be added to obj/Module2.a. My first idea looked quite similar to my implicit rule for object files:
obj/%.a: $(wildcard obj/%/*.o)
#mkdir -p $(#D);
ar -crs "$#" $^;
This would pass off the work of compiling the object files to the other implicit rule. However, the issue here is that the % character is not expanded.
Is there a way to access the % from the implicit rule within the wildcard function call?
You can do it with Secondary Expansion:
.SECONDEXPANSION:
obj/%.a: $$(wildcard obj/%/*.o)
#mkdir -p $(#D);
ar -crs "$#" $^;
Related
I am trying to use make to build a set of markdown documents from specific and shared files.
PKGS := foo bar baz app
DEFAULT := DEPENDENCIES.md RULES.md
BUILD := .build
VPATH := .:packages
$(BUILD)/%.history.md: $(DEFAULT) $(wildcard %/RULES.*.md) $(foreach f,$(DEFAULT),%/$(f))
echo "!! BUILDING NOW $#"
echo $^
echo
with the following directory structure:
.
├── ABSTRACT.md
├── DEPENDENCIES.md
├── IMPLEMENTATION.md
├── Makefile
├── packages
│ ├── foo
│ │ ├── ABSTRACT.md
│ │ ├── DEPENDENCIES.md
│ │ ├── IMPLEMENTATION.md
│ │ ├── RULES.0.md -> ../COMMON.md
│ │ └── RULES.md
│ ├── COMMON.md
│ ├── bar
│ │ ├── ABSTRACT.md
│ │ ├── DEPENDENCIES.md
│ │ ├── IMPLEMENTATION.md
│ │ └── RULES.md
│ ├── baz
│ │ ├── ABSTRACT.md
│ │ ├── DEPENDENCIES.md
│ │ ├── IMPLEMENTATION.md
│ │ └── RULES.md
│ └── app
│ ├── ABSTRACT.md
│ ├── DEPENDENCIES.md
│ ├── IMPLEMENTATION.md
│ ├── RULES.0.md -> ../COMMON.md
│ └── RULES.md
└── RULES.md
However, $(wildcard %/RULES.*.md) always seems to expand into an empty list.
When executing make .build/foo.history.md I expected it to evaluate the prerequisites as DEPENDENCIES.md RULES.md packages/foo/RULES.0.md packages/foo/DEPENDENCIES.md packages/foo/RULES.md but instead I see this:
make .build/foo.history.md
echo "!! BUILDING NOW .build/foo.history.md"
!! BUILDING NOW .build/foo.history.md
echo DEPENDENCIES.md RULES.md packages/foo/DEPENDENCIES.md packages/foo/RULES.md
DEPENDENCIES.md RULES.md packages/foo/DEPENDENCIES.md packages/foo/RULES.md
echo
I've also tried adding some extra targets to better understand the process:
# Pattern rule with `%` as a directory clearly works.
%.static: %/RULES.0.md
echo $# -> $^
%.semi: %/RULES.*.md
echo $# -> $^
%.x: $(wildcard packages/%/RULES.*.md)
echo $# -> $^
# Wildcards using * also clear work.
app.y: $(wildcard packages/app/RULES.*.md)
echo $# -> $^
# Combination of both fails.
# I expect that if the target is `foo.z` for the prerequisites to expand into `foo/RULES.*.md`,
# searching within the VPATH.
%.z: $(wildcard %/RULES.*.md)
echo $# -> $^
Which results in this:
$ make app.static --just-print
echo app.static -> packages/app/RULES.0.md
$ make app.semi --just-print
make: *** No rule to make target 'app.semi'. Stop.
$ make app.x --just-print
echo app.x ->
$ make app.y --just-print
echo app.y -> packages/app/RULES.0.md
$ make app.z --just-print
echo app.z ->
I've also tried using .SECONDEXPANSION, but I couldn't get it to work with that either and I keep seeing other people recommending against it.
Additionally, it does work for $(foreach f,$(DEFAULT),%/$(f)) expanding correctly into packages/app/DEPENDENCIES.md packages/app/RULES.md which adds confusion as to why it doesn't work in $(wildcard).
The high-level problem is that you're trying to use too many unfamiliar tools at once. Let's take VPATH out of the picture for now.
The wildcard function will work, with secondary expansion:
.SECONDEXPANSION:
$(BUILD)/%.history.md: $$(wildcard packages/%/RULES.*.md)
echo "!! BUILDING NOW $#"
echo $^
echo
Notice that we must escape the $ in the prerequisite list with another $. I don't know why people advised against secondary expansion, but I see no easier way to get the effect you want.
Now about that VPATH. There isn't much need for it, since you want to search only one directory (packages/). Also, it doesn't work well with wildcard because VPATH tells Make where to look for prerequisites; it doesn't tell wildcard where to look for matches. And Make doesn't know what prerequisite to look for until after wildcard has done its work, which it can't because it doesn't know where to look. I advise you to do without it.
I have a project structure that looks something like this:
.
└── src
├── Module1
│ ├── source1.cc
│ ├── source2.cc
│ └── source3.cc
├── Module2
│ ├── source1.cc
│ ├── source2.cc
│ └── source3.cc
└── Module3
├── source1.cc
├── source2.cc
└── source3.cc
I have an implicit rule that will create an object file for each source file, and maintain the directory structure (e.g. src/Module1/source2.cc would compile to obj/Module1/source2.o).
However, I would now like to have an implicit rule to create archive files for each module. For example, each object file compiled from src/Module2 would be added to obj/Module2.a. My first idea looked quite similar to my implicit rule for object files:
obj/%.a: $(wildcard obj/%/*.o)
#mkdir -p $(#D);
ar -crs "$#" $^;
This would pass off the work of compiling the object files to the other implicit rule. However, the issue here is that the % character is not expanded.
Is there a way to access the % from the implicit rule within the wildcard function call?
You can do it with Secondary Expansion:
.SECONDEXPANSION:
obj/%.a: $$(wildcard obj/%/*.o)
#mkdir -p $(#D);
ar -crs "$#" $^;
I'm using GNU Make on Arch Linux to generate PDFs from LilyPond source files. I have a directory structure as follows:
scores/
├── makefile
├── out
│ ├── others-songs
│ │ ├── ...
│ │ ├── ...
│ │ └── 失恋阵线联盟
│ │ ├── 失恋阵线联盟.edition.log
│ │ ├── 失恋阵线联盟.oll.log
│ │ └── 失恋阵线联盟.pdf
│ └── ...
├── src
│ ├── others-songs
│ │ ├── ...
│ │ ├── ...
│ │ └── 失恋阵线联盟
│ │ ├── chorus.ily
│ │ ├── verse.ily
│ │ ├── words.ily
│ │ └── 失恋阵线联盟.ly
│ └── ...
The PDFs in the out directory depend on the .ily and .ly files in the corresponding directory in src. The following implicit pattern rule works if the .ly file is modified, but not if any of the other files are:
LY = $(shell find src -iname '*.ly')
PDF = $(subst src,out,$(LY:.ly=.pdf))
pdf: $(PDF)
out/%.pdf: src/%.ly
#mkdir -p $(dir $#)
#lilypond --include=$(lib) \
-dpoint-and-click=\#f \
-o $(basename $#) $<
I tried doing several different things, like appending $(<D)*.ily to the prerequisites, but it wasn't successful. I've looked at the GNU make manual online for help, but I didn't come up with anything that that I was able to apply to my particular situation.
How can I write a pattern rule that makes each PDF depend on all the files in the appropriate corresponding source directory?
EDIT
I may not have been clear enough with my first question. The behavior I want is if any of the files in the source directories are changed, the corresponding PDF is updated. For example, if chorus.ily is changed, then 失恋阵线联盟.ly is made.
A working example:
LY := $(shell find src -iname '*.ly')
PDF := ${LY:src/%.ly=out/%.pdf}
pdf: $(PDF)
define pdf_deps
$(1:src/%.ly=out/%.pdf) : $(wildcard $(dir ${1})*)
endef
# Make each pdf depend on all the files in its src directory.
$(foreach ly,${LY},$(eval $(call pdf_deps,${ly})))
out/%.pdf:
#echo "making $# from $^"
Usage:
$ ls -1R
.:
Makefile
out
src
./out:
./src:
A
B
./src/A:
a.ly
./src/B:
b.1
b.ly
$ make
making out/B/b.pdf from src/B/b.ly src/B/b.1
mkdir -p out/B
touch out/B/b.pdf
making out/A/a.pdf from src/A/a.ly
mkdir -p out/A
touch out/A/a.pdf
$ make
make: Nothing to be done for 'pdf'.
$ touch src/B/b.ly
$ make
making out/B/b.pdf from src/B/b.ly src/B/b.1
mkdir -p out/B
touch out/B/b.pdf
$ make
make: Nothing to be done for 'pdf'.
$ touch src/B/b.1
$ make
making out/B/b.pdf from src/B/b.ly src/B/b.1
mkdir -p out/B
touch out/B/b.pdf
$ make
make: Nothing to be done for 'pdf'.
I am trying to create a makefile to compile LaTeX files in subdirectories. The desired directory structure should look like this, except instead of simply a, b, and c, there can be any number of subdirectories under the latex folder.
├── latex
│ ├── a
│ │ ├── output
│ │ │ └── stuff.pdf
│ │ └── stuff.tex
│ ├── b
│ │ ├── blah.tex
│ │ └── output
│ │ └── blah.pdf
│ └── c
│ ├── asdf.tex
│ └── output
│ └── asdf.pdf
└── makefile
I want to do this with only one makefile in the latex directory that will automatically compile the tex files in every subdirectory.
My current makefile looks like this:
TEX_COMMAND = pdflatex
TEX_FILES = $(wildcard **/*.tex)
OUTPUT_DIRECTORIES = $(addsuffix output/,$(wildcard */))
PDF_FILES = $(join $(dir $(TEX_FILES)),$(addprefix output/,$(notdir $(TEX_FILES:tex=pdf))))
all: mkdir $(PDF_FILES)
mkdir:
#mkdir -p $(OUTPUT_DIRECTORIES)
$(PDF_FILES): $(TEX_FILES)
#$(TEX_COMMAND) -file-line-error -halt-on-error -output-directory $(dir $#) -aux_directory=$(dir $#) $(subst output/,$(notdir $(#:pdf=tex)),$(dir $#))
#$(TEX_COMMAND) -file-line-error -halt-on-error -output-directory $(dir $#) -aux_directory=$(dir $#) $(subst output/,$(notdir $(#:pdf=tex)),$(dir $#))
clean:
#rm -rf $(OUTPUT_DIRECTORIES)
This will correctly generate the proper pdf, aux, log, toc, etc. files in the output directory in each subdirectory. However, if I change one tex file, then make will cause everything to be recompiled.
I've already looked at many other similar questions. For other questions, the number of subdirectories is known, so you can hardcode them into the makefile. For this situation, the number of subdirectories in the latex folder is constantly changing and being added to, etc, which is why I'm using the wildcard to grab all the tex files. I would prefer not having to create a makefile for each subdirectory and using recursive make.
One of the fundamental shortcomings of Make is its crude handling of wildcards.
In this case, you can use secondary expansion to write a pattern rule that will do what you want:
all: mkdir $(PDF_FILES)
.SECONDEXPANSION:
%.pdf: $$(subst /output/,/, $$(subst .pdf,.tex, $$#))
#echo buiding $# from $<
I have a directory tree like this:
├── dir_a
│ └── file_1.txt
├── dir_b
│ └── dir_c
│ ├── file_2.txt
| └── file_3.txt
└── file_4.txt
I want to mirror this directory structure to hold the results of a command that processes each text file. I.e., the output would look like this:
├── build
│ ├── dir_a
│ │ └── processed_file_1.txt
│ ├── dir_b
│ │ └── dir_c
│ │ ├── processed_file_2.txt
│ | └── processed_file_3.txt
│ └── processed_file_4.txt
├── dir_a
│ └── file_1.txt
├── dir_b
│ └── dir_c
│ ├── file_2.txt
| └── file_3.txt
└── file_4.txt
I'm not very adept with Makefiles, so my question is: how can I get a Makefile to recreate the directory structure and recursively process all text files to place them into the right place inside the build directory? I'll be running this repeatedly as the input files change, so a Makefile that doesn't process unchanged files seems like the right way to go.
Update:
I should also mention that new input files will be added frequently, so I don't want the Makefile to name them explicitly.
It would be easier if you used stems with different suffixes rather than inserting that "processed_" string, but here's an example that works for me here:
OUTPUTS := build/dir_a/processed_file_1.txt \
build/dir_b/dir_c/processed_file_2.txt \
build/dir_b/dir_c/processed_file_3.txt \
build/processed_file_4.txt
all: $(OUTPUTS)
.SECONDEXPANSION:
$(OUTPUTS): build/% : $$(subst processed_file_,file_,%)
mkdir -p $(dir $#)
cp $< $#
clean:
rm -rf build
You could remove the complication of .SECONDEXPANSION by changing the end of the filename instead of the beginning:
OUTPUTS := build/dir_a/file_1.out \
build/dir_b/dir_c/file_2.out \
build/dir_b/dir_c/file_3.out \
build/file_4.out
all: $(OUTPUTS)
$(OUTPUTS) : build/%.out : %.txt
mkdir -p $(dir $#)
cp $< $#
clean:
rm -rf build
As Carl suggested, you could use secondary expansion, but in conjunction with order-only prerequisites.
BUILD_DIR = build
IN_FILES := dir_a/file_1.out \
dir_b/dir_c/file_2.out \
dir_b/dir_c/file_3.out \
file_4.out
OUT_FILES := $(IN_FILES:%=$(BUILD_DIR)/%)
all: $(OUT_FILES)
.SECONDEXPANSION:
$(OUT_FILES) : $(BUILD_DIR)/%.out : %.txt | $$(#D)/.
# your text processing rule here...
%/. :
mkdir -p $*
| $$(#D) means:
during the secondary expansion calculate the value of $(#D) automatic variable (which is the directory part of the target), and
add the order-only dependency on it, that is ensure that the directory exists, but don't consider remaking the target if it is older than the directory (which is an often case)