I have a bunch of different source files in my static HTML blog. The outermost extensions explain the format to be processed next.
Example: Source file article.html.md.gz (with target article.html) should be processed by gunzip, then by my markdown processor.
Further details:
The order of the extensions may vary
Sometimes an extension is not used (article.html.gz)
I know how to process all different extensions
I know that the final form is always article.html
Ideally I would have liked to just write rules as follows:
...
all-articles: $(ALL_HTML_FILES)
%: %.gz
gunzip ...
%: %.md
markdown ...
%: %.zip
unzip ...
And let make figure out the path to take based on the sequence of extensions.
From the documentation however, I understand that there are constraints on match-all rules, and the above is not possible.
What's the best way forward? Can make handle this situation at all?
Extensions are made up examples. My actual source files make more sense :-)
I'm on holiday so I'll bite.
I'm not a fan of pattern rules, they are too restricted and yet too arbitrary at the same time for my tastes. You can achieve what you want quite nicely in pure make:
.DELETE_ON_ERROR:
all: # Default target
files := a.html.md.gz b.html.gz
cmds<.gz> = gzip -d <$< >$#
cmds<.md> = mdtool $< -o $#
define rule-text # 1:suffix 2:basename
$(if $(filter undefined,$(flavor cmds<$1>)),$(error Cannot handle $1 files: [$2$1]))
$2: $2$1 ; $(value cmds<$1>)
all: $2
endef
emit-rule = $(eval $(call rule-text,$1,$2))# 1:suffix 2:basename
emit-hierachy = $(if $(suffix $2),$(call emit-rule,$1,$2)$(call emit-hierachy,$(suffix $2),$(basename $2)))# 1:suffix 2:basename
emit-rules = $(foreach _,$1,$(call emit-hierachy,$(suffix $_),$(basename $_)))# 1:list of source files
$(call emit-rules,${files})
.PHONY: all
all: ; : $# Success
The key here is to set $files to your list of files.
This list is then passed to emit-rules.
emit-rules passes each file one-at-a-time to emit-hierachy.
emit-hierachy strips off each extension in turn,
generates the appropriate make syntax, which it passes to $(eval …).
emit-hierachy carries on until the file has only one extension left.
Thus a.html.md.gz becomes this make syntax:
a.html.md: a.html.md.gz ; gunzip <$< >$#
a.html: a.html.md ; mdtool $< -o $#
all: a.html
Similarly, b.html.gz becomes:
b.html: b.html.gz ; gunzip <$< >$#
all: b.html
Neato, or what?
If you give emit-rules a file with an unrecognised extension (c.html.pp say),
you get a compile-time error:
1:20: *** Cannot handle .pp files: [c.html.pp]. Stop.
Compile-time? Yeah, before any shell commands are run.
You can tell make how to handle .pp files by defining cmds<.pp> :-)
For extra points it's also parallel safe. So you can use -j9 on your 8 CPU laptop, and -j33 on your 32 CPU workstation. Modern life eh?
Related
I have two lists of files as prerequisites
input_i.xx
config_j.yy
and I need to run all of their combinations. A single one looks like this:
input1_config3.output: input1.xx config3.yy
run_script $^
Also in reality, their names are not numbered, but I already have their stems defined in INPUTS and CONFIGS. With that, I can generate all the targets together
TARGETS:=$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(input)_$(config).output))
But I have difficulty with the prerequisites. It seems I need to
get basename
split on _
add the extensions .xx and .yy
.SECONDEXPANSION
$(TARGETS): $(basename $#)
run_script $^
Can someone show me how to do that? Not sure if this the proper way, maybe a bottom-up way is easier?
make is not really suitable for keeping track of an M x N matrix of results. The fundamental problem is that you can't have two stems in a rule, so you can't say something like
# BROKEN
input%{X}_config%{Y}.output: input%{X}.xx config%{Y}.yy
As a rough approximation, you could use a recursive make rule to set a couple of parameters, and take it from there, but this is rather clumsy.
.PHONY: all
all:
$(MAKE) -$(MAKEFLAGS) X=1 Y=6 input1_config6.output
$(MAKE) -$(MAKEFLAGS) X=1 Y=7 input1_config7.output
$(MAKE) -$(MAKEFLAGS) X=2 Y=6 input2_config6.output
:
input$X_config$Y.output: input$X.xx config$Y.yy
run_script $^
It would be a lot easier if you provided a complete sample example with a complete set of targets and prerequisites and exactly what you wanted to happen.
Using .SECONDEXPANSION might work, but you're not using it correctly; please re-read the documentation. The critical aspect of .SECONDEXPANSION is that you have to escape the variables that you want to avoid expanding until the second pass. In your example you've not escaped anything, so .SECONDEXPANSION isn't actually doing anything at all here. However, as #tripleee points out it's not easy to use multiple variable values in a single target.
To do this more easily you'll probably want to use eval. Something like this:
define DECLARE
$1_$2.output: $1.xx $2.yy
TARGETS += $1_$2.output
endef
TARGETS :=
$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(eval $(call DECLARE,$(input),$(config)))))
$(TARGETS):
run_script $^
I have another solution using include and bash for loop.
include trees.mk
trees.mk:
#for input in $(INPUTS); do \
for config in $(CONFIGS); do \
echo $${input}_$$config.output : $${input}.xx $$config.yy; \
echo -e '\t run_scipt $$^ ';\
done \
done > $#
At the beginning, trees.mk doesn't exist. The double for loops write out the rule to the target using file redirection >$#.
I got this idea from Managing Projects with GNU Make, Third Edition By Robert Mecklenburg, on
page 56
I want to create a makefile to be included in others, kind of as a "library", let's call it library.mak:
TARGET_FILES = a.txt b.txt c.txt d.txt
SRC_FOLDER = somewhere/else
.PHONY: all
all:
for target in ${TARGET_FILES} ; do \
echo -e "\nMaking $${target}:" ;\
${MAKE} ${MFLAGS} --no-print-directory $${target} ;\
done
.PHONY: ${TARGET_FILES}
${TARGET_FILES}:
cp ${SRC_FOLDER}/$# $#
The two rules are there to make all, as well as one specific target, respectively; using a default rule (which is the purpose of library.mak).
In my "user makefile" called Makefile, I want to then do this:
include library.mak
# special handling of c.txt
c.txt:
grep -v 'all except this' ${SRC_FOLDER}/$# > $#
As you can see, the user wants to be able to override the behaviour for some special cases. While this works, it always greets the user with the dreaded warning: overriding recipe for target and warning: ignoring old commands for target messages, even though the behaviour is as intended.
So here's the question: Can this be done in a different way that avoids these warnings, or is there a means to suppress them?
The warning says you overwrite a recipe for an explicit rule. And this is really wrong. It makes sense to use a pattern rule instead, like: %.txt: ${SRC_FOLDER}/%.txt. Then it's ok to have some explicit rule (c.txt: ${SRC_FOLDER}/c.txt) overwriting a recipe. Of course, it's not 100% the same, but, I think that should not be a problem. Anyway, putting an explicit rule into a reusable file is a crime.
Next, you constantly use "phonies" and rules w/o prerequisites even when you definitely should have them. This is bad. You're trying to make "make" work as a non-branching shell script. That's not only inefficient but also is a misuse. Basically, make can be viewed as "shell extension" capable of "branching on file timestamps in a sophisticated way". If you don't need it, don't use make at all.
Next, I see absolutely no reason to go into recursive make. Whatever, you're going to do, I believe, you can do without it. And even if you really need recursion, then write simply $(MAKE). $(MFLAGS) shoud not be used anymore (read this).
P.S. There's yet another (more flexible) option to define parameterized variables (macros). For example:
define nl :=
endef
define myrule.text
$1: $2/$1
#cp $$< $$#
endef
myrule = $(eval $(foreach foo$0,$1,$(call myrule.text,$(foo$0),$2)$(nl)))
...
# user makefile
# for everything except c.txt call predefined macro
$(call myrule,$(filter-out c.txt,$(TARGET_FILES)),$(SRC_FOLDER))
# for c.txt only
c.txt: $(SRC_FOLDER)/c.txt
grep -v 'all except this' $< >$#
But as you see, it's a bit of a "cryptic" stuff, and I don't feel like recommending it for a beginner.
Some code from Makefile:
tempDir := ...
javaSources := $(wildcard src/java/**/%.java)
javaClasses := $(subst src/java, $(tempDir)/java/classes, $(subst .java,.class, $(javaSources)))
$(javaClasses): $(javaSources)
mkdir -p $(tempDir)/java/classes || true
javac \
-d $(tempDir)/java/classes \
-cp $(tempDir)/java/classes \
$?
How to create a pattern rule (like here) to preserve in / out order?
#MadScientist
First, your wildcard won't work. GNU make uses only basic shell globbing, which means it can't understand advanced globbing like ** meaning "search all subdirectories". Second, % is not a shell globbing character at all so you're just looking for files that are literally named %.java.
Instead you probably want something like this:
javaSources := $(shell find src/java -name '*.java')
Next, to create the javaClasses content you really don't want to use subst because it substitutes everywhere which can give false matches (e.g., $(subst .x,.y,foo.xbar) will yield foo.ybar which is probably not what you want).
Something like this is simpler to understand:
javaClasses := $(patsubst src/java/%.java,$(tempdir)/java/classes/%.class,$(javaSources))
Finally, you are repeating exactly the same error you made in the previous question, where you tried to list all the targets and all the prerequisites in the same rule. Just as I said for that question, that is not right.
The answer is exactly the same as in the previous question: you should write a pattern rule that describes how to build one single target from one single source file.
And again you need an all target or similar which depends on all the outputs.
In complement to MadScientist answer, you should probably use a pattern rule like:
$(tempDir)/java/classes/%.class: src/java/%.java
mkdir -p $(dir $#)
javac -d $(dir $#) -cp $(dir $#) $<
(not sure what -cp should be in this case, it depends on your specific project). And as MadScientist also suggested, you will need:
.PHONY: all
all: $(javaClasses)
such that you can call make all to compile all the source files that need to be. Put it before any other explicit target if you want all to be the default goal (the goal make selects if you just call make), or use the .DEFAULT_GOAL special variable:
.DEFAULT_GOAL := all
I have a data file that is processed by a script to produce multiple output files. Each of these output files is then processed further. Which files are created depends on the contents of the input file, so I can't list them explicitly. I can't quite figure out how to refer to the various files that are generated in a makefile.
Currently, I have something like this:
final.out: *.out2
merge_files final.out $(sort $^)
%.out2: %.out1
convert_files $?
%.out1: data.in
extract_data data.in
This fails with No rule to make target '*.out2', needed by 'final.out'. I assume this is because the .out2 files don't exist yet and therefore the wildcard expression isn't replaced the way I would like it to. I have tried to use the wildcard function but that fails because the list of prerequisites ends up being empty.
Any pointers would be much appreciated.
EDIT: fixed the list of prerequisites in second pass.
You apparently cannot compute the list of intermediate files before running the extract_data command. In this case a solution consists in running make twice. One first time to generate the *.out1 files and a second time to finish the job. You can use an empty dummy file to mark whether the
extract_data command shall be run again or not:
ifeq ($(FIRST_PASS_DONE),)
final.out: .dummy
$(MAKE) FIRST_PASS_DONE=yes
.dummy: data.in
extract_data $<
else
OUT1 := $(wildcard *.out1)
OUT2 := $(patsubst %.out1,%.out2,$(OUT1))
final.out: $(OUT2)
merge_files $# $(sort $^)
%.out2: %.out1
convert_files $?
endif
Unfortunately your question is missing some details I would ask immediately if some SW developer would present this makefile for review:
does extract_files provide the list of files?
does convert_files convert one file or multiple? The example seems to imply that it converts multiple.
then I have to question the decision to break up extract, convert and merge into separate rules as you will not benefit from parallel build anyway
The following is the approach I would choose. I'm going to use a tar file as an example for an input file that results in multiple output files
generate a makefile fragment for the sorted list of files
use the tar option v to print files while they are extracted
convert each line into a makefile variable assignment
include the fragment to define $(DATA_FILES)
if the fragment needs to be regenerated, make will restart after it has generated it
use static pattern rule for the conversion
use the converted file list as dependency for the final target
.PHONY: all
all: final.out
# extract files and created sorted list of files in $(DATA_FILES)
Makefile.data_files: data.tar
set -o pipefail; tar xvf $< | sort | sed 's/^/DATA_FILES += /' >$#
DATA_FILES :=
include Makefile.data_files
CONVERTED_FILES := $(DATA_FILES:%.out1=%.out2)
$(CONVERTED_FILES): %.out2: %.out1
convert_files $< >$#
final.out: $(CONVERTED_FILES)
merge_files final.out $^
UPDATE if extract_data doesn't provide the list of files, you could modify my example like this. But of course that depends on that there are no other files that match *.out1 in your directory.
# extract files and created sorted list of files in $(DATA_FILES)
Makefile.data_files: data.in
set -o pipefail; \
extract_data $< && \
(ls *.out1 | sort | sed 's/^/DATA_FILES += /') >$#
So some anonymous developers have decided to use a ridiculous convention of using spaces in their folder names that contain their source files. I would change these folders not to use spaces but sadly I don't make the rules around here so that's not an option (though I wish it were).
LUAC = luac
SRC_DIR = .
SOURCE = \
stupid/naming\ convention/a.lua \
stupid/naming\ convention/very\ annoying/b.lua \
vpath .lua $(SRC_DIR)
OUT_DIR = ../out/
OUTPUT = $(patsubst %.lua, $(OUT_DIR)/%.luac, $(SOURCE))
all: $(OUTPUT)
$(OUT_DIR)/%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#"
.PHONY: all
Simple Makefile. All it's meant to do is compile all the Lua files that I have and put them into an output directory.
No matter I do it keeps wanting to split the SOURCE string on the spaces in the folder, so I end with a beautiful error like this:
make: *** No rule to make target `stupid/naming ', needed by `all'. Stop.
Is there a way to fix this without renaming the folders?
Thanks in advance.
The very short, but IMO ultimately correct, answer is that make (not just GNU make, but all POSIX-style make implementations) does not support pathnames containing whitespace. If you want to use make, your "anonymous developers" simply cannot use them. If they insist that this is an absolute requirement you should switch to a different build tool altogether, that does support whitespace in filenames.
Yes, it's barely possible to create a makefile that will work with filenames containing whitespace, but you will essentially have to rewrite all your makefiles from scratch, and you will not be able to use many of the features of GNU make so your makefiles will be long, difficult to read, and difficult to maintain.
Just tell them to get over themselves. Or if they really can't, try having them create their workspace in a pathname without any whitespace in the names, then create a symbolic link containing whitespace pointing to the real workspace (the other way around won't work in all situations).
Unfortunately, GNU Make's functions that deal with space-separated list do not
respect the escaping of the space. The only exception is wildcard.
Edit:
Here's my workaround:
LUAC = luac
SRC_DIR = .
SOURCE = \
stupid/naming\ convention/a.lua \
stupid/naming\ convention/very\ annoying/b.lua \
vpath .lua $(SRC_DIR)
OUT_DIR = ../out/
OUTPUT = $(patsubst %.lua,%.luac,$(SOURCE))
all: $(OUTPUT)
%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#""
.PHONY: all
I tried to output it first like that:
%.luac: %.lua
#echo "$<"
#echo "$#""
Output looks as follows:
stupid/naming convention/a.lua
../out/stupid/naming convention/a.luac
stupid/naming convention/very annoying/b.lua
../out/stupid/naming convention/very annoying/b.luac
If you look at this excellent write up: http://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them, the author suggests that this is mostly a difficult task. But his substitution functions could get you going in case you really can't avoid the spaces.
Putting this into your makefile would look like this (sorry if I changed some of your paths, but this works on my Cygwin installation):
LUAC = luac
s+ = $(subst \\ ,+,$1)
+s = $(subst +,\ ,$1)
SRC_DIR = .
SOURCE := stupid/naming\\ convention/a.lua
SOURCE := $(call s+,$(SOURCE))
vpath .lua $(SRC_DIR)
OUT_DIR = out/
OUTPUT = $(patsubst %.lua, $(OUT_DIR)/%.luac, $(SOURCE))
all: $(call +s,$(OUTPUT))
$(OUT_DIR)/%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#"
.PHONY: all
I know that's not a complete answer, but maybe an encouragement that it actually is possible. But I agree with the other posters that if you can actually avoid spaces altogether, you will have a much easier life!
Another strategy which works when you are generating your Makefile automatically is this one, also used in Perl's ExtUtils::MakeMaker: to separate the name formatted to be usable in recipes, versus it being usable as a dependency. The example here has a THISFILE and a THISFILEDEP.
AWKWARD_DIR = sub dir
AWKWARD_DIRDEP = sub\ dir
THISFILE = $(AWKWARD_DIR)/d1
THISFILEDEP = $(AWKWARD_DIRDEP)/d1
AWKWARD_DIR_EXISTS = $(AWKWARD_DIR)/.exists
AWKWARD_DIR_EXISTSDEP = $(AWKWARD_DIRDEP)/.exists
TARGET = $(AWKWARD_DIR)/t1
TARGETDEP = $(AWKWARD_DIRDEP)/t1
MAKEFILE = spacemake.mk
$(TARGETDEP): $(THISFILEDEP) $(AWKWARD_DIR_EXISTSDEP)
cat "$(THISFILE)" >"$(TARGET)"
$(THISFILEDEP): $(AWKWARD_DIR_EXISTSDEP)
echo "yo" >"$(THISFILE)"
$(AWKWARD_DIR_EXISTSDEP): $(MAKEFILE)
#echo MAKEFILE = $(MAKEFILE)
-mkdir "$(AWKWARD_DIR)"
touch "$(AWKWARD_DIR_EXISTS)"
You can try it by placing it in a file called e.g. spacemake.mk, then run it with gmake -f spacemake.mk.