I'm using a makefile to build some documents from markdown. Since I have various possible outputs, I'm using a pattern rule. And since the input documents may or may not have associated tables, they have a wildcard dependency for csv files with the same stem filename:
tables = $(wildcard $*_*.csv)
%.docx: %.md $(tables)
#echo building $#
#touch $#
This works fine the first time:
$ touch thing.md
$ touch thing_table.csv
$ make thing.docx
building thing.docx
$ ls thing.docx
thing.docx
However, if the table file is updated, make still thinks everything's up to date:
$ touch thing_table.csv
$ make thing.docx
make: `thing.docx' is up to date.
I think this must have something to do with the order in which make evaluates things, but I don't understand it well enough to figure out how to make this work.
I can do sort of what I want to do using a more "literal" pattern rule:
%.docx: %.md %_*.csv
#echo building $#
#touch $#
But of course this fails in the absence of appropriately-named csv files:
$ rm thing.docx thing_table.csv
$ make thing.docx
make: *** No rule to make target `thing.docx'. Stop.
Is there a way to specify a target dependency such that if a target is foo.docx the dependencies would be foo_*.csv, but have those dependencies be optional (i.e. the target still works if the files don't exist) AND have it appropriately update if those optional dependencies change?
Ok, it looks like secondary expansion will work here:
.SECONDEXPANSION:
%.docx: %.md $$(wildcard $$*_*.csv)
#echo building $#
#touch $#
Related
I'm trying to simply copy files that are modified using make. Here is the entire Makefile:
FILES = www/foo.html www/bar.html www/zap.php
all: $(FILES)
$(FILES): src/$#
cp src/$# $#
clean:
rm $(FILES)
After modifying a file src/www/bar.html, make does not copy the file:
$ make
make: Nothing to be done for 'all'.
$ make www/bar.html
make: 'www/bar.html' is up to date.
Why does make not see the prerequisite has been modified and that the file needs to be copied?
If I run make clean, make it works (copies all files).
src/$# is not well-defined. You want
$(FILES): %: src/%
which declares a pattern rule, and restricts its scope to the files in $(FILES). (You might want or even need to remove this restriction.)
Consider the following makefile:
-include target.d
build:
%.d: %.c
#echo ">>> %.d: %.c"
touch $#
target.d: dependency
# #echo ">>> target.d: dependency"
dependency:
#echo ">>> dependency"
.PHONY: build
If you place this in a directory with two (empty) files, target.d and target.c and run make, then on one VM I use it produces the following output:
$ make
>>> dependency
>>> %.d: %.c
touch target.d
>>> dependency
>>> %.d: %.c
touch target.d
make: Nothing to be done for `build'.
On another VM, it loops infinitely. Both VMs are running Centos7, and both of them are using the same version of GNU make (3.82).
(Note: if you uncomment the commented out line below the target.d target, then both produce the exact same output; that behavior makes sense to me, at least)
I am aware that by adding the recipe to the target.d: dependency makes it prioritise that recipe over the generic one, while simply adding a dependency. But what I don't understand is why one system causes an infinite loop to occur while another extremely similar system does not.
What is the reason for this strange behaviour?
(EDIT: I found out I can simplify the makefile and still see the same behaviour)
I think the issue here is with having a target called target.d that is the same as the include file target.d such that when you edit target.d by touching it this somehow invokes an "undefined" behaviour. It seems like in this case it is triggering a re-call of the makefile. I can distil your problem down to this:
-include target.d
target.d: dep
touch target.d
dep:
#echo dep
With the only files that exist being: makefile. target.d is created on the first run of make.
$ ls
makefile
$ make
dep
touch target.d
dep
touch target.d
dep
touch target.d
:
etc
:
$ ls
makefile target.d
I do not believe it is correct to have a target that is also an item you include in your makefile. Usually with .d (e.g. auto generated dep files with gcc) you would generate these files in a compile rule like %.o: %.c ... which creates the .o and .d files. Then you have your include %.d type line. But you should not have a target to generate .d files directly. I do not think this is valid - please someone correct me if I am wrong here.
E.g. makefile that I think you intend:
-include target.d
.PHONY: build
build: target.o
# Simulates a compile line that generates and object file (.o) and dependency file (.d)
target.o: target.c dependency
touch target.o
touch target.d
.PHONY: dependency
dependency:
#echo ">>> dependency"
output:
$ ls
makefile target.c
$ make
>>> dependency
touch target.o
touch target.d
$ ls
makefile target.c target.d target.o
So as #code_fodder's answer suggested, the entire makefile can be simplified heavily to
-include a
a: b
touch a
b:
#echo b
There is a valid question as to whether or not the inclusion of a makefile that you create is a good idea; As stated in the comments above, this is a distillation of part of a build system that I am working with but do not have control over, so c'est la vie.
So there remains the question as to why this behaves differently on two different systems?
The problem turned out to be the following: one VM was run on a remote server, and the other VM was being run locally. In particular, my host machine is a mac and I was sharing some of the file system between the mac and the VM. It appears that the filesystem that I am using on my Mac only keeps timestamps to second precision, while that on the VMs is kept to sub-millisecond precision. As a consequence the Mac doesn't pick up the "updated" file.
I was able to see this on the VM that shares files with the mac by adding sleep commands of various lengths:
-include a
a: b
sleep 0.8
touch a
b:
#echo b
which would cause it to run a finite, but quasi-random number of times. Or, more simply, running make; make which yields
$ make; make
b
touch a
b
touch a
make: `a' is up to date.
b
touch a
make: `a' is up to date.
i.e. running make twice in short succession lets the second run see the updated timestamp of the first, and so it doesn't trigger a second time.
I have a makefile like the following:
.PHONY: all
all: foo_1.txt foo_2.txt foo_xxx.txt
.PHONY: clean
clean:
rm -f foo_* bar_*
foo_%.txt: bar_%.txt
cp $< $#
#.PRECIOUS: bar_%.txt
bar_%.txt:
touch $#
bar_2.txt:
touch $#
The output of "make all" is folowing
touch bar_1.txt
cp bar_1.txt foo_1.txt
touch bar_2.txt
cp bar_2.txt foo_2.txt
touch bar_xxx.txt
cp bar_xxx.txt foo_xxx.txt
rm bar_xxx.txt bar_1.txt
The intermediate files created by the rule using pattern (bar_xxx.txt, bar_1.txt) are removed on the end. I have found that this behaviour can be supressed by .PRECIOUS (in the code is the line intentionally commented out).
Why are intermediate files created by rule with pattern removed by default and files created by rule without pattern are not?
By the definition of "intermediate files", you can't have an intermediate file created by an explicit rule (a rule "without a pattern").
See the section on Chains of Implicit Rules to understand this feature. If you have specific questions then please update your question.
I have a directory structure that looks like this:
$ tree
.
|-- dir
| `-- subdir
| `-- data
`-- makefile
where data is a file. My makefile looks like this:
all: dir/analysis
.SECONDEXPANSION:
%/analysis: %/subdir
%/analysis: $(addsuffix /data, $$^)
#echo TARGET: $# DEPS: $^
touch $#
When I run make, I would expect the result to look like this:
TARGET: dir/analysis DEPS: dir/subdir/data dir/subdir
touch dir/analysis
Instead, it just reports
make: *** No rule to make target `dir/analysis', needed by `all'. Stop.
If I change the first rule to dir/analysis: dir/subdir then it works as I expected. So I suspect that make ignores the first rule and skips straight to the second when the first rule is %/analysis: %/subdir. It also works as expected when both rules have dir/analysis as their target instead of just the first rule. MadScientists's answer to a different question here seemed to apply to my problem. I tried adding rules like
dummy_target: dir/subdir
mkdir -p dir/subdir
and
dir/subdir:
mkdir -p dir/subdir
to the end of makefile to try to make the dependency on the first rule an explicit target but this didn't change anything. I'm pretty new to make, so I'm probably missing something pretty stupid, but I can't for the life of me figure it out. How do I get the first and second rules to execute in the order they're written? I'm using Make version 3.81 in case that matters.
--EDIT--
If I actually add a command after the first rule, like #echo RULE 1 then that command executes and not the second one.
I think that you understand the makefile:
# Original
all: dir/analysis
.SECONDEXPANSION:
%/analysis: %/subdir
%/analysis: $(addsuffix /data, $$^)
#echo TARGET: $# DEPS: $^
touch $#
like this:
By default we make dir/analysis
We request secondary expansion
We say that whatever/analysis depends on whatever/subdir. So
that makes dir/subdir a prerequisite of dir/analysis
We say also that whatever/analysis depends on the thing we get by
suffixing /data to the prior prerequisites of whatever/analysis. (Secondary
expansion of $$^ gives us the prior prerequisites of the target). And
since the prior prerequisites of dir/analysis are dir/subdir, that now
makes dir/subdir dir/subdir/data the new prerequisites of dir/analysis.
Hence the output of the recipe ought to be:
TARGET: dir/analysis DEPS: dir/subdir/data dir/subdir
touch dir/analysis
This understanding is badly wrong. You're confusing and conflating the operation
of pattern rules with the operation or ordinary rules.
If a makefile says, e.g:
# A
xx: xa
xx: xb
xx: xc
touch $#
or even:
# B
xx: xa
xx: xb
%x: %c
touch $#
then the prerequisites of all the empty rules for the target xx are combined
with those of the (one) non-empty rule when xx is considered as a target. So
# A is equivalent to:
xx: xa xb xc
touch $#
and # B is equivalent to:
xx: xa xb
%x: %c
touch $#
and in both cases the prerequisites of xx are xa xb xc.
However:
%x: %a
%x: %b
touch $#
is not equivalent to:
%x: %a %b
touch $#
If you ever write an empty pattern rule, its effect is simply to delete
any prior pattern rule with the same target and prerequisite patterns. And if you
ever write a non-empty pattern rule, you simply replace
any prior pattern rule with the same target and prerequisite patterns. See
10.5.6 Canceling Implicit Rules
So:
.SECONDEXPANSION:
%/analysis: %/subdir
%/analysis: $(addsuffix /data, $$^)
#echo TARGET: $# DEPS: $^
touch $#
is equivalent to:
.SECONDEXPANSION:
%/analysis: $(addsuffix /data, $$^)
#echo TARGET: $# DEPS: $^
touch $#
Then when this pattern rule is considered with respect to dir/analysis,
there are no prior prerequisites: $^ will expand to the empty string, and
in secondary expansion, so will $$^. So finally the recipe is equivalent to:
# Residual
%/analysis: /data
#echo TARGET: $# DEPS: $^
touch $#
You can satisfy yourself of this by running both of the # Original and
# Residual makefiles in debug mode (-d) and comparing the outputs.
This explains why the outputs in both cases conclude:
make: *** No rule to make target 'dir/analysis', needed by 'all'. Stop.
A pattern rule will only be instantiated to make a target if doing so offers a way for the target to be made. Why else select it? Since the prerequisite /data does not exist, the pattern rule
is not viable and is discarded. Just as if you tried to make foo.o with the makefile:
%.o: %c
touch $#
when there is no foo.c in the directory.
If you really want to see the output you were expecting, then you can get it from:
.PHONY: all clean
all: dir/analysis
%/analysis: %/subdir %/subdir/data
#echo TARGET: $# DEPS: $^
touch $#
clean:
rm -f dir/analysis
But it is hard to believe that is what you really want to see. This makefile
makes the analysis dependent on the data and also on the directory where
the data resides. What can be the point of that?
%/analysis: %/subdir/data
presumably expresses the desired relationship between the data and the analysis:-
If the data doesn't exist then the analysis is out of date and cannot be made (till you get some data).
If the data exists and is older than the analysis then the analysis does not
need to be made.
If the data exists and is newer than the analysis then the analysis can and will be
made.
By making dir/subdir an independent prerequisite all that you achieve is to introduce
a requirement to remake the analysis if it becomes older than the directory in
which the data resides - regardless of the data. So if, e.g. I run touch dir/subdir,
that will require make to re-run the analysis, even though the data hasn't changed.
I can make sense of your motivation only by supposing that in
reality you also want make to make the data, and the directory where it resides, if
they happen not exist when an analysis is required. If that is the situation then
you want a makefile on the lines of:
.PHONY: all clean
all: dir/analysis
.SECONDARY:
%/subdir:
mkdir -p $#
%/subdir/data: | %/subdir
touch $#
%/analysis: %/subdir/data
#echo TARGET: $# DEPS: $^
touch $#
clean:
rm -f dir/analysis dir/subdir/data
Here, the pattern rule:
%/subdir/data: | %/subdir
makes %/subdir an order-only prerequisite
of %/subdir/data. It means that /dir/subdir is not considered when determining whether /dir/subdir/data needs
to be made, but if it is determined that /dir/subdir/data is to be made, then /dir/subdir will be made first. This
way, /dir/subdir will be created, if it does not exist, when you need it to put the data there, but will have no influence
on whether you need to make the data or the analysis.
The special target .SECONDARY is a technicality. It directs make not to automatically delete any subsequent targets
that it deduces to be intermediate artefacts that emerge from chained pattern rules. Without it, in this example make
would deduce that dir/subdir/data and dir/subdir/ are just waste-products of making dir/analysis and auto-delete them
at the end.
If you run this makefile initially with no dir/subdir/data and no dir/subdir/ you get:
$ make
mkdir -p dir/subdir
touch dir/subdir/data
TARGET: dir/analysis DEPS: dir/subdir/data
touch dir/analysis
And subsequently:
$ make clean
rm -f dir/analysis dir/subdir/data
$ make
touch dir/subdir/data
TARGET: dir/analysis DEPS: dir/subdir/data
touch dir/analysis
And then:
$ make
make: Nothing to be done for 'all'.
$ touch dir/subdir
$ make
make: Nothing to be done for 'all'.
I'm trying to write a makefile to produce several output files for each of several sources, using pattern rules.
I have the following Makefile (GNU Make 3.8.1):
all : foo.all bar.all
%.all : %.pdf %.svg
#echo Made $*
%.pdf :
touch $#
%.svg :
touch $#
.PHONY: foo.all bar.all
Since *.all do not represent real output files, I tried marking them as .PHONY. However, running make then doesn't work:
$ ls
Makefile
$ make
make: Nothing to be done for `all'.
According to make -d:
No implicit rule found for `all'.
Considering target file `foo.all'.
File `foo.all' does not exist.
Finished prerequisites of target file `foo.all'.
Must remake target `foo.all'.
Successfully remade target file `foo.all'.
Considering target file `bar.all'.
File `bar.all' does not exist.
Finished prerequisites of target file `bar.all'.
Must remake target `bar.all'.
Successfully remade target file `bar.all'.
Finished prerequisites of target file `all'.
Must remake target `all'.
Successfully remade target file `all'.
make: Nothing to be done for `all'.
which seems to be pretending to run the %.all rules, but skipping the bodies.
But with the .PHONY line commented out, Make runs the targets, but then spontaneously decides to delete the output files:
$ make
touch foo.pdf
touch foo.svg
Made foo
touch bar.pdf
touch bar.svg
Made bar
rm foo.pdf foo.svg bar.pdf bar.svg
According to make -d, it says:
Removing intermediate files...
Minimal example
A minimal example giving anomalous behavior:
%.all: %.out
#echo Made $*
%.out:
touch $#
I expect running make somefile.all to cause it to create the file somefile.out, but it gets deleted:
$ make somefile.all
touch somefile.out
Made somefile
rm somefile.out
Keeping make from deleting intermediary files
I recommend against using .PRECIOUS (see below as to why). Using .SECONDARY would preserve the .out files:
TARGETS=foo bar
all: $(TARGETS:=.all)
%.all: %.out
#echo Made $*
%.out:
touch $#
.SECONDARY: $(TARGETS:=.out)
$(TARGETS:=.all) just appends .all to all names in TARGETS. $(TARGETS:=.out) appends .out. We apparently cannot use %.out as a target of .SECONDARY. These just save having to relist all targets individually.
I prefer to not use .PRECIOUS for this because the documentation says
if make is killed or interrupted during the execution of their recipes, the target is not deleted.
This can leave corrupted files in the file system. Here's an example.
all: foo.all bar.all
%.all: %.out
#echo Made $*
%.out:
sh -e -c 'echo "{1, 2, 3" > $#; FAIL!; echo "}" >> $#'
.PRECIOUS: %.out
The FAIL! command simulates a tool that crashes in the middle of its work. Here's a shell session working with the Makefile above:
$ ls
Makefile
$ make
sh -e -c 'echo "{1, 2, 3" > foo.out; FAIL!; echo "}" >> foo.out'
sh: 1: FAIL!: not found
make: *** [foo.out] Error 127
$ cat foo.out
{1, 2, 3
Yikes... my foo.out file is incomplete. Let's try making again:
$ make
Made foo
sh -e -c 'echo "{1, 2, 3" > bar.out; FAIL!; echo "}" >> bar.out'
sh: 1: FAIL!: not found
make: *** [bar.out] Error 127
$ cat *.out
{1, 2, 3
{1, 2, 3
Make is none the wiser about files left around by earlier runs so when you run make again, it will take the corrupted files at face value. foo.out was not remade (despite the "Made foo" message) because it already exists and the Makefile went straight to trying to make bar.
.SECONDARY makes it so that:
The targets which .SECONDARY depends on are treated as intermediate files, except that they are never automatically deleted.
This means they are never automatically deleted just because they are intermediate files. The default make behavior of deleting targets that were being rebuilt if the tool rebuilding them crashed is not affected.
Using .PHONY with pattern rules
It seems though that .PHONY works only for targets that are explicit, not inferred. I've not found documentation confirming this. However, this works:
TARGETS:=foo bar
TARGETS_all:=$(TARGETS:=.all)
.PHONY: all
all: $(TARGETS_all)
.PHONY: $(TARGETS_all)
$(TARGETS_all): %.all: %.out
#echo Made $*
%.out:
touch $#
.SECONDARY: $(TARGETS:=.out)
In this rule $(TARGETS_all): %.all: %.out $(TARGETS_all): gives the list of targets to which the pattern can be applied. It makes foo.all and bar.all explicit targets. Without this, they would be inferred targets.
You can test that it works by creating file called foo.all in your directory and run make over and over. The foo.all file has no effect on make.
Your somefile.out files are considered intermediate by GNU make, which is why they are automatically deleted in your example. You can instruct GNU make to preserve these files by use the of .PRECIOUS special target, like this:
%.all: %.out
#echo Made $*
%.out:
touch $#
.PRECIOUS: %.out