I have a datafile that depends on other datafiles, but the generating script ignores if they are missing. The script automatically creates the makefile, including the ones that do not exist - that in the case they get created, the depending file is recreated.
data: infile1.raw infile2.raw
touch $#
infile1.raw:
infile2.raw:
Adding empty rules avoids an error, but triggers an unwanted recreation.
Implicit rules work sometimes, but not reliable - %.raw: causes make to complain that there is no rule to make the infile1.raw targets. Explicitly stating that it does not need to be build with %.raw:; removes the error, but causes again a rebuild. I would hope to avoid wildcards such as
data: $(wildcard infile*.raw)
as it very hard to automate this in such a way that not far to many files are matched.
Is there a way to achieve that make ignores missing dependencies?
Based on #user657267's answer, it seems you can simply use wildcard with a fixed file name:
data: $(wildcard infile1.raw) $(wildcard infile2.raw)
touch $#
Note that this will not detect the case where an input file was deleted (e.g. if you concatenate all input files, deleting one will not cause the target to be rebuilt).
If wildcard alone is giving you too many spurious matches, use wildcard with a whitelist
whitelist := infile1.raw infile2.raw
data: $(filter $(whitelist),$(wildcard infile*.raw))
Related
In my project, I have a set of sub-directories that contain package.yaml files, for e.g.:
A/package.yaml
B/package.yaml
C/package.yaml
If I run hpack A/package.yaml, the file A/A.cabal is (re-)generated. The list of such directories can change over time, so I want to use GNU make to find all immediate sub-directories containing package.yaml files and generate the corresponding .cabal files using hpack.
I tried this based on another question, but it didn't work:
HPACK_FILES := $(wildcard */package.yaml)
PKG_DIRS := $(subst /,,$(dir $(HPACK_FILES)))
CABAL_FILES := $(addsuffix .cabal,$(join $(dir $(HPACK_FILES)),$(PKG_DIRS)))
test:
#echo $(CABAL_FILES)
update-cabal: $(CABAL_FILES)
%.cabal: package.yaml
hpack $<
However, make update-cabal says there's nothing to be done. make test however does output the right cabal files. How can I fix this?
Cheers!
The problem is this:
%.cabal: package.yaml
There is no file package.yaml. The files are named things like A/package.yaml. That is not the same thing.
Because the prerequisite doesn't exist, make decides that this pattern rule cannot match and so it goes looking for another rule that might be able to build the target. It doesn't find any rule that can build the target, so make says there's nothing to do because all the output files already exist.
Unfortunately what you want to do is not at all easy with make, because make is most comfortable with input and output files that are tied together by the filename with extensions, or similar. And in particular, it has a really hard time with relationships where the variable part is repeated more than once (as in, A/A.cabal where the A is repeated). There's no easy way to do that in make.
You'll have to use an advanced feature such as eval to do this. Something like:
# How to build a cabal file
%.cabal:
hpack $<
# Declare the prerequisites
$(foreach D,$(dir $(HPACK_FILES)),$(eval $D/$D.cabal: $D/package.yml))
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
At the end of my build, make deletes a file:
Removing intermediate files...
rm some/dir/myfile.inc
I want to keep it to make later builds faster, but I have not been able to. Either one of these two lines should force make to keep all intermediate files, as in this question, but they have no effect:
.SECONDARY:
.PRECIOUS:
If I spell out the exact file path it does keep the file:
.SECONDARY: some/dir/myfile.inc
If but this does not work
.SECONDARY: %.inc
Is the .SECONDARY rules being ignored because of a later rule like this is more specific?
%foo.o %foo.h %foo.inc: some_other_file
dosomething
For a project of mine I am automatically generating makefiles and including them, like this:
all:
#echo 'SUCCESS is $(SUCCESS)'
clean:
rm depend.mk
depend.mk:
#echo 'Creating $#'
#echo 'SUCCESS := 1' > $#
.PHONY: all clean
include depend.mk
This works, but the include line generates a warning message:
$ make
Makefile:13: depend.mk: No such file or directory
Creating depend.mk
SUCCESS is 1
I would like to silence that first warning line saying that depend.mk doesn't exist. I know it doesn't exist since I have a rule written to generate it, so the warning is unnecessary (unless of course there isn't a rule for it). I do NOT want make to ignore the error where the included file doesn't exist and there is no rule for it, so prefixing include with a - to ignore the error will not work for me. I'd like something similar to bash's convention of piping stderr to /dev/null like some_cmd 2>/dev/null but for including in make.
The sample above is a very simplified example of this case. In my actual project there are a lot of automatically generated makefiles (via clang's automatic dependency generation) being included, meaning a fresh run of make will flood my screen with these warning messages.
Is anything like this possible, or am I just going to have to deal with the annoying warning messages?
I've encountered and (re-re-re-re-)solved this problem a number of times myself. Really, the problem is in the thinking surrounding when the dependency files are generated and used.
This link has the detailed description of the "resolution": http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
Basically it comes down to the fact that dependency files are really only necessary for rebuilding, not the initial building of your library/executable. Resultantly you don't need to have a rule for generating dependency files up front (which is in fact less efficient), you instead should generate them during the object file step as intermediate files marked precious (so they're created and tracked as side-effect files that should never be cleaned up automatically). Subsequent builds will then have the files available, which is exactly what you were trying to achieve overall. You can then make it a "-include" on the dependency files, with the foreknowledge that your object file build step will fail if the dependency file generation fails, giving an immediate error, as you've mentioned is preferred, rather than an obscure and indirect one much later.
I've actually done a couple rather large build systems implementing this method, and it does work quite well, including ones that used non-GNU toolchains. To an outside user it appears identical, but internally it performs more efficiently and isn't hiding potentially important errors.
I tried many (many!) things to see if I could prevent or redirect the error message. No luck.
But when I tried -include (include with a leading dash), it didn't give an error, and make with clean, all, depend.mk and 'default' all worked properly and as expected.
Is there a particular reason you didn't want to use the -include variant? Seems to do exactly what you're looking for, and doesn't alter how the Makefile works in any way, just doesn't show the error during the first pass through the Makefile.
I have a Makefile with several rules of this form
protolist.c: $(PROTOCOLS) Makefile src/genmodtable.sh
$(SHELL) $(srcdir)/src/genmodtable.sh \
$# $(filter-out %Makefile %genmodtable.sh, $^)
As the name implies, protolist.c winds up containing a list of all the "protocols" defined by the .c files in $(PROTOCOLS). The contents of this file do formally depend on everything in $(PROTOCOLS), the Makefile, and the generator script, but it's very rare for the file to actually change when one of those .c files is edited. Therefore, genmodtable.sh is coded to not change the timestamp of protolist.c if it's not going to make any change to its contents. This causes Make to skip rebuilding protolist.o and its dependencies when it's not really necessary.
That all works fine; the problem is that, because protolist.c now appears to be out of date with respect to its dependencies, Make thinks it has to try to regenerate protolist.c on every run. This isn't a performance issue -- the script is very fast -- but it is confusing behavior. I dimly recall an idiom, involving timestamp files, that could be used to stop Make from doing this, but I have not been able to reconstruct it or find it described anywhere. Does anyone know what it is?
(Also if anyone can suggest how to get rid of that silly $(filter-out ...) construct, that would be helpful, as that is the only GNUmakeism in this Makefile.)
This appears similar to an issue with Fortran programming and make, involving the files generated when compiling a module. (Not relevant, other than that is where I picked up how to do this.)
What you want is have make compare the timestamp of protolist.o to the timestamp of protolist.c, which remains 'old', and make the decision to run the recipe for protolist.c, depending on the timestamp of, well, a timestamp file, which gets updated each time the recipe is run.
In order to make this work, you have to link the two together with an empty rule.
protolist.o: protolist.c
[...]
protolist.c: protolist.c.time ;
protolist.c.time: $(PROTOCOLS) Makefile src/genmodtable.sh
$(SHELL) $(srcdir)/src/genmodtable.sh \
protolist.c $(filter-out %Makefile %genmodtable.sh, $^)
touch protolist.c.time
In my own makefiles, I have to declare the timestamp files as prerequisites of the special target .PRECIOUS, to prevent make from deleting them, but I'm using pattern rules; I'm not 100% sure, but I think this isn't necessary when using explicit rules, like here.
To avoid the $(filter-out ...) construct, can you not simply replace it with $(PROTOCOLS)?
(Although, personally, I would stick to Paul's First Rule of Makefiles: Don't hassle with writing portable makefiles, use a portable make instead.)