I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
Pretty new to GNU Make. This is a less complex example of something more general I have been trying to get to work.
I have many input files that have similar name format .txt, and I have a shell script that will take the input file and generate an output of the same name but with a different extension .wc. I have written the following Make file.
# name of dependencies
SRC = $(wildcard *.txt)
# get name of targets (substitute .wc for .txt)
TAR = $(SRC:.txt=.wc)
all: $(TAR)
%.wc: %.txt
sh word_count.sh $<
This runs fine, and will generate all the .wc output files. However, if I modify one of the input(dependency) files, they are all rebuilt. So the question is; what is the best way to get GNU Make to only process the modified .txt files in the directory?
In Gnumake, one can include a file as follows:
include some_file
I now wonder, is it possible to get the filepath of some_file from within that file, e.g. by doing $(shell pwd)? (I know that that command in particular doesn't work since it gets the path of the working directory, not the included file.)
You can use the MAKEFILE_LIST variable to obtain that. The last filename in that variable will be the current makefile, as long as you check it before you include any other files. Basically every time make reads a new makefile the name of that makefile is added to the end of the variable, but no value is ever removed from the variable even after the makefile is no longer being parsed:
THIS_MAKEFILE := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
I think you can use the readlink -f <file> command to get the path of the file. You just have to know from within the file what the filename is, which I'm assuming you can hardcode.
Consider having a makefile, which can generate some files using generating lines listed in a file. For example file 001 using first line, 002 using second line, etc.
This file can be changed (it has it's own dependences, but this doesn't metter).
If some lines in this file changed, appropriate files should be remade. But other files shouldn't.
The solution I found is this one: for every file there is flag-file which content is generating line it was made last time. After remaking file with generating lines, i check all this file, and remove them if line changed. So files which have dependences on removed files will be remade, and other files won't. But this works too slow if use msys-make.
Can you suggest any other solution, which doesn't need many extra calls to file system and executable runs.
If I understand your description correctly, what you're describing is a Makefile which depends on another file that is functionally a Makefile but for unknown reasons uses a different format.
Turn that file into Makefile format and include it into the original Makefile. (If you're using GNU Make.)
We have what may be similar to your problem. We have xml files, say foobar.xml which contains the dependencies for foobar.out
<files>
<file>a</file>
<file>b</file>
<file>c</file>
</files>
We decided to adhere to this simple layout, so we don't need to parse xml. We use makepp (because we got fed up with gmake not noticing dependencies like changed commands). This gives us the builtin &sed command (the expression is actually Perl programming, but as you see, you don't need to get into it much). Here's what we do with three simple substitutions for the three kinds of lines:
%.d: %.xml
&sed 's!<files>!$(stem).out: \\! || s!<file>(.+)</file>!$$1 \\! || s!</files>!!' \
$(input) -o $(output)
include foobar.d
This produces foobar.d which we then include:
foobar.out: \
a \
b \
c \
Stackoverflow is swallowing the last empty line here, which avoids having to worry about the trailing backslash.
as a part of build process I need to process some files with m4. These files are given an '.in' extension and are located not only in the top-level dir, but also in subdirs.
I locate them using find and process them in a shell loop.
Is there some makefile syntax magic to write simpler rule to process them save the output into a file (the '.in' extension stripped) in the same directory as the input file?
This is what I have in my makefile now:
PROCESS_FILES=$(shell find . -name \*.in)
WORK_FILES=$(subst .in,,$(PROCESS_FILES))
$(WORK_FILES): $(PROCESS_FILES)
for file in $(PROCESS_FILES); \
do \
m4 $$file > $${file%.*};\
done
You could write a set of rules to convert files with the suffix .in to files without that suffix.
In the classic notation, that would be something like:
# Not functional — see discussion
.SUFFIXES: .in # Add .in as a suffix
M4 = m4
M4FLAGS =
M4SCRIPT = xyz.m4
.in:
${M4} ${M4FLAGS} ${M4SCRIPT} $< > $*
This adds .in as a recognized suffix, and says that you convert the file with a .in suffix to the file without it using the command specified by the three macros. The $< is the name of the file with the .in extension; the $* is the name of the file without the extension.
Unfortunately though, that notation only works when the files to be converted have names such as xyz.in and need to be converted to xyz. It does not work for a case where xyz.h.in needs to be converted to xyz.h.
It would be possible to specify that the suffixes are .h.in and .h, but then the rule for converting between the two starts with .h.in.h: and make gets confused.
However, GNU make has an alternative notation for defining suffix rules which can accommodate this:
.SUFFIXES: .h.in .h # Add .h.in and .h as suffixes
M4 = m4
M4FLAGS =
M4SCRIPT = xyz.m4
%.h: %.h.in
${M4} ${M4FLAGS} ${M4SCRIPT} $< > $*.h
The %.h: %.h.in line is able to deal with the suffix with two dots. The only nuisance is that for each extension such as .c.in or .y.in or .mk.in, you have to provide similar mappings.
(Tested on Mac OS X 10.7.4. If that makefile is xyz.mk, you can create an empty file xyz.h.in and then run make -n -f xyz.mk xyz.h and you'll see the build would run the command m4 xyz.m4 xyz.h.in > xyz.h.)
If you have a source file path/to/file.in, a rule for making % from %.in, and a dependency on the output file path/to/file, that should be all you need.
Perhaps you want to express your dependencies in terms of the output from find. In GNU Make this is easy; in other dialects, perhaps you want to generate and include something like find -name '*.in' | sed 's/\.in$//' >make.dep