Before I start, I'll mention that I'm not using GNU Make in this case for building a C/C++ project.
Makefile:
DEST_DIR = build/
SRC_DIR = src/
$(SRC_DIR)a/ : $(SOMETHING_ELSE)
$(DO_SOMETHING_TO_GENERATE_A_DIR)
$(DEST_DIR)% : $(SRC_DIR)%
cp -r $^ $#
ALL_DEPS += <SOMETHING>
... more code which appends to ALL_DEPS ...
.PHONY: all
all : $(ALL_DEPS)
I've got some files not generated via Make rules in $(SRC_DIR). (For the sake of this example, let's say there's a directory $(SRC_DIR)b/ and a file $(SRC_DIR)c .)
I want to append to ALL_DEPS all targets which represent files or directories in $(DEST_DIR) so that "make all" will run all of the available $(DEST_DIR)% rules.
I thought to do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(wildcard $(SRC_DIR)*)))
But of course, that doesn't catch anything that hasn't yet been made. (i.e. it doesn't append $(DEST_DIR)a/ to the list because $(SRC_DIR)a/ doesn't yet exist when the $(wildcard ...) invocation is evaluated and the shell doesn't include it in the results returned by the $(wildcard ...) invocation.)
So, rather than a function which finds all (currently-existing) files matching a pattern, I need one which finds all targets matching a pattern. Then, I could do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(targetwildcard $(SRC_DIR)*)))
If it matters any, I've got much of the GNU Make code split across multiple files and included by a "master" Makefile. The ALL_DEPS variable is appended to in any of these files which has something to add to it. This is in an attempt to keep the build process modular as opposed to dropping it all in one monster Makefile.
I'm definitely still learning GNU Make, so it's not unlikely that I'm missing something fairly obvious. If I'm just going about this all wrong, please let me know.
Thanks!
It is simply not possible to do what you're trying to do; you're trying to get make to recognise something that doesn't exist.
This is part of the reason why, in general, wildcards are bad (the other being that you can end up including stuff you didn't mean to). The right thing to do here is to explicitly create a list of source files (ls -1 | sed -e 's/\(.*\)/sources+=\1/' > dir.mk) and perform the patsubst transformation on that list.
If you have additional files that are generate as part of the build, then you can append them to that list and their rules will be found as you'd expect.
Related
In my project, I have a set of sub-directories that contain package.yaml files, for e.g.:
A/package.yaml
B/package.yaml
C/package.yaml
If I run hpack A/package.yaml, the file A/A.cabal is (re-)generated. The list of such directories can change over time, so I want to use GNU make to find all immediate sub-directories containing package.yaml files and generate the corresponding .cabal files using hpack.
I tried this based on another question, but it didn't work:
HPACK_FILES := $(wildcard */package.yaml)
PKG_DIRS := $(subst /,,$(dir $(HPACK_FILES)))
CABAL_FILES := $(addsuffix .cabal,$(join $(dir $(HPACK_FILES)),$(PKG_DIRS)))
test:
#echo $(CABAL_FILES)
update-cabal: $(CABAL_FILES)
%.cabal: package.yaml
hpack $<
However, make update-cabal says there's nothing to be done. make test however does output the right cabal files. How can I fix this?
Cheers!
The problem is this:
%.cabal: package.yaml
There is no file package.yaml. The files are named things like A/package.yaml. That is not the same thing.
Because the prerequisite doesn't exist, make decides that this pattern rule cannot match and so it goes looking for another rule that might be able to build the target. It doesn't find any rule that can build the target, so make says there's nothing to do because all the output files already exist.
Unfortunately what you want to do is not at all easy with make, because make is most comfortable with input and output files that are tied together by the filename with extensions, or similar. And in particular, it has a really hard time with relationships where the variable part is repeated more than once (as in, A/A.cabal where the A is repeated). There's no easy way to do that in make.
You'll have to use an advanced feature such as eval to do this. Something like:
# How to build a cabal file
%.cabal:
hpack $<
# Declare the prerequisites
$(foreach D,$(dir $(HPACK_FILES)),$(eval $D/$D.cabal: $D/package.yml))
Is there a way to let make determine the number of files to be recompiled before actually compiling? The problem is this: Consider having a quite big project with hundreds of source files. It would very convenient to have a rough idea of how long compilation will take, but to know that, one needs to know the number of files to be compiled.
The general answer is no, because your build could generate files which themselves are inputs to other rules which generate more files. And so on. However if a rough answer is good enough you can try the --dry-run flag. From the GNU make documentation...
“No-op”. Causes make to print the recipes that are needed to make the targets up to date, but not actually execute them. Note that some recipes are still executed, even with this flag (see How the MAKE Variable Works). Also any recipes needed to update included makefiles are still executed (see How Makefiles Are Remade).
As you can see, despite its name even the --dry-run flag will change the state of your build.
"make -n" will do the dry run. But you can't get the list of files to be rebuilt. May be you can write shell script to store the last modified time of files and get the list of files.
I think a found a decent solution for unix. Here SRC are your source files, HDR your headers and DEP the dependency files (something like DEP:=$(OBJ:.o=.d) )
isInDepFile+=$(shell grep -q $(modifiedFile) $(depFile) 1>&2 2> /dev/null && echo $(depFile))
COMPFILES=
checkDepFiles=$(foreach depFile,$(DEP), $(eval filesToCompile+=$(isInDepFile))) $(thinOutDepFiles)
thinOutDepFiles=$(foreach fileToCompile,$(filesToCompile),$(eval DEP=$(filter-out $(fileToCompile),$(DEP))))
countFilesToCompile: $(SRC) $(HDR)
$(eval modifiedFiles=$?)
$(foreach modifiedFile,$(modifiedFiles), $(call checkDepFiles))
$(eval numOfFilesToCompile = $(words $(filesToCompile)))
$(eval numDepFiles = $(words $(DEP)))
$(eval NumSRCFiles = $(words $(SRC)))
#echo $(NumSRCFiles) sources
#echo $(numDepFiles) files to leave
#echo $(numOfFilesToCompile) files to compile
#touch $#
This first generates a list of modified files within your source and header files lists. Then for each modified file it checks all dependency files for its filename. If a dependency file contains the current file we are looking at, it is added to the list of filesToCompile. It is also removed from the list of dependency files to avoid duplication.
This can be invoked in the main building rule of your project. The advantage of that over the dry run is that it gives you a simple number to work with.
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
I have bunch of C files, each with differnt path, so I have something like
SRC=/path1/path2/file1.c file2.c /usr/joe/files/file3.c
I want to build object file from each C file in build directory, so I did:
SRCBASE=$(notdir $(SRC))
OBJS= $(addprefix $(OBJDIR)/,$(SRCBASE:.c=.o))
This works fine, and I got the OBJS is build/file1.o build/file2.o build/file3.o
My question is how to write the implicit rule to build the object file, I tried to do:
build/%.o : %.c
gcc ....
But that seems now to work, since this rule did not catch the files with the full path.
I've tried to look at the gnu make, but did not find the answer there.
Many thanks.
You can use VPATH to locate files in other directories. Note that this does implicit searches, though, rather than using explicit paths. That is, you specify that you need file1.o through file3.o and that make should look first in ., then in /path1/path2, then in /usr/joe/files; so if . contains a file3.c you will get that one rather than the one in /usr/joe/files (or if not, but if there is one in /path1/path2, you will get that one).
It turns out that path searching is more often useful than explicit location anyway, so usually that's what you want. If you need to override something you just copy or symlink the "desired version" to the first place to be searched.
I have a Makefile with the following type of rule:
%.html:
./generate-images.py > $#
make $(patsubst %.png,%.gif,$(wildcard *.png))
The generate-images script writes not only the HTML file (to stdout) but several .png files to the current directory. The goal here is to convert them to .gif. (not really, but this is an example)
This works if I invoke it directly. The problem is: If I invoke it from another rule where foo.html is a dependency, the wildcard statement fails to find any files. In other words, it just called make with no arguments, which is not what I want here.
What's the deal with the wildcard? Or, is there a better way to do this?
While your problem may be something different, I clearly see one.
The whole text of all commands within the rule is simultaneously processed so that make's functions and variables get expanded. Assume you have no .png files in the directory, and you invoke make so it should regenerate them: a.png and b.png. Then, after you invoke make, the text of the rule would effectively look like this:
file.html:
./generate-images.py > file.html
make
because at the moment of reading the makefile there were no .png files! After the first line is executed, the files will appear, but the next line was already generated to be just "make".
And only when you invoke your makefile for the second time, will it expand to
file.html:
./generate-images.py > file.html
make a.gif b.gif
This is not what you want. So I suggest doing it in The Right Way.
# If you have batch conversion program, this may be helpful
images.stamp: *.png
convert_all_images $?
touch images.stamp
# OR, if you want convert one-by-one with means of make
images.stamp: $(wildcard *.png)
touch images.stamp
%.gif: %.png
convert_one --from=$^ --to=$#
# HTML would look like
%.html:
./generate-images.py > $#
make images.stamp
So when you invoke make all, it generates htmls and converts newly generated images. Note that it will only convert the images that are updated, which is what you want.
Thanks to Beta for pointing out the mess with gif/png extensions.
That sounds like it's evaluating all of the $() expressions as it's processing the Makefile, rather than as it executes each rule. You could add a rule to your makefile like so:
images: $(patsubst %.png,%.gif,$(wildcard *.png))
.PHONY: images
and then change your example snippet to
%.html:
./generate-images.py > $#
make images
so that Make evaluates the glob at the right time. This is something about which checking the manual might be worthwhile.