With GNU Make, I want to take a list of files and create symbolic links of them in another directory (with the same filenames), but also overwrite any existing link with the same filename.
existing_files = $(wildcard dir1/dir2/*.txt)
# The next line shows where I would want to put the symbolic links
symlinks = $(wildcard new_dir1/new_dir2/*.txt)
make_some_links:
# Remove previous symlinks if they share name as existing_files
ifeq ($(notdir $(existing_files)), $(notdir $(symlinks)))
$(info Removing previous symlinks)
#rm $(symlinks)
endif
# Loop to link here
As an aside, does the above condition attempt to match each element of each list, or does it succeed if only 1 of the elements match?
I can't quite figure out how to loop through in Make, including with $(foreach), so I wrote the following loop to show what I mean in something like Python with the same variable names.
for i in len($(existing_files)):
#ln -s $(existing_files)[i] $(symlinks)[i]
Here, the first element in $(list_files) is linked to the first element in $(symlinks). Any insight on how to write this loop in Make, or if there is straight-up a better way to approach this would be very helpful. Thanks.
Let's take this in stages.
First, to create a symbolic link, and remove a preexisting link of that name, if there is one:
ln -fs filename linkname
Now to make a list of the existing files:
existing_files = $(wildcard dir1/dir2/*.txt)
So far, so good. Let's suppose this gives us dir1/dir2/red.txt dir1/dir2/green.txt.
# The next line shows where I would want to put the symbolic links
symlinks = $(wildcard new_dir1/new_dir2/*.txt)
That will give you a list of the things that already exist in that directory, which is probably not what you intend. We must construct the list of links we want from the list of files we have:
filenames := $(notdir $(existing_files))
symlinks := $(addprefix new_dir1/new_dir2/, $(filenames))
Now for a rule or rules to build the symlinks. We could write two explicit rules:
new_dir1/new_dir2/red.txt: dir1/dir2/red.txt
ln -fs dir1/dir2/red.txt new_dir1/new_dir2
new_dir1/new_dir2/green.txt: dir1/dir2/green.txt
ln -fs dir1/dir2/green.txt new_dir1/new_dir2
but that is horribly redundant, and besides we don't know the file names beforehand. First we can remove some of the redundancy by defining a variable and using the automatic variale $<:
DEST_DIR := new_dir1/new_dir2
$(DEST_DIR)/red.txt: dir1/dir2/red.txt
ln -fs $< $(DEST_DIR)
$(DEST_DIR)/green.txt: dir1/dir2/green.txt
ln -fs $< $(DEST_DIR)
Now we can see how to replace these rules with a pattern rule:
$(DEST_DIR)/%.txt: dir1/dir2/%.txt
ln -fs $< $(DEST_DIR)
Now all we need is a master rule that requires the links:
.PHONY: make_some_links
make_some_links: $(symlinks)
In most cases, when you start designing shell loops in make recipes, you are missing important features of make. Make will naturally "loop" over the targets you declare to reach the goals you ask it to reach. And make offers many ways to factorize similar rules.
But in your case where the targets are symbolic links there are several issues to consider:
If you declare the links as regular targets and if some exist already but point to other files than the ones you want, make could consider them as up-to-date and skip them, instead of replacing them. You must thus find a way to force make to re-create all the links or, at least, those that do not point to the correct file. And the last modification times are not sufficient for this.
Even with the -f option ln will not replace an exiting directory (or a link to a directory), instead it will create the link inside the directory. You must thus first delete any directory (or link to a directory) that conflicts with a target link.
The ln command shall be used with the -sr options to create symbolic links relative to link location.
The target directory (new_dir1/new_dir2) shall exist before you try to create the first link.
If you don't care re-creating links that are already correct, things are easy:
declare a phony dummy target (force) and make it a prerequisite of all your links to force make re-creating them,
before creating your links delete existing links, files or directories that conflict with them,
use the proper ln options,
declare the target directory as an order-only prerequisite of the links and add a rule to create it if it is missing.
The following should do the job:
source_dir := dir1/dir2
link_dir := new_dir1/new_dir2
existing_files := $(wildcard $(source_dir)/*.txt)
symlinks := $(patsubst $(source_dir)/%,$(link_dir)/%,$(existing_files))
.PHONY: all force
all: $(symlinks)
$(link_dir)/%: $(source_dir)/% force | $(link_dir)
rm -rf $#
ln -sr $< $#
force:;
$(link_dir):
mkdir -p $#
If you want to avoid re-creating the existing correct links, things are more difficult because make needs to know which ones are correct and which ones are not, and this cannot be based on last modification times... But as I suspect that the performance impact of re-creating the existing correct links is negligible I suggest that we don't design a complex solution to a non-existing problem.
Related
Right now I have the following in my make file to create a symbolic link to a file in my current directory:
MY_FILE := "$(SOME_PATH)/file.txt"
ln -s $(MY_FILE)
What I would like is to do some sort of glob operation to link a bunch of files (or none).
MY_FILES := "$(SOME_PATH)/*.txt"
for file in files:
ln -s $(MY_FILE)
Could anyone point me in the right direction to do this?
Thanks!
The following should do what you want:
MY_FILES := $(wildcard $(SOME_PATH)/*.txt)
LINKS := $(notdir $(MY_FILES))
.PHONY: links clean-links
links: $(LINKS)
$(LINKS): %: $(SOME_PATH)/%
ln -s $<
clean-links:
rm -f $(LINKS)
Explanation:
Make functions. wildcard and notdir are two make functions. Knowing the make functions (at least the most frequently used) really helps writing nice, elegant and efficient make files. Of course, if you have spaces or special characters in your file names you will encounter some problems because most make functions consider spaces as separators. But if you have such files and directory names you should probably use something else than make.
Static pattern rules:
$(LINKS): %: $(SOME_PATH)/%
is a static pattern rule. For each word foo.txt in $(LINKS) it instantiates one single make rule:
foo.txt: $(SOME_PATH)/foo.txt
ln -s $(SOME_PATH)/foo.txt
Knowing how to use pattern rules (static or not) is essential if you want to write compact and generic make files.
Automatic variables. $< is a make automatic variable. In recipes (the commands part of rules) it expands as the first listed pre-requisite. There are many other automatic variables and they are quite handy to write generic rules.
Phony targets. links and clean-links are phony targets because they do not correspond to real files that we want make to create or update. They are kind of short-hands for actions. make links creates all missing links and make clean-links removes them all. As make has no way to guess that these targets are not regular file names we tell it with the .PHONY special target.
Make creates "targets" from "prerequisites". In your case MY_FILES holds prerequisites to create the (link) targets. The rule for a single file might look like this:
file.txt: $(SOME_PATH)/file.txt
ln -s $<
You want multiple files, and in this case we can use this pattern rule:
%.txt: $(SOME_PATH)/%.txt
ln -s $<
You can now create links with the make commands:
make file.txt
make otherfile.txt
Finally we will look into the wildcard function to get all files, and the function notdir function to get the link names from the file names. Following rule will print all text files, and depends on all files in SOME_PATH linked to current working directory:
MY_FILES := $(notdir $(wildcard $(SOME_PATH)/*.txt))
cat: $(MY_FILES)
cat $^
%.txt: $(SOME_PATH)/%.txt
ln -s $<
In my project, I have a set of sub-directories that contain package.yaml files, for e.g.:
A/package.yaml
B/package.yaml
C/package.yaml
If I run hpack A/package.yaml, the file A/A.cabal is (re-)generated. The list of such directories can change over time, so I want to use GNU make to find all immediate sub-directories containing package.yaml files and generate the corresponding .cabal files using hpack.
I tried this based on another question, but it didn't work:
HPACK_FILES := $(wildcard */package.yaml)
PKG_DIRS := $(subst /,,$(dir $(HPACK_FILES)))
CABAL_FILES := $(addsuffix .cabal,$(join $(dir $(HPACK_FILES)),$(PKG_DIRS)))
test:
#echo $(CABAL_FILES)
update-cabal: $(CABAL_FILES)
%.cabal: package.yaml
hpack $<
However, make update-cabal says there's nothing to be done. make test however does output the right cabal files. How can I fix this?
Cheers!
The problem is this:
%.cabal: package.yaml
There is no file package.yaml. The files are named things like A/package.yaml. That is not the same thing.
Because the prerequisite doesn't exist, make decides that this pattern rule cannot match and so it goes looking for another rule that might be able to build the target. It doesn't find any rule that can build the target, so make says there's nothing to do because all the output files already exist.
Unfortunately what you want to do is not at all easy with make, because make is most comfortable with input and output files that are tied together by the filename with extensions, or similar. And in particular, it has a really hard time with relationships where the variable part is repeated more than once (as in, A/A.cabal where the A is repeated). There's no easy way to do that in make.
You'll have to use an advanced feature such as eval to do this. Something like:
# How to build a cabal file
%.cabal:
hpack $<
# Declare the prerequisites
$(foreach D,$(dir $(HPACK_FILES)),$(eval $D/$D.cabal: $D/package.yml))
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
Before I start, I'll mention that I'm not using GNU Make in this case for building a C/C++ project.
Makefile:
DEST_DIR = build/
SRC_DIR = src/
$(SRC_DIR)a/ : $(SOMETHING_ELSE)
$(DO_SOMETHING_TO_GENERATE_A_DIR)
$(DEST_DIR)% : $(SRC_DIR)%
cp -r $^ $#
ALL_DEPS += <SOMETHING>
... more code which appends to ALL_DEPS ...
.PHONY: all
all : $(ALL_DEPS)
I've got some files not generated via Make rules in $(SRC_DIR). (For the sake of this example, let's say there's a directory $(SRC_DIR)b/ and a file $(SRC_DIR)c .)
I want to append to ALL_DEPS all targets which represent files or directories in $(DEST_DIR) so that "make all" will run all of the available $(DEST_DIR)% rules.
I thought to do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(wildcard $(SRC_DIR)*)))
But of course, that doesn't catch anything that hasn't yet been made. (i.e. it doesn't append $(DEST_DIR)a/ to the list because $(SRC_DIR)a/ doesn't yet exist when the $(wildcard ...) invocation is evaluated and the shell doesn't include it in the results returned by the $(wildcard ...) invocation.)
So, rather than a function which finds all (currently-existing) files matching a pattern, I need one which finds all targets matching a pattern. Then, I could do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(targetwildcard $(SRC_DIR)*)))
If it matters any, I've got much of the GNU Make code split across multiple files and included by a "master" Makefile. The ALL_DEPS variable is appended to in any of these files which has something to add to it. This is in an attempt to keep the build process modular as opposed to dropping it all in one monster Makefile.
I'm definitely still learning GNU Make, so it's not unlikely that I'm missing something fairly obvious. If I'm just going about this all wrong, please let me know.
Thanks!
It is simply not possible to do what you're trying to do; you're trying to get make to recognise something that doesn't exist.
This is part of the reason why, in general, wildcards are bad (the other being that you can end up including stuff you didn't mean to). The right thing to do here is to explicitly create a list of source files (ls -1 | sed -e 's/\(.*\)/sources+=\1/' > dir.mk) and perform the patsubst transformation on that list.
If you have additional files that are generate as part of the build, then you can append them to that list and their rules will be found as you'd expect.
I have a list of libraries where each have 2 files (.so, .dll).
How should I create a make rule which would execute the recipe only once if both of the files are missing or if one of them is missing.
LIBS = alib blib
LIBS_SO = $(patsubst %, %.so, $(LIBS))
LIBS_DLL = $(patsubst %, %.dll, $(LIBS))
If I make this target
$(LIBS_SO) $(LIBS_DSS):
cp .....
it copies only once for all of the the possibilities.
If I make this:
all : $(LIBS_SO) $(LIBS_DSS):
$(LIBS_SO) $(LIBS_DSS):
cp .....
I copy in all cases of any missing files.
I want to copy the alib directory if both or one of the files alib.dll / alib.so is missing; the
the same with blib.
You have a consistent typo of LIBS_DSS where you (presumably) meant LIBS_DLL.
Your first 'rule' is a shorthand for:
alib.so:
cp ...
blib.so:
cp ...
alib.dll:
cp ...
blib.dll:
cp ...
So, when asked to build, make builds the first target in the file, which is alib.so. That's why it does it once.
The second version, when fixed to remove the extra colon and the typo, should work:
all: $(LIBS_SO) $(LIBS_DLL)
$(LIBS_SO) $(LIBS_DLL):
cp .....
The default rule is all; to make all, make ensures that each of the files alib.so, blib.so, alib.dll and blib.dll exists and is up to date. It should execute the commands once for each missing target.
You might conceivably run into trouble if you run a parallel make; make -j4 or something similar. It might launch four copy commands almost simultaneously to make each of the targets. But in a non-parallel build, it will ensure alib.so is up to date (and if it isn't, will do the copy). If that copy also copies alib.dll, then it won't recopy when it ensures alib.dll is up to date.
You haven't given us much information, but I think this will do what you want:
all : $(LIBS_SO) $(LIBS_DLL):
%.so %.dll:
cp $* directory ...
If both alib.so and alib.dll are missing, Make will execute this rule only once.
EDIT: Thanks to Jonathan Leffler for catching the typo.