I have a makefile which is designed to get some datasets from the web and unzip them into folders (given by the archive names). Unfortunately, the archives on the website are grouped within a folder structure. For example, if I want dataset.tar.gz, which is part of set DIR1, I call wget with: www.example.com/DIR1/dataset.tar.gz.
This is then downloaded to dataset.tar.gz, which I can then extract with tar -xzf. My makefile for this is currently:
URL = www.example.com
FILES = DIR1/something.url DIR2/another.url
FOLDERS = $(notdir $(patsubst %.url, %, $(FILES)) )
%.url: # do nothing
%.tar.gz : %.url
wget -q $(URL)/$<
% : %.tar.gz
tar -xzf $<
all: $(FOLDERS)
Unfortunately, the target for the rule %.tar.gz : %.url does not resolve to (e.g.) something.tar.gz, but to DIR1/something.tar.gz, even though it produces the expected file.
This causes problems when I rerun the makefile, as it does not detect something.tar.gz (as it is looking for DIR1/something.tar.gz), and redownloads the dataset, wasting time and bandwidth.
Is there any way of stripping prefixes/directories from makefile targets, i.e. removing DIR1 from the rule %.tar.gz : %.url, in order that the makefile correctly checks to see if it is satisfied?
Related
I'm writing a Makefile to build a Latex document depending on plots whose data is generated from some other data by some python script.
It looks like this
% pdf plot needed by final document
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $$(dirname $#) && ../../latexrun $$(basename $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | build
% crunch.py will create data for plots needed by build/tikz-standalone/%.tex
PYTHONPATH=. tools/plots/crunch.py
build:
mkdir -p build build/other_stuff ...
crunch.py generates several data files in build/data which are needed by build/tikz-standalone/%.tex. To create these files it uses other files stored in the variable PLOT_DATA. I could put a list of the intermediate data files in build/data into the Makefile at the position of xy_data. I don't like this as this would require me to update the list whenever a new file is added. What I want is that all data files are recreated whenever crunch.py or $(PLOT_DATA) has changed.
Is there a way to express this in Make?
If you do not want to provide and maintain the list of the generated files you can turn your (implicitly) phony xy_data target into an empty file used as a marker. Simply touch it at the end of the recipe:
BUILDDIRS := build build/other_stuff ...
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $(dir $#) && ../../latexrun $(notdir $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | $(BUILDDIRS)
PYTHONPATH=. tools/plots/crunch.py
touch $#
$(BUILDDIRS):
mkdir -p $#
Note: I also improved a bit some other aspects:
Use of make functions dir and notdir instead of the shell equivalents.
Variable declaration for the build directories to avoid writing the same list several times, which is tedious and error prone.
Explicit list of all build directories as order-only prerequisites instead of just one, which could lead to unexpected results if this single one exists but not some others.
Generic rule for all build directories thanks to the $# automatic variable.
A directory with css files in it and for each file in the directory I want a separate minified .min.css file in the same directory.
So for example my.css get minified into my.min.css in the same directory.
First I tried
css/*.css:
uglifycss $# > $(subst .css,.min.css,$#)
make -B yeaaa victorie its working :D
But after doing make -B again I get a new separate file each time resulting in my.min.css my.min.min.css my.min.min.min.css
Then I tried
.PHONY: clean
css/*.css: clean
uglifycss $# > $(subst .css,.min.css,$#)
clean:
-rm css/*.min.css
DOH! after clean it is still remembering the files it deleted in the first place resulting again in a my.min.min.min.css file
How can I tell make to stop doing my.min.min.min.css?
(make: GNU Make 3.81 OSX)
First get a list of all files with the suffix .css:
ALL := $(wildcard *.css)
then remove files that have the suffix .me.css:
NEW := $(filter-out %.me.css,$(ALL))
and add the suffix to remaining files:
ME := $(patsubst %.css,%.me.css,$(NEW))
Then you add those files as prerequisites to the default target, and add you own recipe that builds those files, in this case a simple echo:
%.me.css:
echo 123 > $#
default: $(ME)
Compared to your approach, this has the benefit that you don't have to use the flag -B, as only the files that need to be built are built. Therefore invoking make is done by simply caling make without any targets or flags (assuming the makefile is named makefile or Makefile):
make
I'm using Automake.
I have a few source files listed in dist_man1_MANS like this:
dist_man1_MANS = some-file.1 some-other-file.1
Now, Automake + configure eventually generate this in Makefile:
dist_man1_MANS = some-file.1 some-other-file.1
# ...
install-man1: $(dist_man1_MANS)
# generated recipe here
Since I'm not prefixing the .1 files with $(srcdir), I assume that, since I run make from the build directory (its current working directory), it should find them in the build directory.
So, I'm doing an out-of-tree build, for example, in /tmp/build:
/path/to/src/configure --prefix=$(pwd)/install
make
make install
and the build succeeds, that is, make finds the man pages and installs them. They are not in the build directory, however. I add this to the generated Makefile:
install-man1: $(dist_man1_MANS)
#echo ">>> $(^)"
#echo "::: $(dist_man1_MANS)"
# generated recipe here
Now, I would assume that both echos print the same thing, because $^ means the names of all the prerequisites, with spaces between them. To my surprise, the output is:
>>> /path/to/src/some-file.1 /path/to/src/some-other-file.1
::: some-file.1 some-other-file.1
So:
How did make find the /path/to/src/ prefix exactly? Where does it come from in this very context?
Why do $^ and $(dist_man1_MANS) differ?
I found the answer.
Automake sets the VPATH variable in its generated Makefile, a special variable for make, to something like:
VPATH = /path/to/src
From the previous links:
4.5.1 VPATH: Search Path for All Prerequisites
The value of the make variable VPATH specifies a list of directories that make should search. Most often, the directories are expected to contain prerequisite files that are not in the current directory; however, make uses VPATH as a search list for both prerequisites and targets of rules.
So make searches for the some-file.1 and some-other-file.1 prerequisites in the current working directory first, then for /path/to/src/some-file.1 and /path/to/src/some-other-file.1 if it cannot find the first ones. In this case I understand why $^ is different from $(dist_man1_MANS): $^ is the list of effective (resolved) prerequisites.
I have a very bizzare problem with GNU make. I have the following files:
a/x.html
b/Makefile
b/c/Makefile
The contents of a/x.html are irrelevant. The contents of b/Makefile are as follows:
SRC=../a
all: x.html
%.html: ${SRC}/%.html
rsync $< $#
The contents of b/c/Makefile are the same, except for the definition of SRC:
SRC=../../a
If I run make in b/c/ the result is as expected:
rsync ../../a/x.html x.html
and x.html gets copied from a/ to b/c/.
However, if I run make in b/ the output I get is several lines of:
make: stat: ../a/../a/.. (repeated many times) ../a/x.html: File name too long
It seems that make is applying the rule for %.html recursively, but why? Is there something obvious I am missing?
To build a target that matches the pattern %.html (i.e. any target name that ends in .html), make applies the rule if it can build the dependency (target built from the original target with ../a/ prepended).
You ask to build x.html. This matches the pattern %.html, so the rule applies: make sees if it can build ../a/x.html.
../a/x.html matches the pattern %.html, so the rule applies: make sees if it can build ../a/../a/x.html.
../../a/x.html matches the pattern %.html, so the rule applies, etc.
The stem character can match any part of a path, including directory separators.
You can see what make is trying by running make -r -d (-d to show debugging output, -r to turn off built-in rules which would cause a huge amount of noise).
When you're in b/c, this stops at step 2 because ../../a/x.html exists but ../../../../a/x.html doesn't.
One way to fix this is to list the files on which you want to act. You can build that list from the list of files that already exist in ../a:
$(notdir $(wildcard ${SRC}/*.html)): %.html: ${SRC}/%.html
rsync $< $#
This has the downside that if the HTML files in ../a are themselves built by a rule in b/Makefile, then running make in b won't built them in a pristine source directory. This shouldn't be a problem though: it would be unusual to have a makefile in b build things outside b.
Another approach which doesn't have this defect is to use an absolute path.
%.html: $(abspath ${SRC})/%.html
rsync $< $#
This seems slightly related to How to write Makefile where target and source files have the same extension?. In that question the extensions are the same, but the input and output files seem to be in the same directory and filenames are being conditionally renamed.
I have a large collection of .txt files in ../src/ that need to be processed, and dumped into ./ (which is a directory called target/) as txt files of the same name. I want to use make, so that only files in ../src/ that have been changed get updated in ./. I would like to get the prototype working before I put the real code in.
My Makefile in ./ is as follows:
DIR = ../src
INPUTS = $(wildcard $(DIR)/*.txt)
OUTPUTS = $(patsubst $(DIR)/%.txt,%.txt,$(INPUTS))
all: $(OUTPUTS)
.PHONY: $(INPUTS)
check:
#echo "DIR = $(DIR)"
#echo "INPUTS = $(INPUTS)"
#echo "OUTPUTS = $(OUTPUTS)"
%.txt: $(DIR)/%.txt
sed -e "s/test/cat/g" "$<" > $#
For now, the contents of ../src/ are test1.txt and test2.txt.
As the Makefile stands now, running make test2.txt generates the file as expected.
target/ $ make test2.txt
sed -e "s/test/cat/g" "../src/test2.txt" > test2.txt
Running make check shows the INPUTS and OUTPUTS correctly.
target/ $ make check
DIR = ../src
INPUTS = ../src/test1.txt ../src/test2.txt
OUTPUTS = test1.txt test2.txt
If I run make all, it generates every file, every time. This is expected with the .PHONY $(INPUTS) line in there.
If I remove the .PHONY $(INPUTS) target, Make gets all bound up in itself trying to find the target to make ../src/test1.txt and keeps prefixing $(DIR) in front of it until it makes too long of a filename and gives up.
make: stat: ../src/../src/../src/ [repeat for a few pages] ../src/../src/test1.txt: File name too long
make: stat: ../src/../src/../src/ [repeat for a few pages] ../src/../src/../src/test1.txt: File name too long
make: *** No rule to make target `../src/../src/../src/[repeat]../src/../src/test1.txt', needed by `../src/[repeat]../src/../src/test1.txt'. Stop.
It never does get to processing test2.txt.
As I was drafting this, I had the idea to remove the ../ from the DIR,
and relocate the Makefile so it was parent to both src/ and target/. That approach seems to work, but isn't ideal. Eventually there would be a chain of these Makefiles, each pulling from one directory to another.
Is there a way to keep the Makefile in 'target/' along with the generated destination files, and base those destination files off of something in a relative path?
Replace
%.txt: $(DIR)/%.txt
with:
${CURDIR}/%.txt: $(DIR)/%.txt
This way %.txt does not match any .txt file in any directory. In other words, you limit this rule's scope to files in ${CURDIR}/ only and this prevents that endless recursion.
See ยง10.5.4 How Patterns Match for more details.
It is also good practice to avoid relative paths:
DIR = $(abspath ../src)