I'm writing a Makefile to build a Latex document depending on plots whose data is generated from some other data by some python script.
It looks like this
% pdf plot needed by final document
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $$(dirname $#) && ../../latexrun $$(basename $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | build
% crunch.py will create data for plots needed by build/tikz-standalone/%.tex
PYTHONPATH=. tools/plots/crunch.py
build:
mkdir -p build build/other_stuff ...
crunch.py generates several data files in build/data which are needed by build/tikz-standalone/%.tex. To create these files it uses other files stored in the variable PLOT_DATA. I could put a list of the intermediate data files in build/data into the Makefile at the position of xy_data. I don't like this as this would require me to update the list whenever a new file is added. What I want is that all data files are recreated whenever crunch.py or $(PLOT_DATA) has changed.
Is there a way to express this in Make?
If you do not want to provide and maintain the list of the generated files you can turn your (implicitly) phony xy_data target into an empty file used as a marker. Simply touch it at the end of the recipe:
BUILDDIRS := build build/other_stuff ...
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $(dir $#) && ../../latexrun $(notdir $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | $(BUILDDIRS)
PYTHONPATH=. tools/plots/crunch.py
touch $#
$(BUILDDIRS):
mkdir -p $#
Note: I also improved a bit some other aspects:
Use of make functions dir and notdir instead of the shell equivalents.
Variable declaration for the build directories to avoid writing the same list several times, which is tedious and error prone.
Explicit list of all build directories as order-only prerequisites instead of just one, which could lead to unexpected results if this single one exists but not some others.
Generic rule for all build directories thanks to the $# automatic variable.
Related
A code generator is executed from GNU make. The generator produces several files (depending on the input), and only touches the files, when their content change. Therefore a sentinel target needs to be used to record the generator execution time:
GEN_READY : $(gen_input_files)
gen.exe $(gen_input_files)
touch GEN_READY
$(gen_output_files): GEN_READY
It works well, except when a generated file is deleted, but the sentinel file is left in place. Since the sentinel is there, and it's up-to-date, the generator is not executed again.
What is the proper solution to force make to re-run the generator in this case?
Here is one way to group them using an archive:
# create archive of output files from input files passed through gen.exe
GEN_READY.tar: $(gen_input_files)
#echo Generate the files
gen.exe $^
#echo Put generated files in archive
tar -c -f $# $(gen_output_files)
#echo Remove intermediate files (recreated by next recipe)
rm $(gen_output_files)
# Extracting individual files for use as prerequisite or restoration
$(gen_output_files): GEN_READY.tar
#echo Extract one member
tar -x -f $< $#
Since tar (and zip for that matter) allows duplicate entries there could be opportunities updating or appending files in archive instead of rewriting if input-output relation allows.
Edit: Simplified solution.
Right now I have this command: rm -f *.x *.o, which removes all the object files and executable files by calling make clean in my shell. But I have one specific object file, say a.o that I do not want to remove. How can I do this without hardcoding it? Note that I can only implement it under C90 because this is an assignment.
Firstly, do not issue wildcard remove commands in a clean: rule (or any other). When the user types make clean, they could lose files that is not even under the control of the build system. Your build system should only clean what it builds. (That is, at least with the "regular strength" clean target; here we should acknowledge the widespread practice of providing additional targets that remove editor backup files, tags, generated configuration materials and whatnot.)
The object files should be listed in some variable like $(OBJS) and should be removed using rm -f ... $(OBJS).
If a certain object file is not to be removed, you can divide things:
DISPOSABLE_OBJS = ...
PRECIOUS_OBJS = ...
OBJS = $(DISPOSABLE_OBJS) $(PRECIOUS_OBJS)
Then in the clean rule, just remove the $(DISPOSABLE_OBJS).
In GNU Make you have additional flexibilities, like rm -f $(filter-out keep1.o keep2.o,$(OBJS)) to remove all $(OBJS) except keep1.o and keep2.o.
I read this: Makefile: Copying files with a rule but couldn't do it.
To make it simple, suppose I have two directories dir1 and dir2. Under dir1 I have three files: rabbit.c, tiger.c and bus.c .
I made Makefile like this:
dir2/rabbit.c:dir1/rabbit.c
dir2/tiger.c:dir1/tiger.c
dir2/bike.c:dir1/bike.c
dir2/%:
cp -f $< $#
I specified the prerequisites in three separate lines and specified the unified recipe for the three targets. I expected when I touch any file under dir1, make will copy that file to dir2. But this happend only for rabbit.c. What is wrong?
ADD(after selecting an answer) :
After realizing what's wrong by Takkat's answer, I fixed it and later modified it further and I think this is the correct simplest Makefile for this case.
.PHONY:all
LIST:=rabbit.c tiger.c bike.c
DSTFILES:=$(addprefix dir2/, $(LIST))
all: $(DSTFILES)
dir2/%:dir1/%
cp -f $< $#
Make chooses a default target in your makefile and, unless you specify differently on the command line, it builds just that target (and all prerequisites required to build that target).
The default target in a makefile is, by default, the first explicit target listed.
So in your makefile the first rule is:
dir2/rabbit.c:dir1/rabbit.c
so the first explicit target is dir2/rabbit.c, so that's all make builds.
If you want to build multiple targets by default, you need a first target that lists all the "real" targets as prerequisites; put this line first in your makefile:
all: dir2/rabbit.c dir2/tiger.c dir2/bike.c
and it will work. It's often considered good practice to declare targets like this, which don't relate to real files on the disk, as phony:
.PHONY: all
I am trying to use makefile to manage my building process in a small project, where the target number and target names are not known beforehand but depends on the input. Specifically, I want to generate a bunch of data files (say .csv files) according to a cities_list.txt file with a list of city names inside. For example, if the contents of the txt file are:
newyork
washington
toronto
then a script called write_data.py would generate three files called newyork.csv, washington.csv and toronto.csv. When the content of the cities_list.txt file changes, I want make to deal with this change cleverly, i.e. only update the new-added cities files.
I was trying to define variable names in target names to make this happen but didn't succeed. I'm now trying to create a bunch of intermediate .name files as below:
all: *.csv
%.name: cities_list.txt
/bin/bash gen_city_files.sh $<
%.csv: %.name write_data.py
python3 write_data.py $<
clean:
rm *.name *.csv
This seems to be very close to success, but it only gives me one .csv file. The reason is obvious, because make can't determine what files should be generated for the all target. How can I let make know that this *.csv should contain all the files where there exists a corresponding *.name file? Or is there any better way to achieve what I wanted to do here?
All right, this should do it. We'd like a variable assignment at the head of the file:
CITY_FILES := newyork.csv washington.csv toronto.csv
There are two ways to do this. This way:
-include cities.mak
# this rule can come later in the makefile, near the bottom
cities.mak: cities_list.txt
#sed 's/^/CITIES := /' $< > $#
and this way:
CITIES := $(shell cat cities_list.txt)
After we've done one of those two, we can construct the list of needed files:
CITY_FILES := $(addsuffix .csv, $(CITIES))
and build them:
# It is convenient to have this be the first rule in the makefile.
all: $(CITY_FILES)
%.csv: write_data.py
python3 $< $*.name
Before I start, I'll mention that I'm not using GNU Make in this case for building a C/C++ project.
Makefile:
DEST_DIR = build/
SRC_DIR = src/
$(SRC_DIR)a/ : $(SOMETHING_ELSE)
$(DO_SOMETHING_TO_GENERATE_A_DIR)
$(DEST_DIR)% : $(SRC_DIR)%
cp -r $^ $#
ALL_DEPS += <SOMETHING>
... more code which appends to ALL_DEPS ...
.PHONY: all
all : $(ALL_DEPS)
I've got some files not generated via Make rules in $(SRC_DIR). (For the sake of this example, let's say there's a directory $(SRC_DIR)b/ and a file $(SRC_DIR)c .)
I want to append to ALL_DEPS all targets which represent files or directories in $(DEST_DIR) so that "make all" will run all of the available $(DEST_DIR)% rules.
I thought to do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(wildcard $(SRC_DIR)*)))
But of course, that doesn't catch anything that hasn't yet been made. (i.e. it doesn't append $(DEST_DIR)a/ to the list because $(SRC_DIR)a/ doesn't yet exist when the $(wildcard ...) invocation is evaluated and the shell doesn't include it in the results returned by the $(wildcard ...) invocation.)
So, rather than a function which finds all (currently-existing) files matching a pattern, I need one which finds all targets matching a pattern. Then, I could do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(targetwildcard $(SRC_DIR)*)))
If it matters any, I've got much of the GNU Make code split across multiple files and included by a "master" Makefile. The ALL_DEPS variable is appended to in any of these files which has something to add to it. This is in an attempt to keep the build process modular as opposed to dropping it all in one monster Makefile.
I'm definitely still learning GNU Make, so it's not unlikely that I'm missing something fairly obvious. If I'm just going about this all wrong, please let me know.
Thanks!
It is simply not possible to do what you're trying to do; you're trying to get make to recognise something that doesn't exist.
This is part of the reason why, in general, wildcards are bad (the other being that you can end up including stuff you didn't mean to). The right thing to do here is to explicitly create a list of source files (ls -1 | sed -e 's/\(.*\)/sources+=\1/' > dir.mk) and perform the patsubst transformation on that list.
If you have additional files that are generate as part of the build, then you can append them to that list and their rules will be found as you'd expect.