Pretty new to GNU Make. This is a less complex example of something more general I have been trying to get to work.
I have many input files that have similar name format .txt, and I have a shell script that will take the input file and generate an output of the same name but with a different extension .wc. I have written the following Make file.
# name of dependencies
SRC = $(wildcard *.txt)
# get name of targets (substitute .wc for .txt)
TAR = $(SRC:.txt=.wc)
all: $(TAR)
%.wc: %.txt
sh word_count.sh $<
This runs fine, and will generate all the .wc output files. However, if I modify one of the input(dependency) files, they are all rebuilt. So the question is; what is the best way to get GNU Make to only process the modified .txt files in the directory?
Related
I want to let makefile manage the compilation of figures with metapost.
The source file is file.mp. This generates .mps files file.1, file.2 etc. that are then converted to file-1.pdf, file-2.pdf etc.
Here are my rules:
export TEX = latex
%: %.1
mptopdf $*
%.1: %.mp
mpost $*
So that when I run make file it creates all the files.
However, I am not satisfied with this solution. Namely, I'd like to be able to let only one of the files be compiled (say file-2.pdf) by entering make file-2.
Unfortunately, I don't know how to write the rule for this, although I suspect it might be trivial.
I thought the problem could be solved by extracting the number from the file name given in the command line (i.e. extract 2 from file-2) but it is not clear to me how to do it either.
In my project, I have a set of sub-directories that contain package.yaml files, for e.g.:
A/package.yaml
B/package.yaml
C/package.yaml
If I run hpack A/package.yaml, the file A/A.cabal is (re-)generated. The list of such directories can change over time, so I want to use GNU make to find all immediate sub-directories containing package.yaml files and generate the corresponding .cabal files using hpack.
I tried this based on another question, but it didn't work:
HPACK_FILES := $(wildcard */package.yaml)
PKG_DIRS := $(subst /,,$(dir $(HPACK_FILES)))
CABAL_FILES := $(addsuffix .cabal,$(join $(dir $(HPACK_FILES)),$(PKG_DIRS)))
test:
#echo $(CABAL_FILES)
update-cabal: $(CABAL_FILES)
%.cabal: package.yaml
hpack $<
However, make update-cabal says there's nothing to be done. make test however does output the right cabal files. How can I fix this?
Cheers!
The problem is this:
%.cabal: package.yaml
There is no file package.yaml. The files are named things like A/package.yaml. That is not the same thing.
Because the prerequisite doesn't exist, make decides that this pattern rule cannot match and so it goes looking for another rule that might be able to build the target. It doesn't find any rule that can build the target, so make says there's nothing to do because all the output files already exist.
Unfortunately what you want to do is not at all easy with make, because make is most comfortable with input and output files that are tied together by the filename with extensions, or similar. And in particular, it has a really hard time with relationships where the variable part is repeated more than once (as in, A/A.cabal where the A is repeated). There's no easy way to do that in make.
You'll have to use an advanced feature such as eval to do this. Something like:
# How to build a cabal file
%.cabal:
hpack $<
# Declare the prerequisites
$(foreach D,$(dir $(HPACK_FILES)),$(eval $D/$D.cabal: $D/package.yml))
I have few txt files in a directory. I want to run a shell script only on the files which have been modified. How can I achieve this through Makefile?
Have written the following part but it builds all the txt files in the directory. Would be great to get some pointers on this.
FILENAME:= $(wildcard dir/txts/*/*.txt)
.PHONY: build-txt
build-txt: $(FILENAME)
sh build-txts.sh $^
I'm guessing you want something like this:
files := $(wildcard dir/txts/*/*.txt)
dummies := $(addprefix .mod_,$files)
all:$(dummies)
$(dummies): .mod_% : %
sh build-txts.sh $^
touch $#
For any new text file, it will run the script, and create a .mod counterpart. For any non-new text file, it will check if the timestamp is newer than the .mod files timestamp. If it is, it runs the script, and then touches the .mod (making the .mod newer than the text). For any text file that has not been modified since the last make, the .mod file will be newer and the script will not run. Notice that the .mod files are NOT PHONY targets. They are dummy files who exist solely to mark when the text file was last modified. You can stick them in a dummy directory for easy cleaning as well.
If you need something where you don't want to rebuild the text files by default on a fresh checkout, or your script criteria isn't based on timestamps, you would need something a bit more tricky:
files := $(wildcard dir/txts/*/*.txt)
md5s:= $(addprefix .md5_,$files)
all:$(md5s)
.PHONY:$(md5s)
$(md5s):
( [ -e $# ] && md5sum -c $# ) || \
( sh build-txts.sh $# && md5sum $(#:.md5_=) > $# )
Here, you run the rule for all text files regardless, and you use bash to determine if the file is out of date. If the text file does not exist, or the md5sum is not correct, it runs the script, then updates the md5sum. Because the rules are phony, they always run for all the .md5sum files regardless of whether they already exist.
Using this method, you could submit the .md5 files to your repository, and it would only run the script on those files whose md5 sum changed after checkout.
I am trying to use makefile to manage my building process in a small project, where the target number and target names are not known beforehand but depends on the input. Specifically, I want to generate a bunch of data files (say .csv files) according to a cities_list.txt file with a list of city names inside. For example, if the contents of the txt file are:
newyork
washington
toronto
then a script called write_data.py would generate three files called newyork.csv, washington.csv and toronto.csv. When the content of the cities_list.txt file changes, I want make to deal with this change cleverly, i.e. only update the new-added cities files.
I was trying to define variable names in target names to make this happen but didn't succeed. I'm now trying to create a bunch of intermediate .name files as below:
all: *.csv
%.name: cities_list.txt
/bin/bash gen_city_files.sh $<
%.csv: %.name write_data.py
python3 write_data.py $<
clean:
rm *.name *.csv
This seems to be very close to success, but it only gives me one .csv file. The reason is obvious, because make can't determine what files should be generated for the all target. How can I let make know that this *.csv should contain all the files where there exists a corresponding *.name file? Or is there any better way to achieve what I wanted to do here?
All right, this should do it. We'd like a variable assignment at the head of the file:
CITY_FILES := newyork.csv washington.csv toronto.csv
There are two ways to do this. This way:
-include cities.mak
# this rule can come later in the makefile, near the bottom
cities.mak: cities_list.txt
#sed 's/^/CITIES := /' $< > $#
and this way:
CITIES := $(shell cat cities_list.txt)
After we've done one of those two, we can construct the list of needed files:
CITY_FILES := $(addsuffix .csv, $(CITIES))
and build them:
# It is convenient to have this be the first rule in the makefile.
all: $(CITY_FILES)
%.csv: write_data.py
python3 $< $*.name
as a part of build process I need to process some files with m4. These files are given an '.in' extension and are located not only in the top-level dir, but also in subdirs.
I locate them using find and process them in a shell loop.
Is there some makefile syntax magic to write simpler rule to process them save the output into a file (the '.in' extension stripped) in the same directory as the input file?
This is what I have in my makefile now:
PROCESS_FILES=$(shell find . -name \*.in)
WORK_FILES=$(subst .in,,$(PROCESS_FILES))
$(WORK_FILES): $(PROCESS_FILES)
for file in $(PROCESS_FILES); \
do \
m4 $$file > $${file%.*};\
done
You could write a set of rules to convert files with the suffix .in to files without that suffix.
In the classic notation, that would be something like:
# Not functional — see discussion
.SUFFIXES: .in # Add .in as a suffix
M4 = m4
M4FLAGS =
M4SCRIPT = xyz.m4
.in:
${M4} ${M4FLAGS} ${M4SCRIPT} $< > $*
This adds .in as a recognized suffix, and says that you convert the file with a .in suffix to the file without it using the command specified by the three macros. The $< is the name of the file with the .in extension; the $* is the name of the file without the extension.
Unfortunately though, that notation only works when the files to be converted have names such as xyz.in and need to be converted to xyz. It does not work for a case where xyz.h.in needs to be converted to xyz.h.
It would be possible to specify that the suffixes are .h.in and .h, but then the rule for converting between the two starts with .h.in.h: and make gets confused.
However, GNU make has an alternative notation for defining suffix rules which can accommodate this:
.SUFFIXES: .h.in .h # Add .h.in and .h as suffixes
M4 = m4
M4FLAGS =
M4SCRIPT = xyz.m4
%.h: %.h.in
${M4} ${M4FLAGS} ${M4SCRIPT} $< > $*.h
The %.h: %.h.in line is able to deal with the suffix with two dots. The only nuisance is that for each extension such as .c.in or .y.in or .mk.in, you have to provide similar mappings.
(Tested on Mac OS X 10.7.4. If that makefile is xyz.mk, you can create an empty file xyz.h.in and then run make -n -f xyz.mk xyz.h and you'll see the build would run the command m4 xyz.m4 xyz.h.in > xyz.h.)
If you have a source file path/to/file.in, a rule for making % from %.in, and a dependency on the output file path/to/file, that should be all you need.
Perhaps you want to express your dependencies in terms of the output from find. In GNU Make this is easy; in other dialects, perhaps you want to generate and include something like find -name '*.in' | sed 's/\.in$//' >make.dep