I have few txt files in a directory. I want to run a shell script only on the files which have been modified. How can I achieve this through Makefile?
Have written the following part but it builds all the txt files in the directory. Would be great to get some pointers on this.
FILENAME:= $(wildcard dir/txts/*/*.txt)
.PHONY: build-txt
build-txt: $(FILENAME)
sh build-txts.sh $^
I'm guessing you want something like this:
files := $(wildcard dir/txts/*/*.txt)
dummies := $(addprefix .mod_,$files)
all:$(dummies)
$(dummies): .mod_% : %
sh build-txts.sh $^
touch $#
For any new text file, it will run the script, and create a .mod counterpart. For any non-new text file, it will check if the timestamp is newer than the .mod files timestamp. If it is, it runs the script, and then touches the .mod (making the .mod newer than the text). For any text file that has not been modified since the last make, the .mod file will be newer and the script will not run. Notice that the .mod files are NOT PHONY targets. They are dummy files who exist solely to mark when the text file was last modified. You can stick them in a dummy directory for easy cleaning as well.
If you need something where you don't want to rebuild the text files by default on a fresh checkout, or your script criteria isn't based on timestamps, you would need something a bit more tricky:
files := $(wildcard dir/txts/*/*.txt)
md5s:= $(addprefix .md5_,$files)
all:$(md5s)
.PHONY:$(md5s)
$(md5s):
( [ -e $# ] && md5sum -c $# ) || \
( sh build-txts.sh $# && md5sum $(#:.md5_=) > $# )
Here, you run the rule for all text files regardless, and you use bash to determine if the file is out of date. If the text file does not exist, or the md5sum is not correct, it runs the script, then updates the md5sum. Because the rules are phony, they always run for all the .md5sum files regardless of whether they already exist.
Using this method, you could submit the .md5 files to your repository, and it would only run the script on those files whose md5 sum changed after checkout.
Related
A code generator is executed from GNU make. The generator produces several files (depending on the input), and only touches the files, when their content change. Therefore a sentinel target needs to be used to record the generator execution time:
GEN_READY : $(gen_input_files)
gen.exe $(gen_input_files)
touch GEN_READY
$(gen_output_files): GEN_READY
It works well, except when a generated file is deleted, but the sentinel file is left in place. Since the sentinel is there, and it's up-to-date, the generator is not executed again.
What is the proper solution to force make to re-run the generator in this case?
Here is one way to group them using an archive:
# create archive of output files from input files passed through gen.exe
GEN_READY.tar: $(gen_input_files)
#echo Generate the files
gen.exe $^
#echo Put generated files in archive
tar -c -f $# $(gen_output_files)
#echo Remove intermediate files (recreated by next recipe)
rm $(gen_output_files)
# Extracting individual files for use as prerequisite or restoration
$(gen_output_files): GEN_READY.tar
#echo Extract one member
tar -x -f $< $#
Since tar (and zip for that matter) allows duplicate entries there could be opportunities updating or appending files in archive instead of rewriting if input-output relation allows.
Edit: Simplified solution.
I'm writing a Makefile to build a Latex document depending on plots whose data is generated from some other data by some python script.
It looks like this
% pdf plot needed by final document
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $$(dirname $#) && ../../latexrun $$(basename $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | build
% crunch.py will create data for plots needed by build/tikz-standalone/%.tex
PYTHONPATH=. tools/plots/crunch.py
build:
mkdir -p build build/other_stuff ...
crunch.py generates several data files in build/data which are needed by build/tikz-standalone/%.tex. To create these files it uses other files stored in the variable PLOT_DATA. I could put a list of the intermediate data files in build/data into the Makefile at the position of xy_data. I don't like this as this would require me to update the list whenever a new file is added. What I want is that all data files are recreated whenever crunch.py or $(PLOT_DATA) has changed.
Is there a way to express this in Make?
If you do not want to provide and maintain the list of the generated files you can turn your (implicitly) phony xy_data target into an empty file used as a marker. Simply touch it at the end of the recipe:
BUILDDIRS := build build/other_stuff ...
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $(dir $#) && ../../latexrun $(notdir $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | $(BUILDDIRS)
PYTHONPATH=. tools/plots/crunch.py
touch $#
$(BUILDDIRS):
mkdir -p $#
Note: I also improved a bit some other aspects:
Use of make functions dir and notdir instead of the shell equivalents.
Variable declaration for the build directories to avoid writing the same list several times, which is tedious and error prone.
Explicit list of all build directories as order-only prerequisites instead of just one, which could lead to unexpected results if this single one exists but not some others.
Generic rule for all build directories thanks to the $# automatic variable.
I am trying to use makefile to manage my building process in a small project, where the target number and target names are not known beforehand but depends on the input. Specifically, I want to generate a bunch of data files (say .csv files) according to a cities_list.txt file with a list of city names inside. For example, if the contents of the txt file are:
newyork
washington
toronto
then a script called write_data.py would generate three files called newyork.csv, washington.csv and toronto.csv. When the content of the cities_list.txt file changes, I want make to deal with this change cleverly, i.e. only update the new-added cities files.
I was trying to define variable names in target names to make this happen but didn't succeed. I'm now trying to create a bunch of intermediate .name files as below:
all: *.csv
%.name: cities_list.txt
/bin/bash gen_city_files.sh $<
%.csv: %.name write_data.py
python3 write_data.py $<
clean:
rm *.name *.csv
This seems to be very close to success, but it only gives me one .csv file. The reason is obvious, because make can't determine what files should be generated for the all target. How can I let make know that this *.csv should contain all the files where there exists a corresponding *.name file? Or is there any better way to achieve what I wanted to do here?
All right, this should do it. We'd like a variable assignment at the head of the file:
CITY_FILES := newyork.csv washington.csv toronto.csv
There are two ways to do this. This way:
-include cities.mak
# this rule can come later in the makefile, near the bottom
cities.mak: cities_list.txt
#sed 's/^/CITIES := /' $< > $#
and this way:
CITIES := $(shell cat cities_list.txt)
After we've done one of those two, we can construct the list of needed files:
CITY_FILES := $(addsuffix .csv, $(CITIES))
and build them:
# It is convenient to have this be the first rule in the makefile.
all: $(CITY_FILES)
%.csv: write_data.py
python3 $< $*.name
A directory with css files in it and for each file in the directory I want a separate minified .min.css file in the same directory.
So for example my.css get minified into my.min.css in the same directory.
First I tried
css/*.css:
uglifycss $# > $(subst .css,.min.css,$#)
make -B yeaaa victorie its working :D
But after doing make -B again I get a new separate file each time resulting in my.min.css my.min.min.css my.min.min.min.css
Then I tried
.PHONY: clean
css/*.css: clean
uglifycss $# > $(subst .css,.min.css,$#)
clean:
-rm css/*.min.css
DOH! after clean it is still remembering the files it deleted in the first place resulting again in a my.min.min.min.css file
How can I tell make to stop doing my.min.min.min.css?
(make: GNU Make 3.81 OSX)
First get a list of all files with the suffix .css:
ALL := $(wildcard *.css)
then remove files that have the suffix .me.css:
NEW := $(filter-out %.me.css,$(ALL))
and add the suffix to remaining files:
ME := $(patsubst %.css,%.me.css,$(NEW))
Then you add those files as prerequisites to the default target, and add you own recipe that builds those files, in this case a simple echo:
%.me.css:
echo 123 > $#
default: $(ME)
Compared to your approach, this has the benefit that you don't have to use the flag -B, as only the files that need to be built are built. Therefore invoking make is done by simply caling make without any targets or flags (assuming the makefile is named makefile or Makefile):
make
Pretty new to GNU Make. This is a less complex example of something more general I have been trying to get to work.
I have many input files that have similar name format .txt, and I have a shell script that will take the input file and generate an output of the same name but with a different extension .wc. I have written the following Make file.
# name of dependencies
SRC = $(wildcard *.txt)
# get name of targets (substitute .wc for .txt)
TAR = $(SRC:.txt=.wc)
all: $(TAR)
%.wc: %.txt
sh word_count.sh $<
This runs fine, and will generate all the .wc output files. However, if I modify one of the input(dependency) files, they are all rebuilt. So the question is; what is the best way to get GNU Make to only process the modified .txt files in the directory?