How to use gzip correctly in a makefile? - makefile

Question

I'm a make newbie, so feel free to correct my syntax for this question.
I want to create a makefile that downloads a bunch of compressed files, uncompresses them and does something with them. Here's my toy example:
my_dir/*.txt.gz:
echo "# obtaining data"
mkdir -p my_dir
echo 1 > my_dir/file1.txt
gzip my_dir/file1.txt
echo 2 > my_dir/file2.txt
gzip my_dir/file2.txt
my_dir/*.txt: my_dir/*.txt.gz
echo "# unzipping data"
parallel gzip -d ::: my_dir/*.txt.gz
final.txt: my_dir/*.txt
echo "# doing something with data"
cat my_dir/*.txt > $#
clean_my_dir:
rm -rf my_dir
rm -f final.txt
When I run make final.txt I get what I expect (final.txt contains 1\n2), but if I run make final.txt again (without running make clean_my_dir) I get an error:
make final.txt
echo "# obtaining data"
# obtaining data
mkdir -p my_dir
echo 1 > my_dir/file1.txt
gzip my_dir/file1.txt
echo 2 > my_dir/file2.txt
gzip my_dir/file2.txt
echo "# unzipping data"
# unzipping data
parallel gzip -d ::: my_dir/*.txt.gz
echo "# unzipping data"
# unzipping data
parallel gzip -d ::: my_dir/*.txt.gz
gzip: can't stat: my_dir/*.txt.gz (my_dir/*.txt.gz.gz): No such file or directory
make: *** [my_dir/file2.txt] Error 1
I'm sure I'm making a mistake in the way I set up the targets and the prerequisites: I want final.txt to contain the contents of file1.txt and file2.txt, but these files are initially zipped. How do I tell make to call gzip when there are compressed files and to not do anything when the uncompressed files are available?

Answer 1

After a little experimentation I created to following makefile:
COMPRESSED_DIR = my_dir
UNCOMPRESSED_DIR = temp_dir
DEPS = \
$(UNCOMPRESSED_DIR)/file1.txt \
$(UNCOMPRESSED_DIR)/file2.txt
##
## Recipes how to obtain compressed files
## (may propably not so trivial in your intended application)
##
$(COMPRESSED_DIR)/file1.txt.gz:
#echo ""
#echo "# obtaining $#"
mkdir -p $(COMPRESSED_DIR)
echo 1 > $(COMPRESSED_DIR)/file1.txt
gzip $(COMPRESSED_DIR)/file1.txt
$(COMPRESSED_DIR)/file2.txt.gz:
#echo ""
#echo "# obtaining $#"
mkdir -p $(COMPRESSED_DIR)
echo 2 > $(COMPRESSED_DIR)/file2.txt
gzip $(COMPRESSED_DIR)/file2.txt
## or alternatively:
#$(COMPRESSED_DIR)/%.txt.gz:
# #echo ""
# #echo "# obtaining $#"
# mkdir -p $(COMPRESSED_DIR)
# echo $* > $(COMPRESSED_DIR)/$*.txt
# gzip $(COMPRESSED_DIR)/$*.txt
##
## Recipe to unpack compressed files
##
$(UNCOMPRESSED_DIR)/%.txt: $(COMPRESSED_DIR)/%.txt.gz
#echo ""
#echo "# unpacking $<..."
mkdir -p $(UNCOMPRESSED_DIR)
gzip -c -d $< > $#
##
## Recipe to make final.txt
##
final.txt: $(DEPS)
#echo ""
#echo "# doing something with data"
cat $(UNCOMPRESSED_DIR)/*.txt > final.txt
##
## Recipe to clean
##
clean:
rm -rf $(COMPRESSED_DIR)
rm -rf $(UNCOMPRESSED_DIR)
rm -f final.txt
As pointed out before gzip deletes the txt.gz file and therefore the recipe for creating the txt.gz files is re-executed if making final.txt again.
What I've done is placing the unpacked txt.gz files in a seperate directory and forcing gzip to keep the original file when unpacking. This leads to the following behavior:
If called the first time, the txt.gz files will be created and unpacked and final.txt will be created:
# obtaining my_dir/file1.txt.gz
mkdir -p my_dir
echo 1 > my_dir/file1.txt
gzip my_dir/file1.txt
# unpacking my_dir/file1.txt.gz...
mkdir -p temp_dir
gzip -c -d my_dir/file1.txt.gz > temp_dir/file1.txt
# obtaining my_dir/file2.txt.gz
mkdir -p my_dir
echo 2 > my_dir/file2.txt
gzip my_dir/file2.txt
# unpacking my_dir/file2.txt.gz...
mkdir -p temp_dir
gzip -c -d my_dir/file2.txt.gz > temp_dir/file2.txt
# doing something with data
cat temp_dir/*.txt > final.txt
If called again (without make clean) the output will be make:final.txt' is up to date.`. If a file of $(DEPS) is deleted the corresponding txt.gz file will be re-unpacked. If a .txt.gz file is deleted the .txt.gz file will be re-created and re-unpacked (since it is newer than the existing .txt file).
Alternativly if you've already got all the *.txt.gz files you want to process (and the makefile does not have to create them if they are missing) you can use the following makefile:
COMPRESSED_DIR = my_dir
UNCOMPRESSED_DIR = temp_dir
# get a list of all *.txt.gz files within $(COMPRESSED_DIR)
ARCHIVES = $(wildcard $(COMPRESSED_DIR)/*.txt.gz)
# do some magic to modify $(ARCHIVES) into a list of *.txt files to produce
DEPS = $(subst $(COMPRESSED_DIR),$(UNCOMPRESSED_DIR),$(subst .txt.gz,.txt,$(ARCHIVES)))
##
## Recipe to unpack compressed files
##
$(UNCOMPRESSED_DIR)/%.txt: $(COMPRESSED_DIR)/%.txt.gz
#echo ""
#echo "# unpacking $<..."
mkdir -p $(UNCOMPRESSED_DIR)
gzip -c -d $< > $#
##
## Recipe to make final.txt
##
final.txt: $(DEPS)
#echo "Archives:"
#echo $(ARCHIVES)
#echo ""
#echo "Deps:"
#echo $(DEPS)
#echo ""
#echo "# doing something with data"
cat $(UNCOMPRESSED_DIR)/*.txt > final.txt
##
## Recipe to clean
##
clean:
rm -rf $(UNCOMPRESSED_DIR)
rm -f final.txt
This makefile will uncompress all *.txt.gz files within COMPRESSED_DIR to UNCOMPRESSED_DIR and cat all *.txt files into final.txt.

Answer 2

You can do the gunzipping in one step:
my_dir:
[ -d $# ] || mkdir -p $#
my_dir/%.txt.gz: my_dir
#echo "# obtaining data for $#"
echo $* | gzip -c > $#
final.txt: my_dir/*.txt.gz
#echo "# doing something with data for $#"
gunzip -c $+ > $#
clean:
rm -rf my_dir final.txt
The dependency my_dir/*.txt.gz means that final.txt will be made from whatever matching files happen to be present at the time (and it will fail when none are present). To ensure the files are made when not present, you need to list them explicitly, e.g. (if GNU Make is used)
final.txt: $(foreach x,1 2 3 4 5 6 7 8,my_dir/$x.txt.gz)