I'm using make to write a pipeline for biological data analysis. My project directory is:
PROJECT
- DATA
- SAMPLEA
- A1.FASTQ A2.FASTQ
- SAMPLEB
- B1.FASTQ B2.FASTQ
- RESULTS
- SRC
- makefile
My current makefile uses a wildcard to list the directory of all .FASTQ files in the DATA directory. Using pattern rules each .FASTQ file then goes through a series of recipes with the final output file written to the RESULTS directory. Instead, I would like to create a directory for each SAMPLE where the final output file is written:
PROJECT/RESULTS/SAMPLEA/A1.out
PROJECT/RESULTS/SAMPLEA/A2.out
PROJECT/RESULTS/SAMPLEB/B1.out
PROJECT/RESULTS/SAMPLEB/B2.out
I can do this by having the first recipe make the directory, however this throws an error when the second of the FASTQ files from the same SAMPLE also tries to create the directory. A few posts on stack overflow suggest using the -p flag on mkdir to ignore errors, however this apparently causes problems when I run the makefile in parallel using the -j flag. I thought about forcing a shell script at the start of the makefile to run, to check if the results directories are present, and if not then it should create them, but I'd like to try and solve this issue using just make.
Create directory before executing rule.
DATADIR := $(shell cd DATA; find * -type d)
create_results_dir:= $(shell for i in $(DATADIR); \
do test -d DATA/$$i && mkdir -p RESULTS/$$i; \
done)
all:
#echo do something.
Related
Question Summary
I'm trying to make clean and I'm getting the following output:
Makefile:835: /absolute/path/to/server/server/example/.deps/libfoo-bar.Plo: No such file or directory
Makefile:836: /absolute/path/to/server/server/example/.deps/libfoo-baz.Plo: No such file or directory
Makefile:837: /absolute/path/to/server/server/example/.deps/libfoo-qux.Plo: No such file or directory
Makefile:838: /absolute/path/to/server/server/example/.deps/libfoo-quux.Plo: No such file or directory
Makefile:839: /absolute/path/to/server/server/example/.deps/libfoo-quuz.Plo: No such file or directory
Makefile:840: /absolute/path/to/server/server/example/.deps/libfoo-corge.Plo: No such file or directory
Makefile:841: /absolute/path/to/server/server/example/.deps/libfoo-grault.Plo: No such file or directory
Makefile:842: /absolute/path/to/server/server/example/.deps/libfoo-garply.Plo: No such file or directory
Makefile:843: /absolute/path/to/server/server/example/.deps/libfoo-waldo.Plo: No such file or directory
Makefile:844: /absolute/path/to/server/server/example/.deps/libfoo-fred.Plo: No such file or directory
Makefile:845: /absolute/path/to/server/server/example/.deps/libfoo-plugh.Plo: No such file or directory
Makefile:846: /absolute/path/to/server/server/example/.deps/libfoo-xyzzy.Plo: No such file or directory
Makefile:847: /absolute/path/to/server/server/example/.deps/libfoo-babble.Plo: No such file or directory
Makefile:848: /absolute/path/to/server/server/example/.deps/libfoo-thud.Plo: No such file or directory
Makefile:848: /absolute/path/to/server/server/example/.deps/libfoo-flarp.Plo: No such file or directory
make: *** No rule to make target '/absolute/path/to/server/server/example/.deps/libfoo-flarp.Plo'. Stop.
How can I successfully do a make clean? I don't understand what these .Plo files are, and they are usually something I don't mess with.
Investigation
My Makefile.am
The Makefile.am which is used to generate the Makefile from which I'm trying to run the clean contains this preamble:
include $(top_srcdir)/server/include.am
include $(top_srcdir)/server/tests/include.am
...
The first $(top_srcdir)/server/include.am includes the following:
...
server_libdir = $(exec_prefix)/lib
serverdir = $(top_srcdir)/server
...
and in the specified $(top_srcdir)/server/tests/include.am, I have a lot of things including:
...
server_lib_LTLIBRARIES = libfoo.la
libfoo_la_SOURCES = \
/absolute/path/to/server/server/example/bar.c \
/absolute/path/to/server/server/example/baz.c \
/absolute/path/to/server/server/example/qux.c \
/absolute/path/to/server/server/example/quux.c \
/absolute/path/to/server/server/example/quuz.c \
/absolute/path/to/server/server/example/corge.c \
/absolute/path/to/server/server/example/grault.c \
/absolute/path/to/server/server/example/garply.c \
/absolute/path/to/server/server/example/waldo.c \
/absolute/path/to/server/server/example/fred.c \
/absolute/path/to/server/server/example/plugh.c \
/absolute/path/to/server/server/example/xyzzy.c \
...
Investigating the Makefile Contents
If I look at the Makefile directly and find the lines generating these errors, I see the following:
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-bar.Plo
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-baz.Plo
include /absolute/path/to/server/server/example/$(DEPDIR)/libfoo-qux.Plo
...
Where are the .Plo files generated?
Frustratingly, there seems to be the exact .Plo files that I need generated all over the place, just not in the right place; for example these files exist:
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-bar.Plo
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-baz.Plo
/absolute/path/to/server/tests/absolute/path/to/server/server/example/$(DEPDIR)/libfoo-qux.Plo
...
and there are 7 other directories where these .Plo files are created.
Unsuccessful Attempts
Running config.status directly
I found this message on the GNU mailing list:
Actually, these .P files are created by config.status for each
directory it creates an Automake Makefile in. (BTW, make sure
./config.status --help' lists the relevant Makfiles in the Configuration files' section).
You can request the creation of the .P files for a given directory
by running
% rm -Rf kernel/framework/.deps
% ./config.status kernel/framework/Makefile depfiles
And ran ./config.status as described but the output of make clean hasn't changed
According to this other Stack Overflow question, they were able to generate .Plo files, but I don't know what is different for them.
re-run configure with the --disable-dependency-tracking option
I'm writing a Makefile to build a Latex document depending on plots whose data is generated from some other data by some python script.
It looks like this
% pdf plot needed by final document
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $$(dirname $#) && ../../latexrun $$(basename $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | build
% crunch.py will create data for plots needed by build/tikz-standalone/%.tex
PYTHONPATH=. tools/plots/crunch.py
build:
mkdir -p build build/other_stuff ...
crunch.py generates several data files in build/data which are needed by build/tikz-standalone/%.tex. To create these files it uses other files stored in the variable PLOT_DATA. I could put a list of the intermediate data files in build/data into the Makefile at the position of xy_data. I don't like this as this would require me to update the list whenever a new file is added. What I want is that all data files are recreated whenever crunch.py or $(PLOT_DATA) has changed.
Is there a way to express this in Make?
If you do not want to provide and maintain the list of the generated files you can turn your (implicitly) phony xy_data target into an empty file used as a marker. Simply touch it at the end of the recipe:
BUILDDIRS := build build/other_stuff ...
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $(dir $#) && ../../latexrun $(notdir $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | $(BUILDDIRS)
PYTHONPATH=. tools/plots/crunch.py
touch $#
$(BUILDDIRS):
mkdir -p $#
Note: I also improved a bit some other aspects:
Use of make functions dir and notdir instead of the shell equivalents.
Variable declaration for the build directories to avoid writing the same list several times, which is tedious and error prone.
Explicit list of all build directories as order-only prerequisites instead of just one, which could lead to unexpected results if this single one exists but not some others.
Generic rule for all build directories thanks to the $# automatic variable.
I have few txt files in a directory. I want to run a shell script only on the files which have been modified. How can I achieve this through Makefile?
Have written the following part but it builds all the txt files in the directory. Would be great to get some pointers on this.
FILENAME:= $(wildcard dir/txts/*/*.txt)
.PHONY: build-txt
build-txt: $(FILENAME)
sh build-txts.sh $^
I'm guessing you want something like this:
files := $(wildcard dir/txts/*/*.txt)
dummies := $(addprefix .mod_,$files)
all:$(dummies)
$(dummies): .mod_% : %
sh build-txts.sh $^
touch $#
For any new text file, it will run the script, and create a .mod counterpart. For any non-new text file, it will check if the timestamp is newer than the .mod files timestamp. If it is, it runs the script, and then touches the .mod (making the .mod newer than the text). For any text file that has not been modified since the last make, the .mod file will be newer and the script will not run. Notice that the .mod files are NOT PHONY targets. They are dummy files who exist solely to mark when the text file was last modified. You can stick them in a dummy directory for easy cleaning as well.
If you need something where you don't want to rebuild the text files by default on a fresh checkout, or your script criteria isn't based on timestamps, you would need something a bit more tricky:
files := $(wildcard dir/txts/*/*.txt)
md5s:= $(addprefix .md5_,$files)
all:$(md5s)
.PHONY:$(md5s)
$(md5s):
( [ -e $# ] && md5sum -c $# ) || \
( sh build-txts.sh $# && md5sum $(#:.md5_=) > $# )
Here, you run the rule for all text files regardless, and you use bash to determine if the file is out of date. If the text file does not exist, or the md5sum is not correct, it runs the script, then updates the md5sum. Because the rules are phony, they always run for all the .md5sum files regardless of whether they already exist.
Using this method, you could submit the .md5 files to your repository, and it would only run the script on those files whose md5 sum changed after checkout.
Pretty new to GNU Make. This is a less complex example of something more general I have been trying to get to work.
I have many input files that have similar name format .txt, and I have a shell script that will take the input file and generate an output of the same name but with a different extension .wc. I have written the following Make file.
# name of dependencies
SRC = $(wildcard *.txt)
# get name of targets (substitute .wc for .txt)
TAR = $(SRC:.txt=.wc)
all: $(TAR)
%.wc: %.txt
sh word_count.sh $<
This runs fine, and will generate all the .wc output files. However, if I modify one of the input(dependency) files, they are all rebuilt. So the question is; what is the best way to get GNU Make to only process the modified .txt files in the directory?
This seems slightly related to How to write Makefile where target and source files have the same extension?. In that question the extensions are the same, but the input and output files seem to be in the same directory and filenames are being conditionally renamed.
I have a large collection of .txt files in ../src/ that need to be processed, and dumped into ./ (which is a directory called target/) as txt files of the same name. I want to use make, so that only files in ../src/ that have been changed get updated in ./. I would like to get the prototype working before I put the real code in.
My Makefile in ./ is as follows:
DIR = ../src
INPUTS = $(wildcard $(DIR)/*.txt)
OUTPUTS = $(patsubst $(DIR)/%.txt,%.txt,$(INPUTS))
all: $(OUTPUTS)
.PHONY: $(INPUTS)
check:
#echo "DIR = $(DIR)"
#echo "INPUTS = $(INPUTS)"
#echo "OUTPUTS = $(OUTPUTS)"
%.txt: $(DIR)/%.txt
sed -e "s/test/cat/g" "$<" > $#
For now, the contents of ../src/ are test1.txt and test2.txt.
As the Makefile stands now, running make test2.txt generates the file as expected.
target/ $ make test2.txt
sed -e "s/test/cat/g" "../src/test2.txt" > test2.txt
Running make check shows the INPUTS and OUTPUTS correctly.
target/ $ make check
DIR = ../src
INPUTS = ../src/test1.txt ../src/test2.txt
OUTPUTS = test1.txt test2.txt
If I run make all, it generates every file, every time. This is expected with the .PHONY $(INPUTS) line in there.
If I remove the .PHONY $(INPUTS) target, Make gets all bound up in itself trying to find the target to make ../src/test1.txt and keeps prefixing $(DIR) in front of it until it makes too long of a filename and gives up.
make: stat: ../src/../src/../src/ [repeat for a few pages] ../src/../src/test1.txt: File name too long
make: stat: ../src/../src/../src/ [repeat for a few pages] ../src/../src/../src/test1.txt: File name too long
make: *** No rule to make target `../src/../src/../src/[repeat]../src/../src/test1.txt', needed by `../src/[repeat]../src/../src/test1.txt'. Stop.
It never does get to processing test2.txt.
As I was drafting this, I had the idea to remove the ../ from the DIR,
and relocate the Makefile so it was parent to both src/ and target/. That approach seems to work, but isn't ideal. Eventually there would be a chain of these Makefiles, each pulling from one directory to another.
Is there a way to keep the Makefile in 'target/' along with the generated destination files, and base those destination files off of something in a relative path?
Replace
%.txt: $(DIR)/%.txt
with:
${CURDIR}/%.txt: $(DIR)/%.txt
This way %.txt does not match any .txt file in any directory. In other words, you limit this rule's scope to files in ${CURDIR}/ only and this prevents that endless recursion.
See ยง10.5.4 How Patterns Match for more details.
It is also good practice to avoid relative paths:
DIR = $(abspath ../src)