I have the following file and directory structure which contains a lot of files and directories:
$ tree
input/
├── C-1-28558666
│ ├── MGRF-C1_S10_L001_R1_001.fastq.gz
│ ├── MGRF-C1_S10_L001_R2_001.fastq.gz
│ ├── MGRF-C1_S10_L002_R1_001.fastq.gz
│ ├── MGRF-C1_S10_L002_R2_001.fastq.gz
│ ├── MGRF-C1_S10_L003_R1_001.fastq.gz
│ ├── MGRF-C1_S10_L003_R2_001.fastq.gz
│ ├── MGRF-C1_S10_L004_R1_001.fastq.gz
│ └── MGRF-C1_S10_L004_R2_001.fastq.gz
├── C-2-28577664
│ ├── MGRF-C2_S11_L001_R1_001.fastq.gz
│ ├── MGRF-C2_S11_L001_R2_001.fastq.gz
│ ├── MGRF-C2_S11_L002_R1_001.fastq.gz
│ ├── MGRF-C2_S11_L002_R2_001.fastq.gz
│ ├── MGRF-C2_S11_L003_R1_001.fastq.gz
│ ├── MGRF-C2_S11_L003_R2_001.fastq.gz
│ ├── MGRF-C2_S11_L004_R1_001.fastq.gz
│ └── MGRF-C2_S11_L004_R2_001.fastq.gz
...
Each file name contain R1 or R2. The files with R1 and R2 belongs together and the below commanand uses the two files and a dm6.fasta.bwt file.
bwa mem ref/dm6.fasta.bwt input/C-1-28558666/MGRF-C1_S10_L001_R1_001.fastq.gz input/C-1-28558666/MGRF-C1_S10_L001_R2_001.fastq.gz | samtools view -Sb - > BAMs/C-1-28558666/MGRF-C1_S10_L001.bam
I have tried to write the following Makefile rule:
BAMs/%.bam: $(addsuffix .bwt,${REFERENCE}) $(foreach SIDE,R1 R2, ../MGRF_NGS_KUMARAN-25071046/*/*${SIDE}*.fq.gz )
bwa mem ${REFERENCE} $(filter %.fq.gz,$^) | samtools view -Sb - > #>
However, I have got:
$ make -n
make: *** No targets. Stop.
How is it possible to fix the above Makefile?
Your first problem comes from the fact that pattern rules do not create targets if they are not explicitly requested. So, you need to explicitly list the targets to make. In the following example they are listed in the BAMS make variable and then used as dependencies of the all target, which is the first of the makefile. all is thus the default goal and typing make or make all will build all bams.
Your second problem is to handle the pair of R1 and R2 dependencies of each bam target. The GNU make static pattern rules are really useful in such situations:
REFERENCE := ref/dm6.fasta
DIRS := $(wildcard input/*)
R1S := $(foreach dir,$(DIRS),$(wildcard $(dir)/*_R1_001.fastq.gz))
RR1S := $(patsubst input/%_R1_001.fastq.gz,%,$(R1S))
BAMS := $(patsubst %,BAMs/%.bam,$(RR1S))
all: $(BAMS)
$(BAMS): BAMs/%.bam: $(REFERENCE).bwt input/%_R1_001.fastq.gz input/%_R2_001.fastq.gz
#mkdir -p $(dir $#); \
bwa mem $^ | samtools view -Sb - > $#
target: target-pattern: prerequisite-pattern is a static pattern rule.
GNU make also has several functions (call, eval, foreach) that can be used in more complex examples:
REFERENCE := ref/dm6.fasta
DIRS := $(wildcard input/*)
R1S := $(foreach dir,$(DIRS),$(wildcard $(dir)/*_R1_001.fastq.gz))
RR1S := $(patsubst input/%_R1_001.fastq.gz,%,$(R1S))
BAMS := $(patsubst %,BAMs/%.bam,$(RR1S))
all: $(BAMS)
define BAM_rule
BAMs/$(1).bam: $(REFERENCE).bwt input/$(1)_R1_001.fastq.gz input/$(1)_R2_001.fastq.gz
#mkdir -p $$(dir $$#); \
bwa mem $$^ | samtools view -Sb - > $$#
endef
$(foreach r,$(RR1S),$(eval $(call BAM_rule,$(r))))
Explanations:
The RR1S variable lists all C-1-28558666/MGRF-C1_S10_L001 stems. It is then used to build the BAMS variable that lists the targets. The BAM_rule variable is a skeleton of the rule that builds a bam target from its corresponding R1 and R2 dependencies. $(1) is the parameter that will be expanded when using the call function: $(call BAM_rule,foo) expands BAM_rule and substitutes all $(1) occurrences by foo. Note that some $ signs must be doubled to be preserved by this first expansion. Finally,
$(foreach r,$(RR1S),$(eval $(call BAM_rule,$(r))))
iterates over all stems and uses the eval function to instantiate one BAM_rule per stem, expanded and specialized by call.
The normal make expansion will then transform each instance of BAM_rule into the rule you would write manually for the given target:
BAMs/C-1-28558666/MGRF-C1_S10_L001.bam: ref/dm6.fasta.bwt input/C-1-28558666/MGRF-C1_S10_L001_R1_001.fastq.gz input/C-1-28558666/MGRF-C1_S10_L001_R2_001.fastq.gz
#mkdir -p BAMs/C-1-28558666; \
bwa mem ref/dm6.fasta.bwt input/C-1-28558666/MGRF-C1_S10_L001_R1_001.fastq.gz input/C-1-28558666/MGRF-C1_S10_L001_R2_001.fastq.gz | samtools view -Sb - > BAMs/C-1-28558666/MGRF-C1_S10_L001.bam
Related
I need a Makefile that create for every <file.rst> a <file> folder to then execute
hovercraft on the <file.rst> which need a folder as second argument
$ tree
.
├── a.rst
├── b.rst
└── Makefile
With this Makefile
$ cat Makefile
.PHONY: html
HTML_TARGETS:= $(patsubst %.rst,%.html,$(wildcard *.rst))
html: $(HTML_TARGETS)
%.html: %.rst
#rm -fr $(basename $# .html)
#mkdir -p $(basename $# .html)
#hovercraft -Ns $< $(basename $# .html)
$
It kind of works
.
├── a
│ └── index.html
├── a.rst
├── b
│ └── index.html
├── b.rst
└── Makefile
I fell how baroquish this Makefile is, what could be a better way to write it ?
BTW I fail to add in the Makefile this echo:
#echo output done in $(basename $# .html)/index.html
I get:
output done in a /index.html
output done in b /index.html
^
└─ with an unwanted space
I whould like to print:
output done in a/index.html
output done in b/index.html
If I understand correctly that you want to make a directory "x", then execute hovercraft x.rst x/index.html for every file "x.rst", then this should be a succinct way to do so.
SOURCES := $(wildcard *.rst)
TARGETS := $(SOURCES:.rst=/index.html)
%/index.html: %.rst
mkdir -p $*
hovercraft $< $#
.PHONY: all
all: $(TARGETS)
I need to remove in the following case
.
├── a
│ └── index.html
├── a.rst
├── b
│ └── index.html
├── c
│ └── index.html
└── c.rst
folder a and c not b.
I make it work with this Makefile:
$ cat Makefile
.PHONY: clean
HTML_TARGETS:= $(patsubst %.rst,%.html,$(wildcard *.rst))
clean: $(HTML_TARGETS)
%.html: %.rst
#echo rm $(basename $# .html)
$
$ make
rm a
rm c
$
Is there a better way to write it ? (the patsubst use an unneeded .html sub)
What I mean is why don't you just do something like:
.PHONY: clean
HTML_DIRS := $(patsubst %/,%,$(dir $(wildcard */*.html)))
RST_FILES := $(basename $(wildcard *.rst))
clean:
echo rm -r $(filter $(RST_FILES),$(HTML_DIRS))
I want to use make and create a static pattern rule that has the target in a output directory, and the prerequisite files are in the preceeding directory, and it has to work recursively.
I have a minimal example here:
.
├── anotherdir
│ ├── output
│ │ ├── source3.md
│ │ └── source4.md
│ ├── source3.json
│ └── source4.json
├── output
│ ├── source1.md
│ └── source2.md
├── source1.json
└── source2.json
I want to generate the output directories if they do not exist, and I want to generate *.md files from the *.json using make if they do not exist, or *.json is updated.
So far, I have the following Makefile:
SOURCE_FILES := $(shell find ./ -name "*.json")
OUTPUT_FILES := $(join $(addsuffix output/,$(dir $(SOURCE_FILES))), $(addsuffix .md,$(basename $(notdir $(SOURCE_FILES)))))
.PHONY: all
all: $(OUTPUT_FILES)
$(OUTPUT_FILES): %.md: %.json
mkdir -p $(dir $#)
# Command to create MD file from json file into the output directory here
The actual command to create the MD file from the json file doesn't matter here, because I have a script that I will call that will do this for me. The problem here, is that when I try to even run this at all, I get the following output:
> make all
make: *** No rule to make target 'anotherdir/output/source4.json', needed by 'anotherdir/output/source4.md'. Stop.
Obviously, source4.json is not in anotherdir/output, but rather, it's in the preceeding directory, which is just anotherdir. I don't know how to make it so that the pattern $(OUTPUT_FILES): %.md: %.json will match it properly.
Or is a static pattern rule not good here? I'm not sure what to do to fit my scenario.
EDIT: I tried to do something like this:
$(OUTPUT_FILES): %.md: $(join $(subst output,,$(dir %)), $(addsuffix .json,$(basename $(notdir %))))
and this doesn't work, I still get:
> make all
make: *** No rule to make target 'anotherdir/output/source4.json', needed by 'anotherdir/output/source4.md'. Stop.
Edit 2: to clarify, i start with the following files
.
├── anotherdir
│ ├── source3.json
│ └── source4.json
├── source1.json
└── source2.json
And then when i run make, i want it to generate the output folders like this
.
├── anotherdir
│ ├── output
│ │ ├── source3.md
│ │ └── source4.md
│ ├── source3.json
│ └── source4.json
├── output
│ ├── source1.md
│ └── source2.md
├── source1.json
└── source2.json
I want to use some kind of smart makefile syntax to pick up these files names without me hard coding it in myself. Hence, i looked at the documentation and saw that static pattern rules might be the solution that i want, except that i can't get the right prerequisite pattern down.
I would do it this way:
First, find the source files just as you did (with a small change to prevent the unsightly double-slash):
SOURCE_FILES := $(shell find . -name "*.json")
A pattern file would be nice, if we could use two wildcards at once, but Make can't quite do that. So I recommend using a template:
define template
TDIR := $(dir $(1))output
TARG := $$(TDIR)/$(notdir $(basename $(1))).md
$$(TARG): $(1)
mkdir -p $$#
#echo building $$# from $$<
# Command to create MD file from json file into the output directory here
endef
$(foreach SOURCE,$(SOURCE_FILES),$(eval $(call template,$(SOURCE))))
If this works, all that's left is to construct a list of output files, and a default rule that has all of them as prerequisites:
define template
TDIR := $(dir $(1))output
TARG := $$(TDIR)/$(notdir $(basename $(1))).md
OUTPUT_FILES += $$(TARG)
$$(TARG): $(1)
mkdir -p $$#
#echo building $$# from $$<
# Command to create MD file from json file into the output directory here
endef
all:
$(foreach SOURCE,$(SOURCE_FILES),$(eval $(call template,$(SOURCE))))
all: $(OUTPUT_FILES)
It isn't pretty, but it seems to work.
If it had not been proposed already in another answer I would have suggested foreach-eval-call. For completeness here are different solutions for GNU make (they may work also with other versions of make but I did not check):
Creating the output directories beforehand
If the output directories exist already you can refer to ../%.json in your pattern rule:
SOURCE_FILES := $(shell find . -name "*.json")
OUTPUT_FILES := $(join $(dir $(SOURCE_FILES)),\
$(patsubst %.json,output/%.md,$(notdir $(SOURCE_FILES))))
$(shell mkdir -p $(dir $(OUTPUT_FILES)))
.PHONY: all
all: $(OUTPUT_FILES)
%.md: ../%.json
: json2md $< -o $#
This may look strange but if you read carefully the Pattern match section of the GNU make manual you should quickly understand. The only constraint for this to work is that the output directories exist before make searches pattern rules that match the targets. If one does not exist make will complain that there is no eligible rule to build the target. This is the reason for the:
$(shell mkdir -p $(dir $(OUTPUT_FILES)))
at the beginning of the Makefile. Demonstration:
$ make
: json2md output/../source2.json -o output/source2.md
: json2md output/../source1.json -o output/source1.md
: json2md anotherdir/output/../source4.json -o anotherdir/output/source4.md
: json2md anotherdir/output/../source3.json -o anotherdir/output/source3.md
Using the secondary expansion
Secondary expansion gives you the possibility to use automatic variables in the list of prerequisites. The $$ are needed to escape the first expansion by make.
SOURCE_FILES := $(shell find . -name "*.json")
OUTPUT_FILES := $(join $(dir $(SOURCE_FILES)),\
$(patsubst %.json,output/%.md,$(notdir $(SOURCE_FILES))))
.PHONY: all
all: $(OUTPUT_FILES)
$(sort $(dir $(OUTPUT_FILES))):
mkdir -p $#
.SECONDEXPANSION:
$(OUTPUT_FILES): $$(patsubst %output,%,$$(#D))$$(basename $$(#F)).json | $$(dir $$#)
: json2md $< -o $#
Demonstration:
$ make
mkdir -p output/
mkdir -p anotherdir/output/
: json2md source2.json -o output/source2.md
: json2md source1.json -o output/source1.md
: json2md anotherdir/source4.json -o anotherdir/output/source4.md
: json2md anotherdir/source3.json -o anotherdir/output/source3.md
Note: instead of creating the output directories in the json-to-md rule (which has the drawback of creating them several times), I added them as order-only prerequisites and added a specific rule to create them.
Note: the sort function also removes duplicates.
Using recursive make
Here we invoke make (with always the same Makefile) recursively in each sub-directory (except output, of course). Each invocation handles only the local json files, which makes the paths of prerequisites and targets much simpler.
MF := $(realpath $(lastword $(MAKEFILE_LIST)))
SUB_DIRS := $(filter-out . ./output,$(shell find . -maxdepth 1 -type d))
SOURCE_FILES := $(filter-out $(SUB_DIRS),$(wildcard *.json))
OUTPUT_FILES := $(patsubst %.json,output/%.md,$(SOURCE_FILES))
.PHONY: $(SUB_DIRS) all
all: $(SUB_DIRS) $(OUTPUT_FILES)
$(OUTPUT_FILES): output/%.md: %.json | output
: json2md $< -o $#
output:
mkdir -p $#
$(SUB_DIRS):
$(MAKE) -C $# -f $(MF)
Demonstration:
$ make
make -C anotherdir -f /home/doe/json2md/Makefile
make[1]: Entering directory '/home/doe/json2md/anotherdir'
mkdir -p output
: json2md source4.json -o output/source4.md
: json2md source3.json -o output/source3.md
make[1]: Leaving directory '/home/doe/json2md/anotherdir'
mkdir -p output
: json2md source2.json -o output/source2.md
: json2md source1.json -o output/source1.md
In short: I want to compile sources from different directories, and put object files into current directory.
For example, I have files:
test.c
../../lib1/boot.c
../../lib2/startup.c
../common/utils.c
(also few files .s (assembly) and .cpp, but I hope this is not important).
All of their object-files I want to be in the current directory:
test.o
boot.o
startup.o
utils.o
And I can't figure out how to write such rule in my makefile.
For example,
%o.: %.c
does not work now because make can't find a rule to build boot.o from ../../lib1/boot.c, it can only find rule to build ../../lib1/boot.o from ../../lib1/boot.c.
I tried to use this:
%o.: %.c
(my compilation line, for example "gcc -c $^ -o $#")
%o.: ../../lib1/%.c
(my compilation line)
%o.: ../../lib2/%.c
(my compilation line)
%o.: ../common/%.c
(my compilation line)
and it works. But obviously this is not generic enough, and in addition, some user came to me today and said that his application has also some ../../some_other_lib/common_things.c, hence my makefile failed. I looked through our project, and found many such cases with a lot of different directories involved. With my approach, I'll have to write a separate rule for each such directory, with identical compilation line. This does not seem good to me.
So my question is: how to make some generic compilation rule that puts (and checks) object files in current directory, while operating with sources in different directories?
Thank you.
The directories can be extracted from the CSRC variable with $(dir ...) and this list can then be used in the vpath directive.
vpath %.c $(sort $(dir $(CSRC)))
vpath %.s $(sort $(dir $(SSRC)))
vpath %.cpp $(sort $(dir $(CPPSRC)))
(I've thrown in the sort function to remove duplicates, but that's not absolutely necessary.)
Now the rules can be kept simple and make will search the source files in the list of directories.
$(COBJ) := $(notdir $(CSRC))
$(SOBJ) := $(notdir $(SSRC))
$(CPPOBJ) := $(notdir $(CPPSRC))
.PHONY: all
all: $(EXECUTABLE)
$(EXECUTABLE): $(COBJ) $(SOBJ) $(CPPOBJ)
....
$(COBJ): %.o: %.c
...
$(SOBJ): %.o: %.s
...
$(CPPOBJ): %.o: %.cpp
...
Try to use makefile function notdir as this:
%.o: %.c
gcc -c $< -o $(notdir $#)
$# must be equal to the full path ex: ../../lib2/startup.o ad notdir will trunk it to: startup.o.
With this rule you will be able to compile all your source in the current directory.
Actually, your example is like that:
.
└── common
├── lib1
│ └── boot.c
├── lib2
│ └── startup.c
├── test
│ ├── Makefile
│ └── test.c
└── utils.c
I think i will be better like that:
.
├── common
│ ├── lib1
│ │ ├── Makefile
│ │ ├── obj
│ │ └── src
│ │ └── boot.c
│ ├── lib2
│ │ ├── Makefile
│ │ ├── obj
│ │ └── src
│ │ └── startup.c
│ ├── Makefile
│ ├── obj
│ ├── src
│ │ └── utils.c
│ └── test
│ ├── Makefile
│ ├── obj
│ └── src
│ └── test.c
└── Makefile
For that you need all your Makefiles to call the subdirs Makefiles.
and the src/obj dirs is a separation between your source and objects.
SRC := utils.c
OBJ := $(SRC:%.c=%.o)
NAME := project
SRC_D := src
OBJ_D := obj
SUBDIRS := lib1/ \
lib2/ \
test/
all: $(NAME) $(SUBDIRS)
#for dir in $(SUBDIRS); \
do \
$(MAKE) -C $$dir; \
done
$(NAME): $(OBJ:%.o=$(OBJ_D)/%.o)
$(OBJ_D)/%.o : $(SRC_D)/%.c
gcc -c $< -o $#
OK, took me some time, but finally I found the solution (using some threads on this site by the way):
# Defining compilation rules in a way that object files will be produced in current directory, and not in the directory of source files:
all: <List of my targets>
define my_c_rule
$(subst .c,.o,$(notdir $(1))): $(1)
$(CC) $(CFLAGS) $(CDEFINES) $$^ -o $$#
endef
$(foreach f, $(CSRC), $(eval $(call my_c_rule, $(f))))
$(CSRC) contains list of source files with their paths.
Just need to take into account that if earlier I had something like this:
.c.o:
$(CC) $(CFLAGS) $(CDEFINES) $^ -o $#
all: <List of my targets>
...now I have to put all sentence above the rules which I described in my_c_rule procedure. If I don't do this, make stops after compiling first source file. This is because old "wildcard" rules like .c.o or %.o: %.c do not replace all as a default target (even being written earlier), but non-wildcard rules like boot.o: ../../lib1/boot.c (result of the above macros) do replace the default target in case they are written earlier.
I have a directory tree like this:
├── dir_a
│ └── file_1.txt
├── dir_b
│ └── dir_c
│ ├── file_2.txt
| └── file_3.txt
└── file_4.txt
I want to mirror this directory structure to hold the results of a command that processes each text file. I.e., the output would look like this:
├── build
│ ├── dir_a
│ │ └── processed_file_1.txt
│ ├── dir_b
│ │ └── dir_c
│ │ ├── processed_file_2.txt
│ | └── processed_file_3.txt
│ └── processed_file_4.txt
├── dir_a
│ └── file_1.txt
├── dir_b
│ └── dir_c
│ ├── file_2.txt
| └── file_3.txt
└── file_4.txt
I'm not very adept with Makefiles, so my question is: how can I get a Makefile to recreate the directory structure and recursively process all text files to place them into the right place inside the build directory? I'll be running this repeatedly as the input files change, so a Makefile that doesn't process unchanged files seems like the right way to go.
Update:
I should also mention that new input files will be added frequently, so I don't want the Makefile to name them explicitly.
It would be easier if you used stems with different suffixes rather than inserting that "processed_" string, but here's an example that works for me here:
OUTPUTS := build/dir_a/processed_file_1.txt \
build/dir_b/dir_c/processed_file_2.txt \
build/dir_b/dir_c/processed_file_3.txt \
build/processed_file_4.txt
all: $(OUTPUTS)
.SECONDEXPANSION:
$(OUTPUTS): build/% : $$(subst processed_file_,file_,%)
mkdir -p $(dir $#)
cp $< $#
clean:
rm -rf build
You could remove the complication of .SECONDEXPANSION by changing the end of the filename instead of the beginning:
OUTPUTS := build/dir_a/file_1.out \
build/dir_b/dir_c/file_2.out \
build/dir_b/dir_c/file_3.out \
build/file_4.out
all: $(OUTPUTS)
$(OUTPUTS) : build/%.out : %.txt
mkdir -p $(dir $#)
cp $< $#
clean:
rm -rf build
As Carl suggested, you could use secondary expansion, but in conjunction with order-only prerequisites.
BUILD_DIR = build
IN_FILES := dir_a/file_1.out \
dir_b/dir_c/file_2.out \
dir_b/dir_c/file_3.out \
file_4.out
OUT_FILES := $(IN_FILES:%=$(BUILD_DIR)/%)
all: $(OUT_FILES)
.SECONDEXPANSION:
$(OUT_FILES) : $(BUILD_DIR)/%.out : %.txt | $$(#D)/.
# your text processing rule here...
%/. :
mkdir -p $*
| $$(#D) means:
during the secondary expansion calculate the value of $(#D) automatic variable (which is the directory part of the target), and
add the order-only dependency on it, that is ensure that the directory exists, but don't consider remaking the target if it is older than the directory (which is an often case)