Makefile for merging ROOT files with "hadd" - makefile

I am currently writing a Makefile that has several nearly identical rules/recipes to merge .root files using hadd. Is there a good way to condense these into fewer rules? I'm not sure how useful % can be for the cases data[4-6].done since the numbers 4-6 do not match the 1-4 in the directory path.
Many Thanks
[Makefile]
DIR=/hdfs/store/user/$(USER)
DATA.root : data1.done data2.done data3.done data4.done data5.done data6.done
hadd DATA.root $(addsuffix .root,$(basename $^))
rm $(addsuffix .root,$(basename $^))
rm $^
data1.done :
hadd data1.root $(DIR)/de2012A_loose2Aug-DATA/1/*.root
touch data1.done
data2.done :
hadd data2.root $(DIR)/de2012A_loose2Aug-DATA/2/*.root
touch data2.done
data3.done :
hadd data3.root $(DIR)/de2012B_loose2Aug-DATA/1/*.root
touch data3.done
data4.done :
hadd data4.root $(DIR)/de2012B_loose2Aug-DATA/2/*.root
touch data4.done
data5.done :
hadd data5.root $(DIR)/de2012B_loose2Aug-DATA/3/*.root
touch data5.done
data6.done :
hadd data6.root $(DIR)/de2012B_loose2Aug-DATA/4/*.root
touch data6.done

I'll take a crack at it, but you might reconsider your design.
DIR=/hdfs/store/user/$(USER)
DATA.root : data[1-6].done
hadd DATA.root $(^:.done=.root)
rm $(^:.done=.root)
rm $^
data1.done data2.done : PATHLEFT=A
data3.done data4.done data5.done data6.done : PATHLEFT=B
data1.done : PATHRIGHT=1
data2.done : PATHRIGHT=2
data3.done : PATHRIGHT=1
data4.done : PATHRIGHT=2
data5.done : PATHRIGHT=3
data6.done : PATHRIGHT=4
data%.done :
hadd data1.root $(DIR)/de2012$(PATHLEFT)_loose2Aug-DATA/$(PATHRIGHT)/*.root
touch $#
(Yes, I know, I could squeeze two or three more lines out, but I don't want to make it too cryptic.)

Related

Make: Can we Optimize make file targets?

We are supporting 32 bit and 64 bit build in our workflow.For that We have multiple rules in makefiles which are separate for 32-bit and 64-bit. Let me show pair of rules which are same except for the string “32” vs “64” .
Makefile Snippet:-
$(TGTDIR32)/logdir/set_user.c: $(CURDIR)/slv/set_user.c
$(file_transfer)
$(TGTDIR64)/logdir/set_user.c: $(CURDIR)/slv/set_user.c
$(file_transfer)
If you notice, We have targets with same except for the string “32” vs “64”, i want to replace them by single rule/definition. Because we have hundreds of rules like above in our infrastructure code.
Do we have any simplified way in GNUmake to do that?
Thanks in Advance!
Targets which have the same prerequisites and recipe can simply be combined, like so:
$(TGTDIR32)/logdir/set_user.c $(TGTDIR64)/logdir/set_user.c: $(CURDIR)/slv/set_user.c
$(file_transfer)
or more generally:
THESE_TARGETS := $(TGTDIR32)/logdir/set_user.c $(TGTDIR64)/logdir/set_user.c # More...?
...
$(THESE_TARGETS): $(CURDIR)/slv/set_user.c
$(file_transfer)
If Make decides that any member of $(THESE_TARGETS) is out-of-date with respect to the prerequisites, then it will run the recipe for that target.
This makefile:
.PHONY: all clean
all: a b c
a: d e
touch $#
b: d e
touch $#
c: d e
touch $#
d:
touch $#
e:
touch $#
clean:
$(RM) a b c d e
is equivalent to this one:
.PHONY: all clean
all: a b c
a b c: d e
touch $#
d e:
touch $#
clean:
$(RM) a b c d e
Later
There are some static pattern rules...
The same applies. This makefile with static pattern rules:
.PHONY: default clean
default: a.k b.k
a.k: %.k: %.j
cp -f $< $#
b.k: %.k: %.j
cp -f $< $#
a.j:
touch $#
b.j:
touch $#
clean:
$(RM) a.k b.k a.j b.j
is equivalent to this one:
.PHONY: default clean
JS := a.j b.j
KS := $(JS:.j=.k)
default: $(KS)
$(KS): %.k: %.j
cp -f $< $#
$(JS):
touch $#
clean:
$(RM) $(JS) $(KS)
In my opinion this is an appropriate place to use recursive make, at least for the top-level build.
In this situation you could do something like this:
TGTDIR64 = ...
TGTDIR32 = ...
.PHONY: all all32 all64 build
all: all32 all64
all32:
$(MAKE) TGTDIR=$(TGTDIR32) build
all64:
$(MAKE) TGTDIR=$(TGTDIR64) build
# Things below here should just use TGTDIR
build: $(TGTDIR)/b1 $(TGTDIR)/b2
$(TGTDIR)/logdir/set_user.c: $(CURDIR)/slv/set_user.c
$(file_transfer)
$(HEADERGEN_NOTSPLIT_H_COPY): $(TGTDIR)/%.h: %.h $(copy_file)
...
The two rules are semantically identical, they only use a different way to refer to a "parameterized" target. Why you don't just use one target for this
$(TGTDIR)/logdir/set_user.c: $(CURDIR)/slv/set_user.c
$(file_transfer)
and use a properly configured TGTDIR (I suspect this would be something like "xxxx_32" vs "xxxx_64")?
You could achieve this in several ways; one typical would be
ifdef choose32
TGTDIR=xxxx_32
else
TGTDIR=xxxx_64
endif

use a makefile for processing files according two Suffix Rules

I have a docroot folder containing source files that need to built
.usp -> .so
.htt -> .html
Currently my makefile has the following :
.SUFFIXES: .usp .htt
SOURCES = $(wildcard docroot/*.usp) $(wildcard docroot/*.htt)
OBJECTS = $(SOURCES:.usp=.so) $(SOURCES:.htt=.html)
all : ${OBJECTS}
.PHONY : all
%.usp: %.so
usp_compile_incl.sh -i ~/Projects/Concise-ILE/include $<
%.htt: %.html
gpp -I~/Projects/Concise-ILE/include -C $< -o $#
.PHONY: clean
clean:
rm -f docroot/*.so docroot/*.html
make: *** No rule to make target 'docroot/fortune.so', needed by 'all'. Stop.
SOLUTION as per sauerburger
.SUFFIXES: .usp .htt
SOURCES_USP = $(wildcard docroot/*.usp)
SOURCES_HTT = $(wildcard docroot/*.htt)
OBJECTS = $(SOURCES_USP:.usp=.so) $(SOURCES_HTT:.htt=.html)
all : ${OBJECTS}
.PHONY : all
%.so: %.usp
usp_compile_incl.sh -i ~/Projects/Concise-ILE/include $<
%.html: %.htt
gpp -I~/Projects/Concise-ILE/include -C $< -o $#
The build rules for .so and .html are the wrong way round. This should work:
%.so: %.usp
usp_compile_incl.sh -i ~/Projects/Concise-ILE/include $<
%.html: %.htt
gpp -I~/Projects/Concise-ILE/include -C $< -o $#
The syntax of rules is TARGET: DEPENDENCIES.
You should also split the sources variable
SOURCES_USP = $(wildcard docroot/*.usp)
SOURCES_HTT = $(wildcard docroot/*.htt)
OBJECTS = $(SOURCES_USP:.usp=.so) $(SOURCES_HTT:.htt=.html)
Otherwise you end up with a mixed objects list. The first replacement would also include all *.htt files, and the second would include all *.ups files.

Makefile as computational pipeline and multiple stems

I'm writing a computational pipeline for some processing of neuroimaging data with make.
I chose to use make for this task for the advantages that it provides as incrementality, dependency management and easy parallelization of tasks (the latter two are my main focus).
I started writing the Makefile for a single subject and it worked correctly. Now I tried to extend it to perform the same tasks for all the subjects, but I got stuck on the commented part of the Makefile shown below.
File organization is like this:
aal/
data
subj1/
dti/
data
t1/
data
subj2/
dti/
data
t1/
data
subj3/
dti/
data
t1/
data
and this is what I currently got
LOGOPT := >> log.txt 2>> log.txt
SUBJS := subj1 subj2 subj3
MNI_ROIS := ${wildcard AAL/AAL_to_MNI_*.nii.gz}
DIFF_ROIS := ${subst AAL/AAL_to_MNI_,DTI/aal2diff/AAL_,$(MNI_ROIS)}
all: $(SUBJS)
$(SUBJS): % : %/transf/MNI_to_T1_warp.nii.gz %/transf/T1_to_DTI.mat ${addprefix %/,$(DIFF_ROIS)}
#DTI/aal2diff/AAL_%.nii.gz: AAL/AAL_to_MNI_%.nii.gz DTI/nodif_brain.nii.gz transf/MNI_to_T1_warp.nii.gz transf/T1_to_DTI.mat | transf DTI/aal2diff
#applywarp --in=$< --ref=$(word 2,$^) --warp=$(word 3,$^) --postmat=$(word 4,$^) --out=$# --interp=nn $(LOGOPT)
#fslmaths $# -thrP 50 -bin $# $(LOGOPT)
$(addsuffix /DTI/aal2diff, $(SUBJS)):
mkdir -p $#
$(addsuffix /transf/T1_to_DTI.mat, $(SUBJS)): %/transf/T1_to_DTI.mat : %/transf/DTI_to_T1.mat | %/transf
convert_xfm -inverse $< -omat $# $(LOGOPT)
$(addsuffix /transf/DTI_to_T1.mat, $(SUBJS)): %/transf/DTI_to_T1.mat : %/DTI/nodif_brain.nii.gz %/T1/T1_brain.nii.gz | %/transf
flirt -in $< -ref $(word 2,$^) -omat $# -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 6 -cost corratio $(LOGOPT)
$(addsuffix /transf/MNI_to_T1_warp.nii.gz, $(SUBJS)): %/transf/MNI_to_T1_warp.nii.gz : %/transf/T1_to_MNI_warp.nii.gz %/T1/T1.nii.gz | %/transf
invwarp --warp=$< --ref=$(word 2,$^) --out=$# $(LOGOPT)
$(addsuffix /transf/T1_to_MNI_warp.nii.gz, $(SUBJS)): %/transf/T1_to_MNI_warp.nii.gz : %/T1/T1.nii.gz %/transf/T1_to_MNI.mat | %/transf
fnirt --in=$< --aff=$(word 2,$^) --cout=$# --config=T1_2_MNI152_2mm $(LOGOPT)
$(addsuffix /transf/T1_to_MNI.mat, $(SUBJS)): %/transf/T1_to_MNI.mat : %/T1/T1_brain.nii.gz | %/transf
flirt -in $< -ref /usr/local/fsl/data/standard/MNI152_T1_2mm_brain -omat $# -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -dof 12 -cost corratio $(LOGOPT)
$(addsuffix /transf, $(SUBJS)):
mkdir -p $#
$(addsuffix /T1/T1_brain.nii.gz, $(SUBJS)): %/T1/T1_brain.nii.gz : %/T1/T1.nii.gz
bet $< $# -B -f 0.3 -g 0 -m $(LOGOPT)
$(addsuffix /DTI/nodif_brain.nii.gz, $(SUBJS)): %/DTI/nodif_brain.nii.gz : %/DTI/data.nii.gz
bet $< $# -f 0.3 -g 0 -m $(LOGOPT)
$(addsuffix /DTI/data.nii.gz, $(SUBJS)): %/DTI/data.nii.gz : %/DTI/diffusion.nii.gz
eddy_correct $< $# 0 $(LOGOPT)
Intuitively, what I would like to have is the possibility of having multiple stems in the targets section, say, %1 and %2 in order to write something like
%1/DTI/aal2diff/AAL_%2.nii.gz: AAL/AAL_to_MNI_%2.nii.gz %1/DTI/nodif_brain.nii.gz %1/transf/MNI_to_T1_warp.nii.gz %1/transf/T1_to_DTI.mat | %1/transf %1/DTI/aal2diff
applywarp --in=$< --ref=$(word 2,$^) --warp=$(word 3,$^) --postmat=$(word 4,$^) --out=$# --interp=nn $(LOGOPT)
fslmaths $# -thrP 50 -bin $# $(LOGOPT)
AFAIK, multiple stems aren't possible.
Do you have any suggestion of how can I go on?
Is it too bad to use make with something other than building programs?

How do you do pattern rules with arithmetic in Makefile?

My Makefile contains the following rules:
result-n1 : inputs
foo $^
result-n2 : result-n1
hashandmash $^ > $#
result-n3 : result-n2
hashandmash $^ > $#
# ... [~ 50 more lines omitted for clarity]
result-n20 : result-n19
hashandmash $^ > $#
I would love having just one pattern rule. This would be much clearer and would avoid the hard-coded limit at 20.
But how would you do this? How do you say that result-nx depends on result-nx-1? I only see inelegant solutions (like naming my files in base 1, e.g., result-nIIIIIIII).
PS&FWIW, I use GNU Make 3.81
You cannot do that, directly. You can create a pattern rule for result-% that has no prerequisites, then you can declare the prerequisites separately:
result-n1: inputs
foo $^
result-%:
hashandmash $^ > $#
result-n2: result-n1
result-n3: result-n2
...
result-n20: result-n19

Makefile: using variable with computed name as prerequisite

In Make it's possible to compute variable names in runtime using double-evaluation, like $($(var)).
I'm interested if it's possible somehow to make this work:
one.js_DEPS=a b
two.js_DEPS=c b
all: one.js two.js
%.js: $(%.js_DEPS)
cat $^ > $#
I can make this work by declaring two rules with explicit dependencies, like this:
one.js: $(one.js_DEPS)
But this seems a bit anti-DRY. So I'm wondering if I miss something because make doesn't seem to understand me. I use GNU Make 3.81.
% and implicit rules are somewhat second-class citizens in make, since variable expansion seems to happen before implicit rule expansion. So, in your above Makefile, $(%.js_DEPS) is expanded to the empty string before % is substituted.
Alternative:
%.js: %.js_DEPS
cat $^ > $#
%.js_DEPS :
cat $^ > $#
.INTERMEDIATE: one.js_DEPS two.js_DEPS
or simply:
one.js : a b
two.js : c d
%.js :
cat $^ > $#
This could be done using Secondary Expansion.
JS := one.js two.js
all: $(JS)
.SECONDEXPANSION:
one.js_DEPS := a b
two.js_DEPS := c b
$(JS) : $$($$(#F)_DEPS)
cat $^ > $#
But in fact, the goal can be achieved much simply, without using any GNU Make extensions, as thiton has suggested in his answer:
all: one.js two.js
one.js : a b
two.js : c b
%.js :
cat $^ > $#

Resources