I'm new to Make, and am trying to use it to automate a ml pipeline. I have defined two rules in two files: the first Makefile has file locations and a target to clean a dataset. The second one Makefile.label.surv1 has a target to extract labels using a python script. Below is the code for both:
Makefile
#--------------------------------------------------------------------------
# Survival Analysis - Churn prediction top makefile
#--------------------------------------------------------------------------
# date of the snapshot to consider
SNAP_TRN := 2019-06-18
SNAP_TST := 2019-07-24
# directories
DIR_DATA := data
DIR_BUILD := build
DIR_FEATURE := $(DIR_BUILD)/feature
DIR_METRIC := $(DIR_BUILD)/metric
DIR_MODEL := $(DIR_BUILD)/model
DIR_CONFIG := configs
# data files for training and predict
DATA_TRN := $(DIR_DATA)/processed/
DATA_TST := $(DIR_DATA)/processed/
# NOS feature selection
FEATS := $(DIR_CONFIG)/featimp_churnvol.csv
# Config files
CONFIG_PANEL := $(DIR_CONFIG)/config_panel.yaml
CONFIG_INPUT := $(DIR_CONFIG)/config_inpute.yaml
# Generates a clean dataset (inputted and one hot encoded) and labels for train and test
buildDataset: $(DATA_TRN) $(DATA_TST) $(CONFIG_PANEL) $(CONFIG_INPUT) $(FEATS)
python src/buildDataset.py --train-file $< \
--test-file $(word 1, $^) \
--config-panel $(word 2, $^) \
--config-input $(word 3, $^) \
--feats $(lastword $^)
Makefile.label.surv1
#--------------------------------------------------------------------------
# surv1: survival labels
#--------------------------------------------------------------------------
include Makefile
FEATURE_NAME := surv1
GRAN := weekly
STUDY_DUR := 1
Y_SURV_TRN := $(DIR_DATA)/survival/$(FEATURE_NAME)_train_$(SNAP_TRN)_$(GRAN)_$(STUDY_DUR).pkl
Y_SURV_TST := $(DIR_DATA)/survival/$(FEATURE_NAME)_test_$(SNAP_TST)_$(GRAN)_$(STUDY_DUR).pkl
$(Y_SURV_TRN) $(Y_SURV_TST): $(DATA_TRN) $(DATA_TST) $(CONFIG_PANEL) $(STUDY_DUR) $(GRAN)
python ./src/generate_surv_labels.py --train-file $< \
--test-file $(word 1, $^) \
--train-label-file $(Y_SURV_TRN) \
--test-label-file $(Y_SURV_TST)\
--config-panel $(word 2, $^) \
--study-dur $(word 3, $^) \
--granularity $(lastword $^)
So when I run make -f Makefile.label.surv1, it also re-runs the target buildDataset, which I don't want in this case. In this case I haven't made any changes to buildDataset so I don't understand why make re-runs this target... Is there anyway to prevent a target from re-running others?
If you don't provide a target to build on the make command line, make will build the default target. The default target is the FIRST explicit target defined in your makefile(s).
In your Makefile.label.survival the first thing you do before you define any other target, is include Makefile. That means that if any target is defined in Makefile, it will be considered the first explicit target.
And, indeed, Makefile defines buildDataset and so that is the default target and if you run make without any specific target, that's the target that will be built.
Also, this rule is very likely not right:
$(Y_SURV_TRN) $(Y_SURV_TST): ...
I'm not sure what you're hoping this will do, but if you expect that make will interpret this to mean that one invocation of the recipe will build both these files, that's not what this syntax means.
Related
I am trying build a project with several modules included. the file tree looks like this:
the individual Makefiles for lib_one, lib_two and Main.cpp works fine, generating the 2 DLL and the EXE for the project, but requires enter at each directory and execute the command 'make' for each module.
I want now a Makefile in the top level directory ('/project') which triggers the building for all the other modules (DLLs and EXE) once. Based on some search results I got something like that:
lib_one := ./src/lib_one
lib_two := ./src/lib_two
libraries := $(lib_one) $(lib_two)
player := ./src
.PHONY: all $(player) $(libraries)
all: $(player)
$(player) $(libraries):
$(MAKE) --directory=$#
$(player): $(libraries)
when I execute the command 'make' I got this error:
Makefile:10: ***. Stop.
(the line 10 is this one: $(MAKE) --directory=$#). Besides a way to fix this error, I am looking to a way to accomplish this 3 things:
1)
the object files (*.o) from all the modules (DLLs and EXE) should be stores on a directory build in the same place from the directory src.
2)
the final files (*.dll and *.exe) should be placed in a directory reelease alongside the directories src and build.
3) if item 2 was possible, would be nice if each type of file being placed in a specific directory inside release (bin for *.exe, lib for *.dll and shared for other types). also, even with the exe and dlls in different directories, would be possible run the executable from this directory (making it search for the necessary libraries in ../lib alongside the other usual places?).
#Maxim is likely right for the cause of the error -- spaces vs tabs. As far as a your other questions go, without using non-recursive make, you could do something like:
Makefile:
export base_dir := ${CURDIR}
export obj_dir := ${base_dir}/build
export release_dir := ${base_dir}/release
lib_one := ${base_dir}/src/lib_one
lib_two := ${base_dir}/src/lib_two
libraries := $(lib_one) $(lib_two)
player := ${base_dir}/src
.PHONY: all $(player) $(libraries)
all: $(player)
$(player) $(libraries): | ${obj_dir} ${release_dir}
$(MAKE) --directory=$#
$(player): $(libraries)
${obj_dir} ${release_dir} :
#mkdir -f $#
The sub-makefiles would have access to any exported variable from the parent (see here), so in these you could do stuff like:
${obj_dir}/%.o : %.c
#echo compiling "$^ ==> $#"
$(CC) -c -o $# $^ $(CFLAGS)
which would compile the objects to the right directory.
Beware though -- if you have two components that produce an object file with the same name, because they're being produced in the same directory, you will end up with a potentially hard to debug race condition. Typically each component produces object files in its own unique directory to avoid this very thing.
I want to write a rule that looks something like this:
A_vs_B.txt : A.txt B.txt
but a general rule. The problem is I can't have two %s (as far as I know). I was thinking of just making the target %.txt and then using string functions to parse out the A and B in the dependencies, but that will be fairly complicated. I'm wondering if there's a better way to write a rule like this.
Two choices:
Secondary expansion (your idea of string munging in the dependencies). Something like this (might be better/shorter ways but this is what came to me first).
A_v_B.txt: $$(addsuffix $$(suffix $$(lastword $$(subst _, ,$$#))),$$(firstword $$(subst _, ,$$#)) $$(lastword $$(basename $$(subst _, ,$$#))))
#echo $^
Generated targets/prerequisites. Though how you generate the targets/prerequisites you need depends on where/how the pairs are generated/etc.
H1 := $(REP1)
H2 := $(REP2)
SEP := _vs_
JOIN := $(SEP)
define mktgt
H1 += $R.pr1
H2 += $R.pr2
JOIN += $(SEP)
endef
$(foreach R,$(REP1) $(REP2) $(REP1)$(REP2),$(eval $(mktgt)))
PAIRS := $(join $(join $(H1),$(JOIN)),$(H2))
$(foreach P,$(PAIRS),$(eval $P.txt: $(addsuffix .txt,$(subst _vs_, ,$P))))
# Debugging output
$(foreach P,$(PAIRS),$(info $P.txt: $(addsuffix .txt,$(subst _vs_, ,$P))))
That creates (as you'll see from the debugging output) the target/prerequisite mappings. It doesn't give the targets any recipe. I assume you have a recipe already and that you've assigned it to all the appropriate targets.
If not then adding something like:
$(PAIRS):
#echo 'Use $^ to generate $#'
should work.
I have a bunch of libraries that I want to build as dependencies:
LIBS = libs/foo libs/bar
Each library has an object that I need to compile against, all in a predictable spot. They are:
libs/foo/lib/libfoo-O3.a
libs/bar/lib/libbar-O3.a
I would like to effectively transform my LIBS string into these rules:
build : libs/foo/lib/libfoo-O3.a libs/bar/lib/libbar-O3.a
libs/foo/lib/libfoo-O3.a:
$(MAKE) -C libs/foo
libs/bar/lib/libbar-O3.a:
$(MAKE) -C libs/bar
I know you can't use % twice, so unfortunately libs/%/lib/lib%-O3.a is a non-starter as a target. Is there another way to do this? Something with a define template?
If you can change the LIBS variable to just contain the name, you can do it easily:
LIBS = foo bar
LIBPATHS := $(foreach L,$(LIBS),libs/$L/lib/lib$L-O3.a)
If you can't you can still do this, only slightly less readable:
LIBS = libs/foo libs/bar
LIBPATHS := $(foreach L,$(LIBS),$L/lib/lib$(notdir $L)-O3.a)
Then just add something like:
$(LIBPATHS):
$(MAKE) -C $(firstword $(subst /lib/, ,$#))
My makefile looks something like this:
FOO_OBJECT_FILES := $(OBJDIR)/Foo.cpp.o
BAR_OBJECT_FILES := $(OBJDIR)/Bar.cpp.o $(OBJDIR)Bar.c.o
ALL_OBJECT_FILES := $(FOO_OBJECT_FILES) $(BAR_OBJECT_FILES)
$(BINDIR)/Foo.a: $(FOO_OBJECT_FILES)
# Rules for making a static library out of Foo's object files go here.
$(BINDIR)/Bar.a: $(BAR_OBJECT_FILES)
# This uses the exact same command sequence as the previous rule.
$(BINDIR)/All.a: $(ALL_OBJECT_FILES)
# Ditto.
# ...
When (not if) more targets are added to the project, the developer will have to update at least three things:
The list of the new target's object files
The list of all object files
Targets for making the new target, even if it uses the same rules as the others
Is there a way to simplify this process, or am I stuck with it?
I tried using wildcard rules, but it doesn't look like they work with macros.
$(BINDIR)/%.a: $(%_OBJECT_FILES)
# ...
You could treat the lists of object files as rules, but then the final target rules can't access them directly.
OBJECT_FILES_Foo: $(OBJDIR)/Foo.cpp.o
OBJECT_FILES_Bar: $(OBJDIR)/Bar.cpp.o $(OBJDIR)Bar.c.o
OBJECT_FILES_All: FOO_OBJECT_FILES BAR_OBJECT_FILES
$(BINDIR)/%.a: OBJECT_FILES_%
# This rule can't see into the object file lists to use them to build.
Is there no better way?
There are probably plenty of ways to do this. One such way is the following. All that needs to be done for a new target is add its name to the list of modules, and give the list of dependencies for it.
BINDIR := bin
OBJDIR := obj
MODULES := Foo Bar
Foo_OBJS := $(OBJDIR)/Foo.cpp.o
Bar_OBJS := $(OBJDIR)/Bar.cpp.o $(OBJDIR)/Bar.c.o
#####################################################
# #
# Nothing below here should need to be altered. #
# #
#####################################################
All_OBJS := $(foreach mod, $(MODULES),$($(mod)_OBJS))
define rule
$(BINDIR)/$(1).a: $($(1)_OBJS)
#echo
#echo 'Target: $$#'
#echo 'Deps : $$^'
endef
$(foreach lib, All $(MODULES), $(eval $(call rule,$(lib))))
###########################################
# #
# The next part is just here for testing. #
# #
###########################################
.PHONY: all
all: $(foreach lib, All $(MODULES),$(BINDIR)/$(lib).a)
%.o:
#echo Making $#
You can't do much about 1 and 2, those are arbitrary things that Make cannot possibly deduce. You can improve 3 slightly:
$(BINDIR)/%.a:
# commands for making a static library
# adding a new target:
QUARTZ_OBJECT_FILES := $(OBJDIR)/Quartz.cpp.o $(OBJDIR)Arbitrary.o
ALL_OBJECT_FILES += $(QUARTZ_OBJECT_FILES)
$(BINDIR)/Quartz.a: $(QUARTZ_OBJECT_FILES)
You could use a template to reduce those three lines to one:
$(eval $(call template, QUARTZ_OBJECT_FILES, $(OBJDIR)/Quartz.cpp.o $(OBJDIR)Arbitrary.o))
but is it really worth it?
While the other answers have provided good solutions for manual makefile writing, you could simply use automake to ease the build process.
Given:
programs := apps/prog1 apps/prog2 # the actual list is quite long
sources := src/prog1.cpp src/prog2.cpp # showing only 2 files
Make file has 2 targets release and debug. Each target should build every program in bin/ directory and appends target name to the file name.
For example, building release should create bin/prog1_release and bin/prog2_release.
How to write static pattern rule to do it?
Thanks.
This will do it (in GNUMake 3.81):
BINS := $(patsubst apps/%,bin/%,$(programs)) # bin/prog1 bin/prog2 ...
release_bins := $(addsuffix _release,$(BINS)) # bin/prog1_release ...
debug_bins := $(addsuffix _debug,$(BINS)) # bin/prog1_debug ...
$(release_bins): bin/%_release: src/%.cpp
#build the binaries according to the release rule
$(debug_bins): bin/%_debug: src/%.cpp
#build the binaries according to the debug rule
release: $(release_bins)
debug: $(debug_bins)
.PHONY: release debug
# If it turns out that one of the progs needs something else too:
bin/prog20_debug: somethingElse.cpp
(There are ways to make this slightly more concise, but at the cost of clarity.)