Generic make rules using define - makefile

I'm trying to create a common template to generate rules build a set of test cases and place them in unique target locations but have hit a bit of snag with the define directive. The following is the relevant section of my Makefile:
ROOT=../..
PLATFORM=akyboard_gcc
# include all the test cases for the current platform
# They add to the TEST_CASES variable
TEST_CASES=A B
A_FILES = a1.c a2.c
B_FILES = b1.c b2.c
check: $(TEST_CASES:%=check_%)
define CHECK_template
# build artifact directories
$(1)_BLDDIR=$(ROOT)/build/$(PLATFORM)/$(1)
$(1)_OBJDIR=$$($(1)_BLDDIR)/obj
$(1)_EXEDIR=$$($(1)_BLDDIR)/exe
# prepend src/ to all the files that are part of the test case
$(1)_FILES := $($(1:%=%_FILES):%=src/%)
# add the test runner as one of the files to be compiled
$(1)_FILES += test_runner/$(PLATFORM)/main.c
# construct list of objects generated by sources
$(1)_OBJ = $($(1)_FILES:%.c=$$($(1)_OBJDIR)/%.o)
# This creates a rule such as check_{test_case}:
check_$(1): $$($(1)_OBJ)
#echo 1 $(1)
#echo 2 $$($(1)_FILES)
#echo 3 $$($(1)_OBJ)
#echo 5 $$($(1)_OBJDIR)
#echo 4 $$($(1)_BLDDIR)
#echo 6 $$($(1)_EXEDIR)
$$($(1)_OBJDIR)/%.o: $(ROOT)/%.c
#echo coconut
endef
$(foreach testcase, $(TEST_CASES), \
$(eval $(call CHECK_template,$(testcase))) \
)
Issuing "make check" gives the following error
*** No rule to make target `../../build/akyboard_gcc/A/obj/a1.o', needed by `check_A'. Stop.
If I manually create the target rules below it builds without errors
../../build/akyboard_gcc/A/obj/a1.o:
../../build/akyboard_gcc/A/obj/a2.o:
../../build/akyboard_gcc/B/obj/b1.o:
../../build/akyboard_gcc/B/obj/b2.o:
But changing rules like below causes the build error:
../../build/akyboard_gcc/A/obj/%.o:
Would be grateful for any help.

You can find what wrong with your template by replacing eval with info call.
$(foreach testcase, $(TEST_CASES), $(info $(call CHECK_template,$(testcase))))
And your template is fine. But you probably have a problem with pattern rules.
From GNU make manual:
A pattern rule can be used to build a given file only if there is a
target pattern that matches the file name, and all prerequisites in
that rule either exist or can be built.
Perhaps you don't have required sources in $(ROOT) directory. And make just can't create a rule for object file from pattern.

Related

Allow Makefile both append and override target

I have base Makefile for all my services, in some cases I want to use my default "test" target, in other cases I want to override\add to it. These are the files I have so far (and obviously its not working as expect..).
MakefileBase
test:
./.../run-tests.sh
Makefile
BASE_FILE := /path/to/MakefileBase
include ${BASE_FILE}
test:
#$(MAKE) -f $(BASE_FILE) test # un/comment this line in order to run the default tests.
# echo "custom test"
When I run the test with the first line commented out I get the following
Makefile:10: warning: overriding commands for target `test'
/.../MakefileBase:63: warning: ignoring old commands for target `test'
echo "no tests"
no tests
except of the warning it works as expected, the problem is when I try to use the parent function then I get the following errors:
Makefile:9: warning: overriding commands for target `test'
/.../MakefileBase:63: warning: ignoring old commands for target `test'
make[1]: test: No such file or directory
make[1]: *** No rule to make target `test'. Stop.
make: *** [test] Error 2
Actually, both answers so far are wrong or incomplete:
exit 0 in a rule will just exit the current shell (which runs only the exit 0 command, so it is a no-op in this case). So this won't override.
It's not true that you cannot override a command without warning. If it is not necessary that both targets have the same name, you can do:
MakefileBase
.PHONY: test-base
test-base:
echo base
%: %-base # handles cases where you don't want to override
Makefile1
include MakefileBase
.PHONY: test
test:
echo override
Makefile
include MakefileBase
.PHONY: test
test: test-base
echo append
As with double colon rules, the effects of each targets (on each other) have to be considered, especially if you move away from .PHONY (for example, files considered up-to-date because the other rule just updated them).
BTW, I don't see the problem with your approach (aside from the warning). For me it worked fine.
This is what double-colon rules are for:
test::
./.../run-tests.sh
and:
BASE_FILE := /path/to/MakefileBase
include ${BASE_FILE}
test::
#$(MAKE) -f $(BASE_FILE) test
This will "add to" an existing target. There is no way to override a target with a different recipe without incurring a warning.
If you want to do that the only way is to use variables to hold the recipe then override the variable value. For example:
test_recipe = ./.../run-tests.sh
test:
$(test_recipe)
and:
BASE_FILE := /path/to/MakefileBase
include ${BASE_FILE}
test_recipe = #$(MAKE) -f $(BASE_FILE) test
Hacky, but you can get add, and a limited form of override that can never be deeper than one override. Both use double colon rules.
add: use double colons on both rules
override: use double colons on both rules, appending command exit 0 to the last rule
# "addcmd" echoes "BA", "overridecmd" echoes "B"
addcmd ::
echo "A"
addcmd ::
echo "B"
overridecmd ::
echo "A"
overridecmd ::
echo "B"
exit 0

makefile: from 3 input generate one output

I have this version of makefile
[sbsuser#compute-00-01 415]$ make --version GNU Make 3.81
I have directory SOMATIC where I have 3 file . I want to produce a only one output. This is what I wrote.
`
OUTSOMATIC=SOMATIC
FINAL=FINAL
INPUT=$(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2= $(patsubst $(OUTSOMATIC)/%.vcf,$(FINAL)/%somatic.ensemble.gz,$(INPUT))
$(info lista $(OUTSORT2))
$(info lista $(INPUT))
.PHONY: all
all: $(INPUT) $(OUTSOMATIC) $(OUTSORT2) $(FINAL)
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $(FINAL)/somatic_ensemble.gz /illumina/software/database/database_2016/hg19_primary.fa $^
`
With this script make 3 time the same files. I don't understand how to create only one output from list of input to use in the same time.
What is the best way to do this?
If I change $(FINAL)/%somatic.ensemble.gz: in $(FINAL)/somatic.ensemble.gz I have this error:
make: *** No rule to make target FINAL/415_merge_mutect2.somaticsomatic.ensemble.gz', needed byall'. Stop`
You probably should review the GNU make manual introductory sections where they describe how make works.
Let's look at your makefile; first you define some variables. Let's assume that you have the files SOMATIC/foo.vcf, SOMATIC/bar.vcf, and SOMATIC/baz.vcf. Then the variables you created will have these values, after they are expanded:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf
Now your patsubst finds all words in INPUT that match the pattern SOMATIC/%.vcf and replace that with FINAL/%somatic.ensemble.gz, where the part that matches the % in the input is substituted into the output:
OUTSORT2 = FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz
Now, make sees that you've defined an all target. Since it's the first target in the makefile this is the target that will be run by default. After expansion, it will look like this:
all: SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf SOMATIC FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz FINAL
So, make will try to build every prerequisite of the all target to be sure it's up to date. First it tries to build the SOMATIC/*.vcf files. Those files already exist and make doesn't have any rules about how to rebuild them, so it assumes they're up to date.
Next it tries to build the SOMATIC file. This is a directory and it also has no rule to be built, so make assumes that's up to date as well.
Next make tries to build the target FINAL/foosomatic.ensemble.gz. Make does have a rule that can build it, you've created one:
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java ...
This matches the target you want to build, with a % value of foo, so then make substitutes the % in the prerequisite for foo and finds that SOMATIC/foo.vcf exists and doesn't need to be rebuilt, so it runs your recipe. However your recipe doesn't actually create the target FINAL/foosomatic.ensemble.gz; it creates the target FINAL/somatic_ensemble.gz. So this rule is broken because it tells make it will do one thing, but it does something else.
You should always ensure all your recipes build the file represented by the automatic variable $#; that will ensure that you and make agree on the meaning of your rule. If you want your recipe to build some other file, then your rule is written incorrectly.
Next make does the same thing with the next prerequisite of all: FINAL/barsomatic.ensemble.gz. Since that file doesn't exist, make tries to build it using the pattern rule, but again that creates the same output file.
And again for the third .gz file FINAL/bazsomatic.ensemble.gz. That's why things are run three times.
If you change the pattern rule to an explicit rule building FINAL/somatic.ensemble.gz, which is what you want, then make can't find any way to build the prerequisites of the all target so it gives this error.
Your problem is the creation of OUTSORT2. You want to create only one output file, but you've set OUTSORT2 to contain three different files, so make tries to create all three files. You want this:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = $(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2 = $(FINAL)/somatic.ensemble.gz
.PHONY: all
all: $(OUTSORT2)
$(OUTSORT2): $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $# /illumina/software/database/database_2016/hg19_primary.fa $^

GNU make - transform every prerequisite into target (implicitly)

I have another make-like tool that produces an XML as an artifact after parsing my makefile which I'll then further process with Python.
It'd simplify things for me - a lot - if I could have make consider every single prerequisite to be an actual target because then this other tool
will classify each and every file as a "job".
This is a fragment of my makefile:
.obj/eventlookupmodel.o: C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h \
...
I'd want for make to think I have a dummy rule for each prerequisite such as below:
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h:
#echo target pre= $#
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h:
#echo target pre=$#
C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp :
#echo target pre=$#
C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h:
#echo target pre=$#
I don't care about the exact form of the rule just that each file is considered an actual target.
My method of passing in this rule would be by setting the MAKEFILES variable like so
make all MAKEFILES=Dummy.mk
with Dummy.mk containing this rule so that I do not modify the makefiles.
I've tried the following so far.
Dummy.mk:
%.h:
#echo header xyz = $#
%:
#echo other xyz= $#
This partially works.
I run make all --trace --print-data-base MAKEFILES=Dummy.mk and I can see that
make does "bind" the %.h: rule to the header files. In the --print-data-base section, I see that rule being assigned to the header files.
C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef.h:
# Implicit rule search has been done.
# Implicit/static pattern stem: 'C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef'
# Last modified 2016-05-27 12:39:16
# File has been updated.
# Successfully updated.
# recipe to execute (from '#$(QMAKE) top_builddir=C:/Users/User1/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/ top_srcdir=C:/Users/User1/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile C:/Users/User1/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro
', line 2):
#echo header xyz = $#
However, I do NOT see the "echo header xyz $#"-rule being executed.
Regarding the %: rule, it is neither executed for the .cpp files nor "bound" to them in the --print-data-base section.
However, it is bound and executed for existing targets which have no suffix i.e.
all: library binary
binary: | library
ifs: | library
For the %: rule, the reason for this behavior is because of 10.5.5 Match-Anything Pattern Rules: If you do not mark the match-anything rule as terminal, then it is non-terminal. A non-terminal match-anything rule cannot apply to a file name that indicates a specific type of data. A file name indicates a specific type of data if some non-match-anything implicit rule target matches it.
If I make it non-terminal - no double colon - then the rule doesn't apply to built-in types like .cppunless I un-define the built-in rules that negate my intended %: rule.
If I make it terminal, "it does not apply unless its prerequisites actually exist". But a .h or .cpp doesn't technically have prerequisites; can I just create a dummy file and have that as its prerequisite?
NOTE: This has NOTHING to do with gcc -M generation. Yes the -M option would help in the specific case of header and source files but this question is for more generic targets and prerequisites that already exist in the makefile when make is launched.
This may take a few iterations. Try:
%.h: null
#echo header xyz = $#
%: null
#echo other xyz= $#
null:
#:
Try generating static pattern rules for the header files. See one of the answers to Make ignoring Prerequisite that doesn't exist.
Static pattern rules only apply to an explicit list of target files like this:
$(OBJECTS): %.o: %.c
*recipe here*
where the variable OBJECTS is defined earlier in the makefile to be a list of target files (separated by spaces), for example:
OBJECTS := src/fileA.c src/fileB.c src/fileC.c
Note that you can use the various make utility functions to build that list of target files. For example, $(wildcard pattern), $(addsuffix), etc.
You should also ensure that the recipe "touches" the header file to change the timestamp.
I've found that using static pattern rules instead of pattern rules fixes problems where make doesn’t build prerequisites that don’t exist, or deletes files that you want.
Here is an example of using wildcard to copy files from one directory to another.
# Copy images to build/images
img_files := $(wildcard src/images/*.png src/images/*.gif src/images/*.jpg \
src/images/*.mp3)
build_images := $(subst src/,$(BUILD_DIR)/,$(img_files))
$(build_images): $(BUILD_DIR)/images/% : src/images/%
mkdir -p $(dir $#)
cp -v -a $< $#
There are other make functions like addprefix that could be used to generate a more complex file specification.

nested defines in makefile

I'm very new to creating makefiles, so this question might be trivial. If I'm using bad coding practices, I would appreciate to be pointed at it.
I am trying to create a generic makefile that creates rules based on input variables. At the end of my posting is a "minimal" example.
I got 2 problems.
In line 35 I instantiate the meta_template. I would expect the variable $(Dirs) to be expanded and then the meta_template to be called with each entry of $(Dirs). But it only expands till the second last, ie. the result of make terminates with:
make: *** No rule to make target 'Dir03Target01Tag01', needed by 'Dir03'. Stop.
In line 21 I want to create a shortcut rule, a rule that prerequisites all tags to a given example.
Eg: Dir01Target02: Dir01Target02Tag04 Dir01Target02Tag05
First I was puzzled that I couldn't use $($$(1)Tags) but had to use $$($$(1)Tags) to access the according Tags. (stored in Target01Tags for example).
Using $(addprefix) does not seem to expand as I expected, ie the result of make Dir01Target01 is:
Create Rule Dir01Target01 with Dir01Target01Tag01 Tag02 Tag03
where I expected it to be:
Create Rule Dir01Target01 with Dir01Target01Tag01 Dir01Target01Tag02 Dir01Target01Tag03
edit: I found the answer to question 2. I had $(addprefix $(1)$$(1),$$($$(1)Tags))alter to $$(addprefix $(1)$$(1),$$($$(1)Tags)). I think it's because this needs to be expanded in 2 runs. So after the first run $(addprefix Dirxx$(1),$($(1)Tags)) remains.
Thanks in advance.
# Targets to compile
Targets = Target01 Target02
# Tags to add to Targets
Target01Tags = Tag01 Tag02 Tag03
Target02Tags = Tag04 Tag05
# Some List (in the actual makefile a list of directories $(wildcard NumeratedDir*))
Dirs = Dir01 Dir02 Dir03
all: $(Dirs)
define meta_template
$(1): $(foreach target,$(Targets),$(foreach tag,$($(target)Tags),$(1)$(target)$(tag)))
define compile_meta
# $(1) 1st argument of meta_compile (<Dirs>xx, eg Dir03)
# $$(1) 1st argument of compile_template (<Targets>, eg Target01)
$(1)$$(1): #$(addprefix $(1)$$(1),$($$(1)Tags))
#echo Create Rule $(1)$$(1) with $(addprefix $(1)$$(1),$$($$(1)Tags)) <---- Line21
endef
$(foreach target,$(Targets),$(eval $(call compile_meta,$(target))))
define compile_template
# $(1) 1st argument of meta_compile (<Dirs>xx, eg Dir03)
# $$(1) 1st argument of compile_template (<Targets>, eg Target01)
# $$(2) 2nd argument of compile_template (<Targets>Tags, eg Tag01)
# create rule using arguments (eg Dir03Target01Tag01)
$(1)$$(1)$$(2):
#echo Create Rule $(1)$$(1)$$(2)
endef
$(foreach target,$(Targets),$(foreach tag,$($(target)Tags),$(eval $(call compile_template,$(target),$(tag))))) <--- Line35
endef
$(foreach prog,$(Dirs),$(eval $(call meta_template,$(prog))))

How do I make an easy-to-update makefile?

My makefile looks something like this:
FOO_OBJECT_FILES := $(OBJDIR)/Foo.cpp.o
BAR_OBJECT_FILES := $(OBJDIR)/Bar.cpp.o $(OBJDIR)Bar.c.o
ALL_OBJECT_FILES := $(FOO_OBJECT_FILES) $(BAR_OBJECT_FILES)
$(BINDIR)/Foo.a: $(FOO_OBJECT_FILES)
# Rules for making a static library out of Foo's object files go here.
$(BINDIR)/Bar.a: $(BAR_OBJECT_FILES)
# This uses the exact same command sequence as the previous rule.
$(BINDIR)/All.a: $(ALL_OBJECT_FILES)
# Ditto.
# ...
When (not if) more targets are added to the project, the developer will have to update at least three things:
The list of the new target's object files
The list of all object files
Targets for making the new target, even if it uses the same rules as the others
Is there a way to simplify this process, or am I stuck with it?
I tried using wildcard rules, but it doesn't look like they work with macros.
$(BINDIR)/%.a: $(%_OBJECT_FILES)
# ...
You could treat the lists of object files as rules, but then the final target rules can't access them directly.
OBJECT_FILES_Foo: $(OBJDIR)/Foo.cpp.o
OBJECT_FILES_Bar: $(OBJDIR)/Bar.cpp.o $(OBJDIR)Bar.c.o
OBJECT_FILES_All: FOO_OBJECT_FILES BAR_OBJECT_FILES
$(BINDIR)/%.a: OBJECT_FILES_%
# This rule can't see into the object file lists to use them to build.
Is there no better way?
There are probably plenty of ways to do this. One such way is the following. All that needs to be done for a new target is add its name to the list of modules, and give the list of dependencies for it.
BINDIR := bin
OBJDIR := obj
MODULES := Foo Bar
Foo_OBJS := $(OBJDIR)/Foo.cpp.o
Bar_OBJS := $(OBJDIR)/Bar.cpp.o $(OBJDIR)/Bar.c.o
#####################################################
# #
# Nothing below here should need to be altered. #
# #
#####################################################
All_OBJS := $(foreach mod, $(MODULES),$($(mod)_OBJS))
define rule
$(BINDIR)/$(1).a: $($(1)_OBJS)
#echo
#echo 'Target: $$#'
#echo 'Deps : $$^'
endef
$(foreach lib, All $(MODULES), $(eval $(call rule,$(lib))))
###########################################
# #
# The next part is just here for testing. #
# #
###########################################
.PHONY: all
all: $(foreach lib, All $(MODULES),$(BINDIR)/$(lib).a)
%.o:
#echo Making $#
You can't do much about 1 and 2, those are arbitrary things that Make cannot possibly deduce. You can improve 3 slightly:
$(BINDIR)/%.a:
# commands for making a static library
# adding a new target:
QUARTZ_OBJECT_FILES := $(OBJDIR)/Quartz.cpp.o $(OBJDIR)Arbitrary.o
ALL_OBJECT_FILES += $(QUARTZ_OBJECT_FILES)
$(BINDIR)/Quartz.a: $(QUARTZ_OBJECT_FILES)
You could use a template to reduce those three lines to one:
$(eval $(call template, QUARTZ_OBJECT_FILES, $(OBJDIR)/Quartz.cpp.o $(OBJDIR)Arbitrary.o))
but is it really worth it?
While the other answers have provided good solutions for manual makefile writing, you could simply use automake to ease the build process.

Resources