Multiple build variant targets - makefile

I'm building firmware binaries that need multiple variants. There are #ifdefs in the code for different generations of the board, different peripherals and different parameters. So I have filenames like foo-REVC-LCD-EUR.elf for the European version of the board that has an LCD and is rev. C of the hardware.
I'm trying to reduce boilerplate in the makefile. Right now, it looks something like this:
all: foo.elf \
foo-DEBUG.elf \
foo-LCD.elf \
foo-LCD-DEBUG.elf \
target_unit_tests.elf
foo%elf: \
bar.ao \
baz.ao \
banana.ao
target_unit_tests.elf: \
bar.ao \
bar_tests.ao \
baz.ao \
baz_tests.ao\
banana.ao \
banana_tests.ao
# linker
%.elf: %.ao startup_stm32f0xx.ao system_stm32f0xx.ao
$(ARM_CC) $(ARM_FLAGS) $other_arguments -o $# $^
# compiler, general case
%.ao: %.c
$(ARM_CC) $(ARM_CPPFLAGS) $(ARM_FLAGS) $(CFLAGS) -c -o $# $<
# compiler, build variants for `foo'
foo-%.ao: foo.c
$(ARM_CC) $(ARM_CPPFLAGS) $(ARM_FLAGS) $(CFLAGS) -c -o $# $<
# combinatorial explosion
foo-DEBUG.ao: ARM_CPPFLAGS += -DDEBUG
foo-LCD.ao: ARM_CPPFLAGS += -DLCD
foo-LCD-DEBUG: ARM_CPPFLAGS += -DLCD -DDEBUG
The targets contain flags in their names. How can I replace the last three rules (these are multiplying fast) with a single one that extracts flags from the target name?
Or am I going down an insane path and need to rethink my strategy?

This is a pretty open-ended question, which is not usually approved of on StackOverflow. However to get you started:
First, it's best to build your makefile to place all the generated per-target files into a separate directory for each target. Generated common files (if any) can be placed into a common directory.
Second, consider as much as possible defining your builds by setting variables, then writing generic rules that use these variables. This will allow you to create and modify variants by changing the values of variables, without having to mess with complicated rules.
To accomplish this you might take a look at the Metaprogramming Make blog post series (start with the last one listed, which is the first one written) and consider how capabilities such as constructed macro names can help your situation.
Cheers!

Related

Extract macro parameters from target name in Makefile

I have a bunch of lines like this in my makefile, in different permutations. I want to automate them with a general rule, so if I type $ make foo-WHATEVER, make knows how to build it from foo.c and relevant -D flags.
foo-PARAMA.o: foo.c
foo-PARAMA.o: CPPFLAGS += -DPARAMA
foo-PARAMA-PARAMB.o: foo.c
foo-PARAMA-PARAMB.o: CPPFLAGS += -DPARAMA -DPARAMB
foo-PARAMA-PARAMB-PARAMC.o: foo.c
foo-PARAMA-PARAMB-PARAMC.o: CPPFLAGS += -DPARAMA -DPARAMB -DPARAMC
Well, you can try something like this:
foo-%.o : foo.c
$(CC) $(CPPFLAGS) $(addprefix -D,$(subst -, ,$*)) $(CFLAGS) -o $# -c $<
but my suspicion is that you're really going to want to do this for "any source file" not just foo.c. That's much harder because you can't have multiple patterns in a single target or prerequisite.
For that you'll have to know the list of source files up-front and use eval:
SRCS = foo.c bar.c baz.c biz.c
define make-pattern
$1-%.o : $1.c
$$(CC) $$(CPPFLAGS) $$(addprefix -D,$$(subst -, ,$$*)) $$(CFLAGS) -o $$# -c $$<
endif
$(foreach S,$(SRCS),$(eval $(call make-pattern,$S)))
It looks like you want to do configuration management inside make. The problem is that make itself is not really equipped with such functionality. My guess is that at the time of its invention the extreme diversity of platforms which one day will run some unixoid build system simply wasn't foreseeable - so the main use case was to easily define a rather homogenous build with some small deviations.
Now, its 2019 and make is still around and supposedly with a larger user base than ever before. One possible remedy which has the advantage that it doesn't require any other tool is to use gmtt which is a GNUmake library for exactly this configuration purpose. Below I sketch a hypothetical config scenario:
include gmtt/gmtt.mk
PLATFORM := $(shell uname)
# table of platforms with 2 columns. Don't forget that the leading element of every table must be an integer denoting the number of columns!
define AVAILABLE-PLATFORMS :=
2
CYGWIN?NT-4.? toolX
CYGWIN?NT-10.? toolY
Linux* toolZ
FreeBSD toolXYZ
endef
define ADDITIONAL-FLAGS :=
2
toolX -Dfoo
toolY -Dbar
toolZ -Dbaz
toolXYZ -Dfoo
toolXYZ -Dbar
toolXYZ -Dbaz
endef
# select column 2 from the table line(s) which glob-match the current platform:
USED-TOOL := $(call select,2,$(AVAILABLE-PLATFORMS),$$(call glob-match,$(PLATFORM),$$1))
# now select the right flags for the tool
CFLAGS_OPT := $(call select,2,$(ADDITIONAL-FLAGS),$$(call str-eq,$(USED-TOOL),$$1))
$(info $(USED-TOOL) Options: $(CFLAGS_OPT))
Admittedly, this is a lemma on Greenspun's tenth rule but the other options (composing configuration information with a handful of external tools) aren't so attractive either.

order rule execution in parallel make without introducing dependencies

I have a project that consists a single target that requires ~30 files of type A, that are handled by one pattern rule, and one different target of type B, that is independent of the other 30. Here's some pseudo-code to show what I have, in a very simplified form:
OBJECTS=obj/obj1.o obj/obj2.o ...
SPECIAL=special/specialobj.o
libMyLibrary.so: $(OBJECTS) $(SPECIAL)
g++ -shared -o $# $^
obj/%.o: src/%.cxx
g++ -c -fPIC -o $# $^
$(SPECIAL): special/mySpecialCode.cxx
g++ -c -fPIC -o $# $^ -DFANCY_FLAG
The makefile works fine, and the dependency resolution is flawless. However, somehow, make always decides to build $(SPECIAL) last. While this doesn't matter at all, as it works either way, the compilation of $(SPECIAL) takes significantly longer than anything else in my case, almost as long as all other objects combined. Hence, it would be nice if one could encourage make to start with compiling $(SPECIAL), so that it can be compiled in parallel along the others, which would cut compile time dramatically.
Is it possible to achieve such a thing?
PS: I can live with non-portable versions, as the code is a very dedicated piece of software that will only be compiled and run on a very particular set of machines, the setup of which I know pretty well at development time.
EDIT
Following up on a comment made me realise that in the case I have shown here, the problem can indeed be solved by simply swapping the order $(OBJECTS) $(SPECIAL) to $(SPECIAL) $(OBJECTS) - sometimes, it's that simple.
However, in my actual usecase, this does not work, so I have built an MWE that (1) actually works (with gnumake) and is (2) a little bit closer to how things actually work in my project.
Here, the file that takes so long to compile actually depends on a source file that is only generated in an additional step. You can observe the change in order by swapping the two lines following the comment.
If somebody can provide a way how to fix the behavior also for this case, that would be great!
If it's not possible to change the order to the desired one in this case, I will accept any answer that explains to some details why it's impossible.
OBJECTS=obj/obj1.o obj/obj2.o obj/obj3.o obj/obj4.o obj/obj5.o
SPECIAL=special/specialobj.o
SPECIALSRC=special/mySpecialCode.cxx
OBJ_DIR=obj
SPECIAL_DIR=special
all: libMyLibrary
$(SPECIAL_DIR):
mkdir -p $#
$(OBJ_DIR):
mkdir -p $#
libMyLibrary: $(SPECIAL) $(OBJECTS)
cat $^ > $#
obj/%.o: | $(OBJ_DIR)
#echo creating $#
#echo $# > $#
$(SPECIALSRC): | $(SPECIAL_DIR)
#echo special > $#
# swap the following two lines to observe the change in ordering
# $(SPECIAL): $(SPECIALSRC) | $(SPECIAL_DIR)
$(SPECIAL): | $(SPECIAL_DIR)
#echo starting special
#echo special > $#
#sleep 1s
#echo special done
clean:
rm -rf libMyLibrary $(OBJ_DIR) $(SPECIAL_DIR)

separate builds in separate directories

I'm sure this is a totally normal thing to do, but I can't figure out how to get make to do this.
I have a compiler that generates make dependencies of the usual form:
M/A.o : M/A.hs
M/B.o : M/A.o
So I write a rule to compile %.hs into %.o, add a rule to link the binary, include the dependencies file, and all is well. But I want to have several binary targets with different flags. E.g. I want build/test built with -DTESTING and build/profile built with -prof. So I need to keep the .o files in a separate tree, where they will be compiled with special flags.
The straightforward way I can think of would be to have dependencies that look something like this:
build/test/M/A.o : M/A.hs
build/test/M/B.o : build/test/M/A.o
build/profile/M/A.o : M/A.hs
... etc.
And then rules so that %.hs to build/test/%.o compiles with -DTESTING, etc. I think this would work, but it's clumsy, means preprocessing the deps file to add all that build/whatever/ prefix stuff, and would multiply its size by however many kinds of builds.
VPATH appears to be designed for this sort of thing and my idea was that I could set the VPATH and compiler flags depending on the target, and it almost works, but:
%.o: %.hs
#mkdir -p build/M
cp $< build/$#
VPATH = build
main: M/A.o M/B.o
cat $^ >$#
M/A.o : M/A.hs
M/B.o : M/B.hs
The first time the main target wants to run 'cat M/A.o M/B.o >main' which seems contrary to the gnu make documentation that says $^ should include the include the VPATH directory in which the dependency was found. Curiously, if I remove 'main' and make again, this time it uses the correct path. This is GNU make, 3.81.
What's going on here? Is there a better way to build with different flags? VPATH seems like a clumsy tool, surely there is a better way?
Make is working correctly. It tries cat M/A.o M/B.o >main the first time because it can't find the prerequisites it needs, but it knows a rule for M/A.o' andM/B.o(<em>not</em>build/M/A.o' and build/M/B.o) and expects that that is what the rule will produce. If you remove main and try again, it will find build/M/A.o' andbuild/M/B.o` via VPATH.
Let's modify this makefile in stages. First we change the VPATH so that it can find the .hs files (Make is good at using things there to build things here, not vise-versa, and that's what VPATH is good for), and change the rules slightly:
build/%.o: %.hs
cp $< $#
VPATH = M
main: build/A.o build/B.o
cat $^ > $#
Now for the different object directories.
build/test/%.o build/project/%.o: %.hs
cp $< $#
VPATH = M
test: build/test/A.o build/test/B.o
cat $^ > $#
project: build/project/A.o build/project/B.o
cat $^ > $#
Then we simplify those last two rules, so that it's easy to add more object files and binary targets:
OBJECTS = A.o B.o
test: $(addprefix build/test/,$(OBJECTS))
project: $(addprefix build/project/,$(OBJECTS))
test project:
cat $^ > $#
Now for the different compiler flags:
build/test/%.o: FLAGS += test_flags
build/project/%.o: FLAGS += proj_flags
build/test/%.o build/project/%.o: %.hs
#echo building $# from $^ using flags $(FLAGS)
cp $< $#
Finally the dependencies. This is a little tricky. Suppose you want the dependency B.o : A.hs to apply to however many object you have. This is one approach:
OBJECT_PATHS = build/test/ build/project/
# The following is from the included file generated by the compiler
$(addsuffix B.o,$(OBJECT_PATHS)) : A.hs
To generate lines like that, I'd pipe the raw lines (e.g. B.o: A.hs) through sed 's/\(.*\):\(.*\)/\1:\2/', and note that if you want to put this in a makefile command, don't forget to double the $ signs to preserve them for the shell.
I know that's a lot to absorb. Take it one step at a time and let us know how it works out.
If you haven't solved your problem by now or are experiencing further problems, best give the autotools (automake and autoconf) a chance. They'll quickly build you a Makefile that supports more configurable and flexible out-of-tree builds.

How can I improve the speed of my Makefile?

I am building a multiple binaries in C++ & CUDA with a couple of files in Fortran. I found this question and I'm having a similar problem. A user recently asked me to re-build a three year old version of the repository (before we had performed a massive migration and renaming) and I was shocked to see how quickly it built. It would be impossible / incredibly time consuming to determine exactly which of changes between that version and now caused the build to take so friggin' long.
However, I noticed in an answer's comment to the aforementioned question:
In particular, remember to use := instead of =, as := does the expansion
immediately, which saves time. – Jack Kelly Mar 23 at 22:38
Are there other suggestions that I should be aware of?
Note:
I use the 'include' paradigm where each directory I want to build has a module.mk file which is directly included into the one and only Makefile.
I do use several functions like:
(markdown..)
#
# CUDA Compilation Rules
#
define cuda-compile-rule
$1: $(call generated-source,$2) \
$(call source-dir-to-build-dir, $(subst .cu,.cubin, $2)) \
$(call source-dir-to-build-dir, $(subst .cu,.ptx, $2))
$(NVCC) $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) -o $$# -c $$<
$(call source-dir-to-build-dir, $(subst .cu,.cubin, $2)): $(call generated-source,$2)
$(NVCC) -cubin -Xptxas -v $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) $(SMVERSIONFLAGS) -o $$# $$<
$(call source-dir-to-build-dir, $(subst .cu,.ptx, $2)): $(call generated-source,$2)
$(NVCC) -ptx $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) $(SMVERSIONFLAGS) -o $$# $$<
$(subst .o,.d,$1): $(call generated-source,$2)
$(NVCC) $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $3 $(TARGET_ARCH) $(INCFLAGS) $(DEFINES) -M $$< | \
$(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$#)\1 $$#: ,' > $$#.tmp
$(MV) $$#.tmp $$#
endef
But most of these functions I used in the older version...
Lastly: How can I determine if it's the compilation time or the make time which is really slowing things down?
I didn't want to append the entire Makefile. It's 914 lines, but I'd be happy to update the question with snippets if it would help.
Update: Here is my dependency generation rule & compile rule:
#
# Dependency Generation Rules
#
define dependency-rules
$(subst .o,.d,$1): $2
$(CC) $(CFLAGS) $(DEFINES) $(INCFLAGS) $3 $(TARGET_ARCH) -M $$< | \
$(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$#)\1 $$#: ,' > $$#.tmp
$(MV) $$#.tmp $$#
endef
%.d: %.cpp
$(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< | \
$(SED) 's,\($(notdir $*)\.o\) *:,$(dir $#)\1 $#: ,' > $#.tmp
$(MV) $#.tmp $#
Update 2: Using #Beta's suggestion, I was able to parcel out the dependency generation and Makefile time was roughly 14.2% of the overall compiling time. So I'm going focus on minimizing header inclusion in my C++ code first. Thanks to both of you for your suggestions!!
It shouldn't be all that difficult to determine which changes slowed everything down. You have all the versions over the past three years (I hope), and you say the difference is dramatic. So try the version from two years ago. If it's taking too long, do a binary search. You could even automate this process, have it run overnight and give you a graph in the morning, build time sampled each month for the past 36 months.
If you're using GNUMake (as I hope), `make -n` will print out the commands it would execute, without actually executing them. This will give you all of the Make time with no compilation time.
One of the biggest sources of unnecessary build time (even bigger than recursion, which you aren't using) is unnecessary rebuilding, recompiling/relinking/whatever when you don't really need to. This can be because your makefile doesn't handle dependencies correctly, or because your C++ files `#include` headers recklessly, or something about CUDA or FORTRAN that I wouldn't know. Run Make twice in a row, and see if it does anything on the second pass. Look over the makefile for suspiciously huge prerequisite lists. Have a skillful programmer take a look at a few of the source files, especially the newer ones, and check for unnecessary dependencies.
ElectricMake (emake) is a drop-in replacement for gmake that makes it really, really easy to answer questions like this. emake can generate an annotated build log that includes detailed timing information about every job in the build, and then you can load that into ElectricInsight to generate, for example, the Job Time by Type report:
If you want to give it a try, you can get an eval copy.
(disclaimer: I'm the architect and lead developer of ElectricMake and ElectricInsight!)
I really doubt that make's variable assignment (immediate with := or recursive =) can make significant impact on speed in general. One specific and obvious case when it does serious impact is shell command:
VAR := $(shell ...)
There could be other hidden consuming processes which are not obvious. For example in our environment standard temporary windows directory was on network drive. Thus, when make stored/updated files on that drive (even with 1G LAN) - it was very slow. What you need is to debug makefile(s). This maybe helpful.
According to the mentioned doc you can put debug prints in the form of $(warning Going to do bla-bla-bla) and then watch where the process freezes most of all.

How to write different implicit rules for different file names for GNU Make

I have a directory in which I keep adding different C++ source files, and generic Makefile to compile them. This is the content of the Makefile:
.PHONY: all clean
CXXFLAGS = -pipe -Wall -Wextra -Weffc++ -pedantic -ggdb
SRCS = $(wildcard *.cxx)
OBJS = $(patsubst %.cxx,%.out,$(SRCS))
all: $(OBJS)
clean:
rm -fv $(OBJS)
%.out: %.cxx
$(CXX) $(CXXFLAGS) $^ -o $#
NOTE: As is obvious from above, I am using *.out for executable file extensions (and not for object file).
Also, there are some files which are compiled together:
g++ file_main.cxx file.cxx -o file_main.out
To compile such files, until now I have been adding explicit rules in the Makefile:
file_main.out: file_main.cxx file.cxx
file.out: file_main.out
#echo "Skipping $#"
But now my Makefile has a lot of explicit rules, and I would like to replace them with a simpler implicit rule.
Any idea how to do it?
First, this method of compiling several source files directly into an executable is not a terribly good idea. The more common compile-then-link approach will save a lot of unnecessary compilation.
That said, the way to replace many explicit rules with a simpler rule depends on what the explicit rules have in common. You already have a pattern rule:
%.out: %.cxx
$(CXX) $(CXXFLAGS) $^ -o $#
and if all you want to do is add another source file to a particular target, you don't have to do this:
g++ file_main.cxx file.cxx -o file_main.out
you can get the effect just by adding a prerequisite (in a line by itself):
file_main.out: file.cxx
If you have several targets with that pattern, you can use a pattern rule:
file_main.out another_main.out a_third_main.out: %_main.out : %.cxx
If you have many such targets, you can use a variable:
MAIN_THINGS = file another a_third a_fourth and_yet_another
MAIN_TARGETS = $(addsuffix _main.out, $(MAIN_THINGS))
$(MAIN_TARGETS): %_main.out : %.cxx
And you can add other patterns for other target sets, even overlapping sets. Does that cover your situation?
It seems that you are putting the source code for multiple different programs in the same folder, and this is really the source of your problems. If you separate the source code for your libraries and programs into separate folders (or, better yet, separate projects), then you can skirt this issue by depending on all source files in the given folder. When you have everything intermixed, it is necessary to be explicit.
That said, if your dependencies have consistent, predictable names, then it is possible to eliminate this redundancy by using the eval function. For example, based on the example above:
#
# I'm going to use standard file extensions here,
# slightly deviating from your conventions. I am also
# assuming that there is a variable named PROGNAMES,
# which gives a list of all the programs to be built.
#
define ADD_EXECUTABLE
$(1): $(1).o $(1)_main.o
$(LINK.cc) $(1).o $(1)_main.o -o $(1)
endef
$(foreach progname,$(PROGNAMES),$(eval $(call ADD_EXECUTABLE,$(progname))))
Also, just a few suggestions... you should append to CXXFLAGS rather than overwrite it and you would be better off using standard file extensions (".cpp" for C++ source files, ".o" for object files, no extension for executables). See my Makefile tutorial for tips on making things easier with Make (no pun intended).

Resources