I am building a multiple binaries in C++ & CUDA with a couple of files in Fortran. I found this question and I'm having a similar problem. A user recently asked me to re-build a three year old version of the repository (before we had performed a massive migration and renaming) and I was shocked to see how quickly it built. It would be impossible / incredibly time consuming to determine exactly which of changes between that version and now caused the build to take so friggin' long.
However, I noticed in an answer's comment to the aforementioned question:
In particular, remember to use := instead of =, as := does the expansion
immediately, which saves time. – Jack Kelly Mar 23 at 22:38
Are there other suggestions that I should be aware of?
Note:
I use the 'include' paradigm where each directory I want to build has a module.mk file which is directly included into the one and only Makefile.
I do use several functions like:
(markdown..)
#
# CUDA Compilation Rules
#
define cuda-compile-rule
$1: $(call generated-source,$2) \
$(call source-dir-to-build-dir, $(subst .cu,.cubin, $2)) \
$(call source-dir-to-build-dir, $(subst .cu,.ptx, $2))
$(NVCC) $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) -o $$# -c $$<
$(call source-dir-to-build-dir, $(subst .cu,.cubin, $2)): $(call generated-source,$2)
$(NVCC) -cubin -Xptxas -v $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) $(SMVERSIONFLAGS) -o $$# $$<
$(call source-dir-to-build-dir, $(subst .cu,.ptx, $2)): $(call generated-source,$2)
$(NVCC) -ptx $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) $(SMVERSIONFLAGS) -o $$# $$<
$(subst .o,.d,$1): $(call generated-source,$2)
$(NVCC) $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $3 $(TARGET_ARCH) $(INCFLAGS) $(DEFINES) -M $$< | \
$(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$#)\1 $$#: ,' > $$#.tmp
$(MV) $$#.tmp $$#
endef
But most of these functions I used in the older version...
Lastly: How can I determine if it's the compilation time or the make time which is really slowing things down?
I didn't want to append the entire Makefile. It's 914 lines, but I'd be happy to update the question with snippets if it would help.
Update: Here is my dependency generation rule & compile rule:
#
# Dependency Generation Rules
#
define dependency-rules
$(subst .o,.d,$1): $2
$(CC) $(CFLAGS) $(DEFINES) $(INCFLAGS) $3 $(TARGET_ARCH) -M $$< | \
$(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$#)\1 $$#: ,' > $$#.tmp
$(MV) $$#.tmp $$#
endef
%.d: %.cpp
$(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< | \
$(SED) 's,\($(notdir $*)\.o\) *:,$(dir $#)\1 $#: ,' > $#.tmp
$(MV) $#.tmp $#
Update 2: Using #Beta's suggestion, I was able to parcel out the dependency generation and Makefile time was roughly 14.2% of the overall compiling time. So I'm going focus on minimizing header inclusion in my C++ code first. Thanks to both of you for your suggestions!!
It shouldn't be all that difficult to determine which changes slowed everything down. You have all the versions over the past three years (I hope), and you say the difference is dramatic. So try the version from two years ago. If it's taking too long, do a binary search. You could even automate this process, have it run overnight and give you a graph in the morning, build time sampled each month for the past 36 months.
If you're using GNUMake (as I hope), `make -n` will print out the commands it would execute, without actually executing them. This will give you all of the Make time with no compilation time.
One of the biggest sources of unnecessary build time (even bigger than recursion, which you aren't using) is unnecessary rebuilding, recompiling/relinking/whatever when you don't really need to. This can be because your makefile doesn't handle dependencies correctly, or because your C++ files `#include` headers recklessly, or something about CUDA or FORTRAN that I wouldn't know. Run Make twice in a row, and see if it does anything on the second pass. Look over the makefile for suspiciously huge prerequisite lists. Have a skillful programmer take a look at a few of the source files, especially the newer ones, and check for unnecessary dependencies.
ElectricMake (emake) is a drop-in replacement for gmake that makes it really, really easy to answer questions like this. emake can generate an annotated build log that includes detailed timing information about every job in the build, and then you can load that into ElectricInsight to generate, for example, the Job Time by Type report:
If you want to give it a try, you can get an eval copy.
(disclaimer: I'm the architect and lead developer of ElectricMake and ElectricInsight!)
I really doubt that make's variable assignment (immediate with := or recursive =) can make significant impact on speed in general. One specific and obvious case when it does serious impact is shell command:
VAR := $(shell ...)
There could be other hidden consuming processes which are not obvious. For example in our environment standard temporary windows directory was on network drive. Thus, when make stored/updated files on that drive (even with 1G LAN) - it was very slow. What you need is to debug makefile(s). This maybe helpful.
According to the mentioned doc you can put debug prints in the form of $(warning Going to do bla-bla-bla) and then watch where the process freezes most of all.
Related
I have a generic build system that I import as a Git submodule into my projects, each of which has a wrapper makefile that includes the build system's makefile. Suppose I have something like the following in the build system's makefile:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $<
And prerequisites like the following in the projects' makefiles:
$(BIN_DIR)/foo: $(FOO_OBJ_FILES:%=$(OBJ_DIR)/%)
While this technique works well for libraries, it obviously doesn't work well for executables, because each needs to be linked with a different set of libraries. So my question is how to achieve that without relying of GNU Make-specific features? With GNU Make, I could do something like this:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $< $((nodir $(basename $#))_LINK:%=-l %)
foo_LINK = mylib1 mylib2
$(BIN_DIR)/foo: $(FOO_OBJ_FILES:%=$(OBJ_DIR)/%)
One portable solution that I've thought of but would rather not use because it's a huge hack, is to have a .link file associated with each executable, containing one line like this: -l mylib1 -lmylib2 and then do something like this:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $< `cat $#.link`
So, all these things aside, I need to somehow magically pass the correct set of libraries to the pattern rule or to have a variable expand to the correct set of libraries when the pattern rule gets executed.
Any ideas? Thanks.
PS: I know this seems like more trouble than it's worth but I've simplified things for the purpose of this question. In reality, it saves me from dealing with some pepetition, not just saving me one line.
Okay, I've come up with an answer:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $< $($(F)_LINK:%=-l %)
foo_LINK = mylib1 mylib2
Unfortunately, while there is a proposal for adding nested/computed variables into POSIX make for Issue 8 of the specification, it seems to have been in limbo for the past 10 years or so.
I have a bunch of lines like this in my makefile, in different permutations. I want to automate them with a general rule, so if I type $ make foo-WHATEVER, make knows how to build it from foo.c and relevant -D flags.
foo-PARAMA.o: foo.c
foo-PARAMA.o: CPPFLAGS += -DPARAMA
foo-PARAMA-PARAMB.o: foo.c
foo-PARAMA-PARAMB.o: CPPFLAGS += -DPARAMA -DPARAMB
foo-PARAMA-PARAMB-PARAMC.o: foo.c
foo-PARAMA-PARAMB-PARAMC.o: CPPFLAGS += -DPARAMA -DPARAMB -DPARAMC
Well, you can try something like this:
foo-%.o : foo.c
$(CC) $(CPPFLAGS) $(addprefix -D,$(subst -, ,$*)) $(CFLAGS) -o $# -c $<
but my suspicion is that you're really going to want to do this for "any source file" not just foo.c. That's much harder because you can't have multiple patterns in a single target or prerequisite.
For that you'll have to know the list of source files up-front and use eval:
SRCS = foo.c bar.c baz.c biz.c
define make-pattern
$1-%.o : $1.c
$$(CC) $$(CPPFLAGS) $$(addprefix -D,$$(subst -, ,$$*)) $$(CFLAGS) -o $$# -c $$<
endif
$(foreach S,$(SRCS),$(eval $(call make-pattern,$S)))
It looks like you want to do configuration management inside make. The problem is that make itself is not really equipped with such functionality. My guess is that at the time of its invention the extreme diversity of platforms which one day will run some unixoid build system simply wasn't foreseeable - so the main use case was to easily define a rather homogenous build with some small deviations.
Now, its 2019 and make is still around and supposedly with a larger user base than ever before. One possible remedy which has the advantage that it doesn't require any other tool is to use gmtt which is a GNUmake library for exactly this configuration purpose. Below I sketch a hypothetical config scenario:
include gmtt/gmtt.mk
PLATFORM := $(shell uname)
# table of platforms with 2 columns. Don't forget that the leading element of every table must be an integer denoting the number of columns!
define AVAILABLE-PLATFORMS :=
2
CYGWIN?NT-4.? toolX
CYGWIN?NT-10.? toolY
Linux* toolZ
FreeBSD toolXYZ
endef
define ADDITIONAL-FLAGS :=
2
toolX -Dfoo
toolY -Dbar
toolZ -Dbaz
toolXYZ -Dfoo
toolXYZ -Dbar
toolXYZ -Dbaz
endef
# select column 2 from the table line(s) which glob-match the current platform:
USED-TOOL := $(call select,2,$(AVAILABLE-PLATFORMS),$$(call glob-match,$(PLATFORM),$$1))
# now select the right flags for the tool
CFLAGS_OPT := $(call select,2,$(ADDITIONAL-FLAGS),$$(call str-eq,$(USED-TOOL),$$1))
$(info $(USED-TOOL) Options: $(CFLAGS_OPT))
Admittedly, this is a lemma on Greenspun's tenth rule but the other options (composing configuration information with a handful of external tools) aren't so attractive either.
Make chooses the same source file for different object files. Both are a list of files, only with different filenames. Make switches between the object files but not the source files.
I've already tried some of the answers on StackOverflow with related problems, though those solutions either seem too complicated for what's needed, some don't work and others need the files to be in one directory.
I've also tried compiling the files together in one go (with gcc), but this gives some problems with the linking of the rest of the file.
$(OBJFILES): $(SRCFILES)
$(CC) $(CCFLAGS) -c $< -o $#
$(OBJFILES) contains the following files:
src/kernel.o src/screen/screen_basic.o
And $(SRCFILES) contains these files:
src/kernel.c src/screen/screen_basic.c
Basically, src/kernel.c gets compiled to both src/kernel.o and src/screen/screen_basic.o, while src/screen/screen_basic.c never gets compiled.
What's run by make (I replaced the options for gcc with the variable names to keep it short):
i686-elf-gcc $(CFLAGS) $(WARNINGS) -c src/kernel.c -o src/kernel.o
i686-elf-gcc $(CFLAGS) $(WARNINGS) -c src/kernel.c -o src/screen/screen_basic.o
I don't really know what you need to see what's going wrong. So, the source files (all of them) are at https://github.com/m44rtn/vireo-kernel.
It may be nice to know that this is a rewrite of the project. In the previous 'version' I manually added the file names to the makefile, which worked perfectly, but isn't nice when you have to add new files or when you're moving them around. That makefile is on the master branch (which ISN'T the default branch anymore).
The make version is the newest:
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
So, I expected this to work absolutely great. I thought it would just compile all files in the list. Unfortunately, it didn't. I don't really know what's going wrong here.
It compiles kernel.c to both kernel.o and screen_basic.o. I, of course, had hoped it would compile kernel.c to kernel.o and screen_basic.c to screen_basic.o.
Later on, the two files get linked. However, because they are the same, the linker throws errors because everything is defined twice, which isn't ideal.
I've tried to solve it by compiling every C file in one go, but this gave some issues with linking the Assembly files with the C files (sometimes making it non GRUB multibootable, which is necessarry to have, in my case).
I do not know what's wrong with the makefile for it to behave like this.
All the solutions from stack overflow I've tried:
Some solutions involve throwing all files in the root directory and just using:
%.o: %.c
(..)
However, this project will have a lot of files. This makes having everything in the same directory very annoying, very fast. I think this didn't work as well, but I don't know if that's true or just my brain lying to me. Sorry.
I've heard something about 'static rules':
$(OBJFILES): %.o: %.c
(..)
This didn't work, however I may have used it wrong. I don't know.
I like the makefile to stay the same as much as possible, because it's a very convenient one (it just detects all files automatically).
I really hope I've provided enough information, and that this question wasn't already asked. If it has been, I'm sorry in advance.
If you need more information, please ask! :)
--EDIT--
I'm quite new to make in this way, although I've used it for five years. I've always used it wrong. It is possible that my makefile is very ugly or bad. And I did use an example to write the makefile.
Consider the rule...
$(OBJFILES): $(SRCFILES)
$(CC) $(CCFLAGS) -c $< -o $#
Let's say, for the sake of argument, that we have...
SRCFILES := a.c b.c
OBJFILES := a.o b.o
If you expand the rule manually it becomes...
a.o b.o: a.c b.c
$(CC) $(CCFLAGS) -c $< -o $#
I think (correct me if I'm wrong) you are under the impression that this will be interpreted by make as a.o depends on a.c and, separately, b.o depends on b.c. That's not the case. What it actually states is that both of a.o and b.o depend on both of a.c and b.c.
So, when make tries to update the target a.o it sees the full prerequisite list of a.c and b.c and assigns the first of these, a.c, to the builtin variable $<. That's why you always see the first source file in $(SRCFILES) being compiled.
The best way to solve this probably depends on how you intend to structure your source file hierarchy and object files but you might want to take a look at using one or more vpath directives.
The pattern rule does not put all objects in root directory, consider
CFILES := path/to/a.c b.c
OBJFILES := $(foreach f,$(CFILES),$(f:%.c=%.o))
all: $(OBJFILES)
%.o: %.c
$(CC) $(CCFLAGS) -c $< -o $#
Here is what you get:
cc -c path/to/a.c -o path/to/a.o
cc -c b.c -o b.o
The following is not a recommendation, but kind of an exercise in makefile programming.
If you have $(SRCFILES) and want to compile them one at a time, you can use:
define compile
$1.o: $1.c
$(CC) $(CCFLAGS) -c $$< -o $$#
endef
$(foreach src,$(SRCFILES),$(eval $(call compile, $(src:%.c=%))))
If the correspondence of lists of sources and objects is not by name, but only by placement in list, you can destroy the CFILES list
define compile
src := $(CFILES)
CFILES := $(wordlist 2, $(words $(CFILES)), $(CFILES))
$1: $(src)
$(CC) $(CCFLAGS) -c $$< -o $$#
endef
$(foreach obj,$(OBJFILES),$(eval $(call compile, $(obj))))
Or you may use a helper list, to keep CFILES unchanged:
helperlist := $(CFILES)
define compile
src := $(firstword $(helperlist))
helperlist := $(wordlist 2, $(words $(helperlist)), $(helperlist))
$1: $(src)
$(CC) $(CCFLAGS) -c $$< -o $$#
endef
$(foreach obj,$(OBJFILES),$(eval $(call compile, $(obj))))
I have a project that consists a single target that requires ~30 files of type A, that are handled by one pattern rule, and one different target of type B, that is independent of the other 30. Here's some pseudo-code to show what I have, in a very simplified form:
OBJECTS=obj/obj1.o obj/obj2.o ...
SPECIAL=special/specialobj.o
libMyLibrary.so: $(OBJECTS) $(SPECIAL)
g++ -shared -o $# $^
obj/%.o: src/%.cxx
g++ -c -fPIC -o $# $^
$(SPECIAL): special/mySpecialCode.cxx
g++ -c -fPIC -o $# $^ -DFANCY_FLAG
The makefile works fine, and the dependency resolution is flawless. However, somehow, make always decides to build $(SPECIAL) last. While this doesn't matter at all, as it works either way, the compilation of $(SPECIAL) takes significantly longer than anything else in my case, almost as long as all other objects combined. Hence, it would be nice if one could encourage make to start with compiling $(SPECIAL), so that it can be compiled in parallel along the others, which would cut compile time dramatically.
Is it possible to achieve such a thing?
PS: I can live with non-portable versions, as the code is a very dedicated piece of software that will only be compiled and run on a very particular set of machines, the setup of which I know pretty well at development time.
EDIT
Following up on a comment made me realise that in the case I have shown here, the problem can indeed be solved by simply swapping the order $(OBJECTS) $(SPECIAL) to $(SPECIAL) $(OBJECTS) - sometimes, it's that simple.
However, in my actual usecase, this does not work, so I have built an MWE that (1) actually works (with gnumake) and is (2) a little bit closer to how things actually work in my project.
Here, the file that takes so long to compile actually depends on a source file that is only generated in an additional step. You can observe the change in order by swapping the two lines following the comment.
If somebody can provide a way how to fix the behavior also for this case, that would be great!
If it's not possible to change the order to the desired one in this case, I will accept any answer that explains to some details why it's impossible.
OBJECTS=obj/obj1.o obj/obj2.o obj/obj3.o obj/obj4.o obj/obj5.o
SPECIAL=special/specialobj.o
SPECIALSRC=special/mySpecialCode.cxx
OBJ_DIR=obj
SPECIAL_DIR=special
all: libMyLibrary
$(SPECIAL_DIR):
mkdir -p $#
$(OBJ_DIR):
mkdir -p $#
libMyLibrary: $(SPECIAL) $(OBJECTS)
cat $^ > $#
obj/%.o: | $(OBJ_DIR)
#echo creating $#
#echo $# > $#
$(SPECIALSRC): | $(SPECIAL_DIR)
#echo special > $#
# swap the following two lines to observe the change in ordering
# $(SPECIAL): $(SPECIALSRC) | $(SPECIAL_DIR)
$(SPECIAL): | $(SPECIAL_DIR)
#echo starting special
#echo special > $#
#sleep 1s
#echo special done
clean:
rm -rf libMyLibrary $(OBJ_DIR) $(SPECIAL_DIR)
When I compile large programs (like gcc or clang for example) there is a chance that my computer will overheat, and be forced to shut down.
I would like to resume the make process from where I left off. The problem seems to be that there are half completed/written .o files that are floating around that cause the rest of the built to break (this is especially bad when I specify -j 8)
Is there an easy way to get around this problem (whithout doing a make clean or make distclean or the like)?
Using GNU Make 3.81
Along the same lines as Beta's comment, but more reliable and less confusing IMO, would be to change your compile rule so that you compile to a temporary file, then at the end rename it to the real file. So where you might have something like:
%.o : %.c
$(CC) $(CFLAGS) $(CPPFLAGS) -o $# -c $<
instead you would use something like:
%.o : %.c
$(CC) $(CFLAGS) $(CPPFLAGS) -o $#.tmp -c $< \
&& mv -f $#.tmp $#
You may or may not want to add a "rm -f $#" as well.
As to whether you can make this change "programmatically" or not, it all dependes on your makefile and how it's structured. If it's a well-formed makefile then you can just make these changes in a few places in a few implicit rules, as above.