I have a project that consists a single target that requires ~30 files of type A, that are handled by one pattern rule, and one different target of type B, that is independent of the other 30. Here's some pseudo-code to show what I have, in a very simplified form:
OBJECTS=obj/obj1.o obj/obj2.o ...
SPECIAL=special/specialobj.o
libMyLibrary.so: $(OBJECTS) $(SPECIAL)
g++ -shared -o $# $^
obj/%.o: src/%.cxx
g++ -c -fPIC -o $# $^
$(SPECIAL): special/mySpecialCode.cxx
g++ -c -fPIC -o $# $^ -DFANCY_FLAG
The makefile works fine, and the dependency resolution is flawless. However, somehow, make always decides to build $(SPECIAL) last. While this doesn't matter at all, as it works either way, the compilation of $(SPECIAL) takes significantly longer than anything else in my case, almost as long as all other objects combined. Hence, it would be nice if one could encourage make to start with compiling $(SPECIAL), so that it can be compiled in parallel along the others, which would cut compile time dramatically.
Is it possible to achieve such a thing?
PS: I can live with non-portable versions, as the code is a very dedicated piece of software that will only be compiled and run on a very particular set of machines, the setup of which I know pretty well at development time.
EDIT
Following up on a comment made me realise that in the case I have shown here, the problem can indeed be solved by simply swapping the order $(OBJECTS) $(SPECIAL) to $(SPECIAL) $(OBJECTS) - sometimes, it's that simple.
However, in my actual usecase, this does not work, so I have built an MWE that (1) actually works (with gnumake) and is (2) a little bit closer to how things actually work in my project.
Here, the file that takes so long to compile actually depends on a source file that is only generated in an additional step. You can observe the change in order by swapping the two lines following the comment.
If somebody can provide a way how to fix the behavior also for this case, that would be great!
If it's not possible to change the order to the desired one in this case, I will accept any answer that explains to some details why it's impossible.
OBJECTS=obj/obj1.o obj/obj2.o obj/obj3.o obj/obj4.o obj/obj5.o
SPECIAL=special/specialobj.o
SPECIALSRC=special/mySpecialCode.cxx
OBJ_DIR=obj
SPECIAL_DIR=special
all: libMyLibrary
$(SPECIAL_DIR):
mkdir -p $#
$(OBJ_DIR):
mkdir -p $#
libMyLibrary: $(SPECIAL) $(OBJECTS)
cat $^ > $#
obj/%.o: | $(OBJ_DIR)
#echo creating $#
#echo $# > $#
$(SPECIALSRC): | $(SPECIAL_DIR)
#echo special > $#
# swap the following two lines to observe the change in ordering
# $(SPECIAL): $(SPECIALSRC) | $(SPECIAL_DIR)
$(SPECIAL): | $(SPECIAL_DIR)
#echo starting special
#echo special > $#
#sleep 1s
#echo special done
clean:
rm -rf libMyLibrary $(OBJ_DIR) $(SPECIAL_DIR)
Related
I have a generic build system that I import as a Git submodule into my projects, each of which has a wrapper makefile that includes the build system's makefile. Suppose I have something like the following in the build system's makefile:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $<
And prerequisites like the following in the projects' makefiles:
$(BIN_DIR)/foo: $(FOO_OBJ_FILES:%=$(OBJ_DIR)/%)
While this technique works well for libraries, it obviously doesn't work well for executables, because each needs to be linked with a different set of libraries. So my question is how to achieve that without relying of GNU Make-specific features? With GNU Make, I could do something like this:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $< $((nodir $(basename $#))_LINK:%=-l %)
foo_LINK = mylib1 mylib2
$(BIN_DIR)/foo: $(FOO_OBJ_FILES:%=$(OBJ_DIR)/%)
One portable solution that I've thought of but would rather not use because it's a huge hack, is to have a .link file associated with each executable, containing one line like this: -l mylib1 -lmylib2 and then do something like this:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $< `cat $#.link`
So, all these things aside, I need to somehow magically pass the correct set of libraries to the pattern rule or to have a variable expand to the correct set of libraries when the pattern rule gets executed.
Any ideas? Thanks.
PS: I know this seems like more trouble than it's worth but I've simplified things for the purpose of this question. In reality, it saves me from dealing with some pepetition, not just saving me one line.
Okay, I've come up with an answer:
$(BIN_DIR): | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $< $($(F)_LINK:%=-l %)
foo_LINK = mylib1 mylib2
Unfortunately, while there is a proposal for adding nested/computed variables into POSIX make for Issue 8 of the specification, it seems to have been in limbo for the past 10 years or so.
When I compile large programs (like gcc or clang for example) there is a chance that my computer will overheat, and be forced to shut down.
I would like to resume the make process from where I left off. The problem seems to be that there are half completed/written .o files that are floating around that cause the rest of the built to break (this is especially bad when I specify -j 8)
Is there an easy way to get around this problem (whithout doing a make clean or make distclean or the like)?
Using GNU Make 3.81
Along the same lines as Beta's comment, but more reliable and less confusing IMO, would be to change your compile rule so that you compile to a temporary file, then at the end rename it to the real file. So where you might have something like:
%.o : %.c
$(CC) $(CFLAGS) $(CPPFLAGS) -o $# -c $<
instead you would use something like:
%.o : %.c
$(CC) $(CFLAGS) $(CPPFLAGS) -o $#.tmp -c $< \
&& mv -f $#.tmp $#
You may or may not want to add a "rm -f $#" as well.
As to whether you can make this change "programmatically" or not, it all dependes on your makefile and how it's structured. If it's a well-formed makefile then you can just make these changes in a few places in a few implicit rules, as above.
I'm creating a Makefile for a project. I have the following structure of Makefiles:
./Makefile
./classification/Makefile
./misc/Makefile
./APP/Makefile
./qr/libs/Makefile
I'm doing a recursive make. In each directory, I have a Makefile which generates a Shared Library. So, in the ./classification folder, I'm going to generate classification.so, and thus to the other directories. In general, they have the following structure:
include ../standard_defs.mk
xCFLAGS=$(CFLAGS) -fPIC
SOURCES=help.cpp
OBJECTS=$(SOURCES:.cpp=.o)
EXECUTABLE=misc.so
xxDET=detection/$(EXECUTABLE)
export xxDET;
all: $(SOURCES) $(EXECUTABLE)
$(EXECUTABLE): $(OBJECTS)
$(CC) $(OBJECTS) -shared -o $#
.cpp.o:
$(CC) $(xCFLAGS) $< -c
clean:
rm -f $(OBJECTS) $(EXECUTABLE)
And the main Makefile (./Makefile) has the following structure:
CFLAGS=`pkg-config opencv --cflags`
LDFLAGS=`pkg-config opencv --libs`
include standard_defs.mk
SOURCES=DataFormatDetResult.cpp InputDataFiles.cpp InputImage.cpp \
InputManager.cpp main.cpp maths.cpp misc.cpp
OBJECTS=$(SOURCES:.cpp=.o)
EXECUTABLE=featureExtractor
all: $(LIBS) $(SOURCES) $(EXECUTABLE)
$(EXECUTABLE): $(OBJECTS)
make -C misc
make -C qr/libs
make -C classification
make -C APP
$(CC) $(CFLAGS) $(OBJECTS) -o $# $(LDFLAGS) misc/misc.so qr/libs/ap.so classification/classification.so APP/app.so
.cpp.o:
$(CC) $(CFLAGS) $< -c
clean:
make -C misc clean
make -C qr/libs clean
make -C APP clean
make -C classification clean
rm -f *.o $(EXECUTABLE)
Errors
When I try to compile the main Makefile, I got the following linking error:
classification/classification.so: undefined reference to `Help::InsertHelpType(std::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
classification/classification.so: undefined reference to `Help::Help()'
collect2: ld returned 1 exit status
make: ** [featureExtractor] Erro 1
misc.so contains the class Help and it seems that classification.so does not find it in the final compiling. However, if I run make -C ./classification/, I don't get any errors.
Question 1
How can I solve this linkage problem?
Failed Solutions
I've tried to link misc.so to classification.so, doing this in ./classification/Makefile:
$(EXECUTABLE): $(OBJECTS)
$(CC) $(OBJECTS) -shared -o $# $(LDFLAGS) ../misc/misc.so
But it didn't work. All I've got was a new warning
/usr/bin/ld: warning: ../misc/misc.so, needed by classification/classification.so, not found (try using -rpath or -rpath-link)
Question 2
Is there a better way to do create a Makefile to this project?
EDIT 1: If I run make -C ./classification/, it generates ./classification/classification.so without errors. Then, if I run make in ./, I get the same error.
EDIT 2: When I ran nm --format sysv misc.so | grep Help, I've got:
$ nm --format sysv misc.so | grep Help
HelpTypes |0000000000205120| D | OBJECT|0000000000000038| |.data
_ZN11HelpControl9PrintHelpEv|0000000000002180| T | FUNC|000000000000082e| |.text
_ZN11HelpControlC1Ev|00000000000029b0| T | FUNC|0000000000000187| |.text
_ZN11HelpControlC2Ev|00000000000029b0| T | FUNC|0000000000000187| |.text
_ZN4Help14InsertHelpTypeESs|0000000000001ac0| T | FUNC|0000000000000126| |.text
_ZN4Help9PrintHelpEi|0000000000001bf0| T | FUNC|000000000000020d| |.text
_ZN4HelpC1Ev |0000000000001e00| T | FUNC|000000000000021a| |.text
_ZN4HelpC2Ev |0000000000001e00| T | FUNC|000000000000021a| |.text
_ZN4HelpD1Ev |0000000000003090| W | FUNC|000000000000039b| |.text
_ZN4HelpD2Ev |0000000000003090| W | FUNC|000000000000039b| |.text
_ZNSt8_Rb_treeISsSt4pairIKSs4HelpESt10_Select1stIS3_ESt4lessISsESaIS3_EE8_M_eraseEPSt13_Rb_tree_nodeIS3_E|0000000000003430| W | FUNC|0000000000000526| |.text
Try to get rid of the recursive Makefiles. Although they are somewhat common, recursive make is inherently broken. The main reason, that they are common is because that is what is build when you have automake and autoconf. However autoconf and automake both go to a great length to get the recursive make structure right and I have yet to seem anyone who get's a recursive make structure right without these tools.
There are only a few uses where recursive make is not inherently broken, for example in the way Cmake uses recursive makefiles. But again, these makefiles are build by automated tool, so this too is hard to get right manually.
EDIT: Here is a short summary of the article.
The main problem with recursive makefiles is, that it keeps make from building the full dependency graph, which it needs for building stuff in the right order. Recursive make was originally meant for cases where you would need to build multiple projects at once, without any dependencies between them. As long as there is a dependency across the recursive structure, make will make it very hard to fix the order. In the best case one then is left with a system which can be build, but which break when one tries to rebuild it after editing some of the files. I've seen a lot of cases, where recursive make messed up, so that one part of the application was linked against the libraries compiled from sources before the edit, and another part was linked against libraries compiled from sources after the edit. This leads to sudden breakage of the application, which will miraculously appear after a make clean && make.
In other cases recursive make can mess up the build altogether. This is the case, when the targets are executed in the absolutely wrong order to begin with. This means an ordinary build is not possible. Your example seems like one of those cases. Although I haven't looked at it too closely, it seems that make is not building all needed libraries correctly, because it does not fully know the dependencies.
In almost all cases a parallel make is completely rendered impossible when a recursive make is used. At least I have yet to see a manual recursive make structure which can reliably used with a make -j X.
There are two solutions:
Have the makefiles constructed by one of the tools, such as autotools or cmake. However this requires learning one more tool. Also the usability of these tools is highly debated (at least for autotools).
Get rid of the recursive structure by providing one makefile from which make can derive the full dependency graph. I've seen some people do this actually in a single file even with multiple libraries and automatic source detection, but I wouldn't so I cannot recommend that. Better yet is to have multiple files, one for each subdirectory and then use include to combine them into a large file at the root of the directory tree. This way make can only be called at the root of the tree, but it will always know the full set of dependencies. This way is also recommended by the article.
You haven't given us enough to reproduce the error, so this may take a couple of iterations.
In misc/, we need a way to test the Help class. If you don't have one already, write some simple code to that purpose in misc/:
//test_help.cpp
#include "help.h"
int main()
{
Help H;
return(0);
}
Try it:
make test_help.o help.o
g++ test_help.o help.o -o test_help
./test_help
Then with the library:
make misc.so
g++ test_help.o misc.so -o test_help
./test_help
Then move test_help.cpp to the upper directory and try from there:
make test_help.o
g++ test_help.o misc/misc.so -o test_help
./test_help
Then add a rule to the main Makefile:
test_help: test_help.o
make -C misc
$(CC) $(CFLAGS) $< -o $# misc/misc.so
./$#
and try make clean ; make test_help.
I'm sure this is a totally normal thing to do, but I can't figure out how to get make to do this.
I have a compiler that generates make dependencies of the usual form:
M/A.o : M/A.hs
M/B.o : M/A.o
So I write a rule to compile %.hs into %.o, add a rule to link the binary, include the dependencies file, and all is well. But I want to have several binary targets with different flags. E.g. I want build/test built with -DTESTING and build/profile built with -prof. So I need to keep the .o files in a separate tree, where they will be compiled with special flags.
The straightforward way I can think of would be to have dependencies that look something like this:
build/test/M/A.o : M/A.hs
build/test/M/B.o : build/test/M/A.o
build/profile/M/A.o : M/A.hs
... etc.
And then rules so that %.hs to build/test/%.o compiles with -DTESTING, etc. I think this would work, but it's clumsy, means preprocessing the deps file to add all that build/whatever/ prefix stuff, and would multiply its size by however many kinds of builds.
VPATH appears to be designed for this sort of thing and my idea was that I could set the VPATH and compiler flags depending on the target, and it almost works, but:
%.o: %.hs
#mkdir -p build/M
cp $< build/$#
VPATH = build
main: M/A.o M/B.o
cat $^ >$#
M/A.o : M/A.hs
M/B.o : M/B.hs
The first time the main target wants to run 'cat M/A.o M/B.o >main' which seems contrary to the gnu make documentation that says $^ should include the include the VPATH directory in which the dependency was found. Curiously, if I remove 'main' and make again, this time it uses the correct path. This is GNU make, 3.81.
What's going on here? Is there a better way to build with different flags? VPATH seems like a clumsy tool, surely there is a better way?
Make is working correctly. It tries cat M/A.o M/B.o >main the first time because it can't find the prerequisites it needs, but it knows a rule for M/A.o' andM/B.o(<em>not</em>build/M/A.o' and build/M/B.o) and expects that that is what the rule will produce. If you remove main and try again, it will find build/M/A.o' andbuild/M/B.o` via VPATH.
Let's modify this makefile in stages. First we change the VPATH so that it can find the .hs files (Make is good at using things there to build things here, not vise-versa, and that's what VPATH is good for), and change the rules slightly:
build/%.o: %.hs
cp $< $#
VPATH = M
main: build/A.o build/B.o
cat $^ > $#
Now for the different object directories.
build/test/%.o build/project/%.o: %.hs
cp $< $#
VPATH = M
test: build/test/A.o build/test/B.o
cat $^ > $#
project: build/project/A.o build/project/B.o
cat $^ > $#
Then we simplify those last two rules, so that it's easy to add more object files and binary targets:
OBJECTS = A.o B.o
test: $(addprefix build/test/,$(OBJECTS))
project: $(addprefix build/project/,$(OBJECTS))
test project:
cat $^ > $#
Now for the different compiler flags:
build/test/%.o: FLAGS += test_flags
build/project/%.o: FLAGS += proj_flags
build/test/%.o build/project/%.o: %.hs
#echo building $# from $^ using flags $(FLAGS)
cp $< $#
Finally the dependencies. This is a little tricky. Suppose you want the dependency B.o : A.hs to apply to however many object you have. This is one approach:
OBJECT_PATHS = build/test/ build/project/
# The following is from the included file generated by the compiler
$(addsuffix B.o,$(OBJECT_PATHS)) : A.hs
To generate lines like that, I'd pipe the raw lines (e.g. B.o: A.hs) through sed 's/\(.*\):\(.*\)/\1:\2/', and note that if you want to put this in a makefile command, don't forget to double the $ signs to preserve them for the shell.
I know that's a lot to absorb. Take it one step at a time and let us know how it works out.
If you haven't solved your problem by now or are experiencing further problems, best give the autotools (automake and autoconf) a chance. They'll quickly build you a Makefile that supports more configurable and flexible out-of-tree builds.
How can I run a script, which must execute before all other makefile commands? And it will be nice (but not mandatory) to the script is not executed if there is nothing to build.
I've searched SO and Google, but can't find anything.
I have this workaround:
# myscript.bat output is empty
CHEAT_ARGUMENT = (shell myscript.bat)
CFLAGS += -DCHEAT_ARGUMENT=$(CHEAT_ARGUMENT)
AFLAGS += -DCHEAT_ARGUMENT=$(CHEAT_ARGUMENT)
But it's very ugly. Is there other way to run "pre-build step" in makefile?
I propose two solutions. The first mimics what NetBeans IDE generates:
CC=gcc
.PHONY: all clean
all: post-build
pre-build:
#echo PRE
post-build: main-build
#echo POST
main-build: pre-build
#$(MAKE) --no-print-directory target
target: $(OBJS)
$(CC) -o $# $(OBJS)
clean:
rm -f $(OBJS) target
The second one is inpired by what Eclipse IDE generates:
CC=gcc
.PHONY: all clean
.SECONDARY: main-build
all: pre-build main-build
pre-build:
#echo PRE
post-build:
#echo POST
main-build: target
target: $(OBJS)
$(CC) -o $# $(OBJS)
#$(MAKE) --no-print-directory post-build
clean:
rm -f $(OBJS) target
Note that in the first one, pre and post builds are always called regardless of whether the main build is determined to be up to date or not.
In the second one, the post-build step is not executed if the state of the main build is up to date. While the pre-build step is always executed in both.
Depending on your make version, something like the following should at least avoid running dozens of times if CFLAGS and AFLAGS are evaluated dozens of times:
CHEAT_ARG := $(shell myscript)
Note the colon.
This runs exactly once. Never more than once, but also never less than once. Choose your own tradeoffs.
You could add a special target to your Makefile and have all your build rules depend on that:
run-script:
myscript
.o.c: run-script
$(CC) $(CFLAGS) -o $# $<
.o.S: run-script
$(AS) $(AFLAGS) -o $# $<
Depending on what your script actually does, putting it to run once in a stage before the Makefile (configure stage in autoconf terms) could make even more sense (and be less work).
What you are proposing seems a bit "un-make-like". Why not just run the command in whatever makefile target you need it to go before?
Example, if you need it to run before linking foo:
foo: ${OBJS}
my-command-goes-here
${CC} -o $# ${OBJS} ${LIBS}
Thank you for answers. ndim helped me much, asveikau. The final file is one binary executable, so I can use now something like this:
run-script:
myscript
$(AXF_FILE): run-script $(OBJ_DIRS) $(OBJ_FILES)
$(LINK) #......
It will run myscript once. {AXF_FILE} value depends on myscript and I must run it before. And in this case myscript runs always, not only when rebuild is needed.
After, The Simplest Answer came to my mind:
all: run-script $(AXF_FILE)
That's all ;) (Of course, any target can be used instead of "all")
Edit: this method execute script after $(AXF_FILE) is calculated too. So it's possible to get wrong value of AXF_FILE.
Now only the first answer by ndim works as I need.