Recursive Makefile with Shared Libs - makefile

I'm creating a Makefile for a project. I have the following structure of Makefiles:
./Makefile
./classification/Makefile
./misc/Makefile
./APP/Makefile
./qr/libs/Makefile
I'm doing a recursive make. In each directory, I have a Makefile which generates a Shared Library. So, in the ./classification folder, I'm going to generate classification.so, and thus to the other directories. In general, they have the following structure:
include ../standard_defs.mk
xCFLAGS=$(CFLAGS) -fPIC
SOURCES=help.cpp
OBJECTS=$(SOURCES:.cpp=.o)
EXECUTABLE=misc.so
xxDET=detection/$(EXECUTABLE)
export xxDET;
all: $(SOURCES) $(EXECUTABLE)
$(EXECUTABLE): $(OBJECTS)
$(CC) $(OBJECTS) -shared -o $#
.cpp.o:
$(CC) $(xCFLAGS) $< -c
clean:
rm -f $(OBJECTS) $(EXECUTABLE)
And the main Makefile (./Makefile) has the following structure:
CFLAGS=`pkg-config opencv --cflags`
LDFLAGS=`pkg-config opencv --libs`
include standard_defs.mk
SOURCES=DataFormatDetResult.cpp InputDataFiles.cpp InputImage.cpp \
InputManager.cpp main.cpp maths.cpp misc.cpp
OBJECTS=$(SOURCES:.cpp=.o)
EXECUTABLE=featureExtractor
all: $(LIBS) $(SOURCES) $(EXECUTABLE)
$(EXECUTABLE): $(OBJECTS)
make -C misc
make -C qr/libs
make -C classification
make -C APP
$(CC) $(CFLAGS) $(OBJECTS) -o $# $(LDFLAGS) misc/misc.so qr/libs/ap.so classification/classification.so APP/app.so
.cpp.o:
$(CC) $(CFLAGS) $< -c
clean:
make -C misc clean
make -C qr/libs clean
make -C APP clean
make -C classification clean
rm -f *.o $(EXECUTABLE)
Errors
When I try to compile the main Makefile, I got the following linking error:
classification/classification.so: undefined reference to `Help::InsertHelpType(std::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
classification/classification.so: undefined reference to `Help::Help()'
collect2: ld returned 1 exit status
make: ** [featureExtractor] Erro 1
misc.so contains the class Help and it seems that classification.so does not find it in the final compiling. However, if I run make -C ./classification/, I don't get any errors.
Question 1
How can I solve this linkage problem?
Failed Solutions
I've tried to link misc.so to classification.so, doing this in ./classification/Makefile:
$(EXECUTABLE): $(OBJECTS)
$(CC) $(OBJECTS) -shared -o $# $(LDFLAGS) ../misc/misc.so
But it didn't work. All I've got was a new warning
/usr/bin/ld: warning: ../misc/misc.so, needed by classification/classification.so, not found (try using -rpath or -rpath-link)
Question 2
Is there a better way to do create a Makefile to this project?
EDIT 1: If I run make -C ./classification/, it generates ./classification/classification.so without errors. Then, if I run make in ./, I get the same error.
EDIT 2: When I ran nm --format sysv misc.so | grep Help, I've got:
$ nm --format sysv misc.so | grep Help
HelpTypes |0000000000205120| D | OBJECT|0000000000000038| |.data
_ZN11HelpControl9PrintHelpEv|0000000000002180| T | FUNC|000000000000082e| |.text
_ZN11HelpControlC1Ev|00000000000029b0| T | FUNC|0000000000000187| |.text
_ZN11HelpControlC2Ev|00000000000029b0| T | FUNC|0000000000000187| |.text
_ZN4Help14InsertHelpTypeESs|0000000000001ac0| T | FUNC|0000000000000126| |.text
_ZN4Help9PrintHelpEi|0000000000001bf0| T | FUNC|000000000000020d| |.text
_ZN4HelpC1Ev |0000000000001e00| T | FUNC|000000000000021a| |.text
_ZN4HelpC2Ev |0000000000001e00| T | FUNC|000000000000021a| |.text
_ZN4HelpD1Ev |0000000000003090| W | FUNC|000000000000039b| |.text
_ZN4HelpD2Ev |0000000000003090| W | FUNC|000000000000039b| |.text
_ZNSt8_Rb_treeISsSt4pairIKSs4HelpESt10_Select1stIS3_ESt4lessISsESaIS3_EE8_M_eraseEPSt13_Rb_tree_nodeIS3_E|0000000000003430| W | FUNC|0000000000000526| |.text

Try to get rid of the recursive Makefiles. Although they are somewhat common, recursive make is inherently broken. The main reason, that they are common is because that is what is build when you have automake and autoconf. However autoconf and automake both go to a great length to get the recursive make structure right and I have yet to seem anyone who get's a recursive make structure right without these tools.
There are only a few uses where recursive make is not inherently broken, for example in the way Cmake uses recursive makefiles. But again, these makefiles are build by automated tool, so this too is hard to get right manually.
EDIT: Here is a short summary of the article.
The main problem with recursive makefiles is, that it keeps make from building the full dependency graph, which it needs for building stuff in the right order. Recursive make was originally meant for cases where you would need to build multiple projects at once, without any dependencies between them. As long as there is a dependency across the recursive structure, make will make it very hard to fix the order. In the best case one then is left with a system which can be build, but which break when one tries to rebuild it after editing some of the files. I've seen a lot of cases, where recursive make messed up, so that one part of the application was linked against the libraries compiled from sources before the edit, and another part was linked against libraries compiled from sources after the edit. This leads to sudden breakage of the application, which will miraculously appear after a make clean && make.
In other cases recursive make can mess up the build altogether. This is the case, when the targets are executed in the absolutely wrong order to begin with. This means an ordinary build is not possible. Your example seems like one of those cases. Although I haven't looked at it too closely, it seems that make is not building all needed libraries correctly, because it does not fully know the dependencies.
In almost all cases a parallel make is completely rendered impossible when a recursive make is used. At least I have yet to see a manual recursive make structure which can reliably used with a make -j X.
There are two solutions:
Have the makefiles constructed by one of the tools, such as autotools or cmake. However this requires learning one more tool. Also the usability of these tools is highly debated (at least for autotools).
Get rid of the recursive structure by providing one makefile from which make can derive the full dependency graph. I've seen some people do this actually in a single file even with multiple libraries and automatic source detection, but I wouldn't so I cannot recommend that. Better yet is to have multiple files, one for each subdirectory and then use include to combine them into a large file at the root of the directory tree. This way make can only be called at the root of the tree, but it will always know the full set of dependencies. This way is also recommended by the article.

You haven't given us enough to reproduce the error, so this may take a couple of iterations.
In misc/, we need a way to test the Help class. If you don't have one already, write some simple code to that purpose in misc/:
//test_help.cpp
#include "help.h"
int main()
{
Help H;
return(0);
}
Try it:
make test_help.o help.o
g++ test_help.o help.o -o test_help
./test_help
Then with the library:
make misc.so
g++ test_help.o misc.so -o test_help
./test_help
Then move test_help.cpp to the upper directory and try from there:
make test_help.o
g++ test_help.o misc/misc.so -o test_help
./test_help
Then add a rule to the main Makefile:
test_help: test_help.o
make -C misc
$(CC) $(CFLAGS) $< -o $# misc/misc.so
./$#
and try make clean ; make test_help.

Related

Makefile does not recognize pattern rule

I'm really struggling in understanding why the following makefile won't work:
all: buildFolders main.out
mv main.out build/
-echo "File compiled"
buildFolders:
mkdir -p build src
cp *.c src/
%.s: %.c
gcc -S $< -o $#
%.out: src/%.s
gcc $< -o $#
It is executed in a folder containing only the makefile and a main.c file. It should build the src and build folder, copy the main.c in the src folder and then start compiling the main.out. Unfortunately it throws the error "no rule to make target 'main.out'". Since I have the %.out that matches 'main.out' I don't see why it gives me that error. Instead it should look for the src/main.s file, create it and then use it to generate the main.out.
What am I doing wrong? Thanks
You have a number of problems.
First, listing prerequisites in order doesn't create a dependency relationship. If, for example, you ever wanted to enable parallel builds then this:
all: buildFolders main.out
doesn't force the buildFolders target to be built before main.out. These two targets both must be built before all but this doesn't tell make that there's any relationship between buildFolders and main.out. If buildFolders must be completed before main.out can be built then main.out must list buildFolders as a prerequisite.
Second, you haven't told make how to build a file src/main.c. It's built as a side-effect of the buildFolders target, but make can't know that. You need to explain to make that this file can exist. I recommend adding a rule:
src/%.c: %.c
mkdir -p src
cp $< $#
and removing the buildFolders target altogether.
However, I really question why you want to do this anyway. What's the point of copying the source files in the current directory to some sub-directory to build them? It's dangerous and confusing to have multiple copies of source files lying around because they can get out of sync with each other, then you're building older versions and you spend hours trying to understand why something doesn't work. It's a really bad idea.

Order of libraries and source files from makefile

I should start by saying I'm a bit of a newbie when it comes to gcc and makefiles.
On an Ubuntu machine that I've recently started using, I find that when running gcc, the order in which I put the source files and the libraries/headers makes a difference. On another machine I could do:
gcc -I../include -L../lib myProgram.c -o myProgram
But on the new machine, this will not link the libraries, and I must do:
gcc myProgram.c -o myProgram -I../include -L../lib
Now, I have the following makefile:
SHELL = /bin/sh
CC = gcc -O3
CFLAGS = -I../include
LDFLAGS = -L../lib
PROGS = myProgram
all: $(PROGS)
$(all): $(PROGS).o
$(CC) -o $# $#.o $(LIBS) $(CFLAGS) $(LDFLAGS)
rm -f $#.o
clean:
rm -f *.o $(PROGS)
But when I do "make", the actual gcc command that it runs has the libraries and source files in the wrong order. My question is: what do I need to do in the makefile to force the "-L../libs" option to come after the source files, so that the libraries will link properly?
I've also tried including the "-Wl,--no-as-needed" option, as I thought that an --as-needed flag might be the reason that the order matters in the first place, but this didn't appear to change anything (i.e. it still fails to link the libraries unless "-L../libs" comes after the source files).
The problem was that you thought you were using that rule, but you weren't. You never defined a variable named all, so the target of the second rule actually expanded to nothing. When you commanded Make to build myProgram, Make found no suitable rule in this makefile. Make has a toolbox of implicit rules it can fall back on in such cases; it wanted to build myProgram, it saw a file named myProgram.c, and one of its rules looks something like this:
%: %.c
$(CC) $(LDFLAGS) $^ -o $#
There you have it, linker flags before sources.
You can write your own pattern rule which Make will use instead:
%: %.o
$(CC) -o $# $^ $(LIBS) $(CFLAGS) $(LDFLAGS)
(Note that this builds myProgram from myProgram.o, and lets Make figure out how to build myProgram.o.)
If your executable is to be built from several object files, add a rule like this:
myProgram: other.o yetAnother.o
If you like you can have one more rule (the first) to tell Make what you want built:
all: myProgram myOtherProgram friendsProgram
(A final note: we've all had tight work deadlines. Asking for help once can be faster than learning the tools, but learning the tools is faster than asking for help N times. Determining the value of N is up to you.)

order rule execution in parallel make without introducing dependencies

I have a project that consists a single target that requires ~30 files of type A, that are handled by one pattern rule, and one different target of type B, that is independent of the other 30. Here's some pseudo-code to show what I have, in a very simplified form:
OBJECTS=obj/obj1.o obj/obj2.o ...
SPECIAL=special/specialobj.o
libMyLibrary.so: $(OBJECTS) $(SPECIAL)
g++ -shared -o $# $^
obj/%.o: src/%.cxx
g++ -c -fPIC -o $# $^
$(SPECIAL): special/mySpecialCode.cxx
g++ -c -fPIC -o $# $^ -DFANCY_FLAG
The makefile works fine, and the dependency resolution is flawless. However, somehow, make always decides to build $(SPECIAL) last. While this doesn't matter at all, as it works either way, the compilation of $(SPECIAL) takes significantly longer than anything else in my case, almost as long as all other objects combined. Hence, it would be nice if one could encourage make to start with compiling $(SPECIAL), so that it can be compiled in parallel along the others, which would cut compile time dramatically.
Is it possible to achieve such a thing?
PS: I can live with non-portable versions, as the code is a very dedicated piece of software that will only be compiled and run on a very particular set of machines, the setup of which I know pretty well at development time.
EDIT
Following up on a comment made me realise that in the case I have shown here, the problem can indeed be solved by simply swapping the order $(OBJECTS) $(SPECIAL) to $(SPECIAL) $(OBJECTS) - sometimes, it's that simple.
However, in my actual usecase, this does not work, so I have built an MWE that (1) actually works (with gnumake) and is (2) a little bit closer to how things actually work in my project.
Here, the file that takes so long to compile actually depends on a source file that is only generated in an additional step. You can observe the change in order by swapping the two lines following the comment.
If somebody can provide a way how to fix the behavior also for this case, that would be great!
If it's not possible to change the order to the desired one in this case, I will accept any answer that explains to some details why it's impossible.
OBJECTS=obj/obj1.o obj/obj2.o obj/obj3.o obj/obj4.o obj/obj5.o
SPECIAL=special/specialobj.o
SPECIALSRC=special/mySpecialCode.cxx
OBJ_DIR=obj
SPECIAL_DIR=special
all: libMyLibrary
$(SPECIAL_DIR):
mkdir -p $#
$(OBJ_DIR):
mkdir -p $#
libMyLibrary: $(SPECIAL) $(OBJECTS)
cat $^ > $#
obj/%.o: | $(OBJ_DIR)
#echo creating $#
#echo $# > $#
$(SPECIALSRC): | $(SPECIAL_DIR)
#echo special > $#
# swap the following two lines to observe the change in ordering
# $(SPECIAL): $(SPECIALSRC) | $(SPECIAL_DIR)
$(SPECIAL): | $(SPECIAL_DIR)
#echo starting special
#echo special > $#
#sleep 1s
#echo special done
clean:
rm -rf libMyLibrary $(OBJ_DIR) $(SPECIAL_DIR)

Why doesn't my make file leave behind object files?

I am new to make files and I put this together with a bit of trial & error. This code is used to compile my c++ program.
My main.cpp file in the same folder as the makefile. I have a lib/ folder that contains the headers main depends on.
The following makefile results in a correct and complete compilation of my code. But I was expecting that I would find *.o objects left behind. (Note that I've tried to make both with and without the "clean" rule, and I get the same results both times.)
#
# Makefile
#
CXX = g++
CCFLAGS = -O3 -I/sw/include -L/sw/lib
## /sw/include and /sw/lib contain dependencies for files in my lib/
LDFLAGS = -lpng
OPTS = $(CCFLAGS) $(LDFLAGS)
SOURCES = $(wildcard lib/*.cpp) main.cpp
OBJECTS = $(SOURCES: .cpp = .o)
TARGET = spirals
$(TARGET): $(OBJECTS)
$(CXX) $(OPTS) $^ -o $#
.PHONY: depend
depend:
g++ -MM $(SOURCES) > depend
## generate dependencies list
include depend
.PHONY: clean
clean:
rm -f *.o lib/*.o $(TARGET)
Also, in case it matters, I'm on MacOSX and my program is designed in xcode. (I know that xcode has its own build flow, but I'm designing a command-line program for a linux system and I'd like to test compilation & linking in a bash environment instead of only going through xcode.)
Questions:
Am I correct to expect makefiles to produce *.o files that stick around once the main target has been created?
If so, why doesn't my makefile do this?
If you observe what command your $(TARGET) rule causes to be run:
g++ -O3 -I/sw/include -L/sw/lib -lpng lib/bar.cpp lib/foo.cpp main.cpp -o spirals
you'll see that $(OBJECTS) in fact contains *.cpp files, and there are no *.o files sticking around because you haven't asked for any.
The problem is here:
OBJECTS = $(SOURCES:.cpp=.o)
In your GNU makefile as written, this substitution reference is written with excess spaces, so never matches anything and $(OBJECTS) ends up the same as $(SOURCES). Rewrite it as above and it'll do what you expect.
(Other notes: -lpng needs to go at the end of the link command to work in general, so you should introduce another make variable (traditionally called $(LDLIBS)) so as to arrange that. Especially as someone new to makefiles, you would do better to spell out your dependencies explicitly rather than playing games with $(wildcard) and a computed $(OBJECTS). -I options are needed during compilation while -L options are used during linking, so it would be good to arrange separate $(CXXFLAGS)/$(LDFLAGS) variables used in separate rules so they are only added when required.)

separate builds in separate directories

I'm sure this is a totally normal thing to do, but I can't figure out how to get make to do this.
I have a compiler that generates make dependencies of the usual form:
M/A.o : M/A.hs
M/B.o : M/A.o
So I write a rule to compile %.hs into %.o, add a rule to link the binary, include the dependencies file, and all is well. But I want to have several binary targets with different flags. E.g. I want build/test built with -DTESTING and build/profile built with -prof. So I need to keep the .o files in a separate tree, where they will be compiled with special flags.
The straightforward way I can think of would be to have dependencies that look something like this:
build/test/M/A.o : M/A.hs
build/test/M/B.o : build/test/M/A.o
build/profile/M/A.o : M/A.hs
... etc.
And then rules so that %.hs to build/test/%.o compiles with -DTESTING, etc. I think this would work, but it's clumsy, means preprocessing the deps file to add all that build/whatever/ prefix stuff, and would multiply its size by however many kinds of builds.
VPATH appears to be designed for this sort of thing and my idea was that I could set the VPATH and compiler flags depending on the target, and it almost works, but:
%.o: %.hs
#mkdir -p build/M
cp $< build/$#
VPATH = build
main: M/A.o M/B.o
cat $^ >$#
M/A.o : M/A.hs
M/B.o : M/B.hs
The first time the main target wants to run 'cat M/A.o M/B.o >main' which seems contrary to the gnu make documentation that says $^ should include the include the VPATH directory in which the dependency was found. Curiously, if I remove 'main' and make again, this time it uses the correct path. This is GNU make, 3.81.
What's going on here? Is there a better way to build with different flags? VPATH seems like a clumsy tool, surely there is a better way?
Make is working correctly. It tries cat M/A.o M/B.o >main the first time because it can't find the prerequisites it needs, but it knows a rule for M/A.o' andM/B.o(<em>not</em>build/M/A.o' and build/M/B.o) and expects that that is what the rule will produce. If you remove main and try again, it will find build/M/A.o' andbuild/M/B.o` via VPATH.
Let's modify this makefile in stages. First we change the VPATH so that it can find the .hs files (Make is good at using things there to build things here, not vise-versa, and that's what VPATH is good for), and change the rules slightly:
build/%.o: %.hs
cp $< $#
VPATH = M
main: build/A.o build/B.o
cat $^ > $#
Now for the different object directories.
build/test/%.o build/project/%.o: %.hs
cp $< $#
VPATH = M
test: build/test/A.o build/test/B.o
cat $^ > $#
project: build/project/A.o build/project/B.o
cat $^ > $#
Then we simplify those last two rules, so that it's easy to add more object files and binary targets:
OBJECTS = A.o B.o
test: $(addprefix build/test/,$(OBJECTS))
project: $(addprefix build/project/,$(OBJECTS))
test project:
cat $^ > $#
Now for the different compiler flags:
build/test/%.o: FLAGS += test_flags
build/project/%.o: FLAGS += proj_flags
build/test/%.o build/project/%.o: %.hs
#echo building $# from $^ using flags $(FLAGS)
cp $< $#
Finally the dependencies. This is a little tricky. Suppose you want the dependency B.o : A.hs to apply to however many object you have. This is one approach:
OBJECT_PATHS = build/test/ build/project/
# The following is from the included file generated by the compiler
$(addsuffix B.o,$(OBJECT_PATHS)) : A.hs
To generate lines like that, I'd pipe the raw lines (e.g. B.o: A.hs) through sed 's/\(.*\):\(.*\)/\1:\2/', and note that if you want to put this in a makefile command, don't forget to double the $ signs to preserve them for the shell.
I know that's a lot to absorb. Take it one step at a time and let us know how it works out.
If you haven't solved your problem by now or are experiencing further problems, best give the autotools (automake and autoconf) a chance. They'll quickly build you a Makefile that supports more configurable and flexible out-of-tree builds.

Resources