Automatic dependency resolution using GNU Makefile - gcc

I'm writing a piece of software that utilizes a Makefile for compilation, originally I had a rule setup for each file however this proved to be too cumbersome whenever I added a new file. To try and automate the process I did some research and learnt that GCC/G++ can automatically build Makefile rules with the -M flag.
There are many examples of this being done with a simple directory structure however my ideal directory structure looks like this:
src/
kernel.hpp kernel.cpp
Types/
String.cpp
String.hpp
Drivers/IO-Ports/
CMOS.cpp
CMOS.hpp
...
build/
DEPS/
kernel.d
Types/String.d
...
OBJ/
kernel.o
Types/String.o
...
My current Makefile:
CCHOME=/home/dan/opt/cross/bin
CC=#$(CCHOME)/i586-elf-g++
CFLAGS=-ffreestanding -O2 -Wextra -Wall -fno-exceptions -fno-rtti
KernelName=CPlusKern
QEMU=qemu-system-x86_64 -monitor stdio
SrcDIR=src
SourceDIRS:=$(shell find $(SrcDIR) -type d)
SrcFILES=$(shell find $(SrcDIR) -type f -name *.cpp)
HdrFILES=$(shell find $(SrcDIR) -type f -name *.hpp)
DepDIR=$(BuildDIR)/DEPS
DepFILES0=$(subst $(SrcDIR), $(DepDIR),$(SrcFILES))
DepFILES=$(subst .cpp,.d,$(DepFILES0))
ObjDIR=$(BuildDIR)/OBJ
ObjDIRS=$(subst $(SrcDIR),$(ObjDIR),$(SourceDIRS))
ObjFILES0=$(subst $(SrcDIR), $(ObjDIR),$(SrcFILES))
ObjFILES=$(subst .cpp,.o,$(ObjFILES0))
BuildDIR=build
BuildDIRS=$(BuildDIR) $(ObjDIR) $(DepDIR)
all: assemble compile run
image: assemble compile build-image run-image
debug:
#echo "BuildDIRS: " $(BuildDIRS)
#echo "DepFiles: " $(DepFILES)
#echo "SrcFiles: " $(SrcFILES)
#echo "ObjFiles: " $(ObjFILES)
./src/kernel.o: ./src/kernel.cpp
#echo $(CC) $(CFLAGS) -MMD -MP -MF $< -o $(subst $(SrcDIR), $(ObjDIR),$#)
./src/kernel.cpp:
dir:
#echo "Making Build Dirs..."
#-mkdir -p $(BuildDIRS)
compile: dir $(ObjFILES)
# #echo "Compiling Source Files: " $(SrcFILES)
assemble: dir boot.o
#echo "Assembling Core Files..."
boot.o: $(SrcDIR)/boot.s
#$(CCHOME)/i586-elf-as $(SrcDIR)/boot.s -o $(ObjDIR)/boot.o
build: %.o
echo "Building Kernel Binary..."
#$(CC) -T linker.ld -o $(KernelName).bin -ffreestanding -O2 -nostdlib $(SrcFILES)-lgcc
build-image: build
#echo "Building Kernel Image..."
#cp $(KernelName).bin isodir/boot/$(KernelName).bin
#Scripts/MakeGrub.sh $(KernelName) isodir/boot/grub
grub-mkrescue -o $(KernelName).iso isodir
%.o: %.cpp Makefile
#echo "Building Object $#"
$(CC) $(CFLAGS) -MMD -MP -MF $(subst $(SrcDIR),$(DepDIR),$#) -o $(subst $(SrcDIR), $(ObjDIR),$#)
run:
#echo "Starting QEMU"
#$(QEMU) -kernel $(KernelName).bin
run-image:
#echo "Starting QEMU"
#$(QEMU) -bios OVMF.fd -cdrom $(KernelName).iso
clean:
#echo "Cleaning Build Directories..."
-#rm -R $(BuildDIR) ./isodir
-#$(RM) $(KernelName).bin $(KernelName).iso
I thought this might do the trick however make throws an error:
make: *** No rule to make target `build/OBJ/VGA.o', needed by `compile'. Stop.
I can't determine how to make the rule:
%.o: %.cpp Makefile
$(CC) $(CFLAGS) -MMD -MP -MF $(subst $(SrcDIR),$(DepDIR),$#) -o $(subst $(SrcDIR), $(ObjDIR),$#)
apply to every .cpp file. As far as I know wildcards cannot be used in rule definitions.
I'm not sure if this helps but the path/name of each source file is stored in the $(SrcFILES) variable.
Just to clarify, here is an expanded version of the above rule:
/home/dan/opt/cross/bin/i586-elf-g++ -ffreestanding -O2 -Wextra -Wall -fno-exceptions -fno-rtti -MMD -MP -MF src/kernel.cpp -o build/OBJ/kernel.o
And the generated dependency file for this instance:
kernel.o: src/kernel.cpp src/kernel.hpp src/Globals.hpp \
src/VGATerminal.hpp src/Types/String.hpp src/Types/../Globals.hpp \
src/VGA.hpp src/IO/Read.hpp src/IO/Write.hpp \
src/Drivers/IO-Ports/CMOS.hpp src/Drivers/IO-Ports/../../IO/Read.hpp \
src/Drivers/IO-Ports/../../IO/Write.hpp
This is my first post here so feedback on my question is appreciated :)
Hopefully I can get this out of the way and get back to developing my code.
EDIT:
The rule provided by #Beta worked without a problem, All of my Object files successfully build and are output in the right place. This rule even picked up build/OBJ/Drivers/IO-Ports/CMOS.o and build/OBJ/Drivers/PS2.o.
So now I can happily build all of the Objects individually if I pass the filename however I think I still need dependency resolution so that I don't have to write a rule for each file.

This may take a few iterations.
The first problem is that you have a target, build/OBJ/VGA.o and no corresponding rule. The rule
%.o: %.cpp Makefile
...
doesn't fit; it can build build/OBJ/VGA.o from build/OBJ/VGA.cpp, but there's no such source file.
So for a first step, if the source file is src/whatever/VGA.cpp, try this rule:
build/OBJ/%.o: src/whatever/%.cpp
...
and tell us the result.
EDIT:
Good, now try this:
build/OBJ/%.o: src/%.cpp
...

Related

Makefile can't find input files in specified directory

I am working with the following Makefile:
CFLAGS=-g -O2 -Wall -Wextra -Isrc -rdynamic -DNDEBUG $(OPTFLAGS)
LIBS=-ldl $(OPTLIBS)
PREFIX?=/usr/local
SOURCES=$(wildcard src/lcthw/*.c src/*.c)
OBJECTS=$(patsubst %.c,%.o,$(SOURCES))
TEST_SRC=$(wildcard tests/*_tests.c)
TESTS=$(patsubst %.c,%,$(TEST_SRC))
TARGET=build/libYOUR_LIBRARY.a
SO_TARGET=$(patsubst %.a,%.so,$(TARGET))
all: $(TARGET) $(SO_TARGET) tests
dev: CFLAGS=-g -Wall -Isrc -Wall -Wextra $(OPTFLAGS)
dev: all
$(TARGET): CFLAGS += -fPIC
$(TARGET): build $(OBJECTS)
ar rcs $# $(OBJECTS)
ranlib $#
$(SO_TARGET): $(TARGET) $(OBJECTS)
$(CC) -shared -o $# $(OBJECTS)
build:
#mkdir -p build
#mkdir -p bin
# The Unit Tests
.PHONY: tests
tests: CFLAGS += $(TARGET)
tests: $(TESTS)
sh ./tests/runtests.sh
valgrind:
VALGRIND="valgrind --log-file=/tmp/valgrind-%p.log" $(MAKE)
# The Cleaner
clean:
rm -rf build $(OBJECTS) $(TESTS)
rm -f tests/tests.log
find . -name "*.gc*" -exec rm {} \;
rm -rf `find . -name "*.dSYM" -print`
install: all
install -d $(DESTDIR)/$(PREFIX)/lib/
install $(TARGET) $(DESTDIR)/$(PREFIX)/lib/
And I'm having trouble with this line:
$(CC) -shared -o $# $(OBJECTS)
I am getting the error:
cc: fatal error: no input files
when I try and run any of the following commands: make, make all and make dev. Why can't my makefile find the .c files in the directories I specified?
You need to specify the full paths in this line:
SOURCES=$(wildcard src/lcthw/.c src/.c)
OTW this becomes a constraint on where you can run the make command from. Possibly what's happening is that the wildcards are not being expanded since the directory you are sitting in, $(CURDIR) , makes it hard for it to see these paths.
You should try printing these variables from inside your makefile, by doing:
$(info source=$(SOURCES))
And if your .obj files are placed in a different location than the sources, then you need to account for that by including that place in the $(OBJECTS) as well.

Makefile won't check if updated, and just compiles anyway

My makefile will not check if there has been any updates and just compiles if it has more than a single source file added in. It works fine with just a single source file.
It seems that it's any source file that's not listed as the first one will always be recompiled and linked.
SOURCES=myclass.cpp mylock.cpp
EXECUTABLE=locktest
LIBRARIES=-pthread
CFLAGS=-Wall
CXX=g++
DIR=host/
EXE=$(EXECUTABLE)
OBJECTS=$(SOURCES:%.cpp=$(DIR)%.o)
$(EXE): $(OBJECTS)
$(CXX) -o $# $(OBJECTS) $(LIBRARIES)
$(DIR)%.o: %.cpp $(DIR)
$(CXX) $(CFLAGS) -c $< -o $#
$(DIR):
#mkdir $(DIR)
clean:
#rm $(OBJECTS) $(EXE)
#rmdir $(DIR)
Output shows problem:
stud#pc:~/Desktop/Locktest$ make
g++ -Wall -c myclass.cpp -o host/myclass.o
g++ -Wall -c mylock.cpp -o host/mylock.o
g++ -o locktest host/myclass.o host/mylock.o -pthread
stud#pc:~/Desktop/Locktest$ make
g++ -Wall -c myclass.cpp -o host/myclass.o
g++ -o locktest host/myclass.o host/mylock.o -pthread
As #lijat points out, when you build an object in $(DIR), the operating system updates the modification time of the directory, so that in this rule:
$(DIR)%.o: %.cpp $(DIR)
...
the prerequisite $(DIR) will always appear to be newer than any target except the last target built.
If your version of Make is recent enough, you can get past this by making $(DIR) an order-only prerequisite:
$(DIR)%.o: %.cpp | $(DIR)
...
Does the filesystem update the modified time on the DIR directory when the compiler writes an .o file there
$(DIR)%.o: %.cpp $(DIR)
ensures that all .o files will be recompiled if anything updates the modified time of that directory.

What's the proper lib variable for a makefile?

Running into trouble with libraries in makefiles again. Every time I try to get back into C make gives me a pain with libs.
make -pf /dev/null says the correct vars should be LDLIBS and LOADLIBES but the following doesn't alter the run command at all:
LOADLIBES=testing
LDFLAGS=testing
LDLIBS=testing
Needless to say this gives me errors because the -L flags don't end up in the command. Anyone know what's going on?
Full makefile below (Derivitave of Z Shaw's makefile)
OPTLIBS=$(xml2-config --libs)
OPTFLAGS=$(xml2-config --cflags)
STD=c99
CFLAGS=-std=$(STD) -g -O2 -Wall -Wextra -Isrc -rdynamic -DNDEBUG $(OPTFLAGS)
LDLIBS=-ldl $(OPTLIBS)
PREFIX?=/usr/local
SOURCES=$(wildcard src/**/*.c src/*.c)
OBJECTS=$(patsubst %.c,%.o,$(SOURCES))
TEST_SRC=$(wildcard tests/*_tests.c)
TESTS=$(patsubst %.c,%,$(TEST_SRC))
TARGET=build/lib.a
SO_TARGET=$(patsubst %.a,%.so,$(TARGET))
# The Target Build
all: cls $(TARGET) $(SO_TARGET) tests
dev: CFLAGS=-std=$(STD) -g -Wall -Isrc -Wall -Wextra $(OPTFLAGS)
dev: all
$(TARGET): CFLAGS += -fPIC
$(TARGET): build $(OBJECTS)
ar rcs $# $(OBJECTS)
ranlib $#
$(SO_TARGET): $(TARGET) $(OBJECTS)
$(CC) -shared -o $# $(OBJECTS)
build:
#mkdir -p build
#mkdir -p bin
# The Unit Tests
$(TESTS): $(TARGET)
.PHONY: tests
tests: LDLIBS += $(TARGET)
tests: $(TESTS)
sh ./tests/runtests.sh
valgrind:
VALGRIND="valgrind --log-file=/tmp/valgrind-%p.log" $(MAKE)
# The Cleaner
clean: cls
rm -rf build $(OBJECTS) $(TESTS)
rm -f tests/tests.log
find . -name "*.gc*" -exec rm {} \;
rm -rf `find . -name "*.dSYM" -print`
# The Install
install: all
install -d $(DESTDIR)/$(PREFIX)/lib/
install $(TARGET) $(DESTDIR)/$(PREFIX)/lib/
# The Checker
BADFUNCS='[^_.>a-zA-Z0-9](str(n?cpy|n?cat|xfrm|n?dup|str|pbrk|tok|_)|stpn?cpy|a?sn?printf|byte_)'
check:
#echo Files with potentially dangerous functions.
#egrep $(BADFUNCS) $(SOURCES) || true
# Clear screen for unspammy terminals
cls:
ifdef TERM
clear
endif
You aren't using LDFLAGS, etc in your link command. Make that something along the lines of:
$(SO_TARGET): $(TARGET) $(OBJECTS)
$(CC) -shared $(LDFLAGS) -o $# $(OBJECTS) $(LDLIBS)
It tells the linker to link the dl library, which is located at /usr/lib/libdl.so. -l is the switch to add a library, dl is the name of it (without the lib prefix or .so extension).
This library includes functions for dynamically loading shared libraries.

How to use the include directive in a makefile for a specific target

I want to use the include directive only for a specific target. I do not want to run the other makefiles when the target is not needed because it means the makefiles are generated needlessly.
So is there a way to conditionally use the include directive, which is conditional on a target? Or somehow to make the include directive a prerequisite of a target.
Here's what I have so far:
# Flags
INCDIR = $(CURDIR)/include
CFLAGS = -Wall -Wno-overflow -Wno-uninitialized -pedantic -std=c99 -I$(INCDIR) -O3
LFLAGS = -flat_namespace -dynamiclib -undefined dynamic_lookup
# Directory names
# Set vpath search paths
vpath %.h include
vpath %.c src
vpath %.o build
vpath %.d build
# Get files for the core library
CORE_FILES = $(wildcard src/*.c)
CORE_OBJS = $(patsubst src/%.c, build/%.o, $(CORE_FILES))
CORE_DEPS = $(CORE_OBJS:.o=.d)
# Core library target linking
core : $(CORE_OBJS) | bin
$(CC) $(LFLAGS) -o bin/libcbitcoin.2.0.dylib $(CORE_OBJS)
# Include header prerequisites (How to do only for "core" target?)
include $(CORE_DEPS)
# Makefiles for header dependencies.
$(CORE_DEPS): build/%.d: src/%.c | build
rm -f $#; \
$(CC) -I$(INCDIR) -MM $< -MT '$(#:.d=.o) $#' > $#
# Objects depend on directory
$(CORE_OBS) : | build
# Create build directory
build:
mkdir build
# Create bin directory
bin:
mkdir bin
# Core Compilation
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
# Depencies require include/CBDependencies.h as a prerequisite
build/CBOpenSSLCrypto.o: include/CBDependencies.h
# Crypto library target linking
crypto : build/CBOpenSSLCrypto.o -lcrypto -lssl | bin
$(CC) $(LFLAGS) -o bin/libcbitcoin-crypto.2.0.dylib build/CBOpenSSLCrypto.o -lcrypto -lssl
# Crypto library compile
build/CBOpenSSLCrypto.o: dependencies/crypto/CBOpenSSLCrypto.c
$(CC) -c $(CFLAGS) $< -o $#
#Clean
clean:
rm -f $(CORE_OBJS) $(CORE_DEPS) build/CBOpenSSLCrypto.o
As you should be able to tell I do not need to include the ".d" files for "crypto" but I do for "core" (default goal).
Thank you for any help.
Make is not a procedural language, so treating it as one goes against the grain; your makefiles will be difficult to scale, and it can lead to subtle bugs.
There's a better way by Tom Tromey that's clean, efficient and scalable. The trick is to realize that you can build the dependency file in the same step as the object file. The dependencies simply tell Make when the object is due to be rebuilt; you don't need them when you first build the object, because Make knows that the object must be built. And if the dependencies change, that can only be because something in the source or the old dependencies has changed, so again Make knows that the object must be rebuilt. (This is not obvious, so it may take a little cogitation.)
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
$(CC) -MM -MF build/$*.d $<
-include build/*.d
There's one more hitch: if you alter the code so as to remove a dependency -- and also remove that file -- you won't be able to rebuild, because the old dependency list will still demand a file which can no longer be found. The sophisticated solution is to process the dependency file so as to make each prerequisite (e.g. header) a target in its own right, with no commands, so that it can be assumed to be rebuilt when needed:
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
$(CC) -MM -MF build/$*.d $<
#cp build/$*.d build/$*.P
#sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < build/$*.P >> build/$*.d;
#rm build/$*.P
A cruder method, but almost as foolproof, is to put in catch-all rules for headers and sources:
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
$(CC) -MM -MF build/$*.d $<
%.cc %.h:
EDIT:
To break down the new commands:
The -MM option tells gcc to produce a make rule for the object file, instead of preprocessing or compiling. The default is to send the rule to wherever it would send preprocessed output, which will usually be stdout.
The -MF option, used with -MM, specifies the output file. So -MM -MF build/$*.d will put the rule where we want it.
So the following two commands are (almost always) equivalent:
$(CC) -MM -MF build/$*.d $<
$(CC) -MM $< > build/$*.d
(I've left out the -I$(...) and the possibility of using the -MMD option, because both get a little complicated and are not really the point of the question.)
You can use MAKECMDGOALS.
ifeq (core,$(MAKECMDGOALS))
include $(CORE_DEPS)
endif
You could of course, use ifneq (,$(findstring core,$(MAKECMDGOALS))) if there was a possibility of more than one target.
Note: this is a 'quick and dirty' solution -- I agree with Beta that you shouldn't make this a common practice (this could get messy if you did it in a lot of makefiles...).
John
I can't help breaking the guidelines for what is a good answer.
My answer to the original question is in my opinion, no you cannot include rules that are dependant on the target -all rules are processed before targets are considered. This is a limitation of make (I guess). Then again, good point, there is MAKECMDGOALS, but is this not just a hack in the make utility itself?
The answer from Beta is reasonable and orthodox enough, but you can't call it clean even if it is the best that can be done. It won't work if make has not processed the particular target before and the appropriate build/*.d dependency file is not sitting there.

Making a better Makefile

so I learned what a Makefile was some time ago, created a template Makefile and all I do is copy and alter the same file for every program I'm doing. I changed it a few times, but it's still a very crude Makefile. How should I improve it? This is an example of my current version:
CC = g++
CFLAGS = -std=gnu++0x -m64 -O3 -Wall
IFLAGS = -I/usr/include/igraph
LFLAGS = -ligraph -lgsl -lgslcblas -lm
DFLAGS = -g -pg
# make all
all: run test
# make a fresh compilation from scratch
fresh: clean test
#makes the final executable binary
run: main.o foo1.o foo2.o
$(CC) $(CFLAGS) $(LFLAGS) $^ -o $#
#makes the test executable with debugging and profiling tags
test: test.o foo1.o foo2.o
$(CC) $(DFLAGS) $(CFLAGS) $(LFLAGS) $^ -o $#
#makes teste.o
teste.o: teste.cpp
$(CC) $(CFLAGS) $(IFLAGS) -c $^ -o $#
#makes main.o
main.o: main.cpp
$(CC) $(CFLAGS) $(IFLAGS) -c $^ -o $#
#file foo1
foo1.o: foo1.cpp
$(CC) $(CFLAGS) $(IFLAGS) -c $^ -o $#
#file foo2
foo2.o: foo2.cpp
$(CC) $(CFLAGS) $(IFLAGS) -c $^ -o $#
clean: clean-test clean-o clean-annoying
clean-test:
rm test-rfv
clean-o:
rm *.o -rfv
clean-annoying:
rm *~ -rfv
Just by visually comparing with other makefiles I saw around in the web, this seems to be not a very bright Makefile. I don't know how they work, but I can see there's significantly less boilerplate and more generic code in them.
Can this can be made better, safer, and easier to particularize for each project?
You don't want to name specific files in a makefile if you can get away with it, and 99% of the time you can. This page shows how to develop a very general makefile. The following is my own makefile, based on that page's info:
SHELL := bash
PROG := pathed.exe
OUTDIRS := bin/debug bin/rel obj/debug obj/rel
PROG_REL := bin/rel/$(PROG)
PROG_DEBUG := bin/debug/$(PROG)
SRCFILES := $(wildcard src/*.cpp)
OBJFILES_REL := $(patsubst src/%.cpp,obj/rel/%.o,$(SRCFILES))
OBJFILES_DEBUG := $(patsubst src/%.cpp,obj/debug/%.o,$(SRCFILES))
DEPFILES := $(patsubst src/%.cpp,obj/%.d,$(SRCFILES))
CFLAGS := -Iinc -Wall -Wextra -MMD -MP
DBFLAGS := -g
RELFLAGS :=
CC := g++
.PHONY: default all testmake debug release clean dirs
default: debug
all: dirs clean debug release
dirs:
#mkdir -p $(OUTDIRS)
debug: $(PROG_DEBUG)
release: $(PROG_REL)
testmake:
#echo OBJFILES_REL = $(OBJFILES_REL)
#echo OBJFILES_DEBUG = $(OBJFILES_DEBUG)
#echo SRCFILES = $(SRCFILES)
#echo DEPFILES = $(DEPFILES)
clean:
rm -f $(OBJFILES_REL) $(OBJFILES_DEBUG) $(DEPFILES) $(PROG)
$(PROG_REL): $(OBJFILES_REL)
$(CC) $(OBJFILES_REL) -o $(PROG_REL)
strip $(PROG_REL)
#echo "---- created release binary ----"
$(PROG_DEBUG): $(OBJFILES_DEBUG)
$(CC) $(OBJFILES_DEBUG) -o $(PROG_DEBUG)
#echo "---- created debug binary ----"
-include $(DEPFILES)
obj/rel/%.o: src/%.cpp
$(CC) $(RELFLAGS) $(CFLAGS) -MF $(patsubst obj/rel/%.o, obj/%.d,$#) -c $< -o $#
obj/debug/%.o: src/%.cpp
$(CC) $(DBFLAGS) $(CFLAGS) -MF $(patsubst obj/debug/%.o, obj/%.d,$#) -c $< -o $#
Do NOT use CC for the C++ compiler. The standard convention is that CC is the C compiler, CXX is the C++ compiler. CFLAGS are flags for the C compiler, CXXFLAGS are flags for the C++ compiler, and CPPFLAGS are flags for the pre-processor (eg, -I or -D flags). Use LDFLAGS for -L flags to the linker, and LDLIBS (or LOADLIBES) for -l flags.
Using the standard conventions is good not just because it makes things easier for others to understand, but also because it allows you to take advantage of implicit rules. If make needs to make a .o file from a .c file and you have not provided a rule, it will use a standard rule and honor the settings of CC, CFLAGS, and CPPFLAGS. If CC is a C++ compiler, things will probably not work.

Resources