Using GNU Make with subdirectories - makefile

I was wondering what different approaches of using Make in a project with subdirectories exist, and what are their advantages/drawbacks, but could never see a good summary or cookbook.
I have seen in my researches mainly the "recursive" and "single makefile" approaches, but are there others ?
I also assume that there is not only one "recursive" or "single makefile" approaches but several, so could somebody sum it up ?
For my particular case, I would like a directory architecture looking like this:
.
├── build
│   ├── *.d
│   ├── *.o
| ├── subdir1
| │ ├── *.d
| │ └── *.o
| └── subdir2
| ├── *.d
| ├── *.o
| └── subdir3
| ├── *.d
| └── *.o
├── include
│   ├── *.h
│   └── *.h
├── Makefile
└── src
├── *.c
├── *.h
├── subdir1
│   ├── *.c
│   └── *.h
└── subdir2
├── *.c
├── *.h
└── subdir3
├── *.c
└── *.h
Which solution should I choose ? Possibly one which would allow source files with the same name ?

Your project setup is really basic, so should be your Makefile:
SRC_DIR := src
BLD_DIR := build
SRC := $(shell find $(SRC_DIR) -name "*.c")
OBJ := $(SRC:$(SRC_DIR)/%.c=$(BLD_DIR)/%.o)
DEP := $(OBJ:.o=.d)
CPPFLAGS := -MMD -MP # enable auto-dependency generation
CFLAGS := -Wall -W -pedantic
.PHONY: all clean
all: $(OBJ)
clean:
$(RM) -r $(BLD_DIR)
.SECONDEXPANSION:
$(BLD_DIR)/%.o: $(SRC_DIR)/%.c | $$(#D)/ # First check that the destination directory exists
$(CC) $(CPPFLAGS) $(CFLAGS) -o $# -c $<
%/:
mkdir -p $* # -p flag necessary for recursive directory creation
ifeq "$(MAKECMDGOALS)" ""
-include $(DEP)
endif
The idea here is to list source files recursively using the find command, to supply make with the appropriate pattern rule to compile in the right place and pass the right preprocessor file to your compiler to enable auto-dependency generation.
Tested with GNU Make 4.1 under Windows 8.1 with the GIT Bash shell and the following directory structure:
.
├── Makefile
└── src
├── test.c
├── test1.c
└── subdir1
└── test.c

After reading Recursive Make Considered Harmful, I figured a quite simple and modular way to achieve this, by having files in all subdirectories that would include each other and be included in the main makefile:
CXX := gcc
SRCDIR := src
OBJDIR := build
# These lines are needed to set immediate evaluation for
# these variables, instead of deferred evaluation which is unsuitable.
SRCS :=
SUBDIRS :=
CFLAGS :=
LDFLAGS :=
include $(SRCDIR)/module.mk
OBJS := $(addprefix $(OBJDIR)/, $(SRCS:.c=.o))
SRCS := $(addprefix $(SRCDIR)/, $(SRCS))
DEPS := $(OBJS:.o=.d)
TMPS := $(OBJS) $(OBJS:.o=.d)
CFLAGS += -MD
debug: CFLAGS += -g -g3 -ggdb
CFLAGS += $(addprefix -I./$(SRCDIR)/, $(SUBDIRS))
LDFLAGS += -lsomelib
debug: LDFLAGS += -g -g3 -ggdb
NAME := yolo
all: $(NAME)
debug: re
-include $(DEPS)
$(OBJDIR)/%.o: $(SRCDIR)/%.c
$(COMPILE.c) $(OUTPUT_OPTION) $<
$(NAME): $(OBJS)
#$(CXX) $(OBJS) -o $(NAME) $(LDFLAGS)
$(OBJS): | $(OBJDIR)
$(OBJDIR):
#mkdir -p $(OBJDIR)
#for dir in $(SUBDIRS); \
do \
mkdir -p $(OBJDIR)/$$dir; \
done
clean:
rm -rf $(TMPS)
fclean: clean
rm -rf $(NAME)
rm -rf $(OBJDIR)
re: fclean all
.PHONY: all clean fclean re
And in every subdirectory, a module.mk file (I could have named it anything, but this seemde cool).
For src:
SRCS := main.c file1.c file2.c
SUBDIRS += subdir1 subdir2
include $(SRCDIR)/subdir1/module.mk
include $(SRCDIR)/subdir2/module.mk
For a level 1 subdirectory:
THIS_DIR_L0 := subdir1
MOD_SRC := file3.c file4.c
SRCS += $(addprefix $(THIS_DIR_L0)/, $(MOD_SRC))
SUBDIRS += $(THIS_DIR_L0)/subdir3
include $(SRCDIR)/$(THIS_DIR_L0)/subdir3/module.mk
And for a level 2 subdir (one deeper):
THIS_DIR_L1 := subdir3
MOD_SRC := file5.c file6.c
SRCS += $(addprefix $(THIS_DIR_L0)/$(THIS_DIR_L1)/, $(MOD_SRC))
And so on...
This is quite simple to set up, I find it very modular and it does not use recursive makefiles. It would not be complicated to make librairies and stuff inside your directory structure either.
Anybody having a better idea please tell me.

Related

Write generic Makefile rule for subdirectories

I have a project with a directory tree that looks like this:
.
├── modules
│   ├── mod1
│   │   └── mod1.f90
│   ├── mod2
│   │   └── mod2.f90
│   └── mod.f90
└── src
└── main.f90
├── bin
└── Makefile
main.f90 uses all the modules found in modules and in the 'submodules' that are mod1/ and mod2/.
A simple Makefile I can write to compile the project is something like:
F90 = gfortran
FLAGS = -g -I$(BINDIR)
MODFLAGS = -J$(BINDIR)
BINDIR = bin
SRCDIR = src
MODDIR = modules
SMODDIR = $(dir $(wildcard $(MODDIR)/*/.))
MODFILES = $(wildcard $(MODDIR)/*f90)
MODOBJ = $(join $(addsuffix $(MODFROMBIN)/, $(dir $(MODFILES))), \
$(notdir $(MODFILES:.f90=.o)))
SMODFILES = $(foreach smoddir, $(SMODDIR), $(wildcard $(smoddir)*f90))
SMODOBJ = $(join $(addsuffix $(SMODFROMBIN)/, $(dir $(SMODFILES))), \
$(notdir $(SMODFILES:.f90=.o)))
SRCFILES = $(wildcard $(SRCDIR)/*f90)
SRCOBJ = $(join $(addsuffix $(SRCFROMBIN)/, $(dir $(SRCFILES))), \
$(notdir $(SRCFILES:.f90=.o)))
ALLOBJ = $(MODOBJ) $(SMODOBJ) $(SRCOBJ)
BINOBJ = $(addprefix $(BINDIR)/, $(sort $(notdir $(ALLOBJ))))
all: main
$(MODDIR)/../bin/%.o: $(MODDIR)/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
modules/mod1/../../bin/%.o: modules/mod1/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
modules/mod2/../../bin/%.o: modules/mod2/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
$(SRCDIR)/../bin/%.o: $(SRCDIR)/%.f90
$(F90) $(FLAGS) -c $^ -o $#
main: $(ALLOBJ)
$(F90) $(FLAGS) -o ./bin/main $(BINOBJ)
clean:
#rm bin/*.o bin/*.mod
But now I want to write a generic rule to be able to compile all modules located in directories that are inside modules directory (the code I am working on have more than two submodules and I am not willing to write as many rules as I have subdirectories).
My first try was to write something like this:
$(SMODDIR)/../../%.o: $(SMODDIR)/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
but it fails; from what I understand, it will put all the subdirectories paths and I would actually end up with a rule that would looks like this:
modules/mod1/../../bin/ modules/mod2/../../bin/%.o: modules/mod1/ modules/mod2/%.f90
that indeed looks funny.
As shown in the original Makefile, it is possible to retrieve specificaly with
SMODDIR = $(dir $(wildcard $(MODDIR)/*/.))
SMODFILES = $(foreach smoddir, $(SMODDIR), $(wildcard $(smoddir)*f90))
so I guess it may be possible to use something similar to have more generic rules. I couldn't see however how to use such a syntax to write a rule that makes sense.
Any help would be appreciated!
You have two choices to avoid writing lots of rules.
One is, you can use VPATH and put all your source directories in it, something like this:
VPATH := $(MODDIR) $(SMODDIR) $(SRCDIR)
$(BINDIR)/%.o: %.f90
$(F90) $(FLAGS) -c $< -o $# $(MODFLAGS)
Or you can put all your object files into equivalent subdirectories of $(BINDIR), something like this:
MODOBJ = $(MODFILES:%.f90=$(BINDIR)/%.o)
SMODOBJ = $(SMODFILES:%.f90=$(BINDIR)/%.o)
SRCOBJ = $(SRCFILES:%.f90=$(BINDIR)/%.o)
$(BINDIR)/%.o: %.f90
#mkdir -p $(#D)
$(F90) $(FLAGS) -c $< -o $# $(MODFLAGS)

Getting Make to automatically detect changes in protocol buffers

I want Make to automatically compile protos when I update them, here is what I've got so far:
TARGET=main
BIN_DIR=bin
SRC_DIR=src
OBJ_DIR=obj
PROTO_DIR=protos/
PROTO_COMPILE_DIR=src/$(PROTO_DIR)
CC = g++
CFLAGS = -Wall -std=c++17 -ggdb -pipe -I.
LINKER = g++
LFLAGS = $(CFLAGS) -lprotobuf
SOURCES = $(wildcard src/*.cc) \
$(wildcard src/protos/*.cc) \
$(wildcard src/db_handler/*.cc)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.cc=$(OBJ_DIR)/%.o)
$(BIN_DIR)/$(TARGET): proto $(OBJECTS)
#mkdir -p $(BIN_DIR)/
$(LINKER) $(OBJECTS) $(LFLAGS) -o $#
$(OBJECTS): $(OBJ_DIR)/%.o : $(SRC_DIR)/%.cc
#mkdir -p obj/ obj/protos obj/db_handler
$(CC) $(CFLAGS) -c $< -o $#
.PHONY: proto
proto:
#printf "Compiling protos...\n"
#cd $(PROTO_DIR) && protoc * --grpc_out=../$(PROTO_COMPILE_DIR)\
--cpp_out=../$(PROTO_COMPILE_DIR)\
--plugin=protoc-gen-grpc="/usr/local/bin/grpc_cpp_plugin"\
&& cd ../
It successfully compiles protos; but, it does so every time, even if there are no changes in files. How can I prevent this and compile protos only if protos change?
Edit: Added project structure
├── LICENSE
├── makefile
├── protos
│   ├── client.proto
│   └── person.proto
├── README.md
└── src
├── db_handler
│   ├── db_handler.cc
│   └── db_handler.h
├── main.cc
└── protos
├── client.grpc.pb.cc
├── client.grpc.pb.h
├── client.pb.cc
├── client.pb.h
├── person.grpc.pb.cc
├── person.grpc.pb.h
├── person.pb.cc
└── person.pb.h
You want to add the source files of the protocol buffers to the right (the prerequisites) of the rule that cares about them. This is how Make understands and tracks their time stamps.
By adding the files to the prerequisites, Make will understand that this is the rule which cares about those source files.
PROTO_SOURCES := $(wildcard $(PROTO_DIR)/*.proto)
PROTOS := $(patsubst $(PROTO_DIR)/%.proto,$(PROTO_COMPILE_DIR)/%.cc,$(PROTO_SOURCES))
$(PROTOS): $(PROTO_SOURCES)
#printf "Compiling protos...\n"
#cd $(PROTO_DIR) && protoc * --grpc_out=../$(PROTO_COMPILE_DIR)\
--cpp_out=../$(PROTO_COMPILE_DIR)\
--plugin=protoc-gen-grpc="/usr/local/bin/grpc_cpp_plugin"
However, this $(PROTOS) : $(PROTO_SOURCES) is not good if you use parallel builds; because, Make will try to run the command once for each output file. So, make will run N instances of the command at the same time, which means they may clobber each other.
In order to know the fully correct solution you need to provide more information (for those not familiar with protoc). Is it a requirement that you invoke protoc once with all inputs? Or is it valid to run protoc individually on each input .proto file to get its output? Then you can write a pattern rule that generates one file at a time.
Note 1: As to your other attempt, if you use .PHONY to mark the rule then Make will rebuild the rule every time weather it needs to or not.
Note 2: You don't need the cd .. at the end of the second instruction, since it is run in a sub shell.
Thanks to #FiddlingBits I managed to figure out how to do it correctly
TARGET=main
BIN_DIR=bin
SRC_DIR=src
OBJ_DIR=obj
PROTO_DIR=protos/
PROTO_COMPILE_DIR=src/$(PROTO_DIR)
rm = rm -f
CC = g++
CFLAGS = -Wall -std=c++17 -ggdb -pipe -I.
LINKER = g++
LFLAGS = $(CFLAGS) -lprotobuf
SOURCES = $(wildcard src/*.cc) \
$(wildcard src/protos/*.cc) \
$(wildcard src/db_handler/*.cc)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.cc=$(OBJ_DIR)/%.o)
PROTOS := $($(PROTO_DIR)/%.proto=$(PROTO_COMPILE_DIR)/%.cc)
$(BIN_DIR)/$(TARGET): $(PROTOS) $(OBJECTS)
echo $(PROTOS)
#mkdir -p $(BIN_DIR)/
$(LINKER) $(OBJECTS) $(LFLAGS) -o $#
$(OBJECTS): $(OBJ_DIR)/%.o : $(SRC_DIR)/%.cc
#mkdir -p obj/ obj/protos obj/db_handler
$(CC) $(CFLAGS) -c $< -o $#
$(PROTOS):
#printf "Compiling protos...\n"
#cd $(PROTO_DIR) && protoc * --grpc_out=../$(PROTO_COMPILE_DIR)\
--cpp_out=../$(PROTO_COMPILE_DIR)\
--plugin=protoc-gen-grpc="/usr/local/bin/grpc_cpp_plugin"\
&& cd ../
.PHONY: clean
clean:
#$(rm) -r $(OBJ_DIR)/*
#$(rm) -r $(BIN_DIR)/*
#printf "Cleanup complete!\n"

Generic Makefile build directory error

I have the following directory structure for a dummy C project.
.
├── inc
│   ├── getmsg.c
│   └── getmsg.h
├── makefile
└── src
└── main.c
My current generic Makefile is below,
TARGET = main
# finds all .c files, inc/getmsg.c src/main.c
SOURCES := $(shell find * -name *.c)
# converts all .c files to .o files, inc/getmsg.o src/main.o
OBJECTS := $(SOURCES:.c=.o)
# directories that contain .h files for gcc -I flag, inc/
HEADERS := $(dir $(shell find * -name *.h))
CC = gcc
CFLAGS = -Wall -std=c99 -iquote "$(HEADERS)"
all: $(TARGET)
%.o: %.c
$(CC) $(CFLAGS) -c $< -o $#
$(TARGET): $(OBJECTS)
$(CC) -o $# $^
clean:
rm -rf $(shell find * -name *.o) $(TARGET)
This all compiles fine however it just dumps the .o files into the same directory as its corresponding .c file.
What I would like to do is have all object files put into a build directory. To do this I change the OBJECTS to OBJECTS := $(patsubst %,build/%,$(notdir $(SOURCES:.c=.o))) which lists the files build/getmsg.o build/main.o. Then I set the %.o target to build/%.o: %.c.
This however returns No rule to make target 'build/getmsg.o'. So the make file is unable to build the .o files. What am I missing here?
Try changing
%.o: %.c
to
build/%.o: %.c

Using Makefile with subfolders

I have the following directory structure.
Project/
├── bin/
├── src/
│ ├── main.c
│ ├── util/
│ ├── util.c
│ ├── util.h
├── obj/
├── .depend/
All my source code are in the src folder. In the src root is my main.c file; which includes other files that are on the same level that he (or within a same level folder). I have a Makefile below that works well for all files in the same level of main.c but does not work on files in a subfolder within src
How change my Makefile to allow subfolder within the src folder?
CC := gcc
CFLAGS := -Wall -Wextra
BINDIR := bin
OBJDIR := obj
SRCDIR := src
DEPDIR := .depend
SOURCES := $(wildcard $(SRCDIR)/*.c)
OBJECTS := $(patsubst $(SRCDIR)/%.c, $(OBJDIR)/%.o, $(SOURCES))
DEPENDS := $(patsubst $(SRCDIR)/%.c, $(DEPDIR)/%.d, $(SOURCES))
$(BINDIR)/app: $(OBJECTS) | $(BINDIR)
#$(CC) -o $# $^
-include $(DEPENDS)
$(OBJDIR)/%.o: $(SRCDIR)/%.c | $(OBJDIR)
#$(CC) $(CFLAGS) -c -o $# $<
$(DEPDIR)/%.d: $(SRCDIR)/%.c | $(DEPDIR)
#$(CC) -MM -MG $< | sed 's!^\(.\+\).o:!$(DEPDIR)/\1.d $(OBJDIR)/\1.o:!' > $#
$(DEPDIR) $(BINDIR) $(OBJDIR):
#mkdir $#
clean:
#rm -rf $(BINDIR)/*
#rm -rf $(OBJDIR)/*
.PHONY: clean
EDIT: the .o and .d files do not need to respect the original design of the structure. And I'm using Windows (MinGW)
First, you'll have to change your SOURCES to recursively find the sources. This can be done in pure make:
subdirs = $(filter-out $1,$(sort $(dir $(wildcard $1*/))))
rfind = $(wildcard $1$2) $(foreach d,$(call subdirs,$1),$(call rfind,$d,$2))
SOURCES := $(call rfind,$(SRCDIR)/,*.c)
Everything else will work, except for directory creation. First, change your prerequisites to use $(#D) with secondary expansion:
.SECONDEXPANSION:
$(OBJDIR)/%.o: $(SRCDIR)/%.c | $$(#D)
...
$(DEPDIR)/%.d: $(SRCDIR)/%.c | $$(#D)
...
Then, change your directory creation rule to include all the directories:
$(BINDIR) $(patsubst %/,%,$(sort $(dir $(OBJECTS) $(DEPENDS)))):
#mkdir -p $#
Like the recursive find, it uses sort to deduplicate the directories (otherwise make will warn) and strips the trailing slash (because $(#D) won't have a trailing slash). Note that -p is needed to avoid issues with order and with directories only containing other directories and no sources.
try to set your sources like this:
SOURCES := $(shell find $(SRCDIR) -type f -name "*.c")
if it does not work right a way, try to debug it by just running in your command line
find src -type f -name "*.c"
and see if it outputs the correct list of files and with correct relative path, adjust accordingly.
Note, that this approach works only in unix-like environment, if you are using MinGW from MSYS or Cygwin environment it should work.

How I strip path of source file while writing a compilation rule in makefile?

In short: I want to compile sources from different directories, and put object files into current directory.
For example, I have files:
test.c
../../lib1/boot.c
../../lib2/startup.c
../common/utils.c
(also few files .s (assembly) and .cpp, but I hope this is not important).
All of their object-files I want to be in the current directory:
test.o
boot.o
startup.o
utils.o
And I can't figure out how to write such rule in my makefile.
For example,
%o.: %.c
does not work now because make can't find a rule to build boot.o from ../../lib1/boot.c, it can only find rule to build ../../lib1/boot.o from ../../lib1/boot.c.
I tried to use this:
%o.: %.c
(my compilation line, for example "gcc -c $^ -o $#")
%o.: ../../lib1/%.c
(my compilation line)
%o.: ../../lib2/%.c
(my compilation line)
%o.: ../common/%.c
(my compilation line)
and it works. But obviously this is not generic enough, and in addition, some user came to me today and said that his application has also some ../../some_other_lib/common_things.c, hence my makefile failed. I looked through our project, and found many such cases with a lot of different directories involved. With my approach, I'll have to write a separate rule for each such directory, with identical compilation line. This does not seem good to me.
So my question is: how to make some generic compilation rule that puts (and checks) object files in current directory, while operating with sources in different directories?
Thank you.
The directories can be extracted from the CSRC variable with $(dir ...) and this list can then be used in the vpath directive.
vpath %.c $(sort $(dir $(CSRC)))
vpath %.s $(sort $(dir $(SSRC)))
vpath %.cpp $(sort $(dir $(CPPSRC)))
(I've thrown in the sort function to remove duplicates, but that's not absolutely necessary.)
Now the rules can be kept simple and make will search the source files in the list of directories.
$(COBJ) := $(notdir $(CSRC))
$(SOBJ) := $(notdir $(SSRC))
$(CPPOBJ) := $(notdir $(CPPSRC))
.PHONY: all
all: $(EXECUTABLE)
$(EXECUTABLE): $(COBJ) $(SOBJ) $(CPPOBJ)
....
$(COBJ): %.o: %.c
...
$(SOBJ): %.o: %.s
...
$(CPPOBJ): %.o: %.cpp
...
Try to use makefile function notdir as this:
%.o: %.c
gcc -c $< -o $(notdir $#)
$# must be equal to the full path ex: ../../lib2/startup.o ad notdir will trunk it to: startup.o.
With this rule you will be able to compile all your source in the current directory.
Actually, your example is like that:
.
└── common
├── lib1
│   └── boot.c
├── lib2
│   └── startup.c
├── test
│   ├── Makefile
│   └── test.c
└── utils.c
I think i will be better like that:
.
├── common
│   ├── lib1
│   │   ├── Makefile
│   │   ├── obj
│   │   └── src
│   │   └── boot.c
│   ├── lib2
│   │   ├── Makefile
│   │   ├── obj
│   │   └── src
│   │   └── startup.c
│   ├── Makefile
│   ├── obj
│   ├── src
│   │   └── utils.c
│   └── test
│   ├── Makefile
│   ├── obj
│   └── src
│   └── test.c
└── Makefile
For that you need all your Makefiles to call the subdirs Makefiles.
and the src/obj dirs is a separation between your source and objects.
SRC := utils.c
OBJ := $(SRC:%.c=%.o)
NAME := project
SRC_D := src
OBJ_D := obj
SUBDIRS := lib1/ \
lib2/ \
test/
all: $(NAME) $(SUBDIRS)
#for dir in $(SUBDIRS); \
do \
$(MAKE) -C $$dir; \
done
$(NAME): $(OBJ:%.o=$(OBJ_D)/%.o)
$(OBJ_D)/%.o : $(SRC_D)/%.c
gcc -c $< -o $#
OK, took me some time, but finally I found the solution (using some threads on this site by the way):
# Defining compilation rules in a way that object files will be produced in current directory, and not in the directory of source files:
all: <List of my targets>
define my_c_rule
$(subst .c,.o,$(notdir $(1))): $(1)
$(CC) $(CFLAGS) $(CDEFINES) $$^ -o $$#
endef
$(foreach f, $(CSRC), $(eval $(call my_c_rule, $(f))))
$(CSRC) contains list of source files with their paths.
Just need to take into account that if earlier I had something like this:
.c.o:
$(CC) $(CFLAGS) $(CDEFINES) $^ -o $#
all: <List of my targets>
...now I have to put all sentence above the rules which I described in my_c_rule procedure. If I don't do this, make stops after compiling first source file. This is because old "wildcard" rules like .c.o or %.o: %.c do not replace all as a default target (even being written earlier), but non-wildcard rules like boot.o: ../../lib1/boot.c (result of the above macros) do replace the default target in case they are written earlier.

Resources