I am looking after a system with many hundreds of c files in many folders, there are multiple targets so not all the c files are required in each build.
Because not all files are required I cannot use a pure recursive build. And I don't want to do that, because I would rather not have a slow serial build with mystic dependencies anyway.
What I have come up with broadly is a top level makefile where I pull in a list of folders and include a makefile from each of these folders.
SUB_MAKEFILES := $(foreach subdir,$(SUBDIRS), $(subdir)/subdir.mk)
-include $(SUB_MAKEFILES)
Some folders have specific files to compile so the included file is pretty simple;
srcs += path/a.c path/b.c
deps += path/a.d path/b.d
objs += op/path/a.o op/path/b.o
op/path/%.o: path/%.c path/subdir.mk
compile ...
I do not want to do this dozens of times so I have a generic pattern I use for folders where everything is to be compiled;
PATH155 := src/f1/f2/f3/f4
srcs += $(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')
deps += $(addprefix ${OUT_DIR}, $(patsubst %.c,%.d,$(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')))
objs += $(addprefix ${OUT_DIR}, $(patsubst %.c,%.o,$(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')))
$(OUT_O_DIR)$(PATH155)/%.o: $(PATH155)/%.c $(PATH155)/subdir.mk
gcc -c $(CFLGS) -MF"$(#:%.o=%.d)" -MT"$(#:%.o=%.d)" -o"$#" "$<"
This works, however I would like to avoid having to make up a random unique name for the path where the subdir.mk file resides.
Is there anyway to replace "PATH155673423 := hand/entered/path" with something like "local SUBDIRPWD = $(some function...)".
Then I could just drop in a generic makefile and include it, no error prone typing of paths nor worries that I will get a unique name clash.
It would be nice to have a few less directory scans too, same issue really, need a local variable.
Even some sort of macro for the repeated variable names etc would be better
Maybe there is a way to include the makefiles in a loop instead and set a path variable just before each is included?
Ta
Chris
There is no such thing as variables scoped to a particular makefile, such as you're suggesting. However, you have a lot of options for making this work, from constructed variables to give scope, to using eval to generate functions. For example, something like this:
SUBDIRS := foo bar biz baz
define INCLUDE_FILE
path = $S
include $S
endef
$(foreach S,$(SUBDIRS),$(eval $(INCLUDE_FILE)))
You can find out more about these by looking through this set of blog posts and the associated sections of the GNU make manual.
Related
I am aware of tools like CMake and GNU Autotools but I'm trying to write a universal build system myself, to use for my C and C++ projects. I'll briefly explain how it works and hopefully, someone can suggest either improvements or a better design altogether.
The build system proper lives in one of the project's subdirectories (I import it as a Git submodule). The project's root directory has a wrapper makefile that defines a couple of macros and includes the main makefile from said subdirectory. That does most of the work: it follows the directory organization scheme (i.e., it outputs libraries in lib, binaries in bin, etc.), it handles automatic dependencies for the source code and the DocBook documentation, and provides the de facto standard targets: all, test, clean, install, as well as others.
Here's what a wrapper makefile that builds two binaries, foo and bar, might look like:
# foo-specific macros
FOO_SRC_FILES = foo1.c foo2.c foo3.c
FOO_OBJ_FILES = $(FOO_SRC_FILES:.c=.o)
FOO_BIN_FILE = foo
# bar-specific macros
BAR_SRC_FILES = bar1.c bar2.c
BAR_OBJ_FILES = $(BAR_SRC_FILES:.c=.o)
BAR_BIN_FILE = bar
# Inform the build system about them
SRC_FILES = $(FOO_SRC_FILES) $(BAR_SRC_FILES)
OBJ_FILES = R(BAR_OBJ_FILES) $(BAR_OBJ_FILES)
BIN_FILES = $(FOO_BIN_FILE) $(BAR_BIN_FILE)
# Only install the binaries. If I were building a library, I would instead
# select the "lib" and perhaps "include" directories.
INSTALL = bin
INSTALL_DIR = /usr/share
# Use the build system
include build/build.mk
Now here's the problem. While build.mk can use pattern rules to create dependency and object files, there's only one OBJ_FILES and only one BIN_FILES. So if I put a pattern rule like the following in the build system that looks like this:
$(BIN_DIR)/$(BIN_FILES): $(OBJ_FILES:%=$(OBJ_DIR)/%) $(LIB_FILES:%=$(LIB_DIR)/%) | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $(OBJ_FILES:%=$(OBJ_DIR)/%) -L $(LIB_DIR) $(LIB_FILES:lib%.a=-l %)
then foo would depend on and link with everything that bar does and vice versa. So what I end up doing is asking the user to put these rules in the wrapper makefile, even though they feel like they belong in build.mk:
$(BIN_DIR)/$(FOO_BIN_FILE): $(FOO_OBJ_FILES:%=$(OBJ_DIR)/%) $(FOO_LIB_FILES:%=$(LIB_DIR)/%) | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $(FOO_OBJ_FILES:%=$(OBJ_DIR)/%) -L $(LIB_DIR) $(FOO_LIB_FILES:lib%.a=-l %)
$(BIN_DIR)/$(BAR_BIN_FILE): $(BAR_OBJ_FILES:%=$(OBJ_DIR)/%) $(BAR_LIB_FILES:%=$(LIB_DIR)/%) | $(BIN_DIR)
$(CC) $(LDFLAGS) -o $# $(BAR_OBJ_FILES:%=$(OBJ_DIR)/%) -L $(LIB_DIR) $(BAR_LIB_FILES:lib%.a=-l %)
The same issue applies to libraries as well, of course. The upside is that these rules can be copied and pasted almost verbatim; only the prefixes need to be changed (e.g., FOO or BAR).
Ideas to fix this include:
Asking the user to have separate wrapper makefiles for separate things (e.g., one for foo and another for bar) but that is just terrible.
Changing things up a little bit and then using m4 to do some preprocessing but I don't want to go through that unless a more elegant solution doesn't exist.
I would really appreciate some ideas.
PS: I know that the pattern matching expressions in the last two code samples can be replaced with text functions but those are GNU Make-specific. The style I used is more portable and is in fact on the list of additions for the next version of the POSIX standard.
I have begin to develop a similar system for my own C projects, but the logic I use does rely on some features which I believe are specific to GNU Make.
The main idea is to use a combinaison of $(eval) and $(call), by defining the logic of the build system, and then applying to the project tree.
To do so, I have in each of my directories and subdirectories a piece of Makefile of the following form, which I name Srcs.mk:
SRC := foo.c foo_bar.c bar.c
TARGET := foo_bar
SRC_DIR := src
OBJ_DIR := obj
I define a variable, which is in fact a macro, which is expanded with $(call) and then passed to $(eval). It's defined this way:
define get_local_variables
include Srcs.mk
$1SRC := $(SRC)
$1SRC_DIR := $(SRC_DIR)
$1OBJ_DIR := $(OBJ_DIR)
$1TARGET := $(TARGET)
TARGET :=
SRC :=
SRC_DIR :=
OBJ_DIR :=
$(call get_local_variables, $(DIR)) will expand to the above, with $1 replaced by the content of $(DIR). Then it will be treated as a Makefile fragment by $(eval)
This way, I fill per-directory variables for each of my directory.
I have then a handful or other rules which use this variables, using the same principles.
### Macros ###
obj = $(patsubst %.c,$($1OBJ_DIR)/%.o,$($1SRC))
define standard_rules
$($1TARGET): $(obj)
$$(LINK)
$(obj): $($1OBJ_DIR)/%.o:$($1SRC_DIR)/%.c | $($1OBJ_DIR)
$$(COMPILE)
endef
The variable are computed $(call), then expanded and read as makefile fragments by $(eval).
(I use static pattern rules but that it not intrinsic to the idea).
The whole idea is basically to define directories as a kind of namespace, with data attached to them, and then run function over them.
My actual system is a bit more complicated, but that the whole idea.
If you have a way to emulate $(eval) and $(call) (I think these are specific to GNU make, but not sure), you could try that approach.
You can also implement non recursive make this way, by adding a SUBDIRS variables in each directory and running recursively the same macro which is run on the current one. But it should been done carefully, not to mess it up with the order of expansion and evaluation in make.
So get_local_variables need to be evaluated before the rest of the macros are expanded.
(My project is visible on my Github account if you want to take a look, under make-build-system. But it is far from be complete enough^).
Be aware, though, that this is quite painful to debug when things go wrong. Make (at least, GNU) basically catch the error (when there is one) on the higher $(call) or $(eval) expansion.
I have developed my own non-recursive build system for GNU make, called prorab, where I solved the problem you described as follows.
The approach to solve your problem is somewhat similar to what #VannTen described in his answer, except that I use a macro to clean all state variables before defining build rules for the next binary.
For example, a makefile which builds two binaries could look like this:
include prorab.mk
this_name := AppName
this_ldlibs += -lsomelib1
this_cxxflags += -I../src -DDEBUG
this_srcs := main1.cpp MyClass1.cpp
$(eval $(prorab-build-app))
$(eval $(prorab-clear-this-vars))
this_name := AnotherppName
this_ldlibs += -lsomelib1
this_cxxflags += -I../src -DDEBUG
this_srcs := main2.cpp MyClass2.cpp
$(eval $(prorab-build-app))
So, in this example it will build two binaries: AppName and AnotherppName.
As you can see the build is configured by setting a number of this_-prefixed variables and the calling the $(eval $(prorab-build-app)) which expands to defining all the build, install, clean etc. rules.
Then a call to $(eval $(prorab-clear-this-vars)) clears all this_-prefixed variables, so that those can be defined again from scratch for the next binary, and so on.
Also, the very first line which includes the prorab.mk also cleans all this_-prefixed variables of course, so that the makefiles can be safely included into each other.
You can read more about that build system concepts here https://github.com/cppfw/prorab/blob/master/wiki/HomePage.adoc
Here's a snippet of a GNU Makefile I'm working with. Basically, I have a directory of images that i want copied into a dist directory when running make, but I'd prefer not to list each image individually.
DISTDIR := dist
IMG := $(shell find app/img -type f)
$(subst app/img, $(DISTDIR), $(filter-out %.svg, $(IMG)))): $(filter-out %.svg, $(IMG))
#cp $? $(DISTDIR)
This sort of works, except any time a source file is changed (in app/img), make always remakes the target file. What's the correct way to handle this situation?
It would be easier to answer if you provided some sample results for the find. Let's say that the find operation returns the files: app/img/foo.svg, app/img/bar.jpg, and app/img/baz.png.
In that case the filter-out functions will return the files app/img/bar.jpg and app/img/baz.png. This means that the rule generated will look like this:
$(DISTDIR)/bar.jpg $(DISTDIR)/baz.png: app/img/bar.jpg app/img/baz.png
#cp $? $(DISTDIR)
Here's the thing: when you define N targets in an explicit rule, it's like defining N rules, one for each target, with all the same prerequisites. So the above is identical to writing this:
$(DISTDIR)/bar.jpg: app/img/bar.jpg app/img/baz.png
#cp $? $(DISTDIR)
$(DISTDIR)/baz.png: app/img/bar.jpg app/img/baz.png
#cp $? $(DISTDIR)
Now you can see why you get the behavior that you do: each target lists ALL the files as prerequisites, so whenever ANY file is changed the target is rebuilt.
The answer is that in make, you should generally try to write rules that create exactly one target from just that target's prerequisites. One simple way to do that is with pattern rules; a pattern rule for the above might look like:
DISTDIR := dist
$(DISTDIR)/% : app/img/%
#cp $< $#
After that, all you have to do is declare a target which depends on the files you want created:
IMG := $(shell find app/img -type f)
all: $(IMG:app/img/%=$(DISTDIR)/%)
and make will figure out the rest!
Before I start, I'll mention that I'm not using GNU Make in this case for building a C/C++ project.
Makefile:
DEST_DIR = build/
SRC_DIR = src/
$(SRC_DIR)a/ : $(SOMETHING_ELSE)
$(DO_SOMETHING_TO_GENERATE_A_DIR)
$(DEST_DIR)% : $(SRC_DIR)%
cp -r $^ $#
ALL_DEPS += <SOMETHING>
... more code which appends to ALL_DEPS ...
.PHONY: all
all : $(ALL_DEPS)
I've got some files not generated via Make rules in $(SRC_DIR). (For the sake of this example, let's say there's a directory $(SRC_DIR)b/ and a file $(SRC_DIR)c .)
I want to append to ALL_DEPS all targets which represent files or directories in $(DEST_DIR) so that "make all" will run all of the available $(DEST_DIR)% rules.
I thought to do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(wildcard $(SRC_DIR)*)))
But of course, that doesn't catch anything that hasn't yet been made. (i.e. it doesn't append $(DEST_DIR)a/ to the list because $(SRC_DIR)a/ doesn't yet exist when the $(wildcard ...) invocation is evaluated and the shell doesn't include it in the results returned by the $(wildcard ...) invocation.)
So, rather than a function which finds all (currently-existing) files matching a pattern, I need one which finds all targets matching a pattern. Then, I could do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(targetwildcard $(SRC_DIR)*)))
If it matters any, I've got much of the GNU Make code split across multiple files and included by a "master" Makefile. The ALL_DEPS variable is appended to in any of these files which has something to add to it. This is in an attempt to keep the build process modular as opposed to dropping it all in one monster Makefile.
I'm definitely still learning GNU Make, so it's not unlikely that I'm missing something fairly obvious. If I'm just going about this all wrong, please let me know.
Thanks!
It is simply not possible to do what you're trying to do; you're trying to get make to recognise something that doesn't exist.
This is part of the reason why, in general, wildcards are bad (the other being that you can end up including stuff you didn't mean to). The right thing to do here is to explicitly create a list of source files (ls -1 | sed -e 's/\(.*\)/sources+=\1/' > dir.mk) and perform the patsubst transformation on that list.
If you have additional files that are generate as part of the build, then you can append them to that list and their rules will be found as you'd expect.
I have a following directory structure in my project:
bin/
dist/
include/
├── module_a/
└── module_b/
Makefile
src/
├── module_a/
└── module_b/
Folder include/ contains *.hpp's while *.cpp's are in src/. I would like to compile all sources to bin/ and then link them up together to dist/. Seems a pretty reasonable wish for me.
I would like to know the best practices for a Makefile for this case. All I can find is %.o: %.cpp target, but that doesn't really work, because of different source and binary folder.
I was trying to use something like this:
D_SRC = src
D_BIN=bin
F_CPP := $(shell find $(D_SRC) -iname '*.cpp' -type f)
F_OBJ := $(shell echo $(F_CPP) | sed s:\ :\\n:g | sed s:$(D_SRC):$(D_BIN): | sed 's:^\(.*\)\.cpp$$:\1\.o:')
$(F_OBJ): $(F_SRC)
$(foreach file, $(F_SRC), \
$(GXX) $(CXXFLAGS) -c $(file)\
)
This target doesn't work, because $(F_OBJ) paths start with bin/, while foreach compiles sources to current working dir. I could make it compile to bin/, but that would happen only with a few more sed expressions and it's ugly enough as it is.
It's probably so difficult for me, because I don't know make all that well, but I cannot be the only one with this project setup. In my opinion, it must be a pretty common one. I know I can write a Makefile for each module separately, but is that really the best choice here?
EDIT: I was now wondering what would I achieve with several Makefiles. If one was at root and another one in src/module_a, how would the latter know about the bin/? If you'd execute it with make -f src/module_a/Makefile, it would be the same as executing it from root directory, 'cause it's working directory would be root. Another way, I guess, would be to change directory before executing it, like so: make -C include/module_a, but in that case, how would it find bin/? I wouldn't want to have something like D_BIN = ../../bin in a Makefile.
What I normally do is have a Makefile in the src directory (which can be invoked from the top level Makefile if you like) and then use rules like this:
D_BIN = ../bin
$(D_BIN)/%.o: %.cpp
You could also experiment with just a makefile in the top level dir, and use rules that look like this:
D_BIN = bin
D_SRC = src
$(D_BIN)/%.o: $(D_SRC)/%.cpp
but I have not used such rules, so I don't know the pros/cons vs the way I normally do it. The way I normally do it works fine, I even have rules that build depends like so:
$(D_BIN)/%.d: %.cpp
and the link rule would be like:
../dist/outexe: $(F_OBJ)
Using a foreach is usually frowned upon because it does not make use of all the features built into normal makefile rules (i.e. there is no depends check on a per file basis, either you build everything or nothing), and as such foreach should only be used as a last resort, but in this case you will be able to get it to work without the foreach.
In addition to this there are much easier ways to build your file lists, you don't need to use the shell or sed.
F_CPP = $(wildcard *.cpp)
F_OBJ = $(F_CPP:.cpp=.o)
Update: This is how I normally issue recursive makes:
SUBDIRS = src
.PHONY: $(SUBDIRS)
all: $(SUBDIRS)
$(SUBDIRS):
#echo "Building $#..."
$(MAKE) -C $# $(MFLAGS)
Then indeed in your submake, you would need to use ../bin for example.
However with a project as simple as yours, you might be better off just having one makefile at the root level and using rules like this:
D_BIN = bin
D_SRC = src
$(D_BIN)/%.o: $(D_SRC)/%.cpp
recursive makefiles are ok (ok but not great) if you have a really complex directory structure, where you will be adding/removing/modifying new dir trees as time goes on. But for a simple project where you just want to have separate directories for code and objs, it is probably overkill.
I have a C++ library built using a Makefile. Until recently, all the sources were in a single directory, and the Makefile did something like this
SOURCES = $(wildcard *.cpp)
which worked fine.
Now I've added some sources that are in a subdirectory, say subdir. I know I can do this
SOURCES = $(wildcard *.cpp) $(wildcard subdir/*.cpp)
but I'm looking for a way to avoid specifying subdir manually, that is, make wildcard look into subdirectories, or generating a list of subdirectories somehow and expanding it with several wildcard functions. At this point, having a non-recursive solution (that is, expanding only the first level) would be fine.
I haven't found anything - my best guess is using find -type d to list the subdirectories, but it feels like a hack. Is there any built-in way to do this?
This should do it:
SOURCES = $(wildcard *.cpp) $(wildcard */*.cpp)
If you change you mind and want a recursive solution (i.e. to any depth), it can be done but it involves some of the more powerful Make functions. You know, the ones that allow you to do things you really shouldn't.
EDIT:
Jack Kelly points out that $(wildcard **/*.cpp) works to any depth, at least on some platforms, using GNUMake 3.81. (How he figured that out, I have no idea.)
Recursive wildcards can be done purely in Make, without calling the shell or the find command. Doing the search using only Make means that this solution works on Windows as well, not just *nix.
# Make does not offer a recursive wildcard function, so here's one:
rwildcard=$(wildcard $1$2) $(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2))
# How to recursively find all files with the same name in a given folder
ALL_INDEX_HTMLS := $(call rwildcard,foo/,index.html)
# How to recursively find all files that match a pattern
ALL_HTMLS := $(call rwildcard,foo/,*.html)
The trailing slash in the folder name is required. This rwildcard function does not support multiple wildcards the way that Make's built-in wildcard function does, but adding that support would be straightforward with a couple more uses of foreach.
If you don't want to use recursive makefiles, this might give you some ideas:
subdirs := $(wildcard */)
sources := $(wildcard $(addsuffix *.cpp,$(subdirs)))
objects := $(patsubst %.cpp,%.o,$(sources))
$(objects) : %.o : %.cpp
You can use several rules in wildcard:
SOURCES := $(wildcard *.cpp */*.cpp)
if you need more depth:
SOURCES := $(wildcard *.cpp */*.cpp */*/*.cpp */*/*/*.cpp)
Unfortunately, and unlike what we sometimes read, glob (**) is not supported by makefile and will be interpreted as normal wildcard (*).
For example **/*.cpp match dir/file.cpp but neither file.cpp nor dir/sub/file.cpp.
If you need infinite depth use shell and find:
SOURCES := $(shell find . -name "*.cpp")
Common practice is to put a Makefile in each subdir with sources, then
all: recursive
$(MAKE) -C componentX
# stuff for current dir
or
all: recursive
cd componentX && $(MAKE)
# stuff for current dir
recursive: true
It may be wise to put settings for each Makefile in a Makefile.inc in the root source directory. The recursive target forces make to go into the subdirectories. Make sure that it doesn't recompile anything in a target requiring recursive.
If you can use find shell command, you may define a function to use it.
recurfind = $(shell find $(1) -name '$(2)')
SRCS := $(call recurfind,subdir1,*.c) $(call recurfind,subdir2,*.cc) $(call recurfind,subdir2,*.cu) \
...