Sources from subdirectories in Makefile - makefile

I have a C++ library built using a Makefile. Until recently, all the sources were in a single directory, and the Makefile did something like this
SOURCES = $(wildcard *.cpp)
which worked fine.
Now I've added some sources that are in a subdirectory, say subdir. I know I can do this
SOURCES = $(wildcard *.cpp) $(wildcard subdir/*.cpp)
but I'm looking for a way to avoid specifying subdir manually, that is, make wildcard look into subdirectories, or generating a list of subdirectories somehow and expanding it with several wildcard functions. At this point, having a non-recursive solution (that is, expanding only the first level) would be fine.
I haven't found anything - my best guess is using find -type d to list the subdirectories, but it feels like a hack. Is there any built-in way to do this?

This should do it:
SOURCES = $(wildcard *.cpp) $(wildcard */*.cpp)
If you change you mind and want a recursive solution (i.e. to any depth), it can be done but it involves some of the more powerful Make functions. You know, the ones that allow you to do things you really shouldn't.
EDIT:
Jack Kelly points out that $(wildcard **/*.cpp) works to any depth, at least on some platforms, using GNUMake 3.81. (How he figured that out, I have no idea.)

Recursive wildcards can be done purely in Make, without calling the shell or the find command. Doing the search using only Make means that this solution works on Windows as well, not just *nix.
# Make does not offer a recursive wildcard function, so here's one:
rwildcard=$(wildcard $1$2) $(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2))
# How to recursively find all files with the same name in a given folder
ALL_INDEX_HTMLS := $(call rwildcard,foo/,index.html)
# How to recursively find all files that match a pattern
ALL_HTMLS := $(call rwildcard,foo/,*.html)
The trailing slash in the folder name is required. This rwildcard function does not support multiple wildcards the way that Make's built-in wildcard function does, but adding that support would be straightforward with a couple more uses of foreach.

If you don't want to use recursive makefiles, this might give you some ideas:
subdirs := $(wildcard */)
sources := $(wildcard $(addsuffix *.cpp,$(subdirs)))
objects := $(patsubst %.cpp,%.o,$(sources))
$(objects) : %.o : %.cpp

You can use several rules in wildcard:
SOURCES := $(wildcard *.cpp */*.cpp)
if you need more depth:
SOURCES := $(wildcard *.cpp */*.cpp */*/*.cpp */*/*/*.cpp)
Unfortunately, and unlike what we sometimes read, glob (**) is not supported by makefile and will be interpreted as normal wildcard (*).
For example **/*.cpp match dir/file.cpp but neither file.cpp nor dir/sub/file.cpp.
If you need infinite depth use shell and find:
SOURCES := $(shell find . -name "*.cpp")

Common practice is to put a Makefile in each subdir with sources, then
all: recursive
$(MAKE) -C componentX
# stuff for current dir
or
all: recursive
cd componentX && $(MAKE)
# stuff for current dir
recursive: true
It may be wise to put settings for each Makefile in a Makefile.inc in the root source directory. The recursive target forces make to go into the subdirectories. Make sure that it doesn't recompile anything in a target requiring recursive.

If you can use find shell command, you may define a function to use it.
recurfind = $(shell find $(1) -name '$(2)')
SRCS := $(call recurfind,subdir1,*.c) $(call recurfind,subdir2,*.cc) $(call recurfind,subdir2,*.cu) \
...

Related

Makefile wildcard issue

I am trying to write a makefile to selectively include a certain file.
I have manage to get it working with the following:
OBJS := $(filter-out ./build/./src/class_1.cpp.o, $(OBJS))
OBJS := $(filter-out ./build/./src/class_2.cpp.o, $(OBJS))
OBJS_1 :=./build/./src/class_1.cpp.o $(OBJS)
OBJS_2 := ./build/./src/class_2.cpp.o $(OBJS)
But I would like to make it more generic and use a wildcard.
My understanding is that the following should work:
OBJS := $(filter-out $(wildcard ./build/./src/*), $(OBJS))
I also tried
OBJS := $(filter-out $(wildcard \./build/\./src/*), $(OBJS))
but not sure if the problem is with the special symbols or just a basic understanding.
however it then complains about having too many main functions (class 1 and class 2).
Is there something that I am missing with my understanding of wildcard and filters in make?
If I print out the value of $(OBJS) with the two different approaches, the values are identical, so I am not sure why one solution could work while the other one fails.
However, for some reason OBJS_1 doesn't seem to get the filter that was applied to $(OBJS)
$(wildcard ...) function finds which files actually exist in the filesystem (and some or all .o files might be created during make's execution, thus the result will be different depending on the last command). This is most certainly not what you want.
In your case you need only to perform the string substitution, and the 'match-all' pattern for strings in make is a percent-sign. So it should be:
OBJS:=$(filter-out ./build/./src/%,$(OBJS))
Also note that spaces in the makefile may have significance. For example, $(filter a, b) means that the second argument is " b", not just "b". Even though it wouldn't matter in your case, you should be more cautious with spaces.

How to use patsubst in makefile to do multiple substitutions

I am trying to generate list of object files from source files in my makefile using patsubst
OUT_DIR=Out/
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp
../../../Client2.4/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
OBJ:= $(patsubst %src/%.cpp,${OUT_DIR}$%.o,$(SRC))
I want my OBJ variable to be
OBJ=Out/BrokerModule/BrokerApp.o Out/CommandMsgManager/CConfigModuleInfo.o
after patsubst but above patsubst is not producing the desired result. Please help.
There are some problems with the usage of patsubst, see my suggestion as followed,
OUT_DIR=Out/
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp \
../../../Client2.4/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
# add the definition of src
src=../../../Client2.4/Client/src/
# Modify the definition of OBJ
OBJ:= $(patsubst ${src}%.cpp,${OUT_DIR}%.o,$(SRC))
Filtered out the prepended ${src} and appended .cpp, and keep only
BrokerModule/BrokerApp.cpp & CommandMsgManager/CConfigModuleInfo.cpp.
And % is replaced by the text that matched the % in the previous step.
Patsubst can only handle patterns with one wildcard in it, unluckily. Moreover you are trying to take apart path names not the usual way at the file level. That means, as long as you neither know the prefix nor the postfix parts of the /src/ in your strings, you are out of luck as you can never say 'replace unknown prefix and conserve unknown postfix' (or the other way round).
The usual solution is to 'know' the prefix:
OUT_DIR=Out/
SRC_PATH := ../../../Client2.4/Client/src
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp \
../../../Client2.4/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
OBJ:= $(patsubst $(SRC_PATH)/%,${OUT_DIR}%,$(SRC))
$(info $(OBJ))
Another solution is to use e.g. the GNUmake table toolkit library of make functions (still beta but your problem can be solved):
include gmtt.mk
OUT_DIR=Out
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp \
../../../Client5.6/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
strip-till-last-src = src/$(call implode,$(call down-to,src/,$(call explode,/,$1)))
OBJ:= $(foreach a-path,$(SRC),$(OUT_DIR)/$(call strip-till-last-src,$(a-path)))
$(info $(OBJ))

How does make pass 'VPATH' to the sub-make

Thanks a lot for the comments. In this case actually VPATH is not used.
I write some makefiles to recursively build my project.
Part of the makefile in the root directory looks like this,
VPATH += $(shell find -type d)
export VPATH
#Codes to generate $(ARXIVS), i.e. the list of all local archives,
#is omitted here to save spaces
all : main.o $(ARXIVS)
$(ARXIVS) :
$(MAKE) --directory $(#D)
.PHONY : $(ARXIVS) all
As you can see, I pass the variable VPATH to sub-makes, in order to save some computational cost. And the whole system works.
But the searching paths in a sub-directory is different from the ones in the root directory.
I have print the values of VAPTH in sub-makes, the paths are indeed the ones with respect to the root directory. So VPATH in the sub-makes should be wrong.
But the whole system works. How did this happen?Any one give the suggestion to me. Thanks in advance

Can I have local variables in included makefiles?

I am looking after a system with many hundreds of c files in many folders, there are multiple targets so not all the c files are required in each build.
Because not all files are required I cannot use a pure recursive build. And I don't want to do that, because I would rather not have a slow serial build with mystic dependencies anyway.
What I have come up with broadly is a top level makefile where I pull in a list of folders and include a makefile from each of these folders.
SUB_MAKEFILES := $(foreach subdir,$(SUBDIRS), $(subdir)/subdir.mk)
-include $(SUB_MAKEFILES)
Some folders have specific files to compile so the included file is pretty simple;
srcs += path/a.c path/b.c
deps += path/a.d path/b.d
objs += op/path/a.o op/path/b.o
op/path/%.o: path/%.c path/subdir.mk
compile ...
I do not want to do this dozens of times so I have a generic pattern I use for folders where everything is to be compiled;
PATH155 := src/f1/f2/f3/f4
srcs += $(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')
deps += $(addprefix ${OUT_DIR}, $(patsubst %.c,%.d,$(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')))
objs += $(addprefix ${OUT_DIR}, $(patsubst %.c,%.o,$(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')))
$(OUT_O_DIR)$(PATH155)/%.o: $(PATH155)/%.c $(PATH155)/subdir.mk
gcc -c $(CFLGS) -MF"$(#:%.o=%.d)" -MT"$(#:%.o=%.d)" -o"$#" "$<"
This works, however I would like to avoid having to make up a random unique name for the path where the subdir.mk file resides.
Is there anyway to replace "PATH155673423 := hand/entered/path" with something like "local SUBDIRPWD = $(some function...)".
Then I could just drop in a generic makefile and include it, no error prone typing of paths nor worries that I will get a unique name clash.
It would be nice to have a few less directory scans too, same issue really, need a local variable.
Even some sort of macro for the repeated variable names etc would be better
Maybe there is a way to include the makefiles in a loop instead and set a path variable just before each is included?
Ta
Chris
There is no such thing as variables scoped to a particular makefile, such as you're suggesting. However, you have a lot of options for making this work, from constructed variables to give scope, to using eval to generate functions. For example, something like this:
SUBDIRS := foo bar biz baz
define INCLUDE_FILE
path = $S
include $S
endef
$(foreach S,$(SUBDIRS),$(eval $(INCLUDE_FILE)))
You can find out more about these by looking through this set of blog posts and the associated sections of the GNU make manual.

Dynamic Makefile target

Here's a snippet of a GNU Makefile I'm working with. Basically, I have a directory of images that i want copied into a dist directory when running make, but I'd prefer not to list each image individually.
DISTDIR := dist
IMG := $(shell find app/img -type f)
$(subst app/img, $(DISTDIR), $(filter-out %.svg, $(IMG)))): $(filter-out %.svg, $(IMG))
#cp $? $(DISTDIR)
This sort of works, except any time a source file is changed (in app/img), make always remakes the target file. What's the correct way to handle this situation?
It would be easier to answer if you provided some sample results for the find. Let's say that the find operation returns the files: app/img/foo.svg, app/img/bar.jpg, and app/img/baz.png.
In that case the filter-out functions will return the files app/img/bar.jpg and app/img/baz.png. This means that the rule generated will look like this:
$(DISTDIR)/bar.jpg $(DISTDIR)/baz.png: app/img/bar.jpg app/img/baz.png
#cp $? $(DISTDIR)
Here's the thing: when you define N targets in an explicit rule, it's like defining N rules, one for each target, with all the same prerequisites. So the above is identical to writing this:
$(DISTDIR)/bar.jpg: app/img/bar.jpg app/img/baz.png
#cp $? $(DISTDIR)
$(DISTDIR)/baz.png: app/img/bar.jpg app/img/baz.png
#cp $? $(DISTDIR)
Now you can see why you get the behavior that you do: each target lists ALL the files as prerequisites, so whenever ANY file is changed the target is rebuilt.
The answer is that in make, you should generally try to write rules that create exactly one target from just that target's prerequisites. One simple way to do that is with pattern rules; a pattern rule for the above might look like:
DISTDIR := dist
$(DISTDIR)/% : app/img/%
#cp $< $#
After that, all you have to do is declare a target which depends on the files you want created:
IMG := $(shell find app/img -type f)
all: $(IMG:app/img/%=$(DISTDIR)/%)
and make will figure out the rest!

Resources