Included Makefile's parent directory - makefile

I want to describe each submake's dependencies in a file that a top-level Makefile can include. This is to allows for a recursive make setup (with all of the power of instanced variables and relative pathing) but with all dependencies described in a top-level make to increase compile speed and parallelism.
For instance, let's assume we have a directory tree that looks like this:
project/
|-- lib1
| |-- Makefile
| `-- Makefile.reg
|-- lib2
| |-- Makefile
| `-- Makefile.reg
|-- Makefile
`-- Makefile.reg
The file for project/lib1/Makefile.reg file may look like this:
REG := lib1
include ../Makefile.reg
The file for project/lib2/Makefile.reg file may look like this:
REG := lib2
DEP := lib1
include ../Makefile.reg
The file project/Makefile.reg will look like this:
$(REG)_DIR = ????
$(REG): $(DEP)
$(MAKE) -C $($#_DIR)
And finally, the top-level project/Makefile would look like this:
include $(shell find . -name "Makefile.reg")
Now, the top-level Makefile has all of the dependency information for every target and can intelligently call recursive make and utilize full parallelism while keeping the dependency tree intact.
The issue is that I'm not sure how to let project/Makefile.reg know what the current submake's Makefile.reg's path is. make will always be called from the top-level directory, so $(shell pwd) will always report project/. One solution would be to include the line from this SO answer, however I would like each Makefile.reg to only specify a target and an optional dependency list.
Is there a way to include the directory path discovery in the common project/Makefile.reg rather than putting a new line in every single submake Makefile.reg? In other words, can an included makefile deduce the parent makefile's directory? I see potential in some different parsing of the MAKEFILE_LIST variable.

As the snippet of documentation included in the answer on your linked ticket indicates. The value of MAKEFILE_LIST is updated as make goes. The latest entry in the list is the current makefile which makes the penultimate entry in the list the most recently included makefile before that. If you are willing to assert that the main project/Makefile.reg will only ever be included from a sub-directory makefile then it could simply examine that entry in MAKEFILE_LIST.
Alternatively you could simply define a canned recipe in the main Makefile and call it in each project makefile to define the appropriate targets.
Unfortunately make does make getting the penultimate entry a little more difficult than is pleasant. But the following should work:
FOO=a b c d e f g
BAR=h i j k l m n
BAZ=o p q r s t u
QUX=v w x y z 1 2
penultimateword = $(wordlist $(words $1),$(words $1), x $1)
REG=FOO
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
REG=BAR
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
REG=BAZ
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
REG=QUX
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
all: ;
Inspiration for the above came from the chop function from the fantastic GMSL.

Related

Using Makefiles In Nested Directories

A project that I am working on deals with multiple architectures. The code is divided into architecture specific and general modules (which can be shared between the architectures).
I am planning to organize the code as below
TOP
|
|---- Makefile
|
|---- src (arch independent)
|---- inc (arch independent)
|
|----arch1 (arch specific)
|---- src
|---- inc
|---- Makefile
|----arch2 (arch specific)
|---- src
|---- src
|---- Makefile
The arch specific project would be compiled by the Makefile within the arch directory. This makefile includes the Makefile at the TOP level.
The idea was that in the arch specific makefile will have all the src code from within the arch directory and the top level Makefile defines a variable which is simply the path of the TOP level directory, so that the arch specific Makefile can include code from the src folder in the top level.
However, when including the top level Makefile, the defined path variable (through shell pwd command) is evaluuated at the arch specific directory level.
How can I solve this? Is their a better way of doing this?
Right now it's a 2 level structure. But in future I plan top make it multi level with innermost level being arch specific and the layers become more and more general as one moves out to the top level.
What is your top-level makefile doing? - I mean, can you run make on it standalone? If not then all you really appear to gain from it is the list of source files and header locations - it would be easier to just specify those in a common_make.mk or somthing which only contains:
SOURCES += ../src/test1.c
SOURCES += ../src/test2.c
SOURCES += ../src/test3.c
INC += ../inc
then both of your lower level file include this very basic makefile "snippet" and you don't have to worry about the paths.
Usually I am putting common elements into separate config controlled repos and I have my projects the other way around:
arch1/
|--src
|--inc
|--common (repo)
|--inc
|--src
arch2/
|--src
|--inc
|--common (repo)
|--inc
|--src
So here I have two separate repo's with a shared sub-repo. If you wanted to tie the two archx projects together you can add those to into a super-repo with it's own makefile that just does:
make -C arch1
make -C arch2
I see two options for you
Common makefile template for archX/ directories
# ./Makefile.template
ARCH_DIR := $(PWD)
TOP_DIR := $(ARCH_DIR)/..
BUILD_DIR := $(TOP_DIR)/build
# non-architecture specific source code
NOARCH_SRC_DIR := $(TOP_DIR)/src
NOARCH_INC_DIR := $(TOP_DIR)/inc
# non-architecture specific source code
ARCH_SRC_DIR := $(ARCH_DIR)/src
ARCH_INC_DIR := $(ARCH_DIR)/inc
# other stuff...
CFLAGS := -I$(ARCH_INC_DIR) -I$(NOARCH_INC_DIR)
# directory specific variables, e.g. XXX_SRCS
include $(NOARCH_SRC_DIR)/Makefile.include
include $(ARCH_SRC_DIR)/Makefile.include
# generate object file lists
NOARCH_OBJS := $(NOARCH_SRCS:%.c=$(BUILD_DIR)/%.o)
ARCH_OBJS := $(ARCH_SRCS:%.c=$(BUILD_DIR)/%.o)
# add paths to source files
NOARCH_SRCS := $(addprefix $(NOARCH_SRC_DIR)/, $(NOARCH_SRC_DIR))
ARCH_SRCS := $(addprefix $(ARCH_SRC_DIR)/, $(ARCH_SRC_DIR))
# target-specific pattern rules for C files
$(NOARCH_OBJS): %.o: $(NOARCH_SRC_DIR)/%.c | $(BUILD_DIR)
$(CC) $(CFLAGS) -o $# -c $<
$(ARCH_OBJS): %.o: $(ARCH_SRC_DIR)/%.c | $(BUILD_DIR)
$(CC) $(CFLAGS) -o $# -c $<
... and so ...
ArchX/ directory makefile
# archX/Makefile
# common stuff
include $(PWD)/../Makefile.template
# arch specific targets follow here...
Now you can already run the build using
$ make -C arch1
Top-level directory Makefile
# ./Makefile
ifndef ARCH
$(error you must specifiy ARCH=)
endif
# forward all targets to arch directory
%:
$(MAKE) -C $(ARCH) $(MAKECMDGOAL)
Now you can run the build using
$ make ARCH=arch1
Personally I would not write the top-level makefile at all and use only the former approach.
IMHO you should avoid things like $(shell). But maybe that is just my personal view, take it with a grain of salt...

Can I have local variables in included makefiles?

I am looking after a system with many hundreds of c files in many folders, there are multiple targets so not all the c files are required in each build.
Because not all files are required I cannot use a pure recursive build. And I don't want to do that, because I would rather not have a slow serial build with mystic dependencies anyway.
What I have come up with broadly is a top level makefile where I pull in a list of folders and include a makefile from each of these folders.
SUB_MAKEFILES := $(foreach subdir,$(SUBDIRS), $(subdir)/subdir.mk)
-include $(SUB_MAKEFILES)
Some folders have specific files to compile so the included file is pretty simple;
srcs += path/a.c path/b.c
deps += path/a.d path/b.d
objs += op/path/a.o op/path/b.o
op/path/%.o: path/%.c path/subdir.mk
compile ...
I do not want to do this dozens of times so I have a generic pattern I use for folders where everything is to be compiled;
PATH155 := src/f1/f2/f3/f4
srcs += $(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')
deps += $(addprefix ${OUT_DIR}, $(patsubst %.c,%.d,$(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')))
objs += $(addprefix ${OUT_DIR}, $(patsubst %.c,%.o,$(shell $(FFshell) $(PATH155) -maxdepth 1 -name '*.c')))
$(OUT_O_DIR)$(PATH155)/%.o: $(PATH155)/%.c $(PATH155)/subdir.mk
gcc -c $(CFLGS) -MF"$(#:%.o=%.d)" -MT"$(#:%.o=%.d)" -o"$#" "$<"
This works, however I would like to avoid having to make up a random unique name for the path where the subdir.mk file resides.
Is there anyway to replace "PATH155673423 := hand/entered/path" with something like "local SUBDIRPWD = $(some function...)".
Then I could just drop in a generic makefile and include it, no error prone typing of paths nor worries that I will get a unique name clash.
It would be nice to have a few less directory scans too, same issue really, need a local variable.
Even some sort of macro for the repeated variable names etc would be better
Maybe there is a way to include the makefiles in a loop instead and set a path variable just before each is included?
Ta
Chris
There is no such thing as variables scoped to a particular makefile, such as you're suggesting. However, you have a lot of options for making this work, from constructed variables to give scope, to using eval to generate functions. For example, something like this:
SUBDIRS := foo bar biz baz
define INCLUDE_FILE
path = $S
include $S
endef
$(foreach S,$(SUBDIRS),$(eval $(INCLUDE_FILE)))
You can find out more about these by looking through this set of blog posts and the associated sections of the GNU make manual.

non-recursive make constrict targets

Based on this paper, I'm trying to rework a subset of my build system to be non-recursive. It's actually working pretty well. By default, I have part of my makefile include all the relevant directories via a template:
DIRECTORIES = dirA dirB ... etc ...
define import_template
dir := $(1)
include $(1)/Rules.mk
include Rules.mk
endef
$(foreach DIR,$(DIRECTORIES), \
$(eval $(call import_template,$(DIR))))
Those includes build up a variables like TGT_BIN, a la the paper, that all works.
$ make # does the right thing
However, I want to provide the user the ability to make a subset of those directories. I know I can define DIRECTORIES like:
DIRECTORIES ?= dirA ...
So that:
$ make DIRECTORIES="dirB dirF"
works. But is there a way to write my makefile such that:
$ make -j12 dirB dirF
will do the same thing?
Assuming you have per-directory variables of targets to build (e.g. TGT_dirB, TGT_dirF, etc.) then doing what you want should be as simple as adding:
$(eval $(DIR): $(TGT_$(DIR)))
to the foreach loop like this:
$(eval $(call import_template,$(DIR)))$(eval $(DIR): $(TGT_$(DIR)))
to add all the per-directory targets as pre-requisites of the directory targets.
And adding
.PHONY: $(DIRECTORIES)
somewhere in the makefile to make sure make realizes those are phony targets and don't actually mean the directories themselves.
I would recommend you a non-recursive prorab build system.
It allows you to have independent makefile's in each directory while having a master makefile for everything. So, you'll be able to build only part of your project tree by cd'ing to the right subdir and invoking make.

How do I obtain the location of the previous Makefile/second-last entry of $(MAKEFILE_LIST) [duplicate]

I want to describe each submake's dependencies in a file that a top-level Makefile can include. This is to allows for a recursive make setup (with all of the power of instanced variables and relative pathing) but with all dependencies described in a top-level make to increase compile speed and parallelism.
For instance, let's assume we have a directory tree that looks like this:
project/
|-- lib1
| |-- Makefile
| `-- Makefile.reg
|-- lib2
| |-- Makefile
| `-- Makefile.reg
|-- Makefile
`-- Makefile.reg
The file for project/lib1/Makefile.reg file may look like this:
REG := lib1
include ../Makefile.reg
The file for project/lib2/Makefile.reg file may look like this:
REG := lib2
DEP := lib1
include ../Makefile.reg
The file project/Makefile.reg will look like this:
$(REG)_DIR = ????
$(REG): $(DEP)
$(MAKE) -C $($#_DIR)
And finally, the top-level project/Makefile would look like this:
include $(shell find . -name "Makefile.reg")
Now, the top-level Makefile has all of the dependency information for every target and can intelligently call recursive make and utilize full parallelism while keeping the dependency tree intact.
The issue is that I'm not sure how to let project/Makefile.reg know what the current submake's Makefile.reg's path is. make will always be called from the top-level directory, so $(shell pwd) will always report project/. One solution would be to include the line from this SO answer, however I would like each Makefile.reg to only specify a target and an optional dependency list.
Is there a way to include the directory path discovery in the common project/Makefile.reg rather than putting a new line in every single submake Makefile.reg? In other words, can an included makefile deduce the parent makefile's directory? I see potential in some different parsing of the MAKEFILE_LIST variable.
As the snippet of documentation included in the answer on your linked ticket indicates. The value of MAKEFILE_LIST is updated as make goes. The latest entry in the list is the current makefile which makes the penultimate entry in the list the most recently included makefile before that. If you are willing to assert that the main project/Makefile.reg will only ever be included from a sub-directory makefile then it could simply examine that entry in MAKEFILE_LIST.
Alternatively you could simply define a canned recipe in the main Makefile and call it in each project makefile to define the appropriate targets.
Unfortunately make does make getting the penultimate entry a little more difficult than is pleasant. But the following should work:
FOO=a b c d e f g
BAR=h i j k l m n
BAZ=o p q r s t u
QUX=v w x y z 1 2
penultimateword = $(wordlist $(words $1),$(words $1), x $1)
REG=FOO
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
REG=BAR
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
REG=BAZ
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
REG=QUX
$(REG)_DIR= $(call penultimateword,$($(REG)))
$(info REG_DIR=$($(REG)_DIR))
all: ;
Inspiration for the above came from the chop function from the fantastic GMSL.

Good practices of Makefile

I have a following directory structure in my project:
bin/
dist/
include/
├── module_a/
└── module_b/
Makefile
src/
├── module_a/
└── module_b/
Folder include/ contains *.hpp's while *.cpp's are in src/. I would like to compile all sources to bin/ and then link them up together to dist/. Seems a pretty reasonable wish for me.
I would like to know the best practices for a Makefile for this case. All I can find is %.o: %.cpp target, but that doesn't really work, because of different source and binary folder.
I was trying to use something like this:
D_SRC = src
D_BIN=bin
F_CPP := $(shell find $(D_SRC) -iname '*.cpp' -type f)
F_OBJ := $(shell echo $(F_CPP) | sed s:\ :\\n:g | sed s:$(D_SRC):$(D_BIN): | sed 's:^\(.*\)\.cpp$$:\1\.o:')
$(F_OBJ): $(F_SRC)
$(foreach file, $(F_SRC), \
$(GXX) $(CXXFLAGS) -c $(file)\
)
This target doesn't work, because $(F_OBJ) paths start with bin/, while foreach compiles sources to current working dir. I could make it compile to bin/, but that would happen only with a few more sed expressions and it's ugly enough as it is.
It's probably so difficult for me, because I don't know make all that well, but I cannot be the only one with this project setup. In my opinion, it must be a pretty common one. I know I can write a Makefile for each module separately, but is that really the best choice here?
EDIT: I was now wondering what would I achieve with several Makefiles. If one was at root and another one in src/module_a, how would the latter know about the bin/? If you'd execute it with make -f src/module_a/Makefile, it would be the same as executing it from root directory, 'cause it's working directory would be root. Another way, I guess, would be to change directory before executing it, like so: make -C include/module_a, but in that case, how would it find bin/? I wouldn't want to have something like D_BIN = ../../bin in a Makefile.
What I normally do is have a Makefile in the src directory (which can be invoked from the top level Makefile if you like) and then use rules like this:
D_BIN = ../bin
$(D_BIN)/%.o: %.cpp
You could also experiment with just a makefile in the top level dir, and use rules that look like this:
D_BIN = bin
D_SRC = src
$(D_BIN)/%.o: $(D_SRC)/%.cpp
but I have not used such rules, so I don't know the pros/cons vs the way I normally do it. The way I normally do it works fine, I even have rules that build depends like so:
$(D_BIN)/%.d: %.cpp
and the link rule would be like:
../dist/outexe: $(F_OBJ)
Using a foreach is usually frowned upon because it does not make use of all the features built into normal makefile rules (i.e. there is no depends check on a per file basis, either you build everything or nothing), and as such foreach should only be used as a last resort, but in this case you will be able to get it to work without the foreach.
In addition to this there are much easier ways to build your file lists, you don't need to use the shell or sed.
F_CPP = $(wildcard *.cpp)
F_OBJ = $(F_CPP:.cpp=.o)
Update: This is how I normally issue recursive makes:
SUBDIRS = src
.PHONY: $(SUBDIRS)
all: $(SUBDIRS)
$(SUBDIRS):
#echo "Building $#..."
$(MAKE) -C $# $(MFLAGS)
Then indeed in your submake, you would need to use ../bin for example.
However with a project as simple as yours, you might be better off just having one makefile at the root level and using rules like this:
D_BIN = bin
D_SRC = src
$(D_BIN)/%.o: $(D_SRC)/%.cpp
recursive makefiles are ok (ok but not great) if you have a really complex directory structure, where you will be adding/removing/modifying new dir trees as time goes on. But for a simple project where you just want to have separate directories for code and objs, it is probably overkill.

Resources