How to avoid this recursive Makefile to relink? - makefile

today I'm requesting your help about a Makefile that's driving me crazy. There it is:
# Executable name
NAME = libft.a
# Compiler and archive linker settings
CC = gcc
AR = ar
CFLAGS = -Wall -Wextra -Werror -O3 -g3
ARFLAGS = -rsc
IFLAGS = -I./includes/
# Project layout
SRC_DIR = ./src/
INC_DIR = ./inc/
OBJ_DIR = ./obj/
OBJ = $(shell grep -r .o ./obj | awk '{print $$3}' | tr '\n' ' ')
.PHONY: all clean fclean re
#------------------------------------------------------------------------------#
all: $(OBJ_DIR) $(NAME)
$(OBJ_DIR):
mkdir -p $(OBJ_DIR)
$(NAME): compile $(OBJ) $(INC_DIR)libft.h
#echo "Linking library $(NAME).\n"
#$(AR) $(ARFLAGS) $(NAME) $(OBJ)
#echo " ✧ $(AR) $(ARFLAGS) $(NAME) object files: OK! √\n"
compile:
make -C src/io
make -C src/lists
make -C src/memory
make -C src/strings
make -C src/tests
I've tried multiple combination of dependencies, rules, etc but I just don't get it. Sometimes I got it to stop relinking but in thoses cases it wouldn't re-compile object files because $(OBJ) was empty and wasn't updated after I ran compile.
This version is close to be good, but everytime I run make it executes the recipe $(NAME) and does the ar -rsc %(OBJ) .. How can I put them in dependencies to $(NAME) ?

Well, basically your entire approach here cannot succeed. Just for one example: you are trying to find the object files using grep (honestly I don't understand that shell command at all; what does printing the $3 word from the output of grep -r do??? Don't you just mean find $(OBJ_DIR) -name \*.o here?) This will expand to all the object files found in your subdirectories. But, that shell command runs when your top-level makefile is parsed, and that parsing happens before make runs any rules... so, no object files have been built yet! So, this target doesn't depend on anything. Even after some object files have been built, it only depends on object files that already exist, not on object files that are created during the build.
Really if I were you I'd do this completely differently. However, the simplest way to make your makefile work properly as written is to build $(NAME) using a recursive make as well; change your makefile like this:
all: compile
$(NAME): $(OBJ) $(INC_DIR)libft.h
#echo "Linking library $(NAME).\n"
#$(AR) $(ARFLAGS) $# $^
#echo " ✧ $(AR) $(ARFLAGS) $# object files: OK! √\n"
compile:
mkdir -p $(OBJ_DIR)
$(MAKE) -C src/io
$(MAKE) -C src/lists
$(MAKE) -C src/memory
$(MAKE) -C src/strings
$(MAKE) -C src/tests
$(MAKE) $(NAME)
Here all doesn't depend on $(NAME); instead, the compile step first builds everything then at the end it recursively invokes itself to build $(NAME); at this point we know everything is up to date and we can depend on the object files existing.
Other things: note I used the automatic variable $^ here not $(OBJ); that variable is a simple variable that runs a shell script: it's expensive! Every time you expand the $(OBJ) variable you pay that cost, so you only ever want to do it one time. Alternatively, you can use := to set OBJS instead so it's only invoked once per make instance. That's still one more time than you need but avoiding this will be painful.
I also moved the mkdir into the compile rule. It's cleaner there than as a prerequisite of all.
Finally, you should never invoke sub-makes using the make command directly. Always use the $(MAKE) variable, or various things will not work correctly.

The question was obvioulsy solved by the previous post.
You need to use the $(MAKE) variable to call recursively your make file with the $(NAME) rule instead of putting $(NAME) as a all dependency, after subsequent calls to your underlying Makefiles using the $(MAKE) variable again.

Related

Evolving a Makefile From Flat Directory Structure to Sub-Directory Structure

SEE UPDATES BELOW
Research Done: I'm finding learning how to evolve Makefiles from one situation to another is difficult. There are a ton of questions and answers out there but few of them actually show how a Makefile can evolve as your project changes. They also all seem to use various different techniques and idioms of Makefiles so translating between one question and another can be tricky when you are learning Makefiles for the first time, as I am.
Problem: My problem is that I have a project that started at as a flat directory structure but then is migrating to a structure with sub-directories. What I can't do is get my Makefile to along for the ride.
First I'll show what I created that works and then I show how I want it to evolve and how that doesn't work.
Flat Directory Structure, Working Makefile
I have project directory that has all my C files and one header file plus my Makefile:
project
Makefile
c8_asm.c
c8_dasm.c
c8_terp.c
chip8.h
Here is my Makefile (which works just fine):
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
# Targets
all: c8_dasm c8_asm c8_terp
c8_dasm: c8_dasm.o
$(CC) $(LDLIBS) c8_dasm.o -o $#
c8_asm: c8_asm.o
$(CC) $(LDLIBS) c8_asm.o -o $#
c8_terp: c8_terp.o
$(CC) $(LDLIBS) c8_terp.o -o $#
# Using implicit rules for updating an '.o' file from a correspondingly
# named '.c' file.
c8_dasm.o: chip8.h
c8_asm.o: chip8.h
c8_terp.o: chip8.h
.PHONY: clean
clean:
rm c8_dasm c8_asm c8_terp c8_dasm.o c8_asm.o c8_terp.o
I get all my .o files and my executables are created in the project directory.
Evolving The Project
But what I wanted to do is have my sources files (all .c and .h) in a src directory. I wanted to build into an obj directory and have the executables go in a bin directory. So my project would look like this:
project
src
c8_asm.c
c8_dasm.c
c8_terp.c
chip8.h
Makefile
Sub-Directory Structure, Makefile NOT Working
To accommodate the above, I changed my Makefile accordingly:
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
SRC_DIR = src
OBJ_DIR = obj
BIN_DIR = bin
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
MKDIR_P ?= mkdir -p
# Targets
all: $(BIN_DIR)/c8_dasm $(BIN_DIR)/c8_asm $(BIN_DIR)/c8_terp
$(BIN_DIR)/c8_dasm: $(OBJ_DIR)/c8_dasm.o
$(CC) $(LDLIBS) $(OBJ_DIR)/c8_dasm.o -o $#
$(BIN_DIR)/c8_asm: $(OBJ_DIR)/c8_asm.o
$(CC) $(LDLIBS) $(OBJ_DIR)/c8_asm.o -o $#
$(BIN_DIR)/c8_terp: $(OBJ_DIR)/c8_terp.o
$(MKDIR_P) $(dir $#)
$(CC) $(LDLIBS) $(OBJ_DIR)/c8_terp.o -o $#
$(OBJECTS): $(OBJ_DIR)/%.o : $(SRC_DIR)/%.c
$(MKDIR_P) $(dir $#)
$(CC) $< -o $(OBJ_DIR)/$#
# Using implicit rules for updating an '.o' file from a correspondingly
# named '.c' file.
$(OBJ_DIR)/c8_dasm.o: $(SRC_DIR)/chip8.h
$(OBJ_DIR)/c8_asm.o: $(SRC_DIR)/chip8.h
$(OBJ_DIR)/c8_terp.o: $(SRC_DIR)/chip8.h
.PHONY: clean
clean:
rm -r $(BUILD_DIR)
rm $(OBJECTS)
Upon running this I get the following:
mkdir -p obj/obj/
gcc src/c8_dasm.c -o obj/c8_dasm.o
gcc -lm obj/c8_dasm.o -o bin/c8_dasm
ld: can't link with a main executable file 'obj/c8_dasm.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [bin/c8_dasm] Error 1
I wanted to stop here and get some assistance because I fear I'm making this Makefile for complicated than it need be and I'm trying to avoid getting into bad habits.
I'm hoping to hear opinions about what I'm not conceptualizing correctly here.
FIRST UPDATE
I managed to take it bit by bit and get it mostly working. Here is what I ended up with:
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
# Directories.
SRC_DIR = src
BIN_DIR = bin
$(shell mkdir -p $(BIN_DIR))
# Patterns for files.
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(SRC_DIR)/%.o)
EXECUTABLES := c8_dasm c8_asm c8_terp
# Targets
all: $(EXECUTABLES)
c8_dasm: $(SRC_DIR)/c8_dasm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Disassembler Built"
c8_asm: $(SRC_DIR)/c8_asm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Assembler Built"
c8_terp: $(SRC_DIR)/c8_terp.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Interpreter Built"
# Using implicit rules for updating an '.o' file from a correspondingly
# named '.c' file.
c8_dasm.o: $(SRC_DIR)/chip8.h
c8_asm.o: $(SRC_DIR)/chip8.h
c8_terp.o: $(SRC_DIR)/chip8.h
.PHONY: clean
clean:
rm $(OBJECTS)
rm -r $(BIN_DIR)
Of course, as I'm finding with Make this leads to other obscure problems. For example doing this:
make
make clean
works fine. Meaning all files are generated and the files are cleaned, including the bin directory.
However, if I do this:
make c8_dasm
make clean
This builds fine. But the clean fails to delete the bin directory (although it does delete the object files). This happens regardless of what individual executable I try to build.
No amount of searching is helping me find out why that is.
SECOND UPDATE
I found that problem was solved as well. It just required using the "-f" for the rm statements in the clean target.
THIRD UPDATE
To get the object file directory part working, I tried (from this: path include and src directory makefile) to construct my Makefile as follows:
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
SRC_DIR = src
OBJ_DIR = obj
BIN_DIR = bin
$(shell mkdir -p $(BIN_DIR))
$(shell mkdir -p $(OBJ_DIR))
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
EXECUTABLES := c8_dasm c8_asm c8_terp
all: $(EXECUTABLES)
c8_dasm: $(SRC_DIR)/c8_dasm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Disassembler Built"
c8_asm: $(SRC_DIR)/c8_asm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Assembler Built"
c8_terp: $(SRC_DIR)/c8_terp.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Interpreter Built"
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.c
$(CC) $(CFLAGS) -c $< -o $(BIN_DIR)/$#
.PHONY: clean
clean:
rm -rf $(BIN_DIR)
rm -f $(OBJECTS)
I was able to condense the original three lines using chip8.h into one target but I have no way to know if that's correct. It compiles at least. I also changed the OBJECTS line to reflect the new OBJ_DIR I created.
However, this doesn't put the object files in the right place. It still puts them in the src directory rather than the obj directory.
This is why it makes sense to not do anything complicated with Makefiles. Just put the actual directory names in your commands. Never rely on wildcards.
People using C and C++ and using Makefiles spend too much time trying to get those to work rather than just actually getting things done. That's why you see so many of the questions that you see and why the answers vary so much.
In your specific case, your targets don't always have to contain the directory and that's part of the problem. The rules getting generated don't have an actual target in your file because of the directories you are prepending to everything. You have to think in terms of what is getting generated by each target: meaning, the output. So if c8_dasm is getting output, that's your target. The directory has nothing to do with that. So you need to remove all of your directory substitutions where they aren't needed.
But before doing that, ask yourself this: if your first solution was working, why change it? It's better to not even do directories when you're using Make. Just have everything in the same directory as you started off with. You can even see that this allows your Makefile to be much cleaner.
I believe I may have figured this out. Below is my Makefile. It seems to do what I want. It does the following:
Compiles all object files into the obj directory.
Compiles and links so that executables are generated in the bin directory.
Recognizes if any .c files are changed and recompiles accordingly.
Recognizes if the .h file is changed and recompiles all C files that reference it.
This seems to satisfy all the criteria but I can't tell if I've painted myself into some corner that I can't see yet.
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
SRC_DIR = src
OBJ_DIR = obj
BIN_DIR = bin
$(shell mkdir -p $(BIN_DIR))
$(shell mkdir -p $(OBJ_DIR))
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
EXECUTABLES := c8_dasm c8_asm c8_terp
all: $(EXECUTABLES)
c8_dasm: $(OBJ_DIR)/c8_dasm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Disassembler Built"
c8_asm: $(OBJ_DIR)/c8_asm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Assembler Built"
c8_terp: $(OBJ_DIR)/c8_terp.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Interpreter Built"
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.c $(SRC_DIR)/chip8.h
$(CC) $(CFLAGS) -c $< -o $#
.PHONY: clean
clean:
rm -rf $(BIN_DIR)
rm -rf $(OBJ_DIR)
Stackoverflow is whining about too many comments, so I'll make this another "answer." After our back-and-forth to my original comment, your last comment is correct. That's what I wanted you to see.
Understand that you can't use Make to do what you want to do exactly.
So here's really the answer: You can't create multiple executables AND with only some of the object files applying to each one AND while using a directory structure. Make is in no way capable of handling that.
Right now you're trying to use Make in a way that it wasn't intended for which is why you're running into so many problems. If you keep playing around you're going to run into is a series of errors that say "duplicate symbol" because you will be compiling each of your files multiple times for each executable, assuming you follow most of the advice you'll find.
Check out this How can I create a Makefile for C projects with SRC, OBJ, and BIN subdirectories? to see what I mean. That one works because all object files are being used to create a single executable. But as you've stated, that's not going to be the case for you. And that's what Make can't handle. That's why you're not finding an answer to that.
And while your chip8.h file is now not going to cause problems in terms of allowing you to compile, your Makefile with that third update would not recognize when the chip8.h file itself has changed. You would have to change a .c file to force a recompile so that changes to your .h were recognized. So you either have to stick with your second update or use something other than Make.

Why is makefile exhibiting non-deterministic behaviour?

I have a makefile that is trying to do the following: identify all files under the current directory (all sub-directories included) with .c and .s extensions, for each one compile a non-linked object file and put it into a directory. All C files end up in objects/c, all assembly files end up in objects/ass.
The makefile always works as expected on the first execution (all commands are called in the right order) and no errors are produced.
However if I call make again, half of the time i get "nothing to be done for 'all'.". Which is what you would expect, since no files have been modified. But the other half of the time, make is selecting a random assembly file and compiling that file. That is to say,if I keep doing "make" I sometimes compile file1.s sometimes file2.s. and it keeps randomly swapping between the assembly files add infinitum (it never reaches a "nothing to be done") state.
How is make exhibitting non deterministic behaviour?
This is the smallest makefile I could make that reproduces the error:
SRC_C = $(wildcard *.c) $(wildcard **/*.c)
SRC_ASS = $(wildcard *.s) $(wildcard **/*.s)
OBJECTS_C = $(addprefix $(OBJECT_DIR)c/, $(notdir $(SRC_C:.c=.o)))
OBJECTS_ASS = $(addprefix $(OBJECT_DIR)ass/, $(notdir $(SRC_ASS:.s=.o)))
OBJECTS = $(OBJECTS_C) $(OBJECTS_ASS)
OBJECT_DIR = objects/
all: $(OBJECTS)
%/:
mkdir $#
$(OBJECTS_C): $(OBJECT_DIR) $(OBJECT_DIR)c/
arm-none-eabi-gcc -O0 -march=armv8-a $(wildcard */$(#F:.o=.c)) -nostartfiles -c -o $#
$(OBJECTS_ASS): $(OBJECT_DIR) $(OBJECT_DIR)ass/
arm-none-eabi-as -march=armv8-a $(wildcard */$(#F:.o=.s)) -c -o $#
.PHONY: clean
clean:
rm -rf $(OBJECT_DIR)
You have many errors here.
The biggest is a conceptual one: By flattening all your object files into one directory, there's no way to express proper dependencies using pattern rules, so your object files do not really depend on their respective source files. I'd say: just don't do that! Having object directories is fine, but they should mirror the directory structure of the source tree.
Further errors:
directly depending on directories. This will not work as expected, directories should always be order-only dependencies, as already stated in the comments
Make doesn't support recursive wildcards -- if you really need that, you could write your own function or, assuming you're always building on *nix, just call find instead
Pattern rules for creating directories are not the best idea either -- I'd suggest to collect all needed directories in a variable and loop over that.
Stylistic improvements:
Assign variables that don't need deferred evaluation with :=
Assign variables influencing the build process with ?=, so the user can override them at the command line
Use "standard" variables like CC, AS, CROSS_COMPILE
declare all phony targets in .PHONY.
Your Makefile with these changes applied would look like this:
OBJECT_DIR ?= objects
C_OBJECT_DIR ?= $(OBJECT_DIR)/c
AS_OBJECT_DIR ?= $(OBJECT_DIR)/ass
SRC_C:= $(shell find -name \*.c)
SRC_ASS:= $(shell find -name \*.s)
OBJECTS_C:= $(addprefix $(C_OBJECT_DIR)/, $(SRC_C:.c=.o))
OBJECTS_ASS:= $(addprefix $(AS_OBJECT_DIR)/, $(SRC_ASS:.s=.o))
OBJECTS:= $(OBJECTS_C) $(OBJECTS_ASS)
OUTDIRS:= $(sort $(dir $(OBJECTS)))
CROSS_COMPILE ?= arm-none-eabi-
CC ?= gcc
AS ?= as
CFLAGS ?= -O0 -march=armv8-a -nostartfiles
ASFLAGS ?= -march=armv8-a
all: $(OBJECTS)
$(OUTDIRS):
$(foreach _dir,$#,mkdir -p $(_dir);)
$(C_OBJECT_DIR)/%.o: %.c | $(OUTDIRS)
$(CROSS_COMPILE)$(CC) -c -o $# $(CFLAGS) $<
$(AS_OBJECT_DIR)/%.o: %.s | $(OUTDIRS)
$(CROSS_COMPILE)$(AS) -c -o $# $(ASFLAGS) $<
clean:
rm -rf $(OBJECT_DIR)
.PHONY: all clean
Note there is one important thing missing: automatic dependencies. With this Makefile, each object file depends on its respective source file, but completely misses any headers included. For anything other than a simple toy, you should add that, google for "gnu make gcc automatic dependencies" or something similar (not the scope of this question).

Building C-program "out of source tree" with GNU make

I would like to build a C-project for my microcontroller with the GNU make tool. I would like to do it in a clean way, such that my source code is not cluttered with object files and other stuff after the build. So imagine that I have a project folder, called "myProject" with two folders in it:
- myProject
|
|---+ source
|
'---+ build
The build folder only contains a makefile. The figure below shows what should happen when I run the GNU make tool:
So GNU make should create an object file for each .c source file it can find in the source folder. The object files should be structured in a directory tree that is similar to the structure in the source folder.
GNU make should also make a .d dependency file (in fact, a dependency file is some sort of makefile itself) for each .c source file. The dependency file is described in the GNU make manual chapter 4.14 "Generating Prerequisites Automatically":
For each source file name.c there is a makefile name.d which lists
what files the object file name.o depends on.
From the following Stackoverflow question About the GNU make dependency files *.d, I learned that adding the options -MMD and -MP to the CFLAGS of the GNU gcc compiler can help to automate that.
So now comes the question. Has anyone a sample makefile that performs such out-of-source build? Or some good advices on how to get started?
I'm pretty sure that most people who have written such a makefile, are Linux-people. But the microcontroller project should build also on a Windows machine. Anyway, even if your makefile is Linux-only, it provides a good starting point ;-)
PS: I would like to avoid extra tools like CMake, Autotools, or anything that has to do with an IDE. Just pure GNU make.
I would be very grateful :-)
Updating the dependency files
Please have a look at this question: What is the exact chain of events when GNU make updates the .d files?
Here's the Makefile I've added to the documentation (currently in review so I'll post it here) :
# Set project directory one level above the Makefile directory. $(CURDIR) is a GNU make variable containing the path to the current working directory
PROJDIR := $(realpath $(CURDIR)/..)
SOURCEDIR := $(PROJDIR)/Sources
BUILDDIR := $(PROJDIR)/Build
# Name of the final executable
TARGET = myApp.exe
# Decide whether the commands will be shown or not
VERBOSE = TRUE
# Create the list of directories
DIRS = Folder0 Folder1 Folder2
SOURCEDIRS = $(foreach dir, $(DIRS), $(addprefix $(SOURCEDIR)/, $(dir)))
TARGETDIRS = $(foreach dir, $(DIRS), $(addprefix $(BUILDDIR)/, $(dir)))
# Generate the GCC includes parameters by adding -I before each source folder
INCLUDES = $(foreach dir, $(SOURCEDIRS), $(addprefix -I, $(dir)))
# Add this list to VPATH, the place make will look for the source files
VPATH = $(SOURCEDIRS)
# Create a list of *.c sources in DIRS
SOURCES = $(foreach dir,$(SOURCEDIRS),$(wildcard $(dir)/*.c))
# Define objects for all sources
OBJS := $(subst $(SOURCEDIR),$(BUILDDIR),$(SOURCES:.c=.o))
# Define dependencies files for all objects
DEPS = $(OBJS:.o=.d)
# Name the compiler
CC = gcc
# OS specific part
ifeq ($(OS),Windows_NT)
RM = del /F /Q
RMDIR = -RMDIR /S /Q
MKDIR = -mkdir
ERRIGNORE = 2>NUL || true
SEP=\\
else
RM = rm -rf
RMDIR = rm -rf
MKDIR = mkdir -p
ERRIGNORE = 2>/dev/null
SEP=/
endif
# Remove space after separator
PSEP = $(strip $(SEP))
# Hide or not the calls depending of VERBOSE
ifeq ($(VERBOSE),TRUE)
HIDE =
else
HIDE = #
endif
# Define the function that will generate each rule
define generateRules
$(1)/%.o: %.c
#echo Building $$#
$(HIDE)$(CC) -c $$(INCLUDES) -o $$(subst /,$$(PSEP),$$#) $$(subst /,$$(PSEP),$$<) -MMD
endef
# Indicate to make which targets are not files
.PHONY: all clean directories
all: directories $(TARGET)
$(TARGET): $(OBJS)
$(HIDE)echo Linking $#
$(HIDE)$(CC) $(OBJS) -o $(TARGET)
# Include dependencies
-include $(DEPS)
# Generate rules
$(foreach targetdir, $(TARGETDIRS), $(eval $(call generateRules, $(targetdir))))
directories:
$(HIDE)$(MKDIR) $(subst /,$(PSEP),$(TARGETDIRS)) $(ERRIGNORE)
# Remove all objects, dependencies and executable files generated during the build
clean:
$(HIDE)$(RMDIR) $(subst /,$(PSEP),$(TARGETDIRS)) $(ERRIGNORE)
$(HIDE)$(RM) $(TARGET) $(ERRIGNORE)
#echo Cleaning done !
Main features
Automatic detection of C sources in specified folders
Multiple source folders
Multiple corresponding target folders for object and dependency files
Automatic rule generation for each target folder
Creation of target folders when they don't exist
Dependency management with gcc : Build only what is necessary
Works on Unix and DOS systems
Written for GNU Make
How to use this Makefile
To adapt this Makefile to your project you have to :
Change the TARGET variable to match your target name
Change the name of the Sources and Build folders in SOURCEDIR and BUILDDIR
Change the verbosity level of the Makefile in the Makefile itself or in make call (make all VERBOSE=FALSE)
Change the name of the folders in DIRS to match your sources and build folders
If required, change the compiler and the flags
In this Makefile Folder0, Folder1 and Folder2 are the equivalent to your FolderA, FolderB and FolderC.
Note that I have not had the opportunity to test it on a Unix system at the moment but it works correctly on Windows.
Explanation of a few tricky parts :
Ignoring Windows mkdir errors
ERRIGNORE = 2>NUL || true
This has two effects :
The first one, 2>NUL is to redirect the error output to NUL, so as it does not comes in the console.
The second one, || true prevents the command from rising the error level. This is Windows stuff unrelated with the Makefile, it's here because Windows' mkdir command rises the error level if we try to create an already-existing folder, whereas we don't really care, if it does exist that's fine. The common solution is to use the if not exist structure, but that's not UNIX-compatible so even if it's tricky, I consider my solution more clear.
Creation of OBJS containing all object files with their correct path
OBJS := $(subst $(SOURCEDIR),$(BUILDDIR),$(SOURCES:.c=.o))
Here we want OBJS to contain all the object files with their paths, and we already have SOURCES which contains all the source files with their paths.
$(SOURCES:.c=.o) changes *.c in *.o for all sources, but the path is still the one of the sources.
$(subst $(SOURCEDIR),$(BUILDDIR), ...) will simply subtract the whole source path with the build path, so we finally have a variable that contains the .o files with their paths.
Dealing with Windows and Unix-style path separators
SEP=\\
SEP = /
PSEP = $(strip $(SEP))
This only exist to allow the Makefile to work on Unix and Windows, since Windows uses backslashes in path whereas everyone else uses slashes.
SEP=\\ Here the double backslash is used to escape the backslash character, which make usually treats as an "ignore newline character" to allow writing on multiple lines.
PSEP = $(strip $(SEP)) This will remove the space char of the SEP variable, which has been added automatically.
Automatic generation of rules for each target folder
define generateRules
$(1)/%.o: %.c
#echo Building $$#
$(HIDE)$(CC) -c $$(INCLUDES) -o $$(subst /,$$(PSEP),$$#) $$(subst /,$$(PSEP),$$<) -MMD
endef
That's maybe the trick that is the most related with your usecase. It's a rule template that can be generated with $(eval $(call generateRules, param)) where param is what you can find in the template as $(1).
This will basically fill the Makefile with rules like this for each target folder :
path/to/target/%.o: %.c
#echo Building $#
$(HIDE)$(CC) -c $(INCLUDES) -o $(subst /,$(PSEP),$#) $(subst /,$(PSEP),$<) -MMD
This fairly minimal makefile should do the trick:
VPATH = ../source
OBJS = FolderA/fileA1.o FolderA/fileA2.o FolderB/fileB1.o
CPPFLAGS = -MMD -MP
all: init myProgram
myProgram: $(OBJS)
$(CC) $(LDFLAGS) -o $# $(OBJS) $(LDLIBS)
.PHONY: all init
init:
mkdir -p FolderA
mkdir -p FolderB
-include $(OBJS:%.o=%.d)
The main tricky part is ensuring that FolderA and FolderB exist in the build directory bfore trying to run the compiler that will write into them. The above code will work sequential for builds, but might fail with -j2 the first time it is run, as the compiler in one thread might try to open an output file before the other thread creates the directory. Its also somewhat unclean. Usually with GNU tools you have a configure script that will create those directories (and the makefile) for you before you even try to run make. autoconf and automake can build that for you.
An alternate way that should work for parallel builds would be to redefine the standard rule for compiling C files:
VPATH = ../source
OBJS = FolderA/fileA1.o FolderA/fileA2.o FolderB/fileB1.o
CPPFLAGS = -MMD -MP
myProgram: $(OBJS)
$(CC) $(LDFLAGS) -o $# $(OBJS) $(LDLIBS)
%.o: %.c
mkdir -p $(dir $#)
$(CC) $(CFLAGS) $(CPPFLAGS) -c -o $# $<
-include $(OBJS:%.o=%.d)
Which has the disadvantage that you'll also need to redefine the builtin rules for any other kind of sourcefile you want to compile
Here's a basic one I use all the time, it's pretty much a skeleton as it is but works perfectly fine for simple projects. For more complex projects it certainly needs to be adapted, but I always use this one as a starting point.
APP=app
SRC_DIR=src
INC_DIR=inc
OBJ_DIR=obj
BIN_DIR=bin
CC=gcc
LD=gcc
CFLAGS=-O2 -c -Wall -pedantic -ansi
LFLGAS=
DFLAGS=-g3 -O0 -DDEBUG
INCFLAGS=-I$(INC_DIR)
SOURCES=$(wildcard $(SRC_DIR)/*.c)
HEADERS=$(wildcard $(INC_DIR)/*.h)
OBJECTS=$(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
DEPENDS=$(OBJ_DIR)/.depends
.PHONY: all
all: $(BIN_DIR)/$(APP)
.PHONY: debug
debug: CFLAGS+=$(DFLAGS)
debug: all
$(BIN_DIR)/$(APP): $(OBJECTS) | $(BIN_DIR)
$(LD) $(LFLGAS) -o $# $^
$(OBJ_DIR)/%.o: | $(OBJ_DIR)
$(CC) $(CFLAGS) $(INCFLAGS) -o $# $<
$(DEPENDS): $(SOURCES) | $(OBJ_DIR)
$(CC) $(INCFLAGS) -MM $(SOURCES) | sed -e 's!^!$(OBJ_DIR)/!' >$#
ifneq ($(MAKECMDGOALS),clean)
-include $(DEPENDS)
endif
$(BIN_DIR):
mkdir -p $#
$(OBJ_DIR):
mkdir -p $#
.PHONY: clean
clean:
rm -rf $(BIN_DIR) $(OBJ_DIR)
I would avoid manipulating Makefile directly, and use CMake instead.
Just describe your source files in CMakeLists.txt, as below:
Create file MyProject/source/CMakeLists.txt containing;
project(myProject)
add_executable(myExec FolderA/fileA1.c FolderA/fileA2.c FolderB/fileB1.c)
Under MyProject/build, run
cmake ../source/
You'll get a Makefile now. To build, under the same build/ directory,
make
You may also want to switch to a lightning fast build tool, ninja, simply by adding a switch as following.
cmake -GNinja ..
ninja

Makefile - a command in a command?

I have an embarrassingly simple makefile question but I can't google it due to lack of knowledge - I don't know the words for things I don't know.
Basically, I want to run the makefile in the current directory, look into the ./SRC directory for source files and when everything is finished, move the object files into the ./OBJ directory.
Makefile:
move_obj:
mv -f -t ./OBJ_DIR ./$(OBJ_FILES)
file.o: other_file.h
$(CC) $(CFLAGS) $(CPPFLAGS) -c file.c
move_obj
I want to call "move_obj" after compiling the source files but since I don't know what
result: dependency
evaluation
actually represents (and all makefile introduction guides I've found state "This is what a makefile looks like, off you go then"), I don't know why this isn't working. I assume I need some evaluate command or need to define a function or...?
Thanks for any help in advance.
You can do this by creating another rule for example move, like below
all: $(EXECUTABLE) move
$(EXECUTABLE): $(OBJECTFILES)
$(CC) -o $# $<
$(OBJECTFILES): $(SOURCEFILES)
$(CC) $(CFLAGS) -c -o $# -I $(INCLUDE_PATH) $<
# Move the .o to Object directory #
move:
$(MV) $(OBJECTFILES) $(OBJECT_PATH)
But by doing the above, you will defeat the purpose of the Makefile.
Since your rule is dependent on .o, Make will look for .o in current directory and not find it (because you've moved it) and thus rebuild.
To avoid this, you should output it to ./obj directory and use it from there.
Something like
gcc -g -Wall -o obj/foo.o -c src/foo.c -I ./include
gcc -g -Wall -o obj/main.o -c src/main.c -I ./include
gcc -o exe obj/foo.o obj/main.o -lanylibrary
Below is the makefile doing the same.
C_FLAGS := -g -Wall -Wextra
CC := gcc
RM := rm
LINKFLAGS := -lanylibrary
.PHONY: $(TARGET) clean
VPATH:= ./src/ ./obj/ ./include/
# Path for .c , .h and .o Files
SRC_PATH := ./src/
OBJ_PATH := ./obj/
INC_PATH := -I ./include
# Executable Name
TARGET := exe
# Files to compile
OBJ1 := foo.o \
main.o
OBJ := $(patsubst %,$(OBJ_PATH)%,$(OBJ1))
# Build .o first
$(OBJ_PATH)%.o: $(SRC_PATH)%.c
#echo [CC] $<
#$(CC) $(C_FLAGS) -o $# -c $< $(INC_PATH)
# Build final Binary
$(TARGET): $(OBJ)
#echo [INFO] Creating Binary Executable [$(TARGET)]
#$(CC) -o $# $^ $(LINKFLAGS)
# Clean all the object files and the binary
clean:
#echo "[Cleaning]"
#$(RM) -rfv $(OBJ_PATH)*
#$(RM) -rfv $(TARGET)
Refer to this answer for a better understanding
EDIT:
You can also output your executable to directory, add the following changes to your Makefile.
Ensure that the bin directory is created beforehand, and not deleted by clean.
# Path for .c , .h and .o Files, and ./bin directory
BIN_PATH := ./bin
# Executable Name
TARGET := $(BIN_PATH)/exe
# Clean all the object files and the binary
clean:
#echo "[Cleaning]"
#$(RM) -rfv $(OBJ_PATH)*
#$(RM) -fv $(TARGET)
If you want to build a target(move_obj) after another(file.o), add the move_obj to the dependency list of file.o so that the commands under the move_obj will be executed.
So your Makefile should be:
file.o: other_file.h move_obj
$(CC) $(CFLAGS) $(CPPFLAGS) -c file.c
move_obj:
mv -f -t ./OBJ_DIR ./$(OBJ_FILES)
As Colonel Thirty Two mentioned in the comment section, instead of compiling and then move, you can build the object files in the required directory
file.o: other_file.h
$(CC) $(CFLAGS) $(CPPFLAGS) -c file.c -o ./$(OBJ_FILES)/$#
This is flawed in various ways.
result normally is an actual file that should be present after the recipe is executed. If the file is already there and is not older than any of its dependencies, make does nothing. So instead of creating a file somewhere and then moving it around with another rule, make sure the rule creates it where it should FINALLY be. Otherwise make can never check whether it has to rebuild it (and always will). In this case, use the -o flag of the compiler to directly create it where it should be (e.g. -o $(OBJ_DIR)/file.o)
dependency should list ALL files that are needed to build the result, so make really rebuilds it if ANY of these files changed. In your case, at least file.c is missing from the dependency list
In order to place files in a directory, you should make sure it exists. you could do it like this:
$(OBJ_DIR):
mkdir -p $(OBJ_DIR)
$(OBJ_DIR)/file.o: $(OBJ_DIR) file.c other_file.h
$(CC) $(CFLAGS) $(CPPFLAGS) -c file.c -o $(OBJ_DIR)/file.o
Your move_obj recipe, although not suitable in this case, would be a PHONY target, meaning it does not create a file. If you need such rules, mark them accordingly by mentioning them as dependency of the special target .PHONY:
.PHONY: move_obj
The reason for this is that you could (by accident) have a file named move_obj in your working directory. In that case, make would decide there's nothing to do for move_obj, and this is not what you want. Marking it as phony tells make that this rule does not create its target and the recipe must be executed no matter what.
All in all, your question comes down to misunderstanding a Makefile as kind of a script. It is not. It's a declarative file that tells make what has to be done in order to build files (your evaluation block) and when this needs to be done (your dependency block). It's better not to try to misuse a Makefile as a script.

Makefile with Fortran - src and bin directories

I'm having some trouble understanding how to design my makefile to build my project the way I want to. Specifically, I can't figure out how to keep all source files in a src directory, while putting all binaries in a bin directory except the linked executable, which goes in the project root.
This is my makefile:
# Compiler options
FC := mpif90
FFLAGS := -O3 -g -Wall -Warray-bounds -ffixed-line-length-none -fbounds-check
VPATH := src
BINDIR := bin
# Define file extensions
.SUFFIXES:
.SUFFIXES: .f .o .mod
# All modules
OBJS := $(BINDIR)/ratecoeffs.o $(BINDIR)/interpolation.o $(BINDIR)/io.o $(BINDIR)/eedf.o $(BINDIR)/single_particle.o $(BINDIR)/physics.o $(BINDIR)/random.o $(BINDIR)/mpi.o $(BINDIR)/precision.o $(BINDIR)/populations.o
# Build rules
all: runner | $(BINDIR)
$(BINDIR):
mkdir -p $(BINDIR)
$(BINDIR)/%.o: $(VPATH)/%.f | $(BINDIR)
$(FC) $(FFLAGS) -c $^ -o $#
runner: $(OBJS)
clean:
#rm -rf $(BINDIR)
Running make builds everything allright - it finds all source files in src and puts all .o files in bin - but the module files (.mod) that are generated by the compiler are put in the project root instead of in the bin directory. I realize I could just specify a rule to place them there, but that messes with the build order, and will sometimes break the build.
What is the "correct" way to get this behavior?
And yes, I've looked at autotools and automake, but I've never used them before and they seem to be overkill for this project. As I couldn't find any good tutorials on how they work (no, I didn't like the tutorial on gnu.org) I'd prefer if I could avoid having to learn this tool just to get this work...
Assuming your underlying Fortran compiler is gfortran, use the -J command line option.
$(FC) $(FFLAGS) -c $^ -o $# -J$(BINDIR)
With an eye to the future, you may be better off creating a MODDIR or similar variable, that you use instead of BINDIR. Object code (*.o) and mod files have different roles to play in later compilation and linking steps - in larger projects they are often kept separate.
It would be probably more in the sense of the make system to change into the obj-directory and do the compilation from there. Via the VPATH option you can let make to find your source files automatically. You could easily call your makefile recursively from the right directory. Below you find a trivial example which would be straightforward to adapt to your case. Please note, that it only works with GNU make.
ifeq (1,$(RECURSED))
VPATH = $(SRCDIR)
########################################################################
# Project specific makefile
########################################################################
FC = gfortran
FCOPTS =
LN = $(FC)
LNOPTS =
OBJS = accuracy.o eqsolver.o io.o linsolve.o
linsolve: $(OBJS)
$(LN) $(LNOPTS) -o $# $^
%.o: %.f90
$(FC) $(FCOPTS) -c $<
.PHONY: clean realclean
clean:
rm -f *.mod *.o
realclean: clean
rm -f linsolve
accuracy.o:
eqsolver.o: accuracy.o
io.o: accuracy.o
linsolve.o: accuracy.o eqsolver.o io.o
else
########################################################################
# Recusive invokation
########################################################################
BUILDDIR = _build
LOCALGOALS = $(BUILDDIR) distclean
RECURSIVEGOALS = $(filter-out $(LOCALGOALS), $(MAKECMDGOALS))
.PHONY: all $(RECURSIVE_GOALS) distclean
all $(RECURSIVEGOALS): $(BUILDDIR)
+$(MAKE) -C $(BUILDDIR) -f $(CURDIR)/GNUmakefile SRCDIR=$(CURDIR) \
RECURSED=1 $(RECURSIVEGOALS)
$(BUILDDIR):
mkdir $(BUILDDIR)
distclean:
rm -rf $(BUILDDIR)
endif
The principle is simple:
In the first part you write your normal makefile, as if you would create the object files in the source directory. However, additionally you add the VPATH option to make sure the source files are found (as make will be in the directory BUILDDIR when this part of the makefile is processed).
In the second part (which is executed first, when the variable RECURSED is not set yet), you change to the BUILDIR directory and invoke your makefile from there. You pass some helper variables (e.g. the current directory) and all make goals, apart of those, which must be executed from outside BUILDDIR (e.g. distclean and the one creating BUILDDIR itself). The rules for those goals you specify also in the second part.

Resources