I have a makefile that is trying to do the following: identify all files under the current directory (all sub-directories included) with .c and .s extensions, for each one compile a non-linked object file and put it into a directory. All C files end up in objects/c, all assembly files end up in objects/ass.
The makefile always works as expected on the first execution (all commands are called in the right order) and no errors are produced.
However if I call make again, half of the time i get "nothing to be done for 'all'.". Which is what you would expect, since no files have been modified. But the other half of the time, make is selecting a random assembly file and compiling that file. That is to say,if I keep doing "make" I sometimes compile file1.s sometimes file2.s. and it keeps randomly swapping between the assembly files add infinitum (it never reaches a "nothing to be done") state.
How is make exhibitting non deterministic behaviour?
This is the smallest makefile I could make that reproduces the error:
SRC_C = $(wildcard *.c) $(wildcard **/*.c)
SRC_ASS = $(wildcard *.s) $(wildcard **/*.s)
OBJECTS_C = $(addprefix $(OBJECT_DIR)c/, $(notdir $(SRC_C:.c=.o)))
OBJECTS_ASS = $(addprefix $(OBJECT_DIR)ass/, $(notdir $(SRC_ASS:.s=.o)))
OBJECTS = $(OBJECTS_C) $(OBJECTS_ASS)
OBJECT_DIR = objects/
all: $(OBJECTS)
%/:
mkdir $#
$(OBJECTS_C): $(OBJECT_DIR) $(OBJECT_DIR)c/
arm-none-eabi-gcc -O0 -march=armv8-a $(wildcard */$(#F:.o=.c)) -nostartfiles -c -o $#
$(OBJECTS_ASS): $(OBJECT_DIR) $(OBJECT_DIR)ass/
arm-none-eabi-as -march=armv8-a $(wildcard */$(#F:.o=.s)) -c -o $#
.PHONY: clean
clean:
rm -rf $(OBJECT_DIR)
You have many errors here.
The biggest is a conceptual one: By flattening all your object files into one directory, there's no way to express proper dependencies using pattern rules, so your object files do not really depend on their respective source files. I'd say: just don't do that! Having object directories is fine, but they should mirror the directory structure of the source tree.
Further errors:
directly depending on directories. This will not work as expected, directories should always be order-only dependencies, as already stated in the comments
Make doesn't support recursive wildcards -- if you really need that, you could write your own function or, assuming you're always building on *nix, just call find instead
Pattern rules for creating directories are not the best idea either -- I'd suggest to collect all needed directories in a variable and loop over that.
Stylistic improvements:
Assign variables that don't need deferred evaluation with :=
Assign variables influencing the build process with ?=, so the user can override them at the command line
Use "standard" variables like CC, AS, CROSS_COMPILE
declare all phony targets in .PHONY.
Your Makefile with these changes applied would look like this:
OBJECT_DIR ?= objects
C_OBJECT_DIR ?= $(OBJECT_DIR)/c
AS_OBJECT_DIR ?= $(OBJECT_DIR)/ass
SRC_C:= $(shell find -name \*.c)
SRC_ASS:= $(shell find -name \*.s)
OBJECTS_C:= $(addprefix $(C_OBJECT_DIR)/, $(SRC_C:.c=.o))
OBJECTS_ASS:= $(addprefix $(AS_OBJECT_DIR)/, $(SRC_ASS:.s=.o))
OBJECTS:= $(OBJECTS_C) $(OBJECTS_ASS)
OUTDIRS:= $(sort $(dir $(OBJECTS)))
CROSS_COMPILE ?= arm-none-eabi-
CC ?= gcc
AS ?= as
CFLAGS ?= -O0 -march=armv8-a -nostartfiles
ASFLAGS ?= -march=armv8-a
all: $(OBJECTS)
$(OUTDIRS):
$(foreach _dir,$#,mkdir -p $(_dir);)
$(C_OBJECT_DIR)/%.o: %.c | $(OUTDIRS)
$(CROSS_COMPILE)$(CC) -c -o $# $(CFLAGS) $<
$(AS_OBJECT_DIR)/%.o: %.s | $(OUTDIRS)
$(CROSS_COMPILE)$(AS) -c -o $# $(ASFLAGS) $<
clean:
rm -rf $(OBJECT_DIR)
.PHONY: all clean
Note there is one important thing missing: automatic dependencies. With this Makefile, each object file depends on its respective source file, but completely misses any headers included. For anything other than a simple toy, you should add that, google for "gnu make gcc automatic dependencies" or something similar (not the scope of this question).
Related
SEE UPDATES BELOW
Research Done: I'm finding learning how to evolve Makefiles from one situation to another is difficult. There are a ton of questions and answers out there but few of them actually show how a Makefile can evolve as your project changes. They also all seem to use various different techniques and idioms of Makefiles so translating between one question and another can be tricky when you are learning Makefiles for the first time, as I am.
Problem: My problem is that I have a project that started at as a flat directory structure but then is migrating to a structure with sub-directories. What I can't do is get my Makefile to along for the ride.
First I'll show what I created that works and then I show how I want it to evolve and how that doesn't work.
Flat Directory Structure, Working Makefile
I have project directory that has all my C files and one header file plus my Makefile:
project
Makefile
c8_asm.c
c8_dasm.c
c8_terp.c
chip8.h
Here is my Makefile (which works just fine):
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
# Targets
all: c8_dasm c8_asm c8_terp
c8_dasm: c8_dasm.o
$(CC) $(LDLIBS) c8_dasm.o -o $#
c8_asm: c8_asm.o
$(CC) $(LDLIBS) c8_asm.o -o $#
c8_terp: c8_terp.o
$(CC) $(LDLIBS) c8_terp.o -o $#
# Using implicit rules for updating an '.o' file from a correspondingly
# named '.c' file.
c8_dasm.o: chip8.h
c8_asm.o: chip8.h
c8_terp.o: chip8.h
.PHONY: clean
clean:
rm c8_dasm c8_asm c8_terp c8_dasm.o c8_asm.o c8_terp.o
I get all my .o files and my executables are created in the project directory.
Evolving The Project
But what I wanted to do is have my sources files (all .c and .h) in a src directory. I wanted to build into an obj directory and have the executables go in a bin directory. So my project would look like this:
project
src
c8_asm.c
c8_dasm.c
c8_terp.c
chip8.h
Makefile
Sub-Directory Structure, Makefile NOT Working
To accommodate the above, I changed my Makefile accordingly:
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
SRC_DIR = src
OBJ_DIR = obj
BIN_DIR = bin
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
MKDIR_P ?= mkdir -p
# Targets
all: $(BIN_DIR)/c8_dasm $(BIN_DIR)/c8_asm $(BIN_DIR)/c8_terp
$(BIN_DIR)/c8_dasm: $(OBJ_DIR)/c8_dasm.o
$(CC) $(LDLIBS) $(OBJ_DIR)/c8_dasm.o -o $#
$(BIN_DIR)/c8_asm: $(OBJ_DIR)/c8_asm.o
$(CC) $(LDLIBS) $(OBJ_DIR)/c8_asm.o -o $#
$(BIN_DIR)/c8_terp: $(OBJ_DIR)/c8_terp.o
$(MKDIR_P) $(dir $#)
$(CC) $(LDLIBS) $(OBJ_DIR)/c8_terp.o -o $#
$(OBJECTS): $(OBJ_DIR)/%.o : $(SRC_DIR)/%.c
$(MKDIR_P) $(dir $#)
$(CC) $< -o $(OBJ_DIR)/$#
# Using implicit rules for updating an '.o' file from a correspondingly
# named '.c' file.
$(OBJ_DIR)/c8_dasm.o: $(SRC_DIR)/chip8.h
$(OBJ_DIR)/c8_asm.o: $(SRC_DIR)/chip8.h
$(OBJ_DIR)/c8_terp.o: $(SRC_DIR)/chip8.h
.PHONY: clean
clean:
rm -r $(BUILD_DIR)
rm $(OBJECTS)
Upon running this I get the following:
mkdir -p obj/obj/
gcc src/c8_dasm.c -o obj/c8_dasm.o
gcc -lm obj/c8_dasm.o -o bin/c8_dasm
ld: can't link with a main executable file 'obj/c8_dasm.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [bin/c8_dasm] Error 1
I wanted to stop here and get some assistance because I fear I'm making this Makefile for complicated than it need be and I'm trying to avoid getting into bad habits.
I'm hoping to hear opinions about what I'm not conceptualizing correctly here.
FIRST UPDATE
I managed to take it bit by bit and get it mostly working. Here is what I ended up with:
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
# Directories.
SRC_DIR = src
BIN_DIR = bin
$(shell mkdir -p $(BIN_DIR))
# Patterns for files.
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(SRC_DIR)/%.o)
EXECUTABLES := c8_dasm c8_asm c8_terp
# Targets
all: $(EXECUTABLES)
c8_dasm: $(SRC_DIR)/c8_dasm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Disassembler Built"
c8_asm: $(SRC_DIR)/c8_asm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Assembler Built"
c8_terp: $(SRC_DIR)/c8_terp.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Interpreter Built"
# Using implicit rules for updating an '.o' file from a correspondingly
# named '.c' file.
c8_dasm.o: $(SRC_DIR)/chip8.h
c8_asm.o: $(SRC_DIR)/chip8.h
c8_terp.o: $(SRC_DIR)/chip8.h
.PHONY: clean
clean:
rm $(OBJECTS)
rm -r $(BIN_DIR)
Of course, as I'm finding with Make this leads to other obscure problems. For example doing this:
make
make clean
works fine. Meaning all files are generated and the files are cleaned, including the bin directory.
However, if I do this:
make c8_dasm
make clean
This builds fine. But the clean fails to delete the bin directory (although it does delete the object files). This happens regardless of what individual executable I try to build.
No amount of searching is helping me find out why that is.
SECOND UPDATE
I found that problem was solved as well. It just required using the "-f" for the rm statements in the clean target.
THIRD UPDATE
To get the object file directory part working, I tried (from this: path include and src directory makefile) to construct my Makefile as follows:
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
SRC_DIR = src
OBJ_DIR = obj
BIN_DIR = bin
$(shell mkdir -p $(BIN_DIR))
$(shell mkdir -p $(OBJ_DIR))
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
EXECUTABLES := c8_dasm c8_asm c8_terp
all: $(EXECUTABLES)
c8_dasm: $(SRC_DIR)/c8_dasm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Disassembler Built"
c8_asm: $(SRC_DIR)/c8_asm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Assembler Built"
c8_terp: $(SRC_DIR)/c8_terp.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Interpreter Built"
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.c
$(CC) $(CFLAGS) -c $< -o $(BIN_DIR)/$#
.PHONY: clean
clean:
rm -rf $(BIN_DIR)
rm -f $(OBJECTS)
I was able to condense the original three lines using chip8.h into one target but I have no way to know if that's correct. It compiles at least. I also changed the OBJECTS line to reflect the new OBJ_DIR I created.
However, this doesn't put the object files in the right place. It still puts them in the src directory rather than the obj directory.
This is why it makes sense to not do anything complicated with Makefiles. Just put the actual directory names in your commands. Never rely on wildcards.
People using C and C++ and using Makefiles spend too much time trying to get those to work rather than just actually getting things done. That's why you see so many of the questions that you see and why the answers vary so much.
In your specific case, your targets don't always have to contain the directory and that's part of the problem. The rules getting generated don't have an actual target in your file because of the directories you are prepending to everything. You have to think in terms of what is getting generated by each target: meaning, the output. So if c8_dasm is getting output, that's your target. The directory has nothing to do with that. So you need to remove all of your directory substitutions where they aren't needed.
But before doing that, ask yourself this: if your first solution was working, why change it? It's better to not even do directories when you're using Make. Just have everything in the same directory as you started off with. You can even see that this allows your Makefile to be much cleaner.
I believe I may have figured this out. Below is my Makefile. It seems to do what I want. It does the following:
Compiles all object files into the obj directory.
Compiles and links so that executables are generated in the bin directory.
Recognizes if any .c files are changed and recompiles accordingly.
Recognizes if the .h file is changed and recompiles all C files that reference it.
This seems to satisfy all the criteria but I can't tell if I've painted myself into some corner that I can't see yet.
CC = gcc
CFLAGS += -c -Wall -std=c99
CFLAGS += -D_POSIX_C_SOURCE=200809L
LDLIBS += -lm
SRC_DIR = src
OBJ_DIR = obj
BIN_DIR = bin
$(shell mkdir -p $(BIN_DIR))
$(shell mkdir -p $(OBJ_DIR))
SOURCES := $(wildcard $(SRC_DIR)/*.c)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
EXECUTABLES := c8_dasm c8_asm c8_terp
all: $(EXECUTABLES)
c8_dasm: $(OBJ_DIR)/c8_dasm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Disassembler Built"
c8_asm: $(OBJ_DIR)/c8_asm.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Assembler Built"
c8_terp: $(OBJ_DIR)/c8_terp.o
$(CC) $^ $(LDLIBS) -o $(BIN_DIR)/$#
#echo "C8 Interpreter Built"
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.c $(SRC_DIR)/chip8.h
$(CC) $(CFLAGS) -c $< -o $#
.PHONY: clean
clean:
rm -rf $(BIN_DIR)
rm -rf $(OBJ_DIR)
Stackoverflow is whining about too many comments, so I'll make this another "answer." After our back-and-forth to my original comment, your last comment is correct. That's what I wanted you to see.
Understand that you can't use Make to do what you want to do exactly.
So here's really the answer: You can't create multiple executables AND with only some of the object files applying to each one AND while using a directory structure. Make is in no way capable of handling that.
Right now you're trying to use Make in a way that it wasn't intended for which is why you're running into so many problems. If you keep playing around you're going to run into is a series of errors that say "duplicate symbol" because you will be compiling each of your files multiple times for each executable, assuming you follow most of the advice you'll find.
Check out this How can I create a Makefile for C projects with SRC, OBJ, and BIN subdirectories? to see what I mean. That one works because all object files are being used to create a single executable. But as you've stated, that's not going to be the case for you. And that's what Make can't handle. That's why you're not finding an answer to that.
And while your chip8.h file is now not going to cause problems in terms of allowing you to compile, your Makefile with that third update would not recognize when the chip8.h file itself has changed. You would have to change a .c file to force a recompile so that changes to your .h were recognized. So you either have to stick with your second update or use something other than Make.
This question is different from the one at makefiles - compile all c files at once in the sense that I have one extra requirement: I want to redirect all the object files in a separate directory.
Here is the setup:
I have multiple sources in a directory say src/mylib.
I want the objects files to end up in build/mylib.
Please note also that under mylib there are subdirectories.
The first attempt was as follows:
sources = $(shell find src/ -name ".c")
objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This is where things aren't working as expected
$(objects): build $(sources)
$(cc) $(cflags) -o $# $(word 2, $^))
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
For the makefile above, only one object file was being generated. I guessed this might have something to do with GCC only being able to generate one object file at a time. Regardless of that, checking the values of $# and $(word 2, $^) in the $(objects) target shows that only one file is being considered even though I have multiple files.
So I changed my makefile to the following:
sources = $(shell find src/ -name ".c")
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This works as expected but it appears to me like make is generating all the objects files even though source files did not change. This can be seen by checking the timestamps on new object files after running make again.
$(objects): build $(sources)
$(foreach source, $(sources), $(shell $(cc) $(cflags) -o $(subst src/,build/, $(patsubst %.o,%.c,$(source))) $(source)))
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
The second makefile works as expected but objects files are being rebuilt again which defeats another purpose of using make: only recompile those source files that changed from the last compilation.
Hence my question: how does one generate all object files in a separate directory at once (by this I mean perform the compilation of all sources files in one rule) while making sure that if a source file didn't change the associated object file should not be regenerated.
I am not after speeding up compilation. What I seek is one rule that will generate all objects files such that only updated source files should be recompiled.
The last makefile does the job but there is a recompiling of all source files which defeats another purpose of using make: only changed source files should be recompiled.
EDIT
After reading comments, it appears I have not phrased my question properly. As the details of what I have are already present, I leave the question as it is with additional details below.
The second makefile in the source code above does work. But it does only half the job. The build directory effectively mirrors the src directory.
So if I have say a file as src/mylib/point/point.c, I get build/mylib/point/point.o generated. This is the first part.
The second part is that if point.c does not changes, point.o in the build/mylib/point/ directory must not be regenerated. But after checking timestamps on the object file, I can tell that a new object file replaced the old one after running make again. This is not good because for large projects, compilation time remains O(n) with n being the number of source files to compile.
So this question is about how to preserve the second makefile without make regenerating object files.
From what I can gather from comments, I am asking too much from make. But if anyone knows how to make this happen, I leave the question open.
Makefile:
all:
clean:
src_root := src
src_subdirs := foo foo/bar foo/bar/buz
build_root := build
o_suffix := .o
# Build list of sources. Iterate every subfolder from $(src_subdirs) list
# and fetch all existing files with suffixes matching the list.
source_suffixes := .c .cpp .cxx
sources := $(foreach d,$(addprefix $(src_root)/,$(src_subdirs)),$(wildcard $(addprefix $d/*,$(source_suffixes))))
# If src_subdirs make variable is unset, use 'find' command to build list of sources.
# Note that we use the same list of suffixes but tweak them for use with 'find'
ifeq ($(src_subdirs),)
sources := $(shell find $(src_root) -type f $(foreach s,$(source_suffixes),$(if $(findstring $s,$(firstword $(source_suffixes))),,-o) -name '*$s'))
endif
$(info sources=$(sources))
# Build source -> object file mapping.
# We want map $(src_root) -> $(build_root) and copy directory structure
# of source tree but populated with object files.
objects := $(addsuffix $(o_suffix),$(basename $(patsubst $(src_root)%,$(build_root)%,$(sources))))
$(info objects=$(objects))
# Generate rules for every .o file to depend exactly on corresponding source file.
$(foreach s,$(sources),$(foreach o,$(filter %$(basename $(notdir $s)).o,$(objects)),$(info New rule: $o: $s)$(eval $o: $s)))
# This is how we compile sources:
# First check if directory for the target file exists.
# If it doesn't run 'mkdir' command.
$(objects): ; $(if $(wildcard $(#D)),,mkdir -p $(#D) &&) g++ -c $< -o $#
# Compile all sources.
all: $(objects)
clean: ; rm -rf $(build_root)
.PHONY: clean all
Environment:
$ find
.
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp
Run makefile:
$ make -f /cygdrive/c/stackoverflow/Makefile.sample -j
sources=src/foo/bar/bar.cxx src/foo/bar/buz/buz.c src/foo/bar/foo.c src/foo/foo.cpp
objects=build/foo/bar/bar.o build/foo/bar/buz/buz.o build/foo/bar/foo.o build/foo/foo.o
New rule: build/foo/bar/bar.o: src/foo/bar/bar.cxx
New rule: build/foo/bar/buz/buz.o: src/foo/bar/buz/buz.c
New rule: build/foo/bar/foo.o: src/foo/bar/foo.c
New rule: build/foo/foo.o: src/foo/bar/foo.c
New rule: build/foo/bar/foo.o: src/foo/foo.cpp
New rule: build/foo/foo.o: src/foo/foo.cpp
mkdir -p build/foo/bar && g++ -c src/foo/bar/bar.cxx -o build/foo/bar/bar.o
mkdir -p build/foo/bar/buz && g++ -c src/foo/bar/buz/buz.c -o build/foo/bar/buz/buz.o
mkdir -p build/foo/bar && g++ -c src/foo/bar/foo.c -o build/foo/bar/foo.o
mkdir -p build/foo && g++ -c src/foo/bar/foo.c -o build/foo/foo.o
Environment again:
$ find
.
./build
./build/foo
./build/foo/bar
./build/foo/bar/bar.o
./build/foo/bar/buz
./build/foo/bar/buz/buz.o
./build/foo/bar/foo.o
./build/foo/foo.o
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp
Try running this Makefile with 'src_subdirs=' to exercise another approach to locate sources. Output should be the same.
I finally had some time to experiment with this, so here is what I came up with:
BUILD_DIR = build
SRC_DIR = src
SOURCES = $(shell find $(SRC_DIR)/ -name "*.c")
TARGET = program
OBJECTS = $(SOURCES:$(SRC_DIR)/%.c=$(BUILD_DIR)/%.o)
default: $(TARGET)
.SECONDEXPANSION:
$(OBJECTS) : $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$#)
mkdir -p $(#D)
$(CC) -c -o $# $(CFLAGS) $<
$(TARGET): $(OBJECTS)
$(CC) -o $# $(CFLAGS) $^
.PHONY: default
Points of interest:
I had to change the sources find pattern from ".c" to "*.c", I'm not sure if it depends on the exact shell used, but if you want to stay portable, be sure to use a widely accepted pattern.
The .SECONDEXPANSION: is needed to enable the $$ rules for GNU Make. It is needed to allow target based substitution rules in the prerequisites for the $(OBJECTS).
The prerequisite $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$#) is saying, that the current target depends on a specific source file with the same folder structure and name.
The command mkdir -p $(#D) is ensuring, that the path of the current target is created if it's missing.
If all you want is a single rule to handle all object files, without necessarily needing to "compile all at once" then you could have something like this:
BUILD_DIR = build
SOURCES = ...
TARGET = ...
OBJECTS = $(SOURCES:%.c=$(BUILD_DIR)/%.o)
default: target
target: $(TARGET)
$(TARGET): $(OBJECTS)
$(LD) -o $# $(LDFLAGS) $^ $(LIBS)
$(BUILD_DIR)/%.o: %.c
$(CC) -c -o $# $< $(CFLAGS)
$(BUILD_DIR):
-mkdir $#
[Note: This is written from memory and without testing.]
After reading the GNU make manual again, here is a solution that solves the second problem.
The first attempt was the correct path. And the second attempt has the $(sources) in the prerequisites but does not use it in the commands and this is silly.
So the working makefile follows. It puts object files in a separate directory and it only compiles files that have changed.
sources = $(shell find src/ -name ".c")
$objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This should now work as expected: object files go into their designated directories under "build/" and only updated files will be recompiled.
$(objects): build $(sources)
# After running say "make clean", make will figure out the need to run the first prerequisite.
# If we are doing a clean build, the number of prerequisites will equal the number of new prerequisites.
ifeq ($(words $?), $(words $^))
# Note the use of "$?" instead of "$^". $? is used since it holds prerequisites that are newer than the target while $^ will holds all prerequisites whether they are new or not.
$(foreach source, $(wordlist 2, $(words $?), $?), $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
else
# If we have a few new targets, no need to exclude "build" from prerequisites because the first prerequisite will be a file that changed.
$(foreach source, $?, $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
endif
.PHONY: build
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
.PHONY: clean
clean:
#rm -rf build/
The makefile is heavily commented with changes that made it work. The most important changes were:
Use of $(foreach) to compile each file individually as required by GCC
Use of $? to work only with prerequisites that are newer than the target
Use of conditional to detected whether the first prerequisite has changed depending on circumstances. If we have a clean build (running make for the first time or after running make clean), the number of updated prerequisites will be the same as the number of newer prerequisites compared to the target. In other words $(words $?) == $(words $^) will be true. So we use this fact to exclude the firs prerequisite listed (build in our case) from the list of files to pass to GCC.
Also, when building the executable from the objects files, make sure to use $^ and not $? when selecting prerequisites else you will end up with only newer files in the executable and it will not run.
target = bin/mylib.a
.PHONY: all
all: $(target)
$(target): $(objects)
ar -cvq $# $^ # Notice that we're not using $? else only updated object files will end up in the archive.
I would like to build a C-project for my microcontroller with the GNU make tool. I would like to do it in a clean way, such that my source code is not cluttered with object files and other stuff after the build. So imagine that I have a project folder, called "myProject" with two folders in it:
- myProject
|
|---+ source
|
'---+ build
The build folder only contains a makefile. The figure below shows what should happen when I run the GNU make tool:
So GNU make should create an object file for each .c source file it can find in the source folder. The object files should be structured in a directory tree that is similar to the structure in the source folder.
GNU make should also make a .d dependency file (in fact, a dependency file is some sort of makefile itself) for each .c source file. The dependency file is described in the GNU make manual chapter 4.14 "Generating Prerequisites Automatically":
For each source file name.c there is a makefile name.d which lists
what files the object file name.o depends on.
From the following Stackoverflow question About the GNU make dependency files *.d, I learned that adding the options -MMD and -MP to the CFLAGS of the GNU gcc compiler can help to automate that.
So now comes the question. Has anyone a sample makefile that performs such out-of-source build? Or some good advices on how to get started?
I'm pretty sure that most people who have written such a makefile, are Linux-people. But the microcontroller project should build also on a Windows machine. Anyway, even if your makefile is Linux-only, it provides a good starting point ;-)
PS: I would like to avoid extra tools like CMake, Autotools, or anything that has to do with an IDE. Just pure GNU make.
I would be very grateful :-)
Updating the dependency files
Please have a look at this question: What is the exact chain of events when GNU make updates the .d files?
Here's the Makefile I've added to the documentation (currently in review so I'll post it here) :
# Set project directory one level above the Makefile directory. $(CURDIR) is a GNU make variable containing the path to the current working directory
PROJDIR := $(realpath $(CURDIR)/..)
SOURCEDIR := $(PROJDIR)/Sources
BUILDDIR := $(PROJDIR)/Build
# Name of the final executable
TARGET = myApp.exe
# Decide whether the commands will be shown or not
VERBOSE = TRUE
# Create the list of directories
DIRS = Folder0 Folder1 Folder2
SOURCEDIRS = $(foreach dir, $(DIRS), $(addprefix $(SOURCEDIR)/, $(dir)))
TARGETDIRS = $(foreach dir, $(DIRS), $(addprefix $(BUILDDIR)/, $(dir)))
# Generate the GCC includes parameters by adding -I before each source folder
INCLUDES = $(foreach dir, $(SOURCEDIRS), $(addprefix -I, $(dir)))
# Add this list to VPATH, the place make will look for the source files
VPATH = $(SOURCEDIRS)
# Create a list of *.c sources in DIRS
SOURCES = $(foreach dir,$(SOURCEDIRS),$(wildcard $(dir)/*.c))
# Define objects for all sources
OBJS := $(subst $(SOURCEDIR),$(BUILDDIR),$(SOURCES:.c=.o))
# Define dependencies files for all objects
DEPS = $(OBJS:.o=.d)
# Name the compiler
CC = gcc
# OS specific part
ifeq ($(OS),Windows_NT)
RM = del /F /Q
RMDIR = -RMDIR /S /Q
MKDIR = -mkdir
ERRIGNORE = 2>NUL || true
SEP=\\
else
RM = rm -rf
RMDIR = rm -rf
MKDIR = mkdir -p
ERRIGNORE = 2>/dev/null
SEP=/
endif
# Remove space after separator
PSEP = $(strip $(SEP))
# Hide or not the calls depending of VERBOSE
ifeq ($(VERBOSE),TRUE)
HIDE =
else
HIDE = #
endif
# Define the function that will generate each rule
define generateRules
$(1)/%.o: %.c
#echo Building $$#
$(HIDE)$(CC) -c $$(INCLUDES) -o $$(subst /,$$(PSEP),$$#) $$(subst /,$$(PSEP),$$<) -MMD
endef
# Indicate to make which targets are not files
.PHONY: all clean directories
all: directories $(TARGET)
$(TARGET): $(OBJS)
$(HIDE)echo Linking $#
$(HIDE)$(CC) $(OBJS) -o $(TARGET)
# Include dependencies
-include $(DEPS)
# Generate rules
$(foreach targetdir, $(TARGETDIRS), $(eval $(call generateRules, $(targetdir))))
directories:
$(HIDE)$(MKDIR) $(subst /,$(PSEP),$(TARGETDIRS)) $(ERRIGNORE)
# Remove all objects, dependencies and executable files generated during the build
clean:
$(HIDE)$(RMDIR) $(subst /,$(PSEP),$(TARGETDIRS)) $(ERRIGNORE)
$(HIDE)$(RM) $(TARGET) $(ERRIGNORE)
#echo Cleaning done !
Main features
Automatic detection of C sources in specified folders
Multiple source folders
Multiple corresponding target folders for object and dependency files
Automatic rule generation for each target folder
Creation of target folders when they don't exist
Dependency management with gcc : Build only what is necessary
Works on Unix and DOS systems
Written for GNU Make
How to use this Makefile
To adapt this Makefile to your project you have to :
Change the TARGET variable to match your target name
Change the name of the Sources and Build folders in SOURCEDIR and BUILDDIR
Change the verbosity level of the Makefile in the Makefile itself or in make call (make all VERBOSE=FALSE)
Change the name of the folders in DIRS to match your sources and build folders
If required, change the compiler and the flags
In this Makefile Folder0, Folder1 and Folder2 are the equivalent to your FolderA, FolderB and FolderC.
Note that I have not had the opportunity to test it on a Unix system at the moment but it works correctly on Windows.
Explanation of a few tricky parts :
Ignoring Windows mkdir errors
ERRIGNORE = 2>NUL || true
This has two effects :
The first one, 2>NUL is to redirect the error output to NUL, so as it does not comes in the console.
The second one, || true prevents the command from rising the error level. This is Windows stuff unrelated with the Makefile, it's here because Windows' mkdir command rises the error level if we try to create an already-existing folder, whereas we don't really care, if it does exist that's fine. The common solution is to use the if not exist structure, but that's not UNIX-compatible so even if it's tricky, I consider my solution more clear.
Creation of OBJS containing all object files with their correct path
OBJS := $(subst $(SOURCEDIR),$(BUILDDIR),$(SOURCES:.c=.o))
Here we want OBJS to contain all the object files with their paths, and we already have SOURCES which contains all the source files with their paths.
$(SOURCES:.c=.o) changes *.c in *.o for all sources, but the path is still the one of the sources.
$(subst $(SOURCEDIR),$(BUILDDIR), ...) will simply subtract the whole source path with the build path, so we finally have a variable that contains the .o files with their paths.
Dealing with Windows and Unix-style path separators
SEP=\\
SEP = /
PSEP = $(strip $(SEP))
This only exist to allow the Makefile to work on Unix and Windows, since Windows uses backslashes in path whereas everyone else uses slashes.
SEP=\\ Here the double backslash is used to escape the backslash character, which make usually treats as an "ignore newline character" to allow writing on multiple lines.
PSEP = $(strip $(SEP)) This will remove the space char of the SEP variable, which has been added automatically.
Automatic generation of rules for each target folder
define generateRules
$(1)/%.o: %.c
#echo Building $$#
$(HIDE)$(CC) -c $$(INCLUDES) -o $$(subst /,$$(PSEP),$$#) $$(subst /,$$(PSEP),$$<) -MMD
endef
That's maybe the trick that is the most related with your usecase. It's a rule template that can be generated with $(eval $(call generateRules, param)) where param is what you can find in the template as $(1).
This will basically fill the Makefile with rules like this for each target folder :
path/to/target/%.o: %.c
#echo Building $#
$(HIDE)$(CC) -c $(INCLUDES) -o $(subst /,$(PSEP),$#) $(subst /,$(PSEP),$<) -MMD
This fairly minimal makefile should do the trick:
VPATH = ../source
OBJS = FolderA/fileA1.o FolderA/fileA2.o FolderB/fileB1.o
CPPFLAGS = -MMD -MP
all: init myProgram
myProgram: $(OBJS)
$(CC) $(LDFLAGS) -o $# $(OBJS) $(LDLIBS)
.PHONY: all init
init:
mkdir -p FolderA
mkdir -p FolderB
-include $(OBJS:%.o=%.d)
The main tricky part is ensuring that FolderA and FolderB exist in the build directory bfore trying to run the compiler that will write into them. The above code will work sequential for builds, but might fail with -j2 the first time it is run, as the compiler in one thread might try to open an output file before the other thread creates the directory. Its also somewhat unclean. Usually with GNU tools you have a configure script that will create those directories (and the makefile) for you before you even try to run make. autoconf and automake can build that for you.
An alternate way that should work for parallel builds would be to redefine the standard rule for compiling C files:
VPATH = ../source
OBJS = FolderA/fileA1.o FolderA/fileA2.o FolderB/fileB1.o
CPPFLAGS = -MMD -MP
myProgram: $(OBJS)
$(CC) $(LDFLAGS) -o $# $(OBJS) $(LDLIBS)
%.o: %.c
mkdir -p $(dir $#)
$(CC) $(CFLAGS) $(CPPFLAGS) -c -o $# $<
-include $(OBJS:%.o=%.d)
Which has the disadvantage that you'll also need to redefine the builtin rules for any other kind of sourcefile you want to compile
Here's a basic one I use all the time, it's pretty much a skeleton as it is but works perfectly fine for simple projects. For more complex projects it certainly needs to be adapted, but I always use this one as a starting point.
APP=app
SRC_DIR=src
INC_DIR=inc
OBJ_DIR=obj
BIN_DIR=bin
CC=gcc
LD=gcc
CFLAGS=-O2 -c -Wall -pedantic -ansi
LFLGAS=
DFLAGS=-g3 -O0 -DDEBUG
INCFLAGS=-I$(INC_DIR)
SOURCES=$(wildcard $(SRC_DIR)/*.c)
HEADERS=$(wildcard $(INC_DIR)/*.h)
OBJECTS=$(SOURCES:$(SRC_DIR)/%.c=$(OBJ_DIR)/%.o)
DEPENDS=$(OBJ_DIR)/.depends
.PHONY: all
all: $(BIN_DIR)/$(APP)
.PHONY: debug
debug: CFLAGS+=$(DFLAGS)
debug: all
$(BIN_DIR)/$(APP): $(OBJECTS) | $(BIN_DIR)
$(LD) $(LFLGAS) -o $# $^
$(OBJ_DIR)/%.o: | $(OBJ_DIR)
$(CC) $(CFLAGS) $(INCFLAGS) -o $# $<
$(DEPENDS): $(SOURCES) | $(OBJ_DIR)
$(CC) $(INCFLAGS) -MM $(SOURCES) | sed -e 's!^!$(OBJ_DIR)/!' >$#
ifneq ($(MAKECMDGOALS),clean)
-include $(DEPENDS)
endif
$(BIN_DIR):
mkdir -p $#
$(OBJ_DIR):
mkdir -p $#
.PHONY: clean
clean:
rm -rf $(BIN_DIR) $(OBJ_DIR)
I would avoid manipulating Makefile directly, and use CMake instead.
Just describe your source files in CMakeLists.txt, as below:
Create file MyProject/source/CMakeLists.txt containing;
project(myProject)
add_executable(myExec FolderA/fileA1.c FolderA/fileA2.c FolderB/fileB1.c)
Under MyProject/build, run
cmake ../source/
You'll get a Makefile now. To build, under the same build/ directory,
make
You may also want to switch to a lightning fast build tool, ninja, simply by adding a switch as following.
cmake -GNinja ..
ninja
I have a project structure like this
-Project
--Common
---types.f90
---global.f90
---common_routines.f90
--Program 1
---program1.f90
---module1.f90
---module2.f90
---etc...
--Program 2
--etc...
Where, Common is folder that contains some modules that are shared across all programs. How do I include this modules on my makefile?
I tried this:
FC = gfortran
FCFLAGS = -fbounds-check -O3
FCFLAGS += -I ../Common
all: program1
program1: module1.o module2.o module3.o
program1.o: module1.o module2.o module3.o
module2.o: module1.o
module3.o: module2.o module1.o
%: %.o
$(FC) $(FCFLAGS) -o $# $^
%.o: %.f90
$(FC) $(FCFLAGS) -c $<
clean:
rm -rf *.o *.mod
but I get an undefined reference error to the common modules variables.
I tried FCFLAGS += -I../Common types.o global.o common_routines.o
This will not work because -I is an option to the GNU Fortran preprocessor
to specify a path that the preprocessor shall search for files to be INCLUDE-ed
prior to compilation. You cannot use it to specify a path where object files (*.o)
will be searched for, after compilation, by the linker. It means nothing to
the linker and is not passed to the linker.
For simplicity let's assume that the object files you need to to link for
program1 are just program1/program1.o plus the pre-existing common/types.o,
common/global.o and common/common_routines.o
Then the following Makefile, placed in directory program1, will build it:
OBJS = program1.o ../common/types.o ../common/global.o ../common/common_routines.o
.phony: all
all: program1
program1: program1.o
$(FC) -o $# $(FCFLAGS) $(OBJS)
clean:
rm -f *.o program1
Just list all the required object files to the linker, in this case via $(OBJS)
You might wish to take the precaution of making sure that the common modules
are up to date before you build program1, and you now might think that you can do that
simply be replacing:
program1: program1.o
with:
program1: $(OBJS)
thus prompting make to recompile any of the four object files that is out
of date with respect to the corresponding source file, as a prerequisite of
building program1
make will certainly endeavour to do that, but take care. That way, it will
recompile, say, ../common/types.o from ../common/types.f90 just by its
implicit default recipe for making an .o from an .f90, since this makefile is
not telling it to do any different. But that may not be the way in which
../common/types.f90 is meant to be compiled, if you also have is a makefile in common
that stipulates how to do it in some non-default manner.
In that case, the common object files should always be compiled as per the
makefile in common. Better leave the prerequisites of program1 alone but change the recipe to:
program1: program1.o
$(MAKE) -C ../common
$(FC) -o $# $(FCFLAGS) $(OBJS)
Now, any time program1 needs to be rebuilt, the recipe will preemptively run make in ../common
before it links the four object files. (It's a small inelegance that this $(MAKE) -C ../common
will be invoked even if there's nothing for it to do: this is avoidable by more advanced make usage).
Lastly you might also find a need (if not in this case, then in another) to distinguish
between flags passed to preprocessing and/or flags passed to compilation and/or flags passed to linkage.
Conventionally, these are assigned to distinct make variables, e.g. FPPFLAGS (preprocessor),
FCFLAGS (compiler), LDFLAGS (linker).
The GNU syntax to define additional include directory is -Idir not -I dir (extra space)
Also make sure that common modules are already compiled and include search path points to the directory where you have compiled modules, not source files:
This path is also used to search for .mod files when previously compiled modules are required by a USE statement.
I'm having some trouble understanding how to design my makefile to build my project the way I want to. Specifically, I can't figure out how to keep all source files in a src directory, while putting all binaries in a bin directory except the linked executable, which goes in the project root.
This is my makefile:
# Compiler options
FC := mpif90
FFLAGS := -O3 -g -Wall -Warray-bounds -ffixed-line-length-none -fbounds-check
VPATH := src
BINDIR := bin
# Define file extensions
.SUFFIXES:
.SUFFIXES: .f .o .mod
# All modules
OBJS := $(BINDIR)/ratecoeffs.o $(BINDIR)/interpolation.o $(BINDIR)/io.o $(BINDIR)/eedf.o $(BINDIR)/single_particle.o $(BINDIR)/physics.o $(BINDIR)/random.o $(BINDIR)/mpi.o $(BINDIR)/precision.o $(BINDIR)/populations.o
# Build rules
all: runner | $(BINDIR)
$(BINDIR):
mkdir -p $(BINDIR)
$(BINDIR)/%.o: $(VPATH)/%.f | $(BINDIR)
$(FC) $(FFLAGS) -c $^ -o $#
runner: $(OBJS)
clean:
#rm -rf $(BINDIR)
Running make builds everything allright - it finds all source files in src and puts all .o files in bin - but the module files (.mod) that are generated by the compiler are put in the project root instead of in the bin directory. I realize I could just specify a rule to place them there, but that messes with the build order, and will sometimes break the build.
What is the "correct" way to get this behavior?
And yes, I've looked at autotools and automake, but I've never used them before and they seem to be overkill for this project. As I couldn't find any good tutorials on how they work (no, I didn't like the tutorial on gnu.org) I'd prefer if I could avoid having to learn this tool just to get this work...
Assuming your underlying Fortran compiler is gfortran, use the -J command line option.
$(FC) $(FFLAGS) -c $^ -o $# -J$(BINDIR)
With an eye to the future, you may be better off creating a MODDIR or similar variable, that you use instead of BINDIR. Object code (*.o) and mod files have different roles to play in later compilation and linking steps - in larger projects they are often kept separate.
It would be probably more in the sense of the make system to change into the obj-directory and do the compilation from there. Via the VPATH option you can let make to find your source files automatically. You could easily call your makefile recursively from the right directory. Below you find a trivial example which would be straightforward to adapt to your case. Please note, that it only works with GNU make.
ifeq (1,$(RECURSED))
VPATH = $(SRCDIR)
########################################################################
# Project specific makefile
########################################################################
FC = gfortran
FCOPTS =
LN = $(FC)
LNOPTS =
OBJS = accuracy.o eqsolver.o io.o linsolve.o
linsolve: $(OBJS)
$(LN) $(LNOPTS) -o $# $^
%.o: %.f90
$(FC) $(FCOPTS) -c $<
.PHONY: clean realclean
clean:
rm -f *.mod *.o
realclean: clean
rm -f linsolve
accuracy.o:
eqsolver.o: accuracy.o
io.o: accuracy.o
linsolve.o: accuracy.o eqsolver.o io.o
else
########################################################################
# Recusive invokation
########################################################################
BUILDDIR = _build
LOCALGOALS = $(BUILDDIR) distclean
RECURSIVEGOALS = $(filter-out $(LOCALGOALS), $(MAKECMDGOALS))
.PHONY: all $(RECURSIVE_GOALS) distclean
all $(RECURSIVEGOALS): $(BUILDDIR)
+$(MAKE) -C $(BUILDDIR) -f $(CURDIR)/GNUmakefile SRCDIR=$(CURDIR) \
RECURSED=1 $(RECURSIVEGOALS)
$(BUILDDIR):
mkdir $(BUILDDIR)
distclean:
rm -rf $(BUILDDIR)
endif
The principle is simple:
In the first part you write your normal makefile, as if you would create the object files in the source directory. However, additionally you add the VPATH option to make sure the source files are found (as make will be in the directory BUILDDIR when this part of the makefile is processed).
In the second part (which is executed first, when the variable RECURSED is not set yet), you change to the BUILDIR directory and invoke your makefile from there. You pass some helper variables (e.g. the current directory) and all make goals, apart of those, which must be executed from outside BUILDDIR (e.g. distclean and the one creating BUILDDIR itself). The rules for those goals you specify also in the second part.