I'm trying to understand how to handle header file dependencies in Make rules. Let me give you a specific example.
I'm building application called myap using GNU Make. It consists of various *.h and *.c files.
Directory inc/ contains defs.h and util.h header files.
Directory src/ contains main.c, cmd.c and win.c files.
Directory obj/ contains all generated object files.
I have multiple applications that need different build options. So I don't want to rely on any implicit rules and would like to specify my own rules for all object files, etc.
I would like to specify the following rules:
Object files depend on specific *.h and *.c files. If any of them change, all object files must be regenerated. However, even though *.h files are part of the prerequisites list, I don't want to pass them to the compiler. I only want to compile *.c files.
Executable myapp depends on specific *.o files. If any of them change, executable file must be regenerated.
So far, the following Makefile with a static pattern rule seems to work correctly:
myapp_inc := inc/defs.h inc/util.h
myapp_src := src/main.c src/cmd.c src/win.c
myapp_obj := $(patsubst src/%.c,obj/%.o,$(myapp_src))
myapp_bin := obj/myapp
.PHONY: all
all:
# Create obj/main.o obj/cmd.o and obj/win.o from various *.c files
# If any *.h files in $(myapp_inc) list change, all objects are regenerated.
# If any *.c files in $(myapp_src) list change, all objects are regenerated.
$(myapp_obj): obj/%.o: src/%.c $(myapp_inc) $(myapp_src)
gcc -c -o $# $<
# Create obj/myapp from various *.o files
# If any *.o files in $(myapp_obj) list change, executable is regenerated.
$(myapp_bin): $(myapp_obj)
gcc -o $# $^
all: $(myapp_bin)
.PHONY: clean
clean:
rm -f obj/*
I don't quite understand how Make rules should be written correctly in order to handle such use case. Is the above static pattern rule, the only way that works correctly?
Specifically, I have tried the following combinations, as given in various simple examples on the Internet, and they all failed for various reasons.
This rule causes $< to always pass the name of the first prerequisite, which doesn't work with multiple *.c files:
$(myapp_obj): $(myapp_src) $(myapp_inc)
gcc -c -o $# $<
$ make
gcc -c -o obj/main.o src/main.c
gcc -c -o obj/cmd.o src/main.c
gcc -c -o obj/win.o src/main.c
gcc -o obj/myapp obj/main.o obj/cmd.o obj/win.o
/bin/ld: obj/cmd.o: in function `main':
main.c:(.text+0x0): multiple definition of `main'; obj/main.o:main.c:(.text+0x0): first defined here
/bin/ld: obj/win.o: in function `main':
main.c:(.text+0x0): multiple definition of `main'; obj/main.o:main.c:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status
make: *** [Makefile:18: obj/myapp] Error 1
This rule causes $^ to always pass the names of all prerequisites, which fails:
$(myapp_obj): $(myapp_src) $(myapp_inc)
gcc -c -o $# $^
$ make
gcc -c -o obj/main.o src/main.c src/cmd.c src/win.c inc/defs.h inc/util.h
gcc: fatal error: cannot specify ‘-o’ with ‘-c’, ‘-S’ or ‘-E’ with multiple files
compilation terminated.
make: *** [Makefile:13: obj/main.o] Error 1
Now I understand the difference between $< and $^ variables, but a lot of documentation is not clear on how they should be used when dealing with a list of multiple *.c and *.h files as prerequisites.
What are the recommended usage pattern for this?
Why is it that when using $< only *.c files get passed to the recipe, but not *.h files? Is Make doing some internal filtering? Is this documented anywhere? Is it possible to modify this behavior for custom suffixes?
Is the above static pattern rule, the only way to make objects depend on *.h and *.c files, but exclude *.h files during compilation?
I don't understand the goal of trying to avoid implicit rules. But in any event, it doesn't matter to the recipe you write whether the rule was implicit or explicit: the same automatic variables are set either way. The $< automatic variable is always the first prerequisite, so if you write your rules such that the first prerequisite is the appropriate .c file then you can always use $< in your recipe to mean the .c file and no other files. All the following will work:
%.o : %.c $(headers)
gcc -c -o $# $<
foo.o: foo.c $(headers)
gcc -c -o $# $<
foo.o : %.o : %.c $(headers)
gcc -c -o $# $<
%.o : %.c
gcc -c -o $# $<
$(srcs) : $(headers)
and others.
Does this mean that all of the prerequisites apply, but only those that match the pattern get passed to the recipe?
I don't understand the question, really. The value of variables and the expansion of the recipe happens only AFTER make has decided to run the rule and is not really related (except for some special automatic variables like $?). Once make has decided that the target is out of date and the recipe needs to be run, it will assign the appropriate automatic variables, expand the recipe, then pass the recipe to the shell to be run.
The automatic variables are assigned as described in the manual: $# is the target, $< is the first prerequisite, $^ is all the prerequisites, etc.
ETA
You still haven't really explained why you don't want to use static pattern rules. They are a perfectly fine and reasonable way to do things.
If you explain what you don't like about static pattern rules, or what you wish you could do differently, then we can probably suggest alternatives that meet those requirements.
Specifically, I have tried the following combinations, as given in various simple examples on the Internet,
$(myapp_obj): $(myapp_src) $(myapp_inc)
Wherever you found this as a recommended example on the Internet, you should immediately delete from any bookmarks as that site doesn't know anything about make.
We see this paradigm at least once a week on SO. I've never really understand why people think it will work: I guess they think make is much more "magical" than it is. Consider, what does the above expand to? Suppose myapp_obj contained foo.o bar.o biz.o and myapp_src contained foo.c bar.c biz.c and myapp_inc contained foo.h bar.h, then make sees:
foo.o bar.o biz.o: foo.c bar.c biz.c foo.h bar.h
I suppose some people think make will intuit that the ".o" files should somehow match up with the ".c" files and will generate a bunch of rules that make that true. That's not what make does. The above line is exactly identical to writing this:
foo.o: foo.c bar.c biz.c foo.h bar.h
bar.o: foo.c bar.c biz.c foo.h bar.h
biz.o: foo.c bar.c biz.c foo.h bar.h
That is, if you have multiple targets make creates one copy of the rule for each target, with the same prerequisites and recipe.
This is obviously not what you want, and that's why none of the examples that try to do things this way can ever work properly.
Why is it that when using $< only *.c files get passed to the recipe, but not *.h files? Is Make doing some internal filtering? Is this documented anywhere? Is it possible to modify this behavior for custom suffixes?
None of that is the case. As I described above, the $< expands to the first prerequisite. That's all. It doesn't matter whether the first prerequisite is a .c file, a .h file, or some other file; whatever it is, $< will be that value. If you write your rule as:
foo.o : foo.c foo.h ; $(CC) -c -o $# $<
then your compiler will be invoked with foo.c. If you write your rule as:
foo.o : foo.h foo.c ; $(CC) -c -o $# $<
then your compiler will be invoked with foo.h. There's no magic here.
I have set up my makefile like below, to minimize code duplication
The recipes are a set of blocks that set a variable, and then run the sleeper_agent recipe. They work great when called individually as make xlsx_sleeper for example.
But when I call all_sleepers, only the first one (xlsx_sleeper) gets compiled.
I have tried declaring them as phony (.PHONY: all_sleepers xlsx_sleeper docx_sleeper pptx_sleeper pdf_sleeper png_sleeper), which changes nothing
and adding a .FORCE rule to the sleeper_agent rule, which results in no such file or directory:
.FORCE:
sleeper_agent: .FORCE [...]
Here is my makefile:
all_sleepers: xlsx_sleeper docx_sleeper pptx_sleeper png_sleeper pdf_sleeper
sleeper_agent: $(OBJ)/sleeper_agent.o $(OBJ)/identities.o
windres icons/$(ext)/resource.rc -O coff -o obj/$(ext).res
$(CC) -o $(BIN)/sleeper_$(ext).exe $^ $(OBJ)/$(ext).res $(CFLAGS) $(LIBS)
xlsx_sleeper: ext=xlsx
xlsx_sleeper: sleeper_agent
docx_sleeper: ext=docx
docx_sleeper: sleeper_agent
pptx_sleeper: ext=pptx
pptx_sleeper: sleeper_agent
png_sleeper: ext=png
png_sleeper: sleeper_agent
pdf_sleeper: ext=pdf
pdf_sleeper: sleeper_agent
Your problem is that make does not see any reason why it should rebuild the sleeper_agent target several times. You should probably stick to the make philosophy:
Try to have real files as targets ($(BIN)/sleeper_xlsx.exe).
Use phony (non-file) targets only:
To give symbolic names to other targets or groups of targets (all_sleepers, xlsx_sleeper, ...)
For rules that don't produce files (clean, help...)
Declare phony targets as such (.PHONY: ...)
Example using static pattern rules, automatic variables and the patsubst make function:
SLEEPER := xlsx docx pptx png pdf
EXE := $(patsubst %,$(BIN)/sleeper_%.exe,$(SLEEPER))
SHORT := $(patsubst %,%_sleeper,$(SLEEPER))
.PHONY: all_sleepers $(SHORT) clean_sleepers
all_sleepers: $(EXE)
$(SHORT): %_sleeper: $(BIN)/sleeper_%.exe
$(EXE): $(BIN)/sleeper_%.exe: $(OBJ)/sleeper_agent.o $(OBJ)/identities.o
windres icons/$*/resource.rc -O coff -o obj/$*.res
$(CC) -o $# $^ $(OBJ)/$*.res $(CFLAGS) $(LIBS)
clean_sleepers:
rm -f $(EXE)
And then you should be able to run:
make all_sleepers
to build them all or:
make xlsx_sleeper
to build only one of them. EXE is the list of real executable files and a static pattern rule explains how to build them. In its recipe the $* automatic variable expands as the string matching the % wildcard. SHORT is the list of xxxx_sleeper shortcuts and another static pattern rule explains for each of them to which real executable it corresponds. all_sleepers and the xxxx_sleeper shortcuts (plus the clean_sleepers I added as example) are properly declared as phony because there are no such real files.
I have a series of directories organized like this:
foo/
foo.file1 foo.file2
bar/
bar.file1 bar.file2
baz/
baz.file1 baz.file2
Right now I'm processing these files using a script that does all the checking for file existence etc but I thought that perhaps I could use a Makefile for it (since said script is very fragile), to avoid reprocessing files that did not change.
The problem is that each directory is independent, and I'd need to do, for example:
foo.file1.processed: foo.file1
run_random_program foo.file1 -o foo.file1.processed
for each of the 71 directories that are in total in that path. This looks like being extremely tedious and I wonder if there's something that would prevent me from writing all of this by hand.
Is such a thing possible?
EDIT: Some examples that show what I have in mind, had I a single Makefile for each directory:
file1.cds.callable: file1.callable
long_script_name -i $< -o $#
file1.rds: file1.cds.callable
another_long_script_name $< additional_file_in_folder $#
file1.csv: file1.rds
yet_another_script $< $#
Seems like pattern rules are exactly what you need:
# These are the original source files (based on the example)
CALLABLE := $(wildcard */*.callable)
# These are the final targets
TARGETS := $(CALLABLE:%.callable=%.csv)
all: $(TARGETS)
%.csv : %.rds
yet_another_script $< $#
%.rds: %.cds.callable
another_long_script_name $< additional_file_in_folder $#
%.cds.callable: %.callable
long_script_name -i $< -o $#
Googling doesn't help much in understand how the % variable is being used in the Makefile snippet below.
_OBJ = a.o b.o c.o
OBJ = $(patsubst %,$(OBJDIR)/%,$(_OBJ))
$(OBJDIR)/%.o: $(SRCDIR)/%.c $(INC)
$(CC) -c -o $# $< $(CFLAGS)
Can anybody please help me out? Does it mean that if there are five .c files present under $(SRCDIR), that many *.o rules are being populated behind the scene?
That is an "implicit rule". The % makes the word $(OBJDIR)/%.o a pattern. It tells make that if it's trying to build a target and the name of the target matches that pattern (where the % can substitute for one or more characters--any characters), AND that a file that matches the pattern $(SRCDIR)/%.c (where the % here has the same value as in the target) either already exists or can be built, then make can use this recipe to build that target.
So, if make wants to build a file foo/bar.o and the variable OBJDIR has the value foo, then this pattern foo/%.o will match that file with the % matching bar (this is called the stem in the GNU make documentation).
Then if SRCDIR has the value blah and make can find (or create) a file named blah/bar.c, then this implicit rule can be used to build the target foo/bar.o by running this recipe ($(CC) -c -o $# $< $(CFLAGS)).
This question is different from the one at makefiles - compile all c files at once in the sense that I have one extra requirement: I want to redirect all the object files in a separate directory.
Here is the setup:
I have multiple sources in a directory say src/mylib.
I want the objects files to end up in build/mylib.
Please note also that under mylib there are subdirectories.
The first attempt was as follows:
sources = $(shell find src/ -name ".c")
objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This is where things aren't working as expected
$(objects): build $(sources)
$(cc) $(cflags) -o $# $(word 2, $^))
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
For the makefile above, only one object file was being generated. I guessed this might have something to do with GCC only being able to generate one object file at a time. Regardless of that, checking the values of $# and $(word 2, $^) in the $(objects) target shows that only one file is being considered even though I have multiple files.
So I changed my makefile to the following:
sources = $(shell find src/ -name ".c")
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This works as expected but it appears to me like make is generating all the objects files even though source files did not change. This can be seen by checking the timestamps on new object files after running make again.
$(objects): build $(sources)
$(foreach source, $(sources), $(shell $(cc) $(cflags) -o $(subst src/,build/, $(patsubst %.o,%.c,$(source))) $(source)))
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
The second makefile works as expected but objects files are being rebuilt again which defeats another purpose of using make: only recompile those source files that changed from the last compilation.
Hence my question: how does one generate all object files in a separate directory at once (by this I mean perform the compilation of all sources files in one rule) while making sure that if a source file didn't change the associated object file should not be regenerated.
I am not after speeding up compilation. What I seek is one rule that will generate all objects files such that only updated source files should be recompiled.
The last makefile does the job but there is a recompiling of all source files which defeats another purpose of using make: only changed source files should be recompiled.
EDIT
After reading comments, it appears I have not phrased my question properly. As the details of what I have are already present, I leave the question as it is with additional details below.
The second makefile in the source code above does work. But it does only half the job. The build directory effectively mirrors the src directory.
So if I have say a file as src/mylib/point/point.c, I get build/mylib/point/point.o generated. This is the first part.
The second part is that if point.c does not changes, point.o in the build/mylib/point/ directory must not be regenerated. But after checking timestamps on the object file, I can tell that a new object file replaced the old one after running make again. This is not good because for large projects, compilation time remains O(n) with n being the number of source files to compile.
So this question is about how to preserve the second makefile without make regenerating object files.
From what I can gather from comments, I am asking too much from make. But if anyone knows how to make this happen, I leave the question open.
Makefile:
all:
clean:
src_root := src
src_subdirs := foo foo/bar foo/bar/buz
build_root := build
o_suffix := .o
# Build list of sources. Iterate every subfolder from $(src_subdirs) list
# and fetch all existing files with suffixes matching the list.
source_suffixes := .c .cpp .cxx
sources := $(foreach d,$(addprefix $(src_root)/,$(src_subdirs)),$(wildcard $(addprefix $d/*,$(source_suffixes))))
# If src_subdirs make variable is unset, use 'find' command to build list of sources.
# Note that we use the same list of suffixes but tweak them for use with 'find'
ifeq ($(src_subdirs),)
sources := $(shell find $(src_root) -type f $(foreach s,$(source_suffixes),$(if $(findstring $s,$(firstword $(source_suffixes))),,-o) -name '*$s'))
endif
$(info sources=$(sources))
# Build source -> object file mapping.
# We want map $(src_root) -> $(build_root) and copy directory structure
# of source tree but populated with object files.
objects := $(addsuffix $(o_suffix),$(basename $(patsubst $(src_root)%,$(build_root)%,$(sources))))
$(info objects=$(objects))
# Generate rules for every .o file to depend exactly on corresponding source file.
$(foreach s,$(sources),$(foreach o,$(filter %$(basename $(notdir $s)).o,$(objects)),$(info New rule: $o: $s)$(eval $o: $s)))
# This is how we compile sources:
# First check if directory for the target file exists.
# If it doesn't run 'mkdir' command.
$(objects): ; $(if $(wildcard $(#D)),,mkdir -p $(#D) &&) g++ -c $< -o $#
# Compile all sources.
all: $(objects)
clean: ; rm -rf $(build_root)
.PHONY: clean all
Environment:
$ find
.
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp
Run makefile:
$ make -f /cygdrive/c/stackoverflow/Makefile.sample -j
sources=src/foo/bar/bar.cxx src/foo/bar/buz/buz.c src/foo/bar/foo.c src/foo/foo.cpp
objects=build/foo/bar/bar.o build/foo/bar/buz/buz.o build/foo/bar/foo.o build/foo/foo.o
New rule: build/foo/bar/bar.o: src/foo/bar/bar.cxx
New rule: build/foo/bar/buz/buz.o: src/foo/bar/buz/buz.c
New rule: build/foo/bar/foo.o: src/foo/bar/foo.c
New rule: build/foo/foo.o: src/foo/bar/foo.c
New rule: build/foo/bar/foo.o: src/foo/foo.cpp
New rule: build/foo/foo.o: src/foo/foo.cpp
mkdir -p build/foo/bar && g++ -c src/foo/bar/bar.cxx -o build/foo/bar/bar.o
mkdir -p build/foo/bar/buz && g++ -c src/foo/bar/buz/buz.c -o build/foo/bar/buz/buz.o
mkdir -p build/foo/bar && g++ -c src/foo/bar/foo.c -o build/foo/bar/foo.o
mkdir -p build/foo && g++ -c src/foo/bar/foo.c -o build/foo/foo.o
Environment again:
$ find
.
./build
./build/foo
./build/foo/bar
./build/foo/bar/bar.o
./build/foo/bar/buz
./build/foo/bar/buz/buz.o
./build/foo/bar/foo.o
./build/foo/foo.o
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp
Try running this Makefile with 'src_subdirs=' to exercise another approach to locate sources. Output should be the same.
I finally had some time to experiment with this, so here is what I came up with:
BUILD_DIR = build
SRC_DIR = src
SOURCES = $(shell find $(SRC_DIR)/ -name "*.c")
TARGET = program
OBJECTS = $(SOURCES:$(SRC_DIR)/%.c=$(BUILD_DIR)/%.o)
default: $(TARGET)
.SECONDEXPANSION:
$(OBJECTS) : $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$#)
mkdir -p $(#D)
$(CC) -c -o $# $(CFLAGS) $<
$(TARGET): $(OBJECTS)
$(CC) -o $# $(CFLAGS) $^
.PHONY: default
Points of interest:
I had to change the sources find pattern from ".c" to "*.c", I'm not sure if it depends on the exact shell used, but if you want to stay portable, be sure to use a widely accepted pattern.
The .SECONDEXPANSION: is needed to enable the $$ rules for GNU Make. It is needed to allow target based substitution rules in the prerequisites for the $(OBJECTS).
The prerequisite $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$#) is saying, that the current target depends on a specific source file with the same folder structure and name.
The command mkdir -p $(#D) is ensuring, that the path of the current target is created if it's missing.
If all you want is a single rule to handle all object files, without necessarily needing to "compile all at once" then you could have something like this:
BUILD_DIR = build
SOURCES = ...
TARGET = ...
OBJECTS = $(SOURCES:%.c=$(BUILD_DIR)/%.o)
default: target
target: $(TARGET)
$(TARGET): $(OBJECTS)
$(LD) -o $# $(LDFLAGS) $^ $(LIBS)
$(BUILD_DIR)/%.o: %.c
$(CC) -c -o $# $< $(CFLAGS)
$(BUILD_DIR):
-mkdir $#
[Note: This is written from memory and without testing.]
After reading the GNU make manual again, here is a solution that solves the second problem.
The first attempt was the correct path. And the second attempt has the $(sources) in the prerequisites but does not use it in the commands and this is silly.
So the working makefile follows. It puts object files in a separate directory and it only compiles files that have changed.
sources = $(shell find src/ -name ".c")
$objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This should now work as expected: object files go into their designated directories under "build/" and only updated files will be recompiled.
$(objects): build $(sources)
# After running say "make clean", make will figure out the need to run the first prerequisite.
# If we are doing a clean build, the number of prerequisites will equal the number of new prerequisites.
ifeq ($(words $?), $(words $^))
# Note the use of "$?" instead of "$^". $? is used since it holds prerequisites that are newer than the target while $^ will holds all prerequisites whether they are new or not.
$(foreach source, $(wordlist 2, $(words $?), $?), $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
else
# If we have a few new targets, no need to exclude "build" from prerequisites because the first prerequisite will be a file that changed.
$(foreach source, $?, $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
endif
.PHONY: build
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
.PHONY: clean
clean:
#rm -rf build/
The makefile is heavily commented with changes that made it work. The most important changes were:
Use of $(foreach) to compile each file individually as required by GCC
Use of $? to work only with prerequisites that are newer than the target
Use of conditional to detected whether the first prerequisite has changed depending on circumstances. If we have a clean build (running make for the first time or after running make clean), the number of updated prerequisites will be the same as the number of newer prerequisites compared to the target. In other words $(words $?) == $(words $^) will be true. So we use this fact to exclude the firs prerequisite listed (build in our case) from the list of files to pass to GCC.
Also, when building the executable from the objects files, make sure to use $^ and not $? when selecting prerequisites else you will end up with only newer files in the executable and it will not run.
target = bin/mylib.a
.PHONY: all
all: $(target)
$(target): $(objects)
ar -cvq $# $^ # Notice that we're not using $? else only updated object files will end up in the archive.