When trying to organize the compilation output into a build directory, make keeps removing object files. The Makefile is:
MPI_INSTALLATION=/home/gkaf/Software/MPI
IDIR=$(MPI_INSTALLATION)/include
LDIR=$(MPI_INSTALLATION)/lib
CC=gcc
CFLAGS=-I$(IDIR)
LDFLAGS=-L$(LDIR) -Wl,-rpath=$(LDIR)
BUILD_DIR=build
OBJ_DIR=$(BUILD_DIR)/obj
BIN_DIR=$(BUILD_DIR)/bin
SRC_DIR=src
LIBS=-lmpi
.PHONY: all
all: test-mpi
.PHONY: test-mpi
test-mpi: prepare $(BIN_DIR)/test-mpi
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.c
$(CC) -c -o $# $< $(CFLAGS)
$(BIN_DIR)/%: $(OBJ_DIR)/%.o
$(CC) -o $# $^ $(LDFLAGS) $(LIBS)
prepare: $(BUILD_DIR) $(OBJ_DIR) $(BIN_DIR)
$(BUILD_DIR):
mkdir -p $(BUILD_DIR)
$(OBJ_DIR):
mkdir -p $(OBJ_DIR)
$(BIN_DIR):
mkdir -p $(BIN_DIR)
.PHONY: clean
clean:
rm -rf $(BUILD_DIR)
The object file build/obj/test-mpi.o generated during the compilation is deleted after the executable build/bin/test-mpi is created.
I believe that make treats build/obj/test-mpi.o as an intermediate file. However, I was expecting that build/obj/test-mpi.o would not be treated as an intermediate file since it appears explicitly in the target $(OBJ_DIR)/%.o: $(SRC_DIR)/%.c and gnu make documentation states that "ordinarily, a file cannot be intermediate if it is mentioned in the makefile as a target or prerequisite".
This behavior has been reported in a similar issue, but I believe that in both cases the files should not be treated as intermediate since they appear in a target. Am I missing something?
I believe that make treats build/obj/test-mpi.o as an intermediate file.
Yes, that looks right.
However, I was expecting that build/obj/test-mpi.o would not be treated as an intermediate file since it appears explicitly in the target $(OBJ_DIR)/%.o: $(SRC_DIR)/%.c [...]
Given that $(OBJ_DIR) expands to build/obj, the pattern $(OBJ_DIR)/%.o matches build/obj/test-mpi.o. That's the opposite of build/obj/test-mpi.o appearing explicitly.
Even so, you have read the GNU make documentation correctly: make would not consider build/obj/test-mpi.o to be an intermediate file if it was mentioned as a target or prerequisite of some other rule. But it isn't. If make builds that file at all, it is entirely make's idea, notwithstanding the fact that you set the stage for it to come to that decision. This is exactly what it means to be an intermediate file.
Am I missing something?
Apparently you are missing what it means for a file to be "mentioned in" a makefile, as the GNU docs put it. It means that the file's name appears literally in the makefile text, after macro expansion, as a target or prerequisite of a rule. Example:
$(BIN_DIR)/test-mpi: $(OBJ_DIR)/test-mpi.o
or
$(OBJ_DIR)/test-mpi.o: $(SRC_DIR)/test-mpi.c
Matching a target or prerequisite pattern of an implicit ("pattern") rule does not suffice. In fact, it is exactly files that are generated as intermediates in chains of implicit rules that make aims to remove. Implicit rules defined in the makefile are not distinguished from make's built-in implicit rules in this regard.
However, although files such as the one you asked about are definitely intermediate files as GNU make defines that term, make has an additional capability here that might serve your purposes. If you want to use a pattern to specify intermediate targets that you want to preserve, then you can do so by designating the pattern as a prerequisite of the special target .PRECIOUS, like so:
.PRECIOUS: $(OBJ_DIR)/%.o
Intermediate files matching such a pattern will be spared from the automatic deletion to which they otherwise they would be subject.
Related
I have the following rules
define compile_c
$(ECHO) "CC $<"
$(Q)$(CC) $(CFLAGS) -c -MD -o $# $<
## The following fixes the dependency file.
## See http://make.paulandlesley.org/autodep.html for details.
## Regex adjusted from the above to play better with Windows paths, etc.
#$(CP) $(#:.o=.d) $(#:.o=.P); \
$(SED) -e 's/#.*//' -e 's/^.*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $(#:.o=.d) >> $(#:.o=.P); \
$(RM) -f $(#:.o=.d)
endef
vpath %.c . $(TOP)
$(BUILD)/%.o: %.c $(BUILD)/%.pp
$(call compile_c)
vpath %.c . $(TOP)
$(BUILD)/%.pp: %.c
$(ECHO) "PreProcess $<"
$(Q)$(CC) $(CFLAGS) -E -Wp,-C,-dD,-dI -o $# $<
When the build finishes, GNU make says
Removing intermediate files... and deletes all the .pp files which I do NOT want.
Why is it doing this?
How do I stop it?
Since you're using GNU Make, you can make the following adjustment to your Makefile:
.PRECIOUS: $(BUILD)/%.pp # ADD THIS LINE
$(BUILD)/%.pp: %.c
$(ECHO) "PreProcess $<"
$(Q)$(CC) $(CFLAGS) -E -Wp,-C,-dD,-dI -o $# $<
The documentation has this to say about .PRECIOUS directives:
The targets which .PRECIOUS depends on are given the following special treatment: if make is killed or interrupted during the execution of their recipes, the target is not deleted.
[...]
Also, if the target is an intermediate file, it will not be deleted after it is no longer needed, as is normally done.
[...]
You can also list the target pattern of an implicit rule (such as ‘%.o’) as a prerequisite file of the special target .PRECIOUS to preserve intermediate files created by rules whose target patterns match that file's name.
This has the benefit of not creating an unwanted additional rule. It's also clearer what you're trying to do: keep the precious intermediate files that might be expensive to recreate.
I think the best solution is to use the .SECONDARY special target. Just add this line:
.SECONDARY:
Quoting the manual:
.SECONDARY with no prerequisites causes all targets to be treated as secondary (i.e., no target is removed because it is considered intermediate).
Why is this better than making the targets prerequisites of a throw-away target? That's more clutter, and has to be done explicitly for every set of files that might be generated with pattern rules.
Why is this better than .PRECIOUS? That causes files to be retained even if their recipe fails when using .DELETE_ON_ERROR. The latter is important to avoid failing recipes leaving behind bad outputs that are then treated as current by subsequent make invocations. IMO, you always want .DELETE_ON_ERROR, but .PRECIOUS breaks it.
If you search for "gnu make intermediate files" you'll immediately find the answer as to why it's happening, in the GNU make manual section Chains of Implicit Rules.
It also tells you how to avoid it: a file cannot be intermediate if it is mentioned in the makefile as a target or prerequisite.
So, just list your .pp files as a prerequisite of some rule, somewhere. It doesn't have to be a rule that's ever invoked. You don't give enough of your makefile here for us to provide a complete answer, but it would be something like:
all_pps: $(ALL_OBJECTS:.o=.pp)
assuming you had a variable ALL_OBJECTS containing all your .o files.
Here is a detail that finally got PRECIOUS working for me. The pattern that you give to PRECIOUS has to be exactly the pattern that is being used in the rule that creates the intermediate file. I want to save all files prefixed by moc_. At first I used
.PRECIOUS: moc_%
to no avail. Then I tried
.PRECIOUS: moc_%.cpp
and this did the trick.
I've seen a few approaches to making output directories in Make.
These include making all directories ahead of time outside of any rule, and
making an object's destination directory as part of the object's rule.
Both of these approaches involve making directories that likely already exist.
Am I missing any gotchas or drawbacks that explain why I haven't seen the below approach?
.SECONDEXPANSION:
$(OBJDIR)%.o: %.c | $$(#D)/
# Compile command
.PRECIOUS: %/
%/:
# mkdir Command
make is very good at dealing with files. make is not very good at dealing with directories.
So treating directories as implementation detail internal to the target rule makes sense, because then make never has to consider the directory at all:
MKDIR_P = mkdir -p
$(objdir)%.o: %.c
#$(MKDIR_P) $(#D)
$(COMPILE.c) -o $# -c $<
Note that the processing and IO required for the mkdir -p can be neglected next to the processing and IO required for the compilation.
The problem with directories is that (contrary to any other target) you don't care for their timestamp, you only need them to exist. Many Makefiles get directories somehow wrong, and creating them over and over again is what you observe, so make will never detect "Nothing to be done for ...".
In fact, the only thing you need for correct handling of directories with GNU make is an "order only dependency", like shown in your example. The trailing slash normally isn't needed (you seem to use it in order to have a pattern rule, I'm not sure whether this works), and you don't need .PRECIOUS either. Your trick with .SECONDEXPANSION looks quite neat, I guess this will work, given the pattern rule indeed works that way (didn't try).
For an alternative, most Makefiles that handle directories correctly take a simpler approach by concatenating all needed output directories for a rule in a single variable and use this variable as a target for another rule, e.g. like in this simplified example:
MODULES:=src/main
OBJDIR?=obj
OBJS:=$(addprefix $(OBJDIR)/,$(addsuffix .c,$(MODULES)))
DIRS:=$(sort $(addprefix $(OBJDIR)/,$(dir $(OBJS))))
TARGET:= myprogram
all: $(TARGET)
myprogram: $(OBJS)
$(CC) -o$# $^
$(DIRS):
mkdir -p $(DIRS)
$(OBJDIR)/%.o: %.c Makefile | $(DIRS)
$(CC) -c -o$# $<
clean:
rm -fr $(OBJDIR)
.PHONY: all clean
I want to make a language depend target. In Particular: I have one source-file and I want to create different Objects which where add to the corresponding language folder. That single source file will differ in the C-Flags, the compiler will get. As long as I used it in a static way, it works quite fine.
de/info.o en/info.o es/info.o : info.c
$(ECHO) (DEP) $< for $#
Now I thought, it would be great if it is a bit more dynamic, in case i'll add a new language depending file. So I used a wildcard as followed:
de/%.o en/%.o es/%.o : %.c
$(ECHO) (DEP) $< for $#
But now it just make the first target and ignores the rest. The Make-Debug prints the following thing:
Successfully remade target file `de/info.o'.
Considering target file `en/info.o'.
File `en/info.o' was considered already.
Just in case: No, the objects do not exist. So there is no target, but an existing dependencie, so make should execute the rules.
EDIT: Found a solution for that Problem.
define FOO
$(1)/%.o : %.c
$(ECHO) $$< for $(1)
endef
$(foreach lang,$(LANGUAGE_LIST), $(eval $(call FOO,$(lang))))
Inspired by: http://www.gnu.org/software/make/manual/make.html#Eval-Function
Pattern rules work differently than implicit rules. While an implicit rule such as
a b c: d
command
is equivalent to the longer notation
a: d
command
b: d
command
c: d
command
this does NOT hold for pattern rules. Pattern rules with multiple targets are explicitly required to build all of their targets in a single invocation of command. Thus you would have to write
$ cat GNUmakefile
all: de/x.o en/x.o es/x.o
de/%.o: %.c
#echo $# from $<
en/%.o: %.c
#echo $# from $<
es/%.o: %.c
#echo $# from $<
$ gmake
de/x.o from x.c
en/x.o from x.c
es/x.o from x.c
The relevant documentation is found in 10.5.1 Introduction to Pattern Rules of the GNU make manual:
Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and recipe. If a pattern rule has multiple targets, make knows that the rule’s recipe is responsible for making all of the targets. The recipe is executed only once to make all the targets. When searching for a pattern rule to match a target, the target patterns of a rule other than the one that matches the target in need of a rule are incidental: make worries only about giving a recipe and prerequisites to the file presently in question. However, when this file’s recipe is run, the other targets are marked as having been updated themselves.
Here a is my makefile (GNU make) to compile a small OCaml program:
SUFFIXES:=
OCAML=ocamlopt
LD=ocamlopt
OFLAGS=
.PHONY: all clean
all: playground
playground.cmx: playground.ml lstream.cmi
playground: lstream.cmx playground.cmx
%.cmi: %.ml
$(OCAML) $(OFLAGS) -c $<
%.cmx: %.ml
$(OCAML) $(OFLAGS) -c $<
%: %.cmx
$(LD) -o $# $^
playground uses functions from the Lstream module. In this case, the ocaml linker requires the files to link to be specified in order of dependency (eg: ocamlopt -o playground lstream.cmx playground.cmx).
Despite the fact that I defined the playground rule's dependencies in the right order,
make consistently re-orders them and executes ocamlopt -o playground playground.cmx lstream.cmx which causes a linker error.
Is there a way to enforce the correct behaviour ? I would like to avoid specifying the link command explicitely and let make infer it from the dependencies.
Implicit rules always force the pattern matching prerequisite to be first, regardless of the order in which they're defined elsewhere. This is almost always what you want, because in most rules the pattern matching prerequisite is special.
For example when compiling an object file the prerequisites consist of one source file and a bunch of header files; the source file is special and needs to be listed on the command line. Make ensures that for a pattern rule %.o : %.c (for example) the prerequisite matching %.c is first in the list, and so is assigned to the $< automatic variable, and it can be treated differently.
In any event the short answer is no, you cannot modify this behavior. An implicit rule % : %.cmx matching a target playground will always force the prerequisite playground.cmx to be listed first in the prerequisite list. The other prerequisites will maintain their order.
If you really need the prerequisites to maintain their order then I recommend using a static pattern rule:
TARGETS = playground
$(TARGETS) : % :
$(LD) -o $# $^
(you can also use a "match anything" pattern rule but this can be a real performance degrader). Here since you have no pattern in the prerequisite list, nothing will be reordered.
I have a version.c file in my project that contains current revision of the project and some other stuff that is passed as a definition (-D compiler option) from makefile.
I know that to force make to compile version.c always regardless of modification date I can touch version.c.
Is there a makefile only way to achieve this? If I write .PHONY : version.o the object file doesn't get build at all.
EDIT:
Here is my makefile:
export CC = gcc
export MODULES = $(sort \
sys \
cim \
version \
)
export FILES = $(sort \
main.c \
cim.c \
version.c \
)
VPATH = $(MODULES)
OBJS = $(FILES:.c=.o)
INCLUDES = $(addprefix -I,$(MODULES))
all:$(OBJS)
$(CC) $(INCLUDES) $(OBJS) -o main.exe
clean:
rm -rf *.o *.exe
cim.o: cim.c
main.o: main.c cim.o
version.o: version.c
.PHONY: version.o
.c.o :
$(CC) $(CFLAGS) $(INCLUDES) -c $<
The classic way to do it is:
version.o: .FORCE
.FORCE:
(and you might add .PHONY: .FORCE). The file '.FORCE' is presumed not to exist, so it is always 'created', so version.o is always out of date w.r.t it, so version.o is always compiled.
I'm not sure that making version.o into a phony file is correct; it is actually a real file, not a phony one.
Not a makefile way, but easier than touch:
make -B
‘-B’ ‘--always-make’
Consider all targets out-of-date. GNU make proceeds to consider targets and their prerequisites using the normal algorithms; however,
all targets so considered are always remade regardless of the status
of their prerequisites. To avoid infinite recursion, if MAKE_RESTARTS
(see Other Special Variables) is set to a number greater than 0 this
option is disabled when considering whether to remake makefiles (see
How Makefiles Are Remade).
If you want to do this using the FORCE mechanism the correct solution looks like this:
version.o: FORCE
.PHONY: FORCE
FORCE:
By explicitly declaring FORCE to be phony we make sure things will work right even if .SECONDARY: is used (.SECONDARY: will cause FORCE to be considered an intermediate file, and make doesn't rebuilt intermediate files unless they have prerequisites newer than the ultimate target, and FORCE doesn't have any prerequisites, so .PHONY: FORCE is needed).
The other solution (using $(shell touch version.c)) also has a problem: it may cause your editor to think version.c has been updated, and prompt for a reload of the file, which might end up being destructive if you've been editing the file's buffer but haven't yet saved it. If you don't mind this, it can be made even simpler by observing that the touch command is silent, so the assignment to the hack dummy variable isn't needed:
$(shell touch version.c) # This is enough, but will likely confuse your editor
The .PHONY "trick" referred to in the comments on the question generally DOES NOT work. It may look like it does because it will force a relink iff version.o already exists, but the actual object file won't get rebuilt if the .o file rule is an implicit rule (which it usually is). The problem is that make doesn't do the implicit rule search for explicitly phony targets. This make file shows the failure:
fooprog: test.o
cp $< $#
%.o: %.c
cp $< $#
.PHONY: test.o # WRONG
clean:
rm test.o fooprog
If a static pattern rule is used instead of an implicit rule the .PHONY: version.o trick will work. In general using static pattern rules instead of implicit rules cuts out most of the more confusing Make behaviors. But most make files use implicit rules.
The quick hack version when you just need it to work and you don't want to play Make games:
# Hack to get main.c rebuilt
hack := $(shell touch main.c)
Basically just make Make run touch for you.