I'm reading Managing Projects with GNU Make, and found this example in Chapter 2.7 - Automatic Dependency Generation. The Author says their from the GNU manual:
%.d: %c
$(CC) -M $(CPPFLAGS $< > $#.$$$$; \
sed s',\($*\)\.o[ :]*,\1.o $# : ,g' < $#.$$$$ > $#; \
rm -f $#.$$$$
However, I was able to do the same thing with this (note the sed):
-include $(subst .c,.d,$(SOURCES))
%.d: %.c
#$(CC) -M $(CPPFLAGS) $< | sed 's|:| $*.d : |' > $#;
All these lines do is generate the dependencies, then add in the *.d name. They had to change the first line from:
foo.o: bar.h foo.h fubar.h
to
foo.o foo.d : bar.h foo.h fubar.h
Mine is simpler and seems to work quite well, but I assume that the GNU folks had a reason for their sed command. Also:
Why do a redirect of the file into sed? Why not simply take it as a commond line parameter
Why not skip the intermediary file completely?
I know the guys at GNU could have thought of these too, but for some reason, went with the more complex setup. I just want to understand their reasoning, so I can do these on the fly.
Actually even the rule itself is not necessary. There is a great overview of different approaches of generating Make-style dependencies in Advanced Auto-Dependency Generation article written by Paul D. Smith.
After all, the following rule should be enough (in case of using GCC):
%.o: %.c
$(CC) $(CPPFLAGS) $(CFLAGS) -MMD -MP -o $# -c $<
-include $(SOURCES:.c=.d)
UPD.
I have also answered a similar question a bit earlier. It contains an explanation (quotation of GCC manual) of -MMD -MP options.
Addressing the question: Why do a redirect of the file into sed? If you do:
#$(CC) -M $(CPPFLAGS) $< | sed 's|:| $*.d : |' > $#;
and the compilation fails (errors out and generates no output), you will create an empty target file. When make is run again, it will see the newly created empty file and not regenerate it, leading to build errors. Using intermediate files is a common defensive strategy to avoid accidentally created an empty target.
An even simpler solution is to get rid of the sed call completely, as gcc can do everything you need directly:
%.d: %.c
$(CC) -M $(CPPFLAGS) -MF $# $< -MT "$*.o $#"
Related
Some context : I'm trying to build a generic Makefile, with auto-dependency generation (using GCC as described here, non recursive, which could be used from any level of my projects.
For that purpose, the dependency files generated need to be of the following form :
$(DIR)file.o: $(DIR)file.c $(DIR)file.h
$(DIR)file.h
That way, whatever the level of the make invocation, make can properly locate the files.
In order to do so, I have the following implicit rule :
DEPFLAGS = -MT $# -MP -MMD -MF $(word 2,$^).tmp
COMPILE = $(CC) $(DEPFLAGS) $(CFLAGS) $(CPPFLAGS) -c -o $# $<
POSTCOMPILE = sed -E 's|([0-9a-z_.:]*/)?([0-9a-z_.:]+)|$$(DIR)\2|g' $(word 2,$^).tmp > $(word 2,$^)
%.o: %.c %.dep
$(COMPILE)
$(POSTCOMPILE)
$(RM) $(word 2,$^).tmp
touch $#
The sed invocation replace any directory part before the filename by $(DIR)
The problem is that since the dependency file is generated after the object file, it causes the Makefile to relink every time. And so i have to add a touch command to be sure that the object file will be considered up-to-date. Which is not very elegant.
So I have this question : instead of using a temporary file, is there a way to directly redirect the gcc output (which is directed into the .tmp file) into the sed command ?
In others words, is there a way to do the equivalent of the pipe redirection, but for an other stream/file descriptor than stdout ?
If you are in a full fledged linux environment, you can indeed skip the creation of the .tmp file and pipe the dependecies directly into sed.
# This is needed, because the default shell does not support
# process substitution
SHELL = /bin/bash
SED_COMMAND = sed -E 's|([0-9a-z_.:]*/)?([0-9a-z_.:]+)|$$(DIR)\2|g' > $(word 2,$^)
DEPFLAGS = -MT $# -MP -MMD -MF >($(SED_COMMAND))
COMPILE = $(CC) $(DEPFLAGS) $(CFLAGS) $(CPPFLAGS) -c -o $# $<
%.o: %.c %.dep
$(COMPILE)
touch $#
However, this still needs the touch command at the end, for the reason correctly mentioned in a comment by #MadScientist above:
there's no way to be 100% sure that the sed command would finish writing its file before the compiler finished writing the object file: they are running in parallel so you can't guarantee which one will finish first.
I'm trying to write an etags target in make based on the dependency information generated by gcc. Is there some way of getting at all the dependencies in one go? Or, failing that, how would I write a rule that is only executed when the "tags" target is made and that passes all the source and header files to etags? I only want to index the files actually compiled (including headers). I'm aware that I can sed around in the .d files myself but I'm trying to find a more elegant and less brittle solution.
I have the following (excerpt)
DEPFILES = $(OBJFILES:.o=.d)
%.o : %.c
#echo "Compiling $<"
$(NO_ECHO) $(CC) $(CFLAGS) -MMD -MF $(#:.o=.d) -o $# -c $<
%.o : %.S
#echo "Compiling $<"
$(NO_ECHO) $(CC) $(ASFLAGS) -MMD -MF $(#:.o=.d) -o $# -c $<
$(TARGET) : $(OBJFILES)
#echo "Linking $#"
$(NO_ECHO) $(LD) $(LDFLAGS) -o $# $(OBJFILES) $(LIBS:%=-l%)
.PHONY: clean
# Order-only dependency to make Dep/obj-directories if they are not
# present
$(OBJFILES) : | $(ALLPATHS)
$(ALLPATHS):
$(NO_ECHO) mkdir -p $(ALLPATHS)
# Depend on the makefiles
$(OBJFILES) : $(SRC_PATH)/Makefile $(MAKECFG) $(BUILDCFG)
# Include dependency files
sinclude $(DEPFILES)
Edit : The following seems to work but I'd really like to find a more elegant solution
(the double sort/uniq is just for performance).
tags : $(TARGET)
cat $(DEPFILES) | sort | uniq | sed 's/.*:.*$$//' | tr -d '\\' | tr "\n" " " | xargs -n 1 readlink -f | sort | uniq | xargs etags -a
I came here looking for an answer to the same thing as the original question, but went away as didn't think it was adequately answered and found a solution that doesn't require sed etc.
Below I assume a Makefile similar to the Makefile in the original question which generates dep files using the compiler and includes them.
For the compiling rule where the .d files are generated, I modified the compiler options to ask it to also make the tags target depend on the dependancies of the object file. I added -MQ $# -MQ tags to the options. These options explicitly tell the compiler the names of the target for the dependancies.
%.o : %.c
$(CC) $(CFLAGS) -MMD -MF $(#:.o=.d) -o $# -c $< -MQ $# -MQ tags
Now we don't need to explicitly give the tags target a list of dependencies, it will be generated when we compile and will be updated accordingly as the source files change. In my case I am using ctags and this is what the options are I used it with:
tags:
ctags $^ -o $#
The $^ variable is the list of dependancies. This will be a list of source and header files as each .d file will look something like this:
main.o tags: main.c a.h b.h
Hope that helps.
You need to create a mini tag file for each .d file and then use that to update your etags file. Note this appends stuff to the file, rather than removing and replacing, so you might need to remote the tags file occasionally (ctags has a --update option)
TAGFILES = $(OBJFILES:.o=.t)
# prune the .d files a bit to get a list of files for etags to scan
%.t: %.d
cat $< | sed stuff > $#
cat $# | etags -a $#
.PHONY: tags
tags: $(TAGFILES)
I want to use the include directive only for a specific target. I do not want to run the other makefiles when the target is not needed because it means the makefiles are generated needlessly.
So is there a way to conditionally use the include directive, which is conditional on a target? Or somehow to make the include directive a prerequisite of a target.
Here's what I have so far:
# Flags
INCDIR = $(CURDIR)/include
CFLAGS = -Wall -Wno-overflow -Wno-uninitialized -pedantic -std=c99 -I$(INCDIR) -O3
LFLAGS = -flat_namespace -dynamiclib -undefined dynamic_lookup
# Directory names
# Set vpath search paths
vpath %.h include
vpath %.c src
vpath %.o build
vpath %.d build
# Get files for the core library
CORE_FILES = $(wildcard src/*.c)
CORE_OBJS = $(patsubst src/%.c, build/%.o, $(CORE_FILES))
CORE_DEPS = $(CORE_OBJS:.o=.d)
# Core library target linking
core : $(CORE_OBJS) | bin
$(CC) $(LFLAGS) -o bin/libcbitcoin.2.0.dylib $(CORE_OBJS)
# Include header prerequisites (How to do only for "core" target?)
include $(CORE_DEPS)
# Makefiles for header dependencies.
$(CORE_DEPS): build/%.d: src/%.c | build
rm -f $#; \
$(CC) -I$(INCDIR) -MM $< -MT '$(#:.d=.o) $#' > $#
# Objects depend on directory
$(CORE_OBS) : | build
# Create build directory
build:
mkdir build
# Create bin directory
bin:
mkdir bin
# Core Compilation
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
# Depencies require include/CBDependencies.h as a prerequisite
build/CBOpenSSLCrypto.o: include/CBDependencies.h
# Crypto library target linking
crypto : build/CBOpenSSLCrypto.o -lcrypto -lssl | bin
$(CC) $(LFLAGS) -o bin/libcbitcoin-crypto.2.0.dylib build/CBOpenSSLCrypto.o -lcrypto -lssl
# Crypto library compile
build/CBOpenSSLCrypto.o: dependencies/crypto/CBOpenSSLCrypto.c
$(CC) -c $(CFLAGS) $< -o $#
#Clean
clean:
rm -f $(CORE_OBJS) $(CORE_DEPS) build/CBOpenSSLCrypto.o
As you should be able to tell I do not need to include the ".d" files for "crypto" but I do for "core" (default goal).
Thank you for any help.
Make is not a procedural language, so treating it as one goes against the grain; your makefiles will be difficult to scale, and it can lead to subtle bugs.
There's a better way by Tom Tromey that's clean, efficient and scalable. The trick is to realize that you can build the dependency file in the same step as the object file. The dependencies simply tell Make when the object is due to be rebuilt; you don't need them when you first build the object, because Make knows that the object must be built. And if the dependencies change, that can only be because something in the source or the old dependencies has changed, so again Make knows that the object must be rebuilt. (This is not obvious, so it may take a little cogitation.)
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
$(CC) -MM -MF build/$*.d $<
-include build/*.d
There's one more hitch: if you alter the code so as to remove a dependency -- and also remove that file -- you won't be able to rebuild, because the old dependency list will still demand a file which can no longer be found. The sophisticated solution is to process the dependency file so as to make each prerequisite (e.g. header) a target in its own right, with no commands, so that it can be assumed to be rebuilt when needed:
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
$(CC) -MM -MF build/$*.d $<
#cp build/$*.d build/$*.P
#sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < build/$*.P >> build/$*.d;
#rm build/$*.P
A cruder method, but almost as foolproof, is to put in catch-all rules for headers and sources:
$(CORE_OBJS): build/%.o: src/%.c
$(CC) -c $(CFLAGS) $< -o $#
$(CC) -MM -MF build/$*.d $<
%.cc %.h:
EDIT:
To break down the new commands:
The -MM option tells gcc to produce a make rule for the object file, instead of preprocessing or compiling. The default is to send the rule to wherever it would send preprocessed output, which will usually be stdout.
The -MF option, used with -MM, specifies the output file. So -MM -MF build/$*.d will put the rule where we want it.
So the following two commands are (almost always) equivalent:
$(CC) -MM -MF build/$*.d $<
$(CC) -MM $< > build/$*.d
(I've left out the -I$(...) and the possibility of using the -MMD option, because both get a little complicated and are not really the point of the question.)
You can use MAKECMDGOALS.
ifeq (core,$(MAKECMDGOALS))
include $(CORE_DEPS)
endif
You could of course, use ifneq (,$(findstring core,$(MAKECMDGOALS))) if there was a possibility of more than one target.
Note: this is a 'quick and dirty' solution -- I agree with Beta that you shouldn't make this a common practice (this could get messy if you did it in a lot of makefiles...).
John
I can't help breaking the guidelines for what is a good answer.
My answer to the original question is in my opinion, no you cannot include rules that are dependant on the target -all rules are processed before targets are considered. This is a limitation of make (I guess). Then again, good point, there is MAKECMDGOALS, but is this not just a hack in the make utility itself?
The answer from Beta is reasonable and orthodox enough, but you can't call it clean even if it is the best that can be done. It won't work if make has not processed the particular target before and the appropriate build/*.d dependency file is not sitting there.
I like to use the g++ -MM feature to auto-build my dependencies. The way I do this is as follows:
include $(ALLOBJ:%.o=%.d)
%.d: %.cxx
#echo making dependencies for $<
#g++ -MM $(CXXFLAGS) $< -o $#
#sed -i 's,$*\.o,& $# ,g' $#
Basically I can give this rule ALLOBJ, and it will:
convert every .o name to a .d name, and include it,
when it can't find a .d, it will create it from the .cxx file
the final line of the %.d: %.cxx rule will add the name of the .d file to the file itself, so that the dependency will be updated automatically.
The issue arises when I remove a header: the .d file still expects to find it, and make will get upset when it's not there. One solution is to replace include with -include, and to build the dependencies within the compile rule. Unfortunately this requires a dependency generation line for each compile rule, and will also ignore all other include errors (which seems risky). Is there some other simple way to auto-build dependencies which avoids this issue?
Reading the g++ manual a bit more, and thanks to #jackKelly and #Beta's responses above, I found the following solution:
include $(ALLOBJ:%.o=%.d)
%.d: %.cxx
#echo making dependencies for $<
#g++ -MM -MP -MT $*.d -MT $*.o $(CXXFLAGS) $< -o $#
To summarize the flags:
-MM: build dependencies (rather than compiling)
-MP: build 'dummy' targets for all headers. This prevents make from complaining when headers are deleted and thus can't be found.
-MT: specify the targets for the rule. This allows us to tell make the .d file depends on the headers without resorting to the ugly sed rule.
I don't believe my solution is any more correct than #Beta's solution. I tend to use multiple compile rules for C++ files in the same makefile, so having a single dependency rule for all of them is slightly cleaner (in my case) than generating the dependencies in each compile rule.
To restate my answer to the other question, I do it this way:
%.o: %.cpp
#echo making $# and dependencies for $< at the same time
#$(CC) -MD -c $(CXXFLAGS) -o $# $<
#cp $*.d $*.P
#sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $*.P >> $*.d
#rm $*.P
-include $(ALLOBJ:%.o=%.d)
EDIT:
It... It produces the dependency file, but more cleanly and without the sed command:
%.o: %.cpp
#echo making $# and dependencies for $< at the same time
#$(CC) -c $(CXXFLAGS) -o $# $<
#$(CC) -MM -MP $(CXXFLAGS) $< -o $*.d
-include *.d
So now I have to modify the %.o rule in my own makefiles. From now on there'll be a little bit of #JackKelly in everything I compile, mocking me. Oh, this is a black day.
Here's a shortened version of the Makefile I am fighting with on a Linux system:
VPATH=altsrc:src:obj
OBJECTS=\
nondepcode1.o \
nondepcode2.o \
nondepcode3.o \
depcode1.o \
depcode2.o \
depcode3.o
DEP_OBJS= depcode1.o depcode2.o depcode3.o
# Targets
execute: $(OBJECTS)
gfortran -o $# $^ $(LFLAGS)
$(DEP_OBJS): npts.h
obj/%.o: %.f
$(FORTRAN) $(FFLAGS) $< -o $#
obj/%.o: %.f90
$(FORTRAN) $(FFLAGS) $< -o $#
# Rules
.f.o:
$(FORTRAN) $(FFLAGS) -o obj/$# $<
%.o: %.f90
$(FORTRAN) $(FFLAGS) -o obj/$# $<
I was expecting that the three objects that depend on "npts.h" would be automatically updated on a run of make (or gmake) if npts.h was more recent than any of the objects. This just does not happen. Make (and gmake) thinks the objects are up to date. AFAICT, I am doing things the way they are described in the GNU make manual. Anyone have any idea why make/gmake is not doing what I expected? Thanks. BTW, there are tabs at the beginning of all the recipe lines in the actual Makefile. They went away here.
Major egg on my face. As I said, this was a shortened version of the Makefile. I found the bug in the real Makefile. Had some misnamed macros. Not too experienced with the finer points of this stuff. Mea culpa. Very sorry. Thanks for checking.