Make implicit rule fails at first time but succeeds at second time - makefile

I encounter an error relating to implicit rule in make (3.81). The example codes are:
dongli:test02 dongli$ ls -R
Makefile a.F90 b.F90 dir
./dir:
a.t.F90
The dependencies among the codes are:
a.t.F90: a.F90
a.t.o: a.t.F90
b.o: b.F90
b: a.t.o b.o
That is a.F90 is intermediate code, and a.t.F90 will be updated when a.F90 is updated. My make process is:
Test 1 (All codes on position):
-------------------------------------------------------------
Project: >>> test <<<
-------------------------------------------------------------
Creating dependency a.t.o
-------------------------------------------------------------
-----> ./dir/a.t.F90
Creating dependency b.o
-------------------------------------------------------------
-----> b.F90
Creating target 'b'
---> b is created.
-------------------------------------------------------------
Finished
-------------------------------------------------------------
Test 2 (touch a.F90):
dongli:test02 dongli$ touch a.F90
dongli:test02 dongli$ make
-------------------------------------------------------------
Project: >>> test <<<
-------------------------------------------------------------
Processing templates in a.F90
-------------------------------------------------------------
Creating dependency a.t.o
-------------------------------------------------------------
-----> a.t.F90
gfortran: error: a.t.F90: No such file or directory
gfortran: fatal error: no input files
compilation terminated.
make: *** [a.t.o] Error 1
Test 3: (Run make again):
dongli:test02 dongli$ make
-------------------------------------------------------------
Project: >>> test <<<
-------------------------------------------------------------
Creating dependency a.t.o
-------------------------------------------------------------
-----> ./dir/a.t.F90
Creating dependency b.o
-------------------------------------------------------------
-----> b.F90
Creating target 'b'
---> b is created.
-------------------------------------------------------------
Finished
-------------------------------------------------------------
I know there is a bug in make about directory caching (see here), but in my case, dir/a.t.F90 exists all the time. Any idea? Thanks!
Update1:
I use make -d to capture the following information in test 2:
Finished prerequisites of target file `a.t.F90'.
Prerequisite `a.F90' is newer than target `a.t.F90'.
Must remake target `a.t.F90'.
Ignoring VPATH name `./dir/a.t.F90'.
...
Successfully remade target file `a.t.F90'.
Finished prerequisites of target file `a.t.o'.
Prerequisite `a.t.F90' of target `a.t.o' does not exist.
Must remake target `a.t.o'.
Why ./dir/a.t.F90 is ignored when its prerequisite a.F90 is newer than it?
Update2:
I have put the example codes on gist.
Update3:
I found the following relating information:
If a target needs to be rebuilt, GNU make discards the file name found
during the VPATH search for this target, and builds the file locally
using the file name given in the makefile. If a target does not need
to be rebuilt, GNU make uses the file name found during the VPATH
search.

You really need to show the rules that create these targets. You've provided a lot of information about many aspects of your build, but one of the most critical aspects of debugging make issues is seeing the rules.
Failures like this (where the first build fails and the second succeeds) are almost invariably due to rules that do not behave as you have described them to make. If you tell make that a rule builds a file "foo", by creating something like foo : bar, but then the recipe that you write doesn't actually create "foo" but rather "bar/foo", something like:
foo : bar ; cp bar bar/foo
then that is wrong. Another common reason for such issues is that you define a rule that builds "foo.x", but it also builds "foo.y" and you don't tell make about it but then later use "foo.y" as a prerequisite, that can't work. You have to define a rule that tells make that both those files are generated from a single invocation of the recipe, such as:
%.x %.y : %.z ; cp $< $*.x && cp $< $*.y
And finally, you mention VPATH above and it looks like you're trying to use VPATH to find generated files. That will not work. VPATH can only be used to find source files (files make doesn't know how to build and expects to always be present). Without seeing more about your makefiles and how the rules are constructed, that's about all we can say.
Edited to add:
I'm not sure it's really necessary to use so much eval and call here; often people seem to go straight to these very powerful tools when simpler ones would suffice just as well. In any event, the problem you're having is exactly as I suspected in my first comment above; you have this rule:
%.t.F90: %.$(1)
#echo " Processing templates in $$^"
#echo $$(seperator)
#cp $$< dir/$$#
Note the last line, where instead of creating $# you're creating dir/$#; this is precisely the situation I described above. Whenever you have a rule that builds something that is not exactly $#, it's almost 100% certain that rule is wrong. You need to write this as:
dir/%.t.F90: %.$(1)
#echo " Processing templates in $$^"
#echo $$(seperator)
#cp $$< $$#
and maybe more changes to match that target.

I make a workaround that I inserted some scripts in the implicit rule that add the missing directory part of the code, since VPATH has some constraints on this.
%.o: %.$(1)
#echo " Creating dependency $$#"
#echo $$(seperator)
#TEMPLATE_PATTERN='.*\.t\.$(1)'; \ <
if [[ $$$$(dirname $$<) == '.' && '$$<' =~ $$$$TEMPLATE_PATTERN ]]; then \ <
SRC=$(PROJECT_ROOT)/.codemate/processed_codes/$$<; \ <
else \ <
SRC=$$<; \ <
fi; \ <
$(FC) -c $$$$SRC $(OPTIONS) $(FFLAGS) $(INCLUDES)
labeled by <.

Related

Delete targets with recipes failed in Makefile

I tried to use .DELETE_ON_ERROR target in makefile in order to delete both $(OBJ)
and executable files if the recipe fails, but it doesn't work. If I put an error inside any object file than while compiling the pattern rule an error occurs and it stops. The old object file is still on its place but I expect .DELETE_ON_ERROR to remove it.
Can anyone test the code? Can -include $(DEP) or flag -DDBG influence? The goal is to delete both the .o file that failed and the executable.
OUTPUT = executable
CPP := $(shell find $(SRC) -type f -name "*.cpp")
OBJ := $(CPP:.cpp=.o)
DEP := $(OBJ:.o=.d)
CXX := g++
CXXFLAGS =-MMD -MP -DDBG
INCLUDES = -I.
.DELETE_ON_ERROR :
$(OUTPUT): $(OBJ)
$(CXX) $^ -o $#
%.o: %.cpp
$(CXX) $(CXXFLAGS) $(INCLUDES) -c $< -o $#
-include $(DEP)
.PHONY : clean
clean:
rm -rf $(OBJ) $(DEP)
EDIT: According to the Ondrej K. solution to fix this problem you need to add #touch command before compilator in order to make the object files changed (the docs read "delete the target of a rule if it has changed".). So, the code should look like this:
%.o: %.cpp
#touch $#
$(CXX) $(CXXFLAGS) $(INCLUDES) -c $< -o $#
Not sure what failure you're seeing, but I am afraid there really isn't a good way for you to do that. .o files and executable ($(OUTPUT)) are separate rules. If the latter fails, former is already out of consideration. See the documentation:
.DELETE_ON_ERROR:
If .DELETE_ON_ERROR is mentioned as a target anywhere in the makefile, then make will delete the target of a rule if it has changed and its recipe exits with a nonzero exit status, just as it does when it receives a signal. See Errors in Recipes.
In other words, if your the target producing a binary object failed after .o target itself got updated, make would prune the changed file. But if your executable did not link, it won't go back and delete object files.
Not sure it'd be nice, but if you really needed to, you could probably achieve this by refactoring your makefile to basically have direct exec + objs from source prerequisites rule with a single recipe. Obvious downside, such rule would mean single .c file change causing all files being recompiled (basically negating substantial benefit of using make).
EDIT: I'll expand on the comment a bit to clarify. What you seem to want is: in case there is a broken .c file and compilation fails, remove the old .o file. That is quite clearly not how .DELETE_ON_ERROR works though. If the .o file already got updated, and then the rule failed, it would remove it ("delete the target of a rule if it has changed"), but in case of a mentioned syntactical problem, the compiler would fail before it would produced an .o file.
So, if for instance you updated your (pattern) rule for compilation so that it first touches (effectively updates timestamp) on the .o file and then tries to compile. After the compiler call and rule failed make would consider the target of the failed root to have been updated and remove it. Alternatively you could also change to rule to first try to rm the expected '.o' file in which case you actually wouldn't need to use .DELETE_ON_ERROR (and if there is no change in the relevant sources, the rule does not get used, so it's actually not as terrible as it sounds). Either way is not exactly very clean, but leads towards the behavior I understand you're describing.
It is possible that the Compiler crashes while writing the Output file. In this case, there is a corrupt output file that is newer than its sources. Make will stop due to the error, but on next run, it won't recompile the output file as it is newer than ist sources - and the make will fail again and again in the build step.
With the .DELETE_ON_ERROR rule, make will delete the Output file if the compiler (or whatever build step failed) exits with an error after touching (and corrupting) the Output file, so it will be recompiled on next run. (if the Compiler failed without touching the old output file, it will always be recompiled on next run anyway)

gcc - What does ../ (dot dot slash) mean in a variable in a Makefile?

I have searched for hours for an answer to this. I am new to gcc and Makefiles.
I have a Makefile in some source code that looks like this:
CC=gcc
SRCDIR=src
BINDIR=../bin
CFLAGS= -flag
LIBS= -lthing
...
$(BINDIR)/program_name: $(SRCDIR)/program_name.c
$(CC) $(CFLAGS) $(SRCDIR)/program_name.c -o $(BINDIR)/program_name $(LIBS)
I understand what all of this means except what ../ in BINDIR is meant to do. When I make the Makefile, I get the error message:
/usr/bin/ld: cannot open output file ../bin/program_name: No such file or directory
collect2: error: ld returned 1 exit status
Makefile:20: recipe for target '../bin/program_name' failed
make: *** [../bin/program_name] Error 1
My guess is that the original author of this Makefile meant that the bin folder should go in the parent directory of where the Makefile is located. I know when using the Linux CLI command cd that the dot dot means go up a directory. Is that what this is trying to achieve?
To automatically create the $(BINDIR) directory before it is actually needed you must declare it as a prerequisite (dependence) of any target that uses it. But each time its content changes its timestamp also changes. So, declaring it as a regular prerequisite is not the best thing to do because the targets depending on it would be re-built without real reason, just because the content of $(BINDIR) changed.
This is why make also supports order-only prerequisites (OOPs):
$(BINDIR)/program_name: $(SRCDIR)/program_name.c | $(BINDIR)
$(CC) $(CFLAGS) $< -o $# $(LIBS)
$(BINDIR):
mkdir -p $#
Note the | that introduces the list of OOPs. An OOP is built if it does not exist, which causes the targets depending on it to be (re-)built too. But if it exists make does not even consider its last modification time. Even if some target depending on it is older, it is not rebuilt just because of that.
Note: I also used the $< and $# automatic variables. In the rule's recipe they expand as the first prerequisite ($(SRCDIR)/program_name.c) and the target ($(BINDIR)/program_name), respectively. They are highly recommended: less typing, less errors prone, more generic rules... they have many good properties.
Your makefile is missing a rule to create the BINDIR directory - if it doesn't exist, your link line won't be able to put the resulting binary there! A rule like this one should do it:
$(BINDIR):
mkdir -p $(BINDIR)
Just make sure that any other rules (like the one in your question) also depend on this directory!

GNU make - transform every prerequisite into target (implicitly)

I have another make-like tool that produces an XML as an artifact after parsing my makefile which I'll then further process with Python.
It'd simplify things for me - a lot - if I could have make consider every single prerequisite to be an actual target because then this other tool
will classify each and every file as a "job".
This is a fragment of my makefile:
.obj/eventlookupmodel.o: C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h \
...
I'd want for make to think I have a dummy rule for each prerequisite such as below:
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h:
#echo target pre= $#
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h:
#echo target pre=$#
C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp :
#echo target pre=$#
C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h:
#echo target pre=$#
I don't care about the exact form of the rule just that each file is considered an actual target.
My method of passing in this rule would be by setting the MAKEFILES variable like so
make all MAKEFILES=Dummy.mk
with Dummy.mk containing this rule so that I do not modify the makefiles.
I've tried the following so far.
Dummy.mk:
%.h:
#echo header xyz = $#
%:
#echo other xyz= $#
This partially works.
I run make all --trace --print-data-base MAKEFILES=Dummy.mk and I can see that
make does "bind" the %.h: rule to the header files. In the --print-data-base section, I see that rule being assigned to the header files.
C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef.h:
# Implicit rule search has been done.
# Implicit/static pattern stem: 'C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef'
# Last modified 2016-05-27 12:39:16
# File has been updated.
# Successfully updated.
# recipe to execute (from '#$(QMAKE) top_builddir=C:/Users/User1/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/ top_srcdir=C:/Users/User1/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile C:/Users/User1/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro
', line 2):
#echo header xyz = $#
However, I do NOT see the "echo header xyz $#"-rule being executed.
Regarding the %: rule, it is neither executed for the .cpp files nor "bound" to them in the --print-data-base section.
However, it is bound and executed for existing targets which have no suffix i.e.
all: library binary
binary: | library
ifs: | library
For the %: rule, the reason for this behavior is because of 10.5.5 Match-Anything Pattern Rules: If you do not mark the match-anything rule as terminal, then it is non-terminal. A non-terminal match-anything rule cannot apply to a file name that indicates a specific type of data. A file name indicates a specific type of data if some non-match-anything implicit rule target matches it.
If I make it non-terminal - no double colon - then the rule doesn't apply to built-in types like .cppunless I un-define the built-in rules that negate my intended %: rule.
If I make it terminal, "it does not apply unless its prerequisites actually exist". But a .h or .cpp doesn't technically have prerequisites; can I just create a dummy file and have that as its prerequisite?
NOTE: This has NOTHING to do with gcc -M generation. Yes the -M option would help in the specific case of header and source files but this question is for more generic targets and prerequisites that already exist in the makefile when make is launched.
This may take a few iterations. Try:
%.h: null
#echo header xyz = $#
%: null
#echo other xyz= $#
null:
#:
Try generating static pattern rules for the header files. See one of the answers to Make ignoring Prerequisite that doesn't exist.
Static pattern rules only apply to an explicit list of target files like this:
$(OBJECTS): %.o: %.c
*recipe here*
where the variable OBJECTS is defined earlier in the makefile to be a list of target files (separated by spaces), for example:
OBJECTS := src/fileA.c src/fileB.c src/fileC.c
Note that you can use the various make utility functions to build that list of target files. For example, $(wildcard pattern), $(addsuffix), etc.
You should also ensure that the recipe "touches" the header file to change the timestamp.
I've found that using static pattern rules instead of pattern rules fixes problems where make doesn’t build prerequisites that don’t exist, or deletes files that you want.
Here is an example of using wildcard to copy files from one directory to another.
# Copy images to build/images
img_files := $(wildcard src/images/*.png src/images/*.gif src/images/*.jpg \
src/images/*.mp3)
build_images := $(subst src/,$(BUILD_DIR)/,$(img_files))
$(build_images): $(BUILD_DIR)/images/% : src/images/%
mkdir -p $(dir $#)
cp -v -a $< $#
There are other make functions like addprefix that could be used to generate a more complex file specification.

parallel make: two targets depend on the same prerequisite, what happens?

I need to build sources to binary file and two a static library.
Here is an example (I replaced recipes with ';' for brevity):
objects := a.o b.o ...
.PHONY: all build build_lib
all: build build_lib
build: bin/app
build_lib: bin/libapp.a
bin/app: $(objects) ;
bin/libapp.a $(objects) ;
obj/%.o: %.cpp ;
Will there be problems with parallel build? Can two make processes try to rebuild the same *.o file at once, making a broken build?
I guessed that, they can, so I rewritten the code this way:
objects := a.o b.o ...
.PHONY: all build build_lib
all: $(objects) | bin/app bin/libapp.a
build: bin/app
build_lib: bin/libapp.a
bin/app: $(objects) ;
bin/libapp.a $(objects) ;
obj/%.o: %.cpp ;
But the --debug=b output still shows:
Processing target file `all'.
File `all' does not exist.
Processing target file `bin/app'.
File `bin/app' does not exist.
Processing target file `obj/client.o'.
Need to rebuild target `obj/client.o'.
...
File `sb_all' does not exist.
File `bin/app' does not exist.
File `bin/libapp.a' does not exist.
File `sb_all' does not exist.
File `bin/app' does not exist.
File `bin/libapp.a' does not exist.
...
Need to rebuild target `bin/app'.
g++ -lgd ...
Need to rebuild target `bin/libapp.a'.
ar ...
File `all' does not exist.
Target file `all' rebuilt successfully.
So it seems that my $(objects) target does not run before order-only prerequisites, or do I incorrectly read output? And did I need this change anyway?
No, there is no problem with it. Make will not have any problem with parallelism and multiple targets (in the same instance of make) depending on the same prerequisite. If you have recursive instances of make and multiple different make instances try to build the same target you'll have problems.
Order-only doesn't have any impact on parallelism at all. Make will still invoke things in parallel if possible. The only way to impact the order in which rules are run is to declare a prerequisite relationship between those targets. Here you're just saying that both the higher-level targets must be built before the all target, so that doesn't do anything to reduce parallelism.
Luckily as I said above, you don't have to. As long as your makefile correctly defines the dependency relationship between any two targets, make will handle the bigger picture just fine.

Make recompiles non-updated files

I was experimenting with GNU make. Suppose, I have 3 C files with the following structure:
hellomake.c
|
|---------------------
| |
V V
hellofunc.c hellomake.h
makefile:
hellomake: hellomake.c hellofunc.c
gcc -o hellomake hellomake.c hellofunc.c -I.
When I type make for the 1-st time, it creates a program hellomake. When I run it for the 2-nd time, it prints:
make: `hellomake' is up to date.
Everything is working correctly.
I tried to use make for compiling LaTeX files. Suppose, I have 2 TeX files:
1.tex
|
V
1_data.tex
1_data is included into 1.tex internally.
Makefile:
COMMAND = pdflatex
all: 1.tex 1_data.tex
$(COMMAND) 1.tex
But it recompiles 1.tex every time I type make. Even if none of the files were modified.
What's wrong?
makesimply looks at whether or not all exists. Since it doesn't, it attempts to create it (but doesn't even notice that your commands do not create the file!)
Assuming pdflatex really creates a file somewhere, use the file name as the target name. Then it will be recreated only if it is older than the dependencies.
To recapitulate, a Makefile declares a mapping of target files, their dependencies, and how to create them from the dependencies. When a target is missing or out of date with respect to its dependencies, Make runs the commands for recreating it.
(My first attempt at articulating this answer mentioned .PHONY: targets as an aside, but that's not really useful in this context; if you declared .PHONY: all it would run the recipe even if a file named all existed, so that's the opposite behavior of what you are looking for.)

Resources