I want the rules of a target to be executed but all dependent targets shall regard this target as satisfied.
How can I achieve this?
Example:
$(NETWORK_SHARE)/foo.log:
#echo Warning: server offline, still continue ...
#exit 0
foo.csv: $(NETWORK_SHARE)/foo.log
#echo Long export from a (different) server
#echo sleep 20
#echo foo > $#
If $(NETWORK_SHARE)/foo.log exists: foo.csv shall be rebuilt if $(NETWORK_SHARE)/foo.log is newer than foo.csv; otherwise nothing should happen (default)
If $(NETWORK_SHARE)/foo.log does not exist (e.g., server offline, failure, ...) only a message indicating a problem should be printed but foo.csv shall only be built if foo.csv does not exist.
I played around with .PHONY and returning different return values but for case 2, the expensive "export" happens as soon as I execute something on $(NETWORK_SHARE)/foo.log ...
Regards
divB
Looks like instead of using some old file (that someone can accidentally touch), you can use an order-only prerequisite. Here's a quote from the GNU makefile manual (chapter 4.3)
Occasionally, however, you have a situation where you want to impose a
specific ordering on the rules to be invoked without forcing the
target to be updated if one of those rules is executed. In that case,
you want to define order-only prerequisites. Order-only prerequisites
can be specified by placing a pipe symbol (|) in the prerequisites
list: any prerequisites to the left of the pipe symbol are normal; any
prerequisites to the right are order-only:
targets : normal-prerequisites | order-only-prerequisites
Great, thanks to Thiton's answer in my related question (Force make to find out-of-date condition from file) I can now provide a hack to solve this:
.PHONY: always-remake
NETWORK_SHARE = //server/dfs/common/logs
.PHONY: all
all: foo.csv
# file does not exist ...
ifeq "$(wildcard $(NETWORK_SHARE)/foo.log)" ""
old_file: always-remake
#echo Warning: network is not available ....
foo.csv: old_file
#echo Expensive export
#sleep 10
#echo $# > $#
else
foo.csv: $(NETWORK_SHARE)/foo.log
#echo Doing expensive export since log file changed ...
#sleep 10
#echo $# > $#
endif
"old_file" is a dummy file which must exist and should never be newer than any other file (e.g. 1/1/1971, 00:00)
Regards
divB
Related
I want to be able use the result of a target created in a rule in the prerequisite of another rule in GNU make. So for example:
PREREQ = $(shell echo "reading target1" >&2; cat target1)
target1:
echo "prereq" > $#
target2: target1 $(PREREQ)
echo foo > $#
target2 should depend on prereq as read from the target1 file, but that is not in the file until the target1 recipe is executed.
Granted this is very contrived example with I am sure lots of suggestions about how to refactor this particular example but I'm not looking to refactor this example. This is just a simplified example of my more complicated problem where I need to derive prerequisites from the contents of a file that is not created until a recipe in the Makefile is executed.
The question is, [how] can I make expansion of $(PREREQ) (and therefore the execution of the $(shell cat target1) defer until after the target1 rule is actually executed?
Update: I tried .SECONDARYEXPANSION: but that doesn't seem to do the job:
$ make -d target2
...
reading target1
cat: target1: No such file or directory
...
Updating goal targets....
Considering target file 'target2'.
File 'target2' does not exist.
Considering target file 'target1'.
File 'target1' does not exist.
Finished prerequisites of target file 'target1'.
Must remake target 'target1'.
echo "prereq" > target1
[ child management ]
Successfully remade target file 'target1'.
Finished prerequisites of target file 'target2'.
Must remake target 'target2'.
echo foo > target2
[ child management ]
Successfully remade target file 'target2'.
As you can see, "reading target" was only printed once at the very beginning demonstrating that PREREQ is not expanded again due to the .SECONDEXPANSION: and the list of targets Considered for target2 did not include prereq.
Deferring the expansion of the prerequisite $(PREREQ) can be achieved by conditionally creating the target2 and relying on recursion:
ifndef expand-prereq
target2: target1
$(MAKE) --no-print-directory -f $(lastword $(MAKEFILE_LIST)) $# expand-prereq=y
else
target2: target1 $(PREREQ)
echo foo > $#
endif
The first time make runs for this makefile, the variable expand-prereq is not defined and therefore, the first targe2 rule is generated as a result of the conditional. This kind of dummy rule makes possible to update target1 without expanding $(PREREQ).
Matching this rule results in target1 being updated (since target1 is a prerequisite of this rule) and make being called recursively for the same makefile and with target2 as target.
The second time make is (recursively) invoked, the variable expand-prereq was defined by means of the command-line argument expand-prereq=y, so the second target2 rule is generated as a result of the else branch this time. This rule is the one that actually produces the target target2. Note that before this rule can be matched, target1 has been already created as a side effect of the first dummy rule, so the expansion of $(PREREQ) happens after target1 has been created (what you were looking for).
You could write the complete rule for target2 to a separate file and -include it:
Including Other Makefiles
How Makefiles Are Remade
The exact mechanics will depend on your specific use case, and it may well be impossible to achieve what we need using this approach, but it supports a variety of styles for automated dependency generation.
There are several solutions:
GNU make 4.4 has been released! Haven't tried yet, but the release notes claim that secondary expansion only expands the prerequisites when they're considered. Furthermore you can delay execution with the .WAIT special prerequisite. That works fine, I tested. If .WAIT really delays the second expansion of the prerequisites after .WAIT, you're good.
Recursive make. Restart make after the prerequisites for the second rule were updated. This is a bit lame solution, can't recommend it.
Produce the prerequisites into make include file(s). Make automatically restarts after updating include files (re-exec). I'm currently using this method, and it works great. Better than recursive make but still slow, as all the makefiles have to be parsed again. A possible solution is comparing the old and new prerequisites, and only updating the include file, if its content changed. The second rule also needs to be modified to do nothing, if the content changed. Make will run all rules before restarting, but if they don't update their targets, after the restart they'll be executed again, now with the proper prerequisites. An interesting feature of make is that you can define make variables inside a recipe, and use them in other recipes (but not in the prereq list, unless #1 above reall works).
I have another make-like tool that produces an XML as an artifact after parsing my makefile which I'll then further process with Python.
It'd simplify things for me - a lot - if I could have make consider every single prerequisite to be an actual target because then this other tool
will classify each and every file as a "job".
This is a fragment of my makefile:
.obj/eventlookupmodel.o: C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h \
...
I'd want for make to think I have a dummy rule for each prerequisite such as below:
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h:
#echo target pre= $#
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h:
#echo target pre=$#
C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp :
#echo target pre=$#
C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h:
#echo target pre=$#
I don't care about the exact form of the rule just that each file is considered an actual target.
My method of passing in this rule would be by setting the MAKEFILES variable like so
make all MAKEFILES=Dummy.mk
with Dummy.mk containing this rule so that I do not modify the makefiles.
I've tried the following so far.
Dummy.mk:
%.h:
#echo header xyz = $#
%:
#echo other xyz= $#
This partially works.
I run make all --trace --print-data-base MAKEFILES=Dummy.mk and I can see that
make does "bind" the %.h: rule to the header files. In the --print-data-base section, I see that rule being assigned to the header files.
C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef.h:
# Implicit rule search has been done.
# Implicit/static pattern stem: 'C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef'
# Last modified 2016-05-27 12:39:16
# File has been updated.
# Successfully updated.
# recipe to execute (from '#$(QMAKE) top_builddir=C:/Users/User1/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/ top_srcdir=C:/Users/User1/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile C:/Users/User1/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro
', line 2):
#echo header xyz = $#
However, I do NOT see the "echo header xyz $#"-rule being executed.
Regarding the %: rule, it is neither executed for the .cpp files nor "bound" to them in the --print-data-base section.
However, it is bound and executed for existing targets which have no suffix i.e.
all: library binary
binary: | library
ifs: | library
For the %: rule, the reason for this behavior is because of 10.5.5 Match-Anything Pattern Rules: If you do not mark the match-anything rule as terminal, then it is non-terminal. A non-terminal match-anything rule cannot apply to a file name that indicates a specific type of data. A file name indicates a specific type of data if some non-match-anything implicit rule target matches it.
If I make it non-terminal - no double colon - then the rule doesn't apply to built-in types like .cppunless I un-define the built-in rules that negate my intended %: rule.
If I make it terminal, "it does not apply unless its prerequisites actually exist". But a .h or .cpp doesn't technically have prerequisites; can I just create a dummy file and have that as its prerequisite?
NOTE: This has NOTHING to do with gcc -M generation. Yes the -M option would help in the specific case of header and source files but this question is for more generic targets and prerequisites that already exist in the makefile when make is launched.
This may take a few iterations. Try:
%.h: null
#echo header xyz = $#
%: null
#echo other xyz= $#
null:
#:
Try generating static pattern rules for the header files. See one of the answers to Make ignoring Prerequisite that doesn't exist.
Static pattern rules only apply to an explicit list of target files like this:
$(OBJECTS): %.o: %.c
*recipe here*
where the variable OBJECTS is defined earlier in the makefile to be a list of target files (separated by spaces), for example:
OBJECTS := src/fileA.c src/fileB.c src/fileC.c
Note that you can use the various make utility functions to build that list of target files. For example, $(wildcard pattern), $(addsuffix), etc.
You should also ensure that the recipe "touches" the header file to change the timestamp.
I've found that using static pattern rules instead of pattern rules fixes problems where make doesn’t build prerequisites that don’t exist, or deletes files that you want.
Here is an example of using wildcard to copy files from one directory to another.
# Copy images to build/images
img_files := $(wildcard src/images/*.png src/images/*.gif src/images/*.jpg \
src/images/*.mp3)
build_images := $(subst src/,$(BUILD_DIR)/,$(img_files))
$(build_images): $(BUILD_DIR)/images/% : src/images/%
mkdir -p $(dir $#)
cp -v -a $< $#
There are other make functions like addprefix that could be used to generate a more complex file specification.
From the docs:
$?
The names of all the prerequisites that are newer than the target,
with spaces between them.
So, given a makefile:
# Force make to search for 'foo' in the VPATH directory
$(shell rm -rf foo)
# If 'D' is a "regular" file, we remove it first.
$(shell rm -rf D)
$(shell mkdir D)
# Suggest a VPATH-file, for Make to "associate" with 'foo'.
$(shell touch D/foo)
$(shell sleep 1)
# Target 'all' is newer than prerequisite 'D/foo'
$(shell touch all)
VPATH = D
all : foo phony
echo '$?'
foo ::
touch '$#'
.PHONY: phony
.PRECIOUS : D/foo
And running, I get:
$ make -r
touch 'D/foo'
echo 'D/foo phony'
D/foo phony
# Try again, but this time, with parallel-execution mode.
$ make -r -j
touch 'D/foo'
echo 'phony'
phony
Here, we have 2 serious issues:
Given the simple and explicit recipe to "touch" the prerequisite foo, which Make clearly executes - hence will guarantee that foo will be "newer" than all - Make still does not expand $? to D/foo, at-least in the 2nd case above (i.e. for the parallel-execution (-j) mode). Why?
If you come up with an explanation for the above, shouldn't it also explain, why in the 1st case (non-parallel execution), $? - does indeed - get expanded to D/foo.
I guess, I had an assumption, that parallel vs. non-parallel aside, Make will always pause before executing a target, and first check if all of its prerequisites had already finished their respective builds.
So, shouldn't the $? variable be identically expanded for both cases?
I think there are two issues going on here.
The first is that double-colon rules appear to act like phony targets in that they force make to consider the target as "newer" regardless of actual modification time. (This is why the non-parallel version behaves the way it does. Change from foo :: to foo : and you don't get foo in the $? output at all.)
The second thing is that, despite that, using parallel mode seems to force make back into considering modification times of its prerequisites (so the previous behavior is avoided).
This is conjecture and not definitive since I haven't dug through the code to see if this is actually happening but it explains the results here (it also explains the results on the other, nearly identical, question here).
By default make authomatically deletes targets when they are not needed anymore.
For example:
do_it: write_bar write_baz
echo done > $#
.INTERMEDIATE: write_foo write_bar write_baz
write_foo:
echo foo > $#
write_bar: write_foo
cat $< > $#
echo bar >> $#
write_baz: write_foo
echo buz > $#
Here write_foo will be deleted after execution of both write_bar and write_baz.
I want not to simply remove file, but to do some actions just before write_foo deleted.
Can I change command for auto deletion or assign any hook for this action?
Update: autodeletion is applicable only for intermediate rules.
make (at least GNU Make) does not automatically delete successfully built targets (it does delete target if make was killed).
.DEFAULT does not do what you expect it to do. See https://www.gnu.org/software/make/manual/html_node/Special-Targets.html
Invoking make on command line executes first target. Since your first target is write_foo, only write_foo will be executed.
If you invoke make do_it, all targets will be executed (and 3 files will appear in current directory).
Added after question update:
According to https://www.gnu.org/software/make/manual/html_node/Special-Targets.html, including intermediate target in prerequisites for .PRECIOUS special target prevents deleting it.
What does .PHONY mean in a Makefile? I have gone through this, but it is too complicated.
Can somebody explain it to me in simple terms?
By default, Makefile targets are "file targets" - they are used to build files from other files. Make assumes its target is a file, and this makes writing Makefiles relatively easy:
foo: bar
create_one_from_the_other foo bar
However, sometimes you want your Makefile to run commands that do not represent physical files in the file system. Good examples for this are the common targets "clean" and "all". Chances are this isn't the case, but you may potentially have a file named clean in your main directory. In such a case Make will be confused because by default the clean target would be associated with this file and Make will only run it when the file doesn't appear to be up-to-date with regards to its dependencies.
These special targets are called phony and you can explicitly tell Make they're not associated with files, e.g.:
.PHONY: clean
clean:
rm -rf *.o
Now make clean will run as expected even if you do have a file named clean.
In terms of Make, a phony target is simply a target that is always out-of-date, so whenever you ask make <phony_target>, it will run, independent from the state of the file system. Some common make targets that are often phony are: all, install, clean, distclean, TAGS, info, check.
Let's assume you have install target, which is a very common in makefiles. If you do not use .PHONY, and a file named install exists in the same directory as the Makefile, then make install will do nothing. This is because Make interprets the rule to mean "execute such-and-such recipe to create the file named install". Since the file is already there, and its dependencies didn't change, nothing will be done.
However if you make the install target PHONY, it will tell the make tool that the target is fictional, and that make should not expect it to create the actual file. Hence it will not check whether the install file exists, meaning: a) its behavior will not be altered if the file does exist and b) extra stat() will not be called.
Generally all targets in your Makefile which do not produce an output file with the same name as the target name should be PHONY. This typically includes all, install, clean, distclean, and so on.
NOTE: The make tool reads the makefile and checks the modification time-stamps of the files at both the side of ':' symbol in a rule.
Example
In a directory 'test' following files are present:
prerit#vvdn105:~/test$ ls
hello hello.c makefile
In makefile a rule is defined as follows:
hello:hello.c
cc hello.c -o hello
Now assume that file 'hello' is a text file containing some data, which was created after 'hello.c' file. So the modification (or creation) time-stamp of 'hello' will be newer than that of the 'hello.c'. So when we will invoke 'make hello' from command line, it will print as:
make: `hello' is up to date.
Now access the 'hello.c' file and put some white spaces in it, which doesn't affect the code syntax or logic then save and quit. Now the modification time-stamp of hello.c is newer than that of the 'hello'. Now if you invoke 'make hello', it will execute the commands as:
cc hello.c -o hello
And the file 'hello' (text file) will be overwritten with a new binary file 'hello' (result of above compilation command).
If we use .PHONY in makefile as follow:
.PHONY:hello
hello:hello.c
cc hello.c -o hello
and then invoke 'make hello', it will ignore any file present in the pwd 'test' and execute the command every time.
Now suppose, that 'hello' target has no dependencies declared:
hello:
cc hello.c -o hello
and 'hello' file is already present in the pwd 'test', then 'make hello' will always show as:
make: `hello' is up to date.
.PHONY: install
means the word "install" doesn't represent a file name in this
Makefile;
means the Makefile has nothing to do with a file called "install"
in the same directory.
It is a build target that is not a filename.
The special target .PHONY: allows to declare phony targets, so that make will not check them as actual file names: it will work all the time even if such files still exist.
You can put several .PHONY: in your Makefile :
.PHONY: all
all : prog1 prog2
...
.PHONY: clean distclean
clean :
...
distclean :
...
There is another way to declare phony targets : simply put :: without prerequisites :
all :: prog1 prog2
...
clean ::
...
distclean ::
...
The :: has other special meanings, see here, but without prerequisites it always execute the recipes, even if the target already exists, thus acting as a phony target.
The best explanation is the GNU make manual itself: 4.6 Phony Targets section.
.PHONY is one of make's Special Built-in Target Names. There are other targets that you may be interested in, so it's worth skimming through these references.
When it is time to consider a .PHONY target, make will run its recipe
unconditionally, regardless of whether a file with that name exists or
what its last-modification time is.
You may also be interested in make's Standard Targets such as all and clean.
There's also one important tricky treat of ".PHONY" - when a physical target depends on phony target that depends on another physical target:
TARGET1 -> PHONY_FORWARDER1 -> PHONY_FORWARDER2 -> TARGET2
You'd simply expect that if you updated TARGET2, then TARGET1 should be considered stale against TARGET1, so TARGET1 should be rebuild. And it really works this way.
The tricky part is when TARGET2 isn't stale against TARGET1 - in which case you should expect that TARGET1 shouldn't be rebuild.
This surprisingly doesn't work because: the phony target was run anyway (as phony targets normally do), which means that the phony target was considered updated. And because of that TARGET1 is considered stale against the phony target.
Consider:
all: fileall
fileall: file2 filefwd
echo file2 file1 >fileall
file2: file2.src
echo file2.src >file2
file1: file1.src
echo file1.src >file1
echo file1.src >>file1
.PHONY: filefwd
.PHONY: filefwd2
filefwd: filefwd2
filefwd2: file1
#echo "Produced target file1"
prepare:
echo "Some text 1" >> file1.src
echo "Some text 2" >> file2.src
You can play around with this:
first do 'make prepare' to prepare the "source files"
play around with that by touching particular files to see them updated
You can see that fileall depends on file1 indirectly through a phony target - but it always gets rebuilt due to this dependency. If you change the dependency in fileall from filefwd to file, now fileall does not get rebuilt every time, but only when any of dependent targets is stale against it as a file.
I often use them to tell the default target not to fire.
superclean: clean andsomethingelse
blah: superclean
clean:
#echo clean
%:
#echo catcher $#
.PHONY: superclean
Without PHONY, make superclean would fire clean, andsomethingelse, and catcher superclean; but with PHONY, make superclean won't fire the catcher superclean.
We don't have to worry about telling make the clean target is PHONY, because it isn't completely phony. Though it never produces the clean file, it has commands to fire so make will think it's a final target.
However, the superclean target really is phony, so make will try to stack it up with anything else that provides deps for the superclean target — this includes other superclean targets and the % target.
Note that we don't say anything at all about andsomethingelse or blah, so they clearly go to the catcher.
The output looks something like this:
$ make clean
clean
$ make superclean
clean
catcher andsomethingelse
$ make blah
clean
catcher andsomethingelse
catcher blah