When I execute Make, I'd like to know what shell commands / recipes are executed (and perhaps which line in which Makefile invoked these). Is there a way of doing this (print onto the stdout or write to a file) without modifying the Makefiles?
The commands will be echoed to stdout by default, unless you disabled that by prefixing them with #.
A few options you can pass to make:
-n: echos commands that would be run, but without running them. This will also echo commands that are prefixed with #.
-d: print debug output. This will output the process that make goes through in attempting to find rules that match the targets that it's trying to make. There will be a lot of useless information in here, as it will try many default build rules (like how to build .o files from .c).
There are a few ways to cut down on the noise:
-r: disable implicit rules. If you don't depend on any of the default rules , this can be used in conjunction with -d to just print the parts you (mostly) care about
--debug=b: basic debug mode. prints what it's trying to make, but not any information about implicit rules.
None of these print the line numbers of the commands, but make -n --debug=b will print both the targets being built and the commands being run, so it's almost as good. Example below.
$ cat makefile:
c: a b
cat $^ > $#
a:
echo 'foo' > $#
b: a
cat $^ > $#
echo 'bar' >> $#
$ make -n --debug=b:
Reading makefiles...
Updating goal targets....
File 'a' does not exist.
Must remake target 'a'.
echo 'foo' > a
Successfully remade target file 'a'.
File 'b' does not exist.
Must remake target 'b'.
cat a > b
echo 'bar' >> b
Successfully remade target file 'b'.
Prerequisite 'a' is newer than target 'c'.
Prerequisite 'b' is newer than target 'c'.
Must remake target 'c'.
cat a b > c
Successfully remade target file 'c'.
Related
Using make for generic purposes (not compiling)
Suppose I have a set of files with full path names, and I would like to do something with that file.
something --file a/b/c/X > $< # (for example)
And I have made a rule:
something-%:
something --file $*
Which matches fine against, say, "something-foo" but does not catch "something-a/b/c/foo". Is there a way to write a wildcard rule for this latter case?
The manual describes how patterns match:
When the target pattern does not contain a slash (and it usually does not), directory names in the file names are removed from the file name before it is compared with the target prefix and suffix.
In your case when calling as make something-a/b/c/foo, something-a/b/c/ is treated as directory and removed, so the rest does not match your rule. This can be easily checked:
$ cat Makefile
something-%:
echo something --file $<
f%o:
echo $*
Output:
$ make something-OtherDirectory/src/foo -dr
GNU Make 4.2.1
...
Considering target file 'something-OtherDirectory/src/foo'.
File 'something-OtherDirectory/src/foo' does not exist.
Looking for an implicit rule for 'something-OtherDirectory/src/foo'.
Trying pattern rule with stem 'o'.
Found an implicit rule for 'something-OtherDirectory/src/foo'.
Finished prerequisites of target file 'something-OtherDirectory/src/foo'.
Must remake target 'something-OtherDirectory/src/foo'.
echo something-OtherDirectory/src/o
...
Note that it matched the other pattern rule with the stem of o.
You can make it work your way if your pattern does include a slash. For sake of completeness I would also define a prerequisite if your rule is based on a file and declare target as phony if it does not generate a real output file:
$ cat Makefile
.PHONY: something/%
something/%: %
echo something --file $<
Output:
$ make something/OtherDirectory/src/foo.c
echo something --file OtherDirectory/src/foo.c
something --file OtherDirectory/src/foo.c
This is easily a duplicate of this question, but it has not been answered, for what I can see.
Here is my goal: to be able to print the prerequisites to a target.
I have some kind of a solution but it feels like a hack to me.
Say the target is all, and it only has prerequisites, in a file named makefile-1:
all: foo
all: bar baz
I can use another makefile named makefile-2:
all:
ifeq ($(SHOW),yes)
echo $^
else
cat $^
endif
This kind of gives me what I need, when invoked properly:
$ make -f makefile-1 -f makefile-2
cat foo bar baz
$ make -s SHOW=yes -f makefile-1 -f makefile-2
foo bar baz
But I am not sure what to do if the prerequisite actually has a recipe in the original makefile, or if there is a better/cleaner way.
An easier way to show the prerequisites to a target without building the target would probably be to use the -p option (as suggested here) and -q options together, as suggested on the GNU-Make manual page (long name for the option is --print-data-base):
make -qp | grep '^all:'
This will print the database and select only the line which has the target all and its prerequisites. If there is a rule for that target, it would be at the end of the same paragraph (if I am understanding the format of the output correctly).
If multiple makefiles are used, one can specify all of them with the -f option:
make -f makefile-1 -f makefile-2 -qp
This will of course collect prerequisites for the same target from all makefiles, but the target cannot have a rule specified in more than one file.
I was thinking about using Make for small checks for my dev setup. One thing I want is to check that a particular string exists in a file before doing some action. If I wanted to create the entire file it would be trivial
action: filename
...
filename:
echo 'blah' >> filename
But how can this logic be applied to actions, like grep? My dependency isn't that a file exists, it's that the file has correct content.
I'm asking specifically about Make and not other solutions like chef/puppet
You can run any shell commands you want in a make recipe. As many of them as you want also.
So if you need to run grep before doing something else just do that.
Just remember that every line in a recipe is run in its own shell session so they don't share state.
So this:
action: filename
...
filename:
grep -qw blah $# || echo 'blah' > $#
runs grep on filename (via the automatic variable for the current target $#) looking for whole words and quitting on the first match (-q).
If grep finds blah then it will return success and the || will short-circuit and the recipe is done. If grep fails then the || will trigger and the echo will run.
You might be tempted to do things that require the inverse logic do X only if Y is true:
filename:
grep -qw blah $# && echo blah2 > $#
but that doesn't work correctly. When grep fails the && short-circuits and make sees a recipe failure and bails the make process out with an error.
You need this instead.
filename:
! grep -qw blah $# || echo blah2 > $#
to invert the logic and ensure that the "failure" from grep is seen as success as far as make is concerned.
That all being said in this specific example if filename exists at all then that recipe won't ever run as it has no prerequisites so make will always consider it up to date. To work around that you need to give the file a prerequisite that will force it to be considered out of date. Specifically a force target.
Don't follow the advice about .PHONY for this case though. .PHONY targets should never be prerequisites of non-.PHONY targets.
Expanding on what #john wrote I got the following to work:
TEST_FILE=filename
.PHONY: ${TEST_FILE}
string=testing
filecheck=$(shell grep -qw ${string} ${TEST_FILE} || echo ${TEST_FILE})
all: ${filecheck}
${TEST_FILE}:
echo 'changing the file'
echo ${string} >> ${TEST_FILE}
Here the file on which I'm operating is a .PHONY target. I think that's ok because I'm actually not creating the file, just modifying it. This will work if the file does not exist, or exists without the needed string.
You could add a test in the target's recipe (As Etan posted before I could complete this answer...). If you do want to do this using just make logic, you could do something along the lines of:
actions: $(if $(shell grep -q $$string filename && echo y),filename,)
filename:
echo blah >> $#
If filename contains the string, then there will be an actions: filename dependency, and filename will be built when you build actions. Notice, though that this will check whether the string exists in filename at the time the makefile is parsed -- if filename is generated, or modified in this makefile, then it would not effect whether the action is run. If you want to test right before overwriting the file, then you would use a bash if statement in the recipe itself.
I'm using Gnu Make 3.81, and getting an error trying to match a pattern rule that also has a variable in it.
Here's the smallest example I could come up with:
YYMMDD:=$(shell date +%y%m%d)
TMP_DIR:=/tmp/$(YYMMDD)
# create a temporary directory, and put a "source" file in it
$(TMP_DIR):
mkdir $(TMP_DIR)
echo something > $(TMP_DIR)/somefile.orig
# to build an "object" file in the temp dir, process the "source" file
$(TMP_DIR)/%.new: $(TMP_DIR)/%.orig
wc -l $< > $#
atarget: $(TMP_DIR) $(TMP_DIR)/somefile.new
Then when I run make atarget, I get:
mkdir /tmp/141021
echo something > /tmp/141021/somefile.orig
make: *** No rule to make target `/tmp/141021/somefile.new', needed by `atarget'. Stop.
Shouldn't this work? it seems like the pattern rule should match this just fine.
It's because make doesn't know that the .orig file exists: you have a rule that builds $(TMP_DIR) but make doesn't know that this rule also builds $(TMP_DIR)/somefile.orig. So when make is trying to match the pattern rule it will see that the .orig file doesn't exist and it doesn't have any way that it knows how to make that file, so the pattern doesn't match, and after that there's no way to build the .new file.
You should write:
$(TMP_DIR)/%.orig:
mkdir -p $(TMP_DIR)
echo $* > $#
then it will work.
I'm running the following kind of pipeline:
digestA: hugefileB hugefileC
cat $^ > $#
rm $^
hugefileB:
touch $#
hugefileC:
touch $#
The targets hugefileB and hugefileC are very big and take a long time to compute (and need the power of Make). But once digestA has been created, there is no need to keep its dependencies: it deletes those dependencies to free up disk space.
Now, if I invoke 'make' again, hugefileB and hugefileC will be rebuilt, whereas digestA is already ok.
Is there any way to tell 'make' to avoid to re-comile the dependencies ?
NOTE: I don't want to build the two dependencies inside the rules for 'digestA'.
Use "intermediate files" feature of GNU Make:
Intermediate files are remade using their rules just like all other files. But intermediate files are treated differently in two ways.
The first difference is what happens if the intermediate file does not exist. If an ordinary file b does not exist, and make considers a target that depends on b, it invariably creates b and then updates the target from b. But if b is an intermediate file, then make can leave well enough alone. It won't bother updating b, or the ultimate target, unless some prerequisite of b is newer than that target or there is some other reason to update that target.
The second difference is that if make does create b in order to update something else, it deletes b later on after it is no longer needed. Therefore, an intermediate file which did not exist before make also does not exist after make. make reports the deletion to you by printing a rm -f command showing which file it is deleting.
Ordinarily, a file cannot be intermediate if it is mentioned in the makefile as a target or prerequisite. However, you can explicitly mark a file as intermediate by listing it as a prerequisite of the special target .INTERMEDIATE. This takes effect even if the file is mentioned explicitly in some other way.
You can prevent automatic deletion of an intermediate file by marking it as a secondary file. To do this, list it as a prerequisite of the special target .SECONDARY. When a file is secondary, make will not create the file merely because it does not already exist, but make does not automatically delete the file. Marking a file as secondary also marks it as intermediate.
So, adding the following line to the Makefile should be enough:
.INTERMEDIATE : hugefileB hugefileC
Invoking make for the first time:
$ make
touch hugefileB
touch hugefileC
cat hugefileB hugefileC > digestA
rm hugefileB hugefileC
And the next time:
$ make
make: `digestA' is up to date.
If you mark hugefileB and hugefileC as intermediate files, you will get the behavior you want:
digestA: hugefileB hugefileC
cat $^ > $#
hugefileB:
touch $#
hugefileC:
touch $#
.INTERMEDIATE: hugefileB hugefileC
For example:
$ gmake
touch hugefileB
touch hugefileC
cat hugefileB hugefileC > digestA
rm hugefileB hugefileC
$ gmake
gmake: `digestA' is up to date.
$ rm -f digestA
$ gmake
touch hugefileB
touch hugefileC
cat hugefileB hugefileC > digestA
rm hugefileB hugefileC
Note that you do not need the explicit rm $^ command anymore -- gmake automatically deletes intermediate files at the end of the build.
I would recommend you to create pseudo-cache files that are created by the hugefileB and hugeFileC targets.
Then have digestA depend on those cache files, because you know they will not change again until you manually invoke the expensive targets.
See also .PRECIOUS:
.PRECIOUS : hugefileA hugefileB
.PRECIOUS
The targets which .PRECIOUS depends on are given the following special treatment: if make is killed or interrupted during the execution of their recipes, the target is not deleted. See Interrupting or Killing make. Also, if the target is an intermediate file, it will not be deleted after it is no longer needed, as is normally done. See Chains of Implicit Rules. In this latter respect it overlaps with the .SECONDARY special target.
You can also list the target pattern of an implicit rule (such as ‘%.o’) as a prerequisite file of the special target .PRECIOUS to preserve intermediate files created by rules whose target patterns match that file’s name.
Edit: On re-reading the question, I see that you don't want to keep the hugefiles; maybe do this:
digestA : hugefileA hugefileB
grep '^Subject:' %^ > $#
for n in $^; do echo > $$n; done
sleep 1; touch $#
It truncates the hugefiles after using them, then touches the output file a second later, just to ensure that the output is newer than the input and this rule won't run again until the empty hugefiles are removed.
Unfortunately, if only the digest is removed, then running this rule will create an empty digest. You'd probably want to add code to block that.
The correct way is to not delete the files, as that removes the information that make uses to determine whether to rebuild the files.
Recreating them as empty does not help because make will then assume that the empty files are fully built.
If there is a way to merge digests, then you could create one from each of the huge files, which is then kept, and the huge file automatically removed as it is an intermediate.