Let's say I have a make target abc with prerequisites x, y, and z.
abc: x y z
echo 'Do something'
Is there not some simple way to run make and ignore the prerequisites, such that it runs abc without first running x, y, and z?
$ make
echo x updated
x updated
echo y updated
y updated
echo z updated
z updated
echo all updated
all updated
$ make noreq=y
echo all updated
all updated
How it's done:
ifdef noreq
req :=
else
req = $(1)
endif
.PHONY: all x y z
all: $(call req,x y z)
echo $# updated
x y z:
echo $# updated
Note that a GNU Makefile can examine the value of the variable MKCMDGOALS and calculate other variables from that. You could have it that when certain targets are explicitly given on the command line (and thus appear in MKCMDGOALS) then the noreq flags comes into effect, and those targets adjust their behavior with the macro.
CAVEAT: In the noreq mode, the target actually does not have prerequisites, so your rule cannot refer to prerequisites via $^ or $<.
Related
I have a makefile that executes some shell command and I want to store the output to a global variable:
GLOBVAR = a
all:
GLOBVAR=$(shell echo 'X')
$(info $(GLOBVAR))
GLOBVAR is empty. What am I doing wrong?
You are mixing up make and shell variables. In GLOBVAR=$(shell echo 'X') it is a shell variable that you assign, while in $(info $(GLOBVAR)) it is a make variable that you expand.
Try this, instead:
GLOBVAR = $(shell echo 'X')
all:
$(info $(GLOBVAR))
But there are several other issues with your Makefile that you should probably consider.
Using $(shell...) in recipes is not recommended because recipes are already shell scripts. So, if you want to assign a shell variable in a recipe, just:
all:
GLOBVAR="$$(echo 'X')"
Note the $$ to escape the expansion that make performs before passing the recipes to the shell.
The different lines of the recipe are executed in different shells. So, if you want to use in a line a shell variable that was assigned in a previous line you must join them:
all:
GLOBVAR="$$(echo 'X')"; echo $$GLOBVAR
(same remark as before about $$). You can use line continuation if you prefer:
all:
GLOBVAR="$$(echo 'X')"; \
echo $$GLOBVAR
And finally, if you want to assign make variables in recipes you can, with the eval make function, but I strongly discourage you to do so until you perfectly understand when make does what:
$ cat Makefile
.PHONY: all lla
all:
$(eval GLOBVAR = $(shell echo 'X'))
#echo all: $(GLOBVAR)
lla:
#echo lla: $(GLOBVAR)
$ make all
all: X
$ make lla
lla:
$ make all lla
all: X
lla: X
$ make lla all
lla:
all: X
And I let you imagine what the results could be with parallel make... In summary, if you start using make functions in recipes you are probably wandering into dangerous areas.
I am checking for existence of flag that is passed by user to GNUmakefile.
Basically, i am checking whether user has passed -j in my makefile. I have added below if condition. But before that i am trying to display MAKEFLAGS where i can see output is empty for that variable.
ifneq (,$(findstring j,$(MAKEFLAGS)))
....
Am i missing anything here?
Sometimes users may also pass --jobs instead of -j , And also i need to check whether the value passed to -j/--jobs is greater than 2
Is there any easy way in GNUmake for doing so in single if condition ?
The answer to your question depends on what version of GNU make you're using.
If you're using GNU make 4.1 or below, then the answer is "no, it's not possible" from within a makefile (of course you can always write a shell script wrapper around make and check the arguments before invoking make).
If you're using GNU make 4.2 or above, then the answer is "yes, it's quite possible". See this entry from the GNU make NEWS file:
Version 4.2 (22 May 2016)
The amount of parallelism can be determined by querying MAKEFLAGS, even when
the job server is enabled (previously MAKEFLAGS would always contain only
"-j", with no number, when job server was enabled).
This is a tricky question because MAKEFLAGS is a very strange make variable. First of all, with GNU make 4.3, -jN, -j N, --jobs N and --jobs=N are all converted to -jN in MAKEFLAGS, which looks interesting. You could thus try something like:
J := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
to get the N value passed on the command line or the empty string if -j and --jobs have not been used. But then, if you try the following you will see that it is not the whole story:
$ cat Makefile
.PHONY: all
J := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
ifneq ($(J),4)
all:
#echo MAKEFLAGS=$(MAKEFLAGS)
#echo patsubst...=$(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
#echo J=$(J)
else
all:
#echo J=4
endif
$ make -j4
MAKEFLAGS= -j4 -l8 --jobserver-auth=3,4
patsubst...=4
J=
Apparently MAKEFLAGS is not set when the Makefile is parsed (and the J make variable is assigned the empty string) but it is when the recipes are executed. So, using MAKEFLAGS with conditionals does not work. But if you can move your tests in a recipe, something like the following could work:
.PHONY: all
all:
j=$(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS))); \
if [ -n "$$j" ] && [ $$j -gt 2 ]; then \
<do something>; \
else \
<do something else>; \
fi
Or:
.PHONY: all
J = $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
all:
#if [ -n "$(J)" ] && [ $(J) -gt 2 ]; then \
<do something>; \
else \
<do something else>; \
fi
Note the use of the recursively expanded variable assignment (J = ...) instead of simple assignment (J := ...).
I am attempting to do a data pipeline with a Makefile. I have a big file that I want to split in smaller pieces to process in parallel. The number of subsets and the size of each subset is not known beforehand. For example, this is my file
$ for i in {1..100}; do echo $i >> a.txt; done
The first step in Makefile should compute the ranges,... lets make them fixed for now
ranges.txt: a.txt
or i in 0 25 50 75; do echo $$(($$i+1))'\t'$$(($$i+25)) >> $#; done
Next step should read from ranges.txt, and create a target file for each range in ranges.txt, a_1.txt, a_2.txt, a_3.txt, a_4.txt. Where a_1.txt contains lines 1 through 25, a_2.txt lines 26-50, and so on... Can this be done?
You don't say what version of make you're using, but I'll assume GNU make. There are a few ways of doing things like this; I wrote a set of blog posts about metaprogramming in GNU make (by which I mean having make generate its own rules automatically).
If it were me I'd probably use the constructed include files method for this. So, I would have your rule above for ranges.txt instead create a makefile, perhaps ranges.mk. The makefile would contain a set of targets such as a_1.txt, a_2.txt, etc. and would define target-specific variables defining the start and stop values. Then you can -include the generated ranges.mk and make will rebuild it. One thing you haven't described is when you want to recompute the ranges: does this really depend on the contents of a.txt?
Anyway, something like:
.PHONY: all
all:
ranges.mk: a.txt # really? why?
for i in 0 25 50 75; do \
echo 'a_$$i.txt : RANGE_START := $$(($$i+1))'; \
echo 'a_$$i.txt : RANGE_END := $$(($$i+25))'; \
echo 'TARGETS += a_$$i.txt'; \
done > $#
-include ranges.mk
all: $(TARGETS)
$(TARGETS) : a.txt # seems more likely
process --out $# --in $< --start $(RANGE_START) --end $(RANGE_END)
(or whatever command; you don't give any example).
I have a project which includes a code generator which generates several .c and .h files from one input file with just one invocation of the code generator. I have a rule which has the .c and .h files as multiple targets, the input file as the prerequisite, and the recipe is the invocation of the code generator. I then have further rules to compile and link the generated .c files.
This works fine with a -j factor of 1, but if I increase the j factor, I find I get multiple invocations of the code generator, up to the -j factor or the number of expected target files, whichever is smallest. This is bad because multiple invocations of the code generator can cause failures due to the generated code being written multiple times.
I'm not going to post my actual (large) code here, but I have been able to construct a small example which appears to demonstrate the same behavior.
The Makefile looks like this:
output.concat: output5 output4 output3 output2 output1
cat $^ > $#
output1 output2 output3 output4 output5: input
./frob input
clean:
rm -rf output*
Instead of a code generator, for this example I have written a simple shell script, frob which generates multiple output files from one input file:
#!/bin/bash
for i in {1..5}; do
{
echo "This is output${i}, generated from ${1}. input was:"
cat ${1}
} > output${i}
done
When I run this Makefile with non-unity -j factors, I get the following output:
$ make -j2
./frob input
./frob input
cat output5 output4 output3 output2 output1 > output.concat
$
We see ./frob here gets invoked twice, which is bad. Is there some way I can construct this rule such that the recipe only gets invoked once, even with a non-unity -j factor?
I have considered changing the rule so that just one of the expected output files is the target, then adding another rule with no recipe such that its targets are the remaining expected output files, and the prerequisite is the first expected output file. But I'm not sure this would work, because I don't know if I can guarantee the order in which the files are generated, and thus may end up with circular dependencies.
This is how make is defined to work. A rule like this:
foo bar baz : boz ; $(BUILDIT)
is exactly equivalent, to make, to writing these three rules:
foo : boz ; $(BUILDIT)
bar : boz ; $(BUILDIT)
baz : boz ; $(BUILDIT)
There is no way (in GNU make) to define an explicit rule with the characteristics you want; that is that one invocation of the recipe will build all three targets.
However, if your output files and your input file share a common base, you CAN write a pattern rule like this:
%.foo %.bar %.baz : %.boz ; $(BUILDIT)
Strangely, for implicit rules with multiple targets GNU make assumes that a single invocation of the recipe WILL build all the targets, and it will behave exactly as you want.
Correctly generate and update multiple targets a b с in parallel make -j from input files i1 i2:
all: a b c
.INTERMEDIATE: d
a: d
b: d
c: d
d: i1 i2
cat i1 i2 > a
cat i1 i2 > b
cat i1 i2 > c
If any of a,b,c are missing, the pseudo-target d is remade. The file d is never created; the single rule for d avoids several parallel invocations of the recipe.
.INTERMEDIATE ensures that missing file d doesn't trigger the d recipe.
Some other ways for multiple targets in the book "John
Graham-Cumming - GNU Make Book" p.92-96.
#MadScientist's answer is promising - I think I could possibly use that. In the meantime, I have been playing with this some more and come up with a different possible solution, as hinted at in the question. I can split the rule in two as follows:
INPUT_FILE = input
OUTPUT_FILES = output5 output4 output3 output2 output1
OUTPUT_FILE1 = $(firstword $(OUTPUT_FILES))
OUTPUT_FILES_REST = $(wordlist 2,$(words $(OUTPUT_FILES)),$(OUTPUT_FILES))
$(OUTPUT_FILE1): $(INPUT_FILE)
./frob $<
touch $(OUTPUT_FILES_REST)
$(OUTPUT_FILES_REST): $(OUTPUT_FILE1)
Giving only one output file as a target fixes the possible parallelism problem. Then we make this one output file as a prerequisite to the rest of the output files. Importantly in the frob recipe, we touch all the output files with the exception of the first so we are guaranteed that the first will have an older timestamp than all the rest.
As of make 4.3 (Jan 2020) make allows grouped targets. As per docs the following will update all targets only once if any of the targets is missing or outdated:
foo bar biz &: baz boz
echo $^ > foo
echo $^ > bar
echo $^ > biz
Answer by Ivan Zaentsev almost worked for me, with exception of the following issue. Only when running parallel make (-j2 or above), when a prerequisite of the generated file was changed, the generated file was regenerated successfully, however, the subsequent targets that depend on the generated file were not rebuilt.
The workaround I found was to provide a recipe for the generated files (the trivial copy command), besides the dependency on the intermediate target (d):
d: i1 i2
cat i1 i2 > a.gen
cat i1 i2 > b.gen
cat i1 i2 > c.gen
.INTERMEDIATE: d
a.gen : d
b.gen : d
c.gen : d
a: a.gen d
cp $< $#
b: b.gen d
cp $< $#
c: c.gen d
cp $< $#
e: a b c
some_command $# $^
The clue was this debug output from make when running without the workaround (where 'e' was not rebuilt with make -j2, despite a,b,c being rebuilt):
Finished prerequisites of target file `a'.
Prerequisite `d' of target `a' does not exist.
No recipe for `a' and no prerequisites actually changed.
No need to remake target `a'.
Here is the solution that seemed to work for me (credit to #Ivan Zaentsev for the main solution and to #alexei for pointing out the problem with it). It is similar to the original approach with one major change. Instead of generating temporary files (*.gen as suggested), it just touches the files that depend on the INTERMEDIATE file. :
default: f
.INTERMEDIATE: temp
a b c: temp
touch $#
temp: i1 i2
echo "BUILD: a b c"
cat i1 i2 > a
cat i1 i2 > b
cat i1 i2 > c
e: a b c
echo "BUILD: e"
touch $#
f: e
echo "BUILD: f"
touch $#
Considering this (very) simple Makefile,
all: a b
#echo done
a:
touch $#
b:
touch $#
I didn't get the behaviour I expected.
In fact, I wish my rule "all" to be executed only if one of its depencies isn't up to date.
However on first $ make I get
touch a
touch b
done
And on others
done
I undenstand that "up to date" normally qualify a file but...
Is there any way to have rule "all" considered up to date when all its dependies are ?
Thanks a lot,
ps: I know the trick $ touch all, but that's not the point
No. If the target does not exist then make will always run the recipe.
There are hacky ways that your all recipe could check to see if any of the prerequisites had been built and behave differently if so. At least, if you're using GNU make.
ETA:
One way would be to add a $(eval BUILT = 1) to each target, then in the all target you can see if BUILT is set. This requires a moderately recent version of GNU make:
BUILT =
all: a b
#[ -n '$(BUILT)' ] && echo done
a:
touch $# $(eval BUILT = 1)
b:
touch $# $(eval BUILT = 1)