I have a third party makefile, and I'd like one of the targets (T1) to not be built until another, custom target (T2) is built first. Normally, this would be accomplished by making T2 a prerequisite of T1. BUT, T1 uses the $^ in one of its rules.. so, by adding the prerequisite, I end up breaking the build... What I have is this:
T1: x y z T2
$(MAKE) -j $^;
# fails because T2 should not be passed to the make!!!
.PHONY: T2
T2:
#do some linking and prep for T1
Is there a good way to ensure that T2 is run before T1? (Note: the above example is actually simplified by a bit. T1 is actually the vmlinux target within the Linux kernel makefile, so rewriting it is not only difficult, it makes the code non-portable. Also, I can't run T2 before calling make on the kernel due to some other dependencies).
Have T2 as an order-only prerequisite:
T1: x y z | T2
$(MAKE) -j $^;
# Make will run the T2 rule before this one, but T2 will not appear in $^
Could you just call Make in your build script with the two targets in the proper order, e.g.
make T2 T1
That way you don't need to make any modifications to T1.
Related
I have two makefiles
Makefile 1:
all: run1 temp1 clean1
run1:
command1
temp1:
command2
clean1:
command3
Makefile 2:
all: run2 temp2 clean2
run2:
command4
temp2:
command5
clean2:
command6
I want to create a new Makefile which can include both Makefile 1 and 2.
Makefile:
include Makefile_1
include Makefile_2
Now the sequence of targets should be : run1 temp1 clean1 run2 temp2 clean2
Does including the makefile works? Note both makefiles have common 'all' target.
If you include multiple makefiles and they define the same target, then the prerequisites of the targets are appended in the order in which they targets are parsed (of course only one target at most can have a recipe associated with it).
In your example if you write:
include Makefile_1
include Makefile_2
Then, it'll be the same as if you wrote:
all: run1 temp1 clean1 run2 temp2 clean2
Note, the only way to be 100% sure that make will build targets in a specific order is either (a) never using -j to enable parallel builds, or else (b) define proper prerequisite relationships between your targets.
I have some software built using parallel multi-level makefiles and I see that when my main Makefile runs two separate targets from a submakefile that have the same dependency, this dependency is run twice simultaneously and an error is created.
Consider the following main Makefile in the project root folder:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS):
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
clean:
#echo Making $#
$(MAKE) -C folder clean
and the sub-makefile folder/Makefile:
SUBTARGETS = st1 st2 st3 st4 st5 st6 st7 st8
$(SUBTARGETS): dep
#echo Making $#
#sleep 1
#touch $#
dep:
#echo Making $#
#sleep 1
#echo bla >> dep
clean:
rm -f $(SUBTARGETS)
rm -f dep
rm -f dep2dump
Then running make -j8 in the root folder will run targets t1...t8 in parallel, which will then run subtargets st1...st8, which all depend on dependency dep. From the shell output and the contents of the dep file (8 lines) it is obvious that the dep rule is run 8 times, as if the 8 implications of folder/Makefile are completely independent.
I thought submakes coordinated when running in parallel and that they would avoid running the same target twice, but it seems this is not the case.
Can anyone suggest a correct way to solve such a case?
If eventually this is an unavoidable weakness of make, what alternative build tools should I look into?
Thanks
EDIT: The answers by MadScientist and Renaud Pacalet are useful but don't exactly solve my problem because they both require that the author of the top-level makefile has knowledge about the internals of the sub-makefile. I have not explained this requirement explicitly in my original post though.
So to give more details, the use case I am trying to solve is that where the source code in path folder/ is a separate project, eg. a collection of utilities st1...st8 where all (or some) of them have a dependency on library dep, internal to the utilities project in folder. Then I want to be able to use this sub-project (as seamlessly as possible) in various master projects, each of them using just a (possible different) subset of the utilities st1...st8. Additionally, the master project may contain many targets t1...t8, each depending on a different subset of st1...st8, as shown in my example above. Targets t1...t8 need to be able to run separately, building only the required dependencies from subproject (so make t1 only builds st1, etc), thus having to build all st1...st8 for each one of t1...t8 is not desired. On the other hand they also need to be able to run in parallel, eg. by running make all.
Ideally I would not want the author of each master makefile to have to know about internals of sub-project, nor have to include in the sub-makefile all the possible combinations of st1...st8 so that each master project can call just ONE of these to avoid the parallel build issue.
So far I have in mind but not tested the following imperfect solutions:
As Renaud suggested, use something like flock to at least ensure that the multiple runs of dep (by separate sub-make instances) won't happen simultaneously. Cons: requires extra tool (flock or similar) to be installed + dep runs multiple times, so extra work is needed to avoid doing the actual compilation over and over again, otherwise just eat the performance cost.
Include the sub-makefile in the master makefile so that everything runs in one make instance. This requires makes the sub-makefile able to work regardless of the path of the master makefile that includes it. No big issue. Cons: merging / including two makefile from different authors can open a can of worms, i.e. variables with same name, etc.
Modify sub-makefile as described in (2) + In the main project create another makefile, eg. utils.make, that contains a rule for the targets of sub-makefile needed and includes the sub-makefile. So utils.make will be (assuming this master project only needs st1, st5 and st7:
utils: st1 st5 st7
include foldes/Makefile
Then the master makefile will have a utils-ext rule as dependency of each of t1...t8 that will be:
utils-ext:
$(MAKE) -f rules.make utils
to build all the utils needed. This keeps the two main makefiles separate but has all utils / subtargets built when building any single one of t1...t8, which is suboptimal.
You could try to move the dep dependency to your top Makefile:
.PHONY: all $(TARGETS) clean dep
all: $(TARGETS)
$(TARGETS): dep
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
dep:
$(MAKE) -C folder s#
The only decent solution to your problem is to have ONE instance of make build all the sub-directory targets you want. Having the parent make invoke multiple sub-makes in parallel in the same directory, unless every invocation uses a completely disjoint set of targets, is a guaranteed fail situation. So if you have multiple things you want to do in the submake you should collect them all in one invocation of the sub-make and let the sub-make's parallelism handle it for you.
You could do something like this:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS): .submake ;
.submake:
$(MAKE) -C folder $(addprefix s,$(MAKECMDGOALS))
Then in the sub-make add this so that when invoked with no arguments it builds everything:
all: $(SUBTARGETS)
Here, if you run make then the sub-make is invoked with no arguments and builds all the things in parallel. If you invoke make t1 t2 then the submake is invoked with the arguments st1 st2.
Alternatively, you can re-architect your makefiles so that you don't use recursive make at all, and one instance of make knows all the different rules and dependency relationships.
Is it possible to learn about leaf prerequisites of a target?
In this case, the leaf prerequisites of t1 are t3 and foo.txt, while t2 is intermediary:
t1: t2 t3
cat t2 t3 > t1
t2: foo.txt
cat $< > $#
In this example, t2 is strictly an intermediate repository, and t3 already exists (it is not generate-able).
How can I obtain t3 and foo.txt, as leaf-targets of t1, from make? Is it reasonable to expect that such facility exists?
Use case: I am compiling a LaTeX document to PDF, and would like to pdfattach all the leaf prerequisites without listing them explicitly in the Makefile.
You could do
make --dry-run --debug=a t1 | grep "Considering target"
and write a convoluted script to parse the output (the last line, and any entry with a greater indent than it's subsequent line would be leaves). It sounds like you want to do access this list from within the make invocation though, but I don't think there is any built in support for this.
(you could do a recursive make call from within a recipe -- not efficient, but may work).
Inside the rules the prerequisites can be accessed via automatic variables, such as $< and $^.
However, make must know the prerequisites in order to rebuild correctly, so there must be a list of files in a make variable. That list of files can be specified explicitly or be read from a file or another command.
What is the best way to simulate Make's grouped targets feature for older versions of Make that do not have the feature?
UPDATE
It looks like a static pattern rule might work:
"Pattern rules are always treated as grouped targets ... regardless of whether they use the : or &: separator"
Grouped targets for pattern rules have been around forever (you don't need &: for that, it's the default way they work). Grouped targets with &: for non-pattern rules was introduced in GNU make 4.3 which was released in Jan 2020.
There is no GNU make 3.0. If you have the default GNU make on MacOS you have something like GNU make 3.81 or something like that. You'll never get a newer version because Apple will never add a version of GNU software that is under GPLv3 so on MacOS you're stuck with old bash, old make, etc. etc. You can use homebrew or macports to install newer versions on your own of course (or build them yourself).
Using static pattern rules won't work. The naming is unfortunate but static pattern rules are actually just a shortcut for writing explicit rules, not pattern rules, and they don't support grouped targets.
If you can't use real pattern rules (your outputs and inputs are not related through some common portion of their filenames) and you can't use GNU make 4.3, then you can use this idiom; if your rule is like this:
t1 t2 t3 ... : p1 p2 p3 ...
command that builds all of t1 t2 t3 ...
then you can change it to this:
t1 t2 t3 ... : .sentinel ;
.sentinel: p1 p2 p3 ...
command that builds all of t1 t2 t3 ...
#touch $#
Don't forget the semicolon after the .sentinel. Or if you prefer you can do something like this for that rule: it's equivalent:
t1 t2 t3 ... : .sentinel
#: do nothing
(: is the shell's "do nothing" operator) Of course you still need the second rule as well.
In a makefile, the dependency line is of the form -
abc: x y z
All three of the components (x,y,z) are themselves targets in dependency lines further down in the makefile.
If make abc is invoked, in what order will the three targets x,y,z be executed?
By default, the order of execution is the same as specified in the prerequisites list, unless there are any dependencies defined between these prerequisites.
abc: x y z
The order is x y z.
abc: x y z
y : z
The order would be x z y.
But ideally, you should design your Makefiles so that it wouldn't rely on the order in which prerequisites are specified. That is, if y should be executed after z, there must be a y : z dependence.
And keep in mind that GNU Make can execute some recipes in parallel, see Mat's answer.
You really shouldn't depend on the order in which they are executed - all else being equal, all three recipes for those prerequisites could run in parallel.
The only hard rule is that all prerequisites must be met before the target recipe is run.
If there are no dependencies between x, y and z, and no parallel execution, GNU make appears to run them in the order you specified them, but this is not guaranteed in the docs.
The POSIX description of make includes a rationale which says:
The make utilities in most historical implementations process the prerequisites of a target in left-to-right order, and the makefile format requires this. It supports the standard idiom used in many makefiles that produce yacc programs; for example:
foo: y.tab.o lex.o main.o
$(CC) $(CFLAGS) -o $# t.tab.o lex.o main.o
In this example, if make chose any arbitrary order, the lex.o might not be made with the correct y.tab.h. Although there may be better ways to express this relationship, it is widely used historically. Implementations that desire to update prerequisites in parallel should require an explicit extension to make or the makefile format to accomplish it, as described previously.
(I believe the t.tab.o in the $(CC) line is a typo for y.tab.o, but that is what the rationale actually says.)
Thus, the observed behaviour that pre-requisites are processed from left to right has validation here, though it is only in the Rationale section, not in the main description. The Rationale also mentions issues with parallel make etc.
From https://stackoverflow.com/a/22638294/636849, you can add the pipe symbol:
abc: | x y z
From make manual:
Order-only prerequisites can be specified by placing a pipe symbol (|) in the prerequisites list: any prerequisites to the left of the pipe symbol are normal; any prerequisites to the right are order-only:
targets : normal-prerequisites | order-only-prerequisites