I have a Make-based project where the top-level target requires multiple vmlinux binaries (linux kernels) as pre-requisites, so it looks something like:
all: bigfile
bigfile: bigfile.cfg a/vmlinux b/vmlinux c/vmlinux foo bar baz
sometool -obigfile -ibigfile.cfg # other inputs referenced from within the config
and each linux rule looks more or less like:
a/vmlinux: a/.config
$(MAKE) -C $(A_LINUX_SRC) O=$(PWD)/a vmlinux
a/.config
mkdir -p a
$(MAKE) -C $(A_LINUX_SRC) O=$(PWD)/a $(A_LINUX_DEFCONFIG)
Similarly for b and c linux kernels. Note each may have the same or different source trees, and almost certainly will have different defconfigs.
This works for clean builds but I don't really like the recursive call to make. Depending on how I tweak the above few lines, I seem to end up with either one of:
unnecessary recursive makes into linux trees even when nothing changes (which takes 7 seconds to do nothing)
if I edit linux sources, kernel's aren't re-generated unless I explicitly touch the .config or something.
Ideally, I'd like my top level Makefile to be aware of the interior dependancy graph of each linux kernel and "do the right thing" under all circumstances. (i.e the recursive-make-considered-harmful argument).
Although I expect the top-level Linux Makefile won't be happy to be included by some one else, especially multiple times with different configs and src trees!
(I have control over baz/makefile.inc bar/makefile.inc so they can be written to play nice when included by the top level)
Or am I out of luck, and will just have to remember to touch .configs to trigger a decent into each linux build dir?
Thanks,
Dave
EDIT:
the 7-second useless decent into a linux tree looks like this on my mahchine:
$ time make
make -C /home/davidm/md/tests/linux O=/home/davidm/md/tests/linux_a vmlinux
make[1]: Entering directory `/home/davidm/linux-2.6.38'
Using /home/davidm/linux-2.6.38 as source for kernel
GEN /home/davidm/md/tests/linux_a/Makefile
CHK include/linux/version.h
CHK include/generated/utsrelease.h
make[3]: `include/generated/mach-types.h' is up to date.
CALL /home/davidm/md/linux-2.6.38/scripts/checksyscalls.sh
CHK include/generated/compile.h
make[1]: Leaving directory `/home/davidm/md/linux-2.6.38'
real 0m6.577s
user 0m2.930s
sys 0m1.360s
In order for this to work correctly, you really do have to recurse into the kernel source directories on every build. 7 seconds really isn't that bad for checking whether any file in the massive kernel tree has changed...
Including the kernel makefile in your build somehow wouldn't actually help, because the kernel build itself uses recursive make.
Perhaps something like this:
a/.config
mkdir -p a
$(MAKE) -C $(A_LINUX_SRC) O=$(PWD)/a $(A_LINUX_DEFCONFIG)
.PHONY: kernel-a-build
kernel-a-build: a/.config
$(MAKE) -C $(A_LINUX_SRC) O=$(PWD)/a vmlinux
bigfile: kernel-a-build
Since kernel-a-build is a "phony" target (it doesn't correspond to a physical file), it will be run on every single build, allowing the kernel makefile to notice changes to the source files.
Related
I have a simple Makefile that builds an archive, libfoo.a, from a single object file, foo.o, like this:
CC=gcc
CFLAGS=-g -Wall
AR=ar
libfoo.a: libfoo.a(foo.o)
foo.o: foo.c
The first time I run make, it compiles the C file, then creates an archive with the object file:
$ make
gcc -g -Wall -c -o foo.o foo.c
ar rv libfoo.a foo.o
ar: creating libfoo.a
a - foo.o
However, if I run make again immediately (without touching foo.o), it still tries to update the archive with ar r (insert foo.o with replacement):
$ make
ar rv libfoo.a foo.o
r - foo.o
Why does Make do this when it shouldn't have to? (If another target depends on libfoo.a, that target will be rebuilt as well, etc.)
According to the output of make -d, it seems to be checking for the non-existent file named libfoo.a(foo.o), and apparently decides to rerun ar r because of that. But is this supposed to happen? Or am I missing something in my Makefile?
You are seeing this because the people who put together your Linux distribution (in particular the people that built the ar program you're using) made a silly decision.
An archive file like libfoo.a contains within it a manifest of the object files contained in the archive, along with the time that the object was added to the archive. That's how make can know if the object is out of date with respect to the archive (make works by comparing timestamps, it has no other way to know if a file is out of date).
In recent times it's become all the rage to have "deterministic builds", where after a build is complete you can do a byte-for-byte comparison between it and some previous build, to tell if anything has changed. When you want to perform deterministic builds it's obviously a non-starter to have your build outputs (like archive files) contain timestamps since these will never be the same.
So, the GNU binutils folks added a new option to ar, the -D option, to enable a "deterministic mode" where a timestamp of 0 is always put into the archive so that file comparisons will succeed. Obviously, doing this will break make's handling of archives since it will always assume the object is out of date.
That's all fine: if you want deterministic builds you add that extra -D option to ar, and you can't use the archive feature in make, and that's just the way it is.
But unfortunately, it went further than that. The GNU binutils developers unwisely (IMO) provided a configuration parameter that allowed the "deterministic mode" to be specified as the default mode, instead of requiring it to be specified via an extra flag.
Then the maintainers of some Linux distros made an even bigger mistake, by adding that configuration option when they built binutils for their distributions.
You are apparently the victim of one of these incorrect Linux distributions and that's why make's archive management doesn't work for your distribution.
You can fix it by adding the -U option, to force timestamps to be used in your archives, when you invoke ar:
ARFLAGS += -U
Or, you could get your Linux distribution to undo this bad mistake and remove that special configuration parameter from their binutils build. Or you could use a different distribution that doesn't have this mistake.
I have no problem with deterministic builds, I think they're a great thing. But it loses features and so it should be an opt-in capability, not an on-by-default capability.
I am learning makefiles, and can't just wrap my head around this problem i am having, and would like to understand how/why this fail.
I have half a dozen erlang files in a src directory. I want to compile these into a ebin directory, without having to define a rule for each and every one of them. According to the Gnu make documentation, pattern rules should be right up my alley.
However, with the following makefile, all I get from make is make: *** No targets. Stop. Why is that?
ebin/%.beam: src/%.erl
mkdir -p ebin
erlc -o ebin $<
Edit: Based on this answer, I now understand that i would have to explicitly declare the targets, for instance by using make ebin/cmplx.beam. However, i still do not understand how i should write my makefile to get my desired behaviour - since I have half a dozen targets (and in other projects even more), this seems like an unnecessary hassle. Is there not a way to define targets based on the source file names?
The target rule tells make that whenever it needs to produce a beam file in the ebin directory, and there exists a corresponding erl file in the src directory, it can use erlc.
However, this doesn't tell make that this is what it needs to do. You could explicitly tell make what it needs to do by giving it a target on the command line:
make ebin/foo.beam
If you don't give a target on the command line, make will pick the first non-pattern rule in the makefile as its target. However, your makefile doesn't have any non-pattern rules, so there is no target.
What you probably want is that for each existing erl file in src, make should consider the corresponding beam file in ebin to be a target. You can achieve that by calling wildcard and patsubst:
erl_files=$(wildcard src/*.erl)
beam_files=$(patsubst src/%.erl,ebin/%.beam,$(erl_files))
ebin/%.beam: src/%.erl
mkdir -p ebin
erlc -o ebin $<
all: $(beam_files)
(The indented lines need to be actual physical tabs, not spaces.)
That way, running make will rebuild all beam files that are out of date. all gets chosen as the default target, and it in turn depends on all beam existing or potential, each of which in turn depends on the corresponding erl file.
This trick is described in the GNU make manual.
The Problem:
Is it possible to give a target a different name or alias, such that it can be invoked using either the original target name or the alias.
For example something like
/very/long/path/my_binary: dep_a dep_b dep_c
# Compile
# Desired command
ALIAS my_binary = /very/long/path/my_binary
# NOTE: Notice the use of 'my_binary' in the dependencies
data1: my_binary datafile
# Build data file using compiled my_binary
Attempt 1: .PHONY
I have tried using a .PHONY target:
.PHONY: my_binary
my_binary: /very/long/path/my_binary
This works great when invoked from the command-line:
# Runs rule 'my_binary' and then *only* runs rule '/very/long/path/my_binary'
# if the rule '/very/long/path/my_binary' needs updating.
make my_binary
However, this does not work well when the alias my_binary is listed as a dependency:
# *Always* thinks that rule 'data1' needs updating, because it always thinks that
# the .PHONY target 'my_binary' "needs updating". As a result, 'data1' is
# rebuilt every time.
make /very/long/path/my_binary
Possible hack?
A possible hack is to use an empty target as suggested in an answer to this question, but that would require introducing fake files with names corresponding to the alias:
my_binary: /very/long/path/my_binary
touch my_binary
This will clutter the main directory with files! Placing the fake files in a sub-directory would defeat the purpose, as the alias would have to be referred to as 'directory/my_binary'
Okay, I needed something similar. The path to my output artifacts were quite long, but I wanted short target names and also benefit easily from bash-completion.
Here is what I'm came up with:
os := [arbitrary long path to an artifact]
platform := [arbitrary long path to a differ artifact]
packer := [common parts of my packer build command]
.PHONY: all
all: $(platform)
.PHONY: platform
platform: $(platform)
$(platform): platform.json $(os)
#$(packer) $<
.PHONY: os
os: $(os)
$(os): os.json
#$(packer) $<
.PHONY: clean
clean:
rm -fr build/
With the Makefile above you can say:
$ make os
$ make platform
Which will be aliases for the long artifact names. I've made the snippet above quite long, because it's important to see the relationships between the .PHONY aliases and the real targets. I hope that works for you.
Note: I did not delete the clean target from the above example, because many people does not make that a .PHONY target. However, semantically it should be.
I don't think there's any way to do it so that you can use the alias from within your makefile as well as the command line, except by creating those temporary files.
Why can't you just set a variable in the makefile, like:
my_binary = /very/long/path/my_binary
then use $(my_binary) everywhere in the makefile? I don't see any point in creating a real alias target for use inside the makefile.
I had a somewhat similar need. I wanted users of my makefile to be able to enter any of the following to accomplish the same result, such that the following were effectively synonyms of each other:
make hit list
make hitlist
make hit_list
What I did in my makefile was the following:
hit_list:
#echo Got here
<the real recipe goes here>
hit: hit_list
hitlist: hit_list
.PHONY: list
list:
#echo > /dev/null
Then, when I tested it using any of the commands "make hit list", "make hitlist", or "make hit_list", I got identical results, as intended.
By extension, if one of your targets was the one with the long name but you used this approach whereby a simple short name identified the target with the long name as a prerequisite, I think that you should be able to say "make short_name" and accomplish what you're asking about.
This differs from your Approach 1 in that none of the synonyms is defined as a phony target (considering that "make hit list" is a command to make two targets, the second being effectively a noop), so the complication that you described would not arise.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.
I trying to make parallel make work on our build server. I am running into a very frequent problem here that two instances of make trying to make two different targets, say A and B, nearly simultaneously try to make a target which is required by both, say C.
As both instances try to make C together in different instances, C make fails for either of them since one making of C requires some files to be moved here and there and either one of these instances ends up moving or deleting an already created file.
Is there is common construct that I can use to prevent a re-entry into a makefile if the target is already being made ?
Update:
Ok let me put it this way :
My application requires A.lo and B.lo to be present. These A.lo and B.lo are libraries which also link against C.lo.
So the rules look like
app.d : A.lo B.lo (other lo s)
(do linking)
In some other directory say A (which will house A.lo) :
A.lo : C.lo (other .o s and .lo s)
(do linking)
In some other directory say B (which will house B.lo) :
B.lo : C.lo (other .o s and .lo s)
(do linking)
So in effect while making app.d make forks off two parallel makes for targets A.lo and B.lo.
Upon entering directories A and B make forks off another two threads for target C.lo independently and at times both of these are linking C.lo at the same time, which causes one of them to fail with some weird errors like file not recognized (since it may be written onto by other linker instance)
How should I go about solving this? It is not possible to create A.lo and B.lo without C.lo linked against them.
This may sound a little, well, obvious, but the simplest solution is just to arrange for C to be built explicitly before either A or B. That way when the recursive makes for A and B run, C will already be up-to-date, and neither will bother trying to build it.
If your toplevel makefile looks like this:
all: buildA buildB
buildA:
$(MAKE) -C A
buildB:
$(MAKE) -C B
You can extend it like this:
all: buildA buildB
buildA: buildC
$(MAKE) -C A
buildB: buildC
$(MAKE) -C B
buildC:
$(MAKE) -C C
You didn't give a lot of detail about your specific build, so probably this won't just drop-in and work, but hopefully you get the idea.
I solve this problem using the "mkdir" technique:
SHELL = /bin/bash
all: targetC targetD
targetC: targetA
............
targetD: targetB
............
targetA targetB: source
-mkdir source.lock && ( $(command) < source ; rm -r source.lock )
$(SHELL) -c "while [ -d source.lock ] ; do sleep 0.1 ; done"
I would be happy to see a more elegant solution, though.