We have rather complicated makefiles that make race condition debugging a nightmare due to the fact that multiple targets and being run at the same time. I'd like to prefix the logs with the target name and possibly with a timestamp as well.
Is this possible? We are using bash exclusively as the shell.
I'm not aware of any mechanism to automatically prepend the target to every line of output, but there is a make option --output-sync which groups all output from a target together. This makes it much simpler to debug multi-threaded makes. Note that nothing is output for a target until the target has completed, so if you have a long running target, it will delay the output.
As far as prepending the time, you can pipe your output to gawk:
make | gawk '{ print strftime("[%H:%M:%S]"), $0 }'
Related
I'm working on a rather large project in pure C, and for various reasons we are not using CMake.
We have a good system which uses various shell commands to require very little maintanence. It will automatically find new headers and C files and add the headers to the dependencies and will compile the C files and include them in the output. However, we've gotten to the point where any changes we make to the header files becomes a problem because it requires a recompilation of the entire project, rather than just the C files that include it directly or indirectly.
I've worked on a system to ensure that the only header files which are added as dependencies are ones which are required by the C file itself (recursively up the tree).
I had thought I would be able to solve this myself, but it seems that make has rather inconsistent rules for expansion of variables.
The solution I came up with was to try to use ag and get all the includes out of the file. This works as intended, and I pipe it into tr so that I can ensure that no newlines are added, thereby messing up make. The issue that I'm having though is in determining which file to search in. This is the recipe that it's using for the section in question:
$(GAMEOBJDIR)/%.o : $(GAMEDIR)/%.c $(shell ag -o '(?<=(^#include "))(.*?)(?=("$$))' $(GAMEDIR)/%.c | tr '\n' ' ')
$(CC) $(CFLAGS) $(if $(RELEASE),$(RELFLAGS),$(DBFLAGS)) -c -o $# $<
The problem is that $(GAMEDIR)/%.c is expanding to src/game/%.c and not src/game/<filename>.c. I'm unsure how to fix this issue based on the expansion rules of make.
Once I can figure this out I'll be able to make sure this walks up the chain of header files too, but until I can get this figured out I have no reason to work on that.
Make's rules for expansion are completely consistent... but sometimes they aren't what people want. That's not the same thing :)
Rules for expansion are explained in the manual. In your case you're working with a prerequisite list which means the variables are expanded as the makefile is parsed. That includes the shell command and all other variables. Because you are working with a pattern rule, it means that at the time the makefile is parsed it doesn't match any particular file, it's just defining the pattern rule itself. So, % can't be expanded here. It will be expanded later, when make starts walking the dependency graph and trying to locate ways to build targets.
In general your method cannot work efficiently. You could defer the expansion of these parts of the prerequisites list using Secondary Expansion. However, that means every time make wants to TRY to use this pattern it will run these commands--that will just be slow.
Have you considered using a more standard way to manage detecting prerequisites? Try reading this description for example to see if it would work for you.
Is there a way to suppress the silence '#' in the rules?
I want to see what the Makefile does but all rules from a given Makefile are defined with the '#' in front like:
go:
#do something here...
I search for an option to see the output as # was not there.
You can run make with the -n flag. You will be able to see which commands are executed. Without actually executing them though!
From GNU make manual:
When make is given the flag ‘-n’ or ‘--just-print’ it only echoes most
recipes, without executing them. See Summary of Options. In this case
even the recipe lines starting with ‘#’ are printed. This flag is
useful for finding out which recipes make thinks are necessary without
actually doing them.
Aside from using -n as suggested you could (if the makefile is under your control to edit) look at using something like my automake-inspired silent make rules stuff.
Which lets you override the silencing at make run time dynamically.
You might be willing to do this:
At the top of the makefile, put silence ?= #
Change all the #-prefixes to $(silence).
Then make will observe the silences by default and make silence= will
be verbose.
If you don't want to do that, there's a pretty good chance that:
cat Makefile | sed 's/\t#/\t/g' | make -f -
will do the trick.
Is there an option to output the "preprocessed" makefile, something equivalent to the GCC's -E option?
I have a project comprised of an hierarchy of dozens of modules, each with its makefile. The build is invoked from a master makefile. That master makefile contains includes, variable definitions, command line option dependent variables, etc.
So, essentially, I am looking for the processed makefile, including all substitutions.
Not that I'm aware of. The closest thing you can get to this is the output from make -qp (or similar) which will dump the make database out at you.
Part of the problem with this request is that many of the substitutions/etc. happen as targets are processed and the list of targets isn't necessarily known without actually attempting a build (at least to an extent) so it isn't necessarily possible to fully expand/etc. a makefile in-place.
The make -d output is also useful for certain incidental information related to how make has processed the makefiles but doesn't contain makefile contents directly.
Remake might also be able to provide some extra useful information.
If you are looking for the computed value of some assembled/etc. global make variable then this blog post by Eric Melski is likely to be very helpful.
tl;dr It adds a target like this to the Makefile (though there's more magic in the blog post so I suggest reading it).
print-%:
#echo '$*=$($*)'
#echo ' origin = $(origin $*)'
#echo ' flavor = $(flavor $*)'
#echo ' value = $(value $*)'
Though in personal use I replaced that first line with something more like this
#echo '$*=$(subst ','\'',$($*))'
to keep the quoting of the result correct.
Because GNU make allows variables to be as large as memory allows, it has no problem building massive dependency lists. However, if you want to actually use these lists of files in a recipe (sequence of shell commands for building a target), you run into a problem: the command might exceed the shell's command line length limit, producing an error such as "Argument list too long".
For example, suppose I want to concatenate several files contained in the list $(INPUTS) to produce a file combined.txt. Ordinarily, I could use:
combined.txt: $(INPUTS)
cat $^ > $#
But if $(INPUTS) contains many thousands of files, as it does in my case, the call to cat is too long and fails. Is there a way to get around this problem in general? It's safe to assume that there exists some sequence of commands that have identical behaviour to the one enormous command -- in this case, a series of cat commands, one per input file, that use >> to append to combined.txt would work. But how can make be persuaded to generate those commands?
In looking for the answer, about the best suggestion I could find was to break up the list into a series of smaller lists and process them using shell for loops. But you can't always do that, and even when you can it's a messy hack: for example, it's not obvious how to get the usual make behaviour of stopping as soon as a command fails. Luckily, after much searching and experimentation, it turns out that a general solution does exist.
Subshells and newlines
make recipes invoke a separate subshell for each line in the recipe. This behaviour can be annoying and counterintuitive: for example, a cd command on one line will not affect subsequent commands because they are run in separate subshells. Nevertheless it's actually what we need to get make to perform actions on very long lists of files.
Ordinarily, if you build a "multiline" list of files with a regular variable assignment that uses backslashes to break the statement over multiple lines, make removes all newlines:
# The following two statements are equivalent
FILES := a b c
FILES := \
a \
b \
c
However, using the define directive, it's possible to build variable values that contain newlines. What's more, if you substitute such a variable into a recipe, each line will indeed be run using a separate subshell, so that for example running make test from /home/jbloggs with the makefile below (and assuming no file called test exists) will produce the output /home/jbloggs, because the effect of the cd .. command is lost when its subshell ends:
define CMDS
cd ..
pwd
endef
test:
$(CMDS)
If we create a variable that contains newlines using define, it can be concatenated with other text as usual, and processed using all the usual make functions. This, combined with the $(foreach) function, allows us to get what we want:
# Just a single newline! Note 2 blank lines are needed.
define NL
endef
combined.txt: $(INPUTS)
rm $#
$(foreach f,$(INPUTS),cat $(f) >> $#$(NL))
We ask $(foreach) to convert each filename into a newline-terminated command, which will be executed in its own subshell. For more complicated needs, you could instead write out the list of filenames to a file with a series of echo commands and then use xargs.
Notes
The define directive is described as optionally taking a =, := or += token on the end of the first line to determine which variable flavour is to be created -- but note that that only works on versions of GNU make 3.82 and up! You may well be running the popular version 3.81, as I was, which silently assigns nothing to the variable if you add one of these tokens, leading to much frustration. See here for more.
All recipe lines must begin with a literal tab character, not the 8 spaces I have used here.
Is it possible to include Makefiles dynamically? For example depending on some environment variable? I have the following Makefiles:
makefile
app1.1.mak
app1.2.mak
And there is an environment variable APP_VER which could be set to 1.1.0.1, 1.1.0.2, 1.2.0.1, 1.2.0.2.
But there will be only two different makefiles for 1.1 and 1.2 lines.
I have tried to write the following Makefile:
MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
include makefile$(MAK_VER).mak
all: PROD
echo MAK_VER=$(MAK_VER)
But it does not work:
$ make all
"makefile$(echo", line 0: make: Cannot open makefile$(echo
make: Fatal errors encountered -- cannot continue.
UPDATE:
As far as I understand make includes files before it calculates macros.
That's why it tries to execute the following statement
include makefile.mak
instead of
include makefile1.1.mak
You have two problems: your method of obtaining the version is too complicated, and your include line has a flaw. Try this:
include app$(APP_VER).mak
If APP_VER is an environmental variable, then this will work. If you also want to include the makefile called makefile (that is, if makefile is not the one we're writing), then try this:
include makefile app$(APP_VER).mak
Please note that this is considered a bad idea. If the makefile depends on environmental variables, it will work for some users and not others, which is considered bad behavior.
EDIT:
This should do it:
MAK_VER := $(subst ., ,$(APP_VER))
MAK_VER := $(word 1, $(MAK_VER)).$(word 2, $(MAK_VER))
include makefile app$(MAK_VER).mak
Try this:
MAK_VER=$(shell echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
MAK_FILE=makefile$(MAK_VER).mak
include $(MAK_FILE)
all:
echo $(MAK_VER)
echo $(MAK_FILE)
Modifying the outline solution
Have four makefiles:
makefile
app1.1.mak
app1.2.mak
appdummy.mak
The app.dummy.mak makefile can be empty - a symlink to /dev/null if you like. Both app.1.1.mak and app.1.2.mak are unchanged from their current content.
The main makefile changes a little:
MAK_VER = dummy
include makefile$(MAK_VER).mak
dummy:
${MAKE} MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/') all
all: PROD
...as now...
If you type make, it will read the (empty) dummy makefile, and then try to build the dummy target because it appears first. To build the dummy target, it will run make again, with APP_VER=1.1 or APP_VER=1.2 on the command line:
make APP_VER=1.1 all
Macros set on the command line cannot be changed within the makefile, so this overrides the line in the makefile. The second invocation of make, therefore, will read the correct version-specific makefile, and then build all.
This technique has limitations, most noticeably that it is fiddly to arrange for each and every target to be treated like this. There are ways around it, but usually not worth it.
Project organization
More seriously, I think you need to review what you're doing altogether. You are, presumably, using a version control system (VCS) to manage the source code. Also, presumably, there are some (significant) differences between the version 1.1 and 1.2 source code. So, to be able to do a build for version 1.1, you have to switch from the version 1.1 maintenance branch to the version 1.2 development branch, or something along those lines. So, why isn't the makefile just versioned for 1.1 or 1.2? If you switch between versions, you need to clean out all the derived files (object files, libraries, executables, etc) that may have been built with the wrong source. You have to change the source code over. So why not change the makefile too?
A build script to invoke make
I also observe that since you have the environment variable APP_VER driving your process, that you can finesse the problem by requiring a standardized 'make invoker' that sorts out the APP_VER value and invokes make correctly. Imagine that the script is called build:
#!/bin/sh
: ${APP_VER:=1.2.0.1} # Latest version is default
case $APP_VER in
[0-9].[0-9].*)
MAK_VER=`echo $APP_VER | sed -e 's/^\(...\).*/\1/'`
;;
*) echo "`basename $0 .sh`: APP_VER ($APP_VER) should start with two digits followed by dots" 1>&2;
exit 1;;
esac
exec make MAK_VER=$MAK_VER "$#"
This script validates that APP_VER is set, giving an appropriate default if it is not. It then processes that value to derive the MAK_VER (or errors out if it is incorrect). You'd need to modify that test after you reach version 10, of course, since you are planning to be so successful that you will reach double-digit version numbers in due course.
Given the correct version information, you can now invoke your makefile with any command line arguments.
The makefile can be quite simple:
MAK_VER = dummy
include app$(MAK_VER).mak
all: PROD
...as now...
The appdummy.mak file now contains a rule:
error:
echo "You must invoke this makefile via the build script" 1>&2
exit 1
It simply points out the correct way to do the build.
Note that you can avoid the APP_VER environment variable if you keep the product version number under the VCS in a file, and the script then reads the version number from the file. And there could be all sorts of other work done by the script, ensuring that correct tools are installed, other environment variables are set, and so on.