Suppose I define several make PHONY commands:
auto-build-foo
auto-build-bar
auto-build-biz
...
auto-build-inf
And I can observe them all with the following keystrokes in bash with autocompletion configured:
$: make auto-build-<tab><tab>
auto-build-foo
auto-build-bar
auto-build-biz
...
auto-build-inf
Then my natural unix instinct is to write:
make auto-build-*
To build them all.
I understand this sort of thing needs to be implemented in make as a feature, a makefile as some sort of rule system, or some custom shell that integrates bash-completion history with some make-specialized interpretation.
But it would cool and useful to get this "out-of-the-box".
Is there such a mechanism that is -- or will be -- in GNU make?
No, GNU Make command line argument do not support wildcard.
But, you can easily do the job in bash:
One by one
$ for t in $(sed -e '/auto-build/!d' -e 's/:.*$//' -e 's/\n/ /' Makefile); do make "$t" ; done
At once
$ my_targets=$(sed -e '/auto-build/!d' -e 's/:.*$//' -e 's/\n/ /' Makefile)
$ make "$my_targets"
From a Makefile perspective, if you create your own, an efficient way could be to encapsulate the targets in a variable each time you write them:
TARGET:=auto-build-foo
$(TARGET):
#echo "$#"
ALL_TARGETS:=$(ALL_TARGETS) $(TARGET)
TARGET:=auto-build-bar
$(TARGET):
#echo "$#"
ALL_TARGETS:=$(ALL_TARGETS) $(TARGET)
auto-all: $(ALL_TARGETS)
.PHONY: auto-all
will give:
$ make auto-all
auto-build-foo
auto-build-bar
I think such a feature doesn't exist because many makefiles are written with an incomplete dependency graph; command line parameters - which tend to be recorded nowhere - are the main reason for this. Targets built for foo may introduce bugs when being reused as-is for bar - the usual procedure is to require an intermediate make clean to prevent this. The problem is that, given the current (and unlikely to change) execution logic of make, within one run there is no way to execute clean more than once, much less in an order which would not defy the purpose of make (partial updates) at all. I think the original use case for makefiles was more around single-purpose project building rather than a cross-over of dependency and batch processing. From a modern build system perspective this looks like a shortcoming, OTOH it is this focused simplicity which makes it the means of choice for many projects.
Related
DEPRECATED_CHECK := $(shell grep "test454" tex/*.tex)
ifneq ($(DEPRECATED_CHECK), )
$(warning \test454 is deprecated. Use \test2 instead)
endif
When I run this I get:
../common/Makefile.include:133: \test454 is deprecated. Use \test2 instead
That's fine, but I'd quite like to have only:
\test454 is deprecated. Use \test2 instead
Is this possible? Some sort of awk function? I think I need something with:
#echo \text454 is deprecated ...
But I don't know how to get this working with the basic purpose of my MWE, as it keeps complaining about missing separators.
Many thanks
You could use $(info ...) instead of $(warning ...). info doesn't prepend the file and line number.
just an aside -- I usually try to do those sort of checks as part of a sanity rule, and make everything depend on that rule instead of doing it at the top level. It gives you more flexibility that way. For example, if you didn't want to run the check when building clean, it becomes simple, or if you wanted to fail the build if a check failed, it becomes simple as well.
EDIT (adding more detail on aside)
Instead of doing an ifneq at the top level of make, you could add a target as so:
sanity_check:
# ! grep -q "test454" tex/*.txt || echo "test454 is depricated"
.PHONY: sanity check
The add dependencies of your main targets to sanity check:
all maintarg1 maintarg2: sanity_check
This way the sanity check will be run before any of your main targets, and will output as desired. This is in my opinion, a cleaner way of doing the test. This way the test is only run if you are building any of your targets, and will not be run, if for example you are making clean, or if your makefile was included by a parent makefile, or in a bunch of other corner cases that might pop up in the future.
Just a quick note on the recipe syntax: the # is a make directive that tells make not to echo the command as it's run. The ! is bash syntax to inverse the return of grep (so ! grep returns false if the text is found, thereby causing the || part of the statement to be evaluated.). The .PHONY: sanity_check tells make to run the rule, even if a file called sanity_check already exists
I have a rule something, that works on the variable VAR. I also have another rule something-all, that needs to run something, with VAR set to each value in vars.
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach VAR,$(vars),something)
This doesn't quite work, I get
noob#work:~/Desktop$ make something-all
something something
make: something: No such file or directory
make: *** [something-all] Error 1
It should probably print hello\nworld.
I used to do this with wildcard rules by retrieving VAR from %, but got the feeling that was the wrong way to do it. This looked like this:
vars = hello world
all: $(foreach VAR,$(vars),something-$(VAR))
something-%:
echo $*
The below should fix your problem
Using foreach (Tried on GNU Make 3.80 on sparc-solaris 2.8 and windows)
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach i, $(vars), $(MAKE) something VAR=$i || exit 1;)
Using shell for-loop (Tried on GNU Make 3.80 and cc make on sparc-solaris 2.8)
vars = hello world
something:
echo $(VAR)
something-all:
for i in $(vars); do $(MAKE) something VAR=$$i || exit 1; done
TL;DR: If you want to program make, drop GNU Make in favor of BSD Make.
This is a personal recommendation. While BSD Make seems more limited than GNU Make, as it offers less programming facilities, it is much easier to program and has a few unique killer features. This is why I propose a solution with GNU Make and another solution for BSD Make:
Doing it in GNU Make
Using GNU Make, you can write a macro to define a target. The canonical way to define a sequence in a Makefile is to add the steps of the sequence as dependencies to a target, as reflected by the snippet below:
vars= hello world
define something_t =
something: something-$(1)
something-$(1):
#echo $(1)
endef
$(foreach _,$(vars),$(eval $(call something_t,$_)))
It is recommended to use this organisation (rather than defining just one target), because you can work on it to make the task easily resumable if you interrupt the sequence. A Makefile describes a job whose advancement is entirely described by the state of the file system. A task is then easily resumable, if each step is associated to a file, usually a compilation object but sometimes also an empty file which is touch'ed to indicate that important checkpoints have been passed.
Using an auxiliary macro is a flexible solution that can be adapted to more complicated tasks than just echoing a name. Note that this does work with newest versions of GNU Make (4.1). On GNU Make 3.81, you should remove the equal sign from the macro definition.
Adapting your example for BSD Make
If this is an option for you, I recommand dropping the use of GNU Make and replace it by BSD Make, which is way easier to program: it has a short and to the point documentation, while the documentation of GNU Make is very verbose and somewhat unclear, BSD Make has industrial-strength examples of complex rulesets (FreeBSD Build system or BSD Owl), and it has a simple and predictable macro language.
vars= hello world
something:
.for _var in ${vars}
echo ${_var}
.endfor
This can evolve to support more complicated tasks, just by replacing the echo by the adapted commands, or using intermediary steps.
Allow the user to override some tasks, also in BSD Make
In this slightly more advanced variation, we allow the user to override our own recipes for building targets something-hello and something-world.
For each item in our list, a target something-* is created it if it does not already exist, and added to the dependencies of something. The whole operation of defining these targets only happens if something has been left undefined. Therefore, users of these macros can:
Override the recipes for something-hello and something-world
Override the full procedure bound to something.
Implementing such customisation possibilities is mandatory if we want to write useful, reusable, macros for Make. Unluckily, customisation of this sort is nearly impossible in GNU Make.
vars = hello world
.if!target(depend)
.for _var in ${vars}
.if!target(something-${_var})
something-${_var}:
echo ${_var}
.endif
something: something-${_var}
.endfor
.endif
Here's one way to do it:
VARS := hello world
THINGS := $(addprefix something-, $(VARS))
allthings: $(THINGS)
something-%:
echo $*
It should be no surprise that
vars := hello world
something-all:
$(foreach VAR,$(vars),something)
tries to run something something. That's exactly what the foreach expands to, since you don't reference VAR in the third expression.
All you need to do is reference VAR and use a command such as echo:
vars := hello world
something-all:
$(foreach VAR,$(vars),echo $(VAR);)
$ make
echo hello; echo world;
hello
world
Note how chaining the commands with a semicolon avoids forking several shells or -- GASP! -- recursive make invocations. It doesn't get more performant than that.
Alternatively, if your command accepts several somethings as arguments,
vars := hello world
something-all:
echo $(foreach VAR,$(vars),$(VAR))
$ make
echo hello world
hello world
But that is equivalent to the super simple echo $(vars). So it might pay off to think outside the box trying to change your requirements to make this simple solution work.
It looks to me like Makefile rules can be roughly classified into "positive" and "negative" ones: "positive" rules create missing or update outdated files, while "negative" ones remove files.
Writing prerequisites for "positive" rules is quite easy: if the target and the prerequisite are file names, make by default runs the recipe if the target is missing or outdated (a missing file in this context may be viewed as an infinitely old file).
However, consider a "negative" rule, for example for target clean. A usual way to write it seems to be something like this:
clean:
rm -f *.log *.synctex.gz *.aux *.out *.toc
This is clearly not the best way to do:
rm is executed even when there is nothing to do,
its error messages and exit status need to be suppressed with -f flag, which has other (possibly undesirable) effects, and
the fact that there were nothing to do for target clean is not reported to the user, unlike what is normal for "positive" targets.
My question is: how to write a Makefile rule that shall be processed by make only if certain files are present? (Like what would be useful for make clean.)
how to write a Makefile rule that shall be processed by make only if certain files are present? (Like what would be useful for make clean.)
You can do it like so:
filenames := a b c
files := $(strip $(foreach f,$(filenames),$(wildcard $(f))))
all: $(filenames)
$(filenames):
touch $#
clean:
ifneq ($(files),)
rm -f $(files)
endif
Example session:
$ make
touch a
touch b
touch c
$ make clean
rm -f a b c
$ make clean
make: Nothing to be done for 'clean'.
Useful perhaps for some purposes, but it strikes me as a strained refinement for make clean.
This can be easily remedied:
clean:
for file in *.log *.synctex.gz *.aux *.out *.toc; do \
if [ -e "$file" ]; then \
rm "$$file" || exit 1; \
else \
printf 'No such file: %s\n' "$file" \
fi \
done
The if statement is necessary unless your shell supports and has enabled nullglob or something similar.
If your printf supports %q you should use that instead of %s to avoid possible corruptions of your terminal when printing weird filenames.
A meta-answer is: are you sure you want to do this?
The other answers suggest to me that the cure is worse than the disease, since one involves an extension to POSIX make (ifneq), and the other uses a compound command which spreads over seven lines. Both of these are sometimes necessary expedients – I'm not criticising either answer – but both are things I avoid in a Makefile if I can. If I found myself wanting to do this in a clean rule, perhaps for the reason you mention in your comment to #MikeKinghans' answer, I'd try quite hard to change the rest of the Makefile to avoid needing this.
Reflecting on your three original points in turn:
rm is executed even when there is nothing to do: so what? The alternatives still need to, for example, expand the *.log *.synctex.gz ... so there's only miniscule efficiency gain to avoiding the rm. Make is a high-level tool which generally does not concern itself with efficiency.
its error messages and exit status need to be suppressed with -f flag: the -f flag doesn't generally suppress errors and the exit status, it merely indicates to rm that a non-existing or non-permissioned file is not to be regarded as an error.
the fact that there were nothing to do for target clean is not reported to the user: should the user really care?
The last point is the most interesting. People asking about make, on Stackoverflow and elsewhere, sometimes make things hard for themselves by trying to use it as a procedural language – make is not Python, or Fortran. Instead, it's a goal programming language (if we want to get fancy about it): you write snippets of rules to achieve sub-goals, so that the user (you, later) doesn't have to care about the details or the directory's current state, but can simply indicate a goal, and the program does whatever's necessary to get there. So whether there's is or isn't anything to do, the user ‘shouldn't’ care.
I think the short version of this answer is: it's idiomatic to keep make rules as simple (and thus as readable and robust) as possible, even at the expense of a little crudity or repetition.
I'm trying to write a Makefile that automatically calls BibTeX on files that match a specific wildcard but don't exist when I first run Make. Specifically, I have the following:
.FORCE:
all: pdf
foo=something
lat: *.tex
pdflatex $(foo).tex
pdf: lat
open $(foo).pdf &
%.aux: .FORCE
bibtex $#
bib: lat $(foo)?.aux
pdflatex $(foo).tex
pdflatex $(foo).tex
open $(foo).pdf &
What I want to happen is that the following will occur when I run make bib:
pdflatex will be called on $(foo).tex, generating files $(foo)1.aux, $(foo)2.aux, etc.
bibtex will be called on $(foo)1.aux, then $(foo)2.aux, etc.
pdflatex will be called twice on $(foo).tex
open will be called on $(foo).pdf
However, this doesn't happen: in particular, it looks as if Make evaluates the prerequisites $(foo)?.aux up-front, at a point where the files $(foo)1.aux, $(foo)2.aux, etc. don't exist. As a result, BibTeX is never called on them. If I rerun make bib, however, things work, because the files now exist after being created on the previous run.
Question: Is forcing Make to re-evaluate prerequisites for a target the right way to fix this? If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat? If not, how can I achieve what I want please?
What I do in my Maiefile for LaTeX files is rename the targets.
That way, you can have different target names, depending on which phase has been used to create them. This is according to the spirit of make's pattern rules, which assume that files with different contents also have different extensions. So I have rules like this:
%.aux1 : %.tex
rm -f $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%.bbl : %.aux1
cp -pf $< $*.aux
bibtex $* || : > $#
%.aux2 : %.bbl
cp -pf $*.aux1 $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%-tex.pdf: %.aux2
cp -pf $< $*.aux
pdflatex -jobname $*-tex $*
You can't do this in a completely straightforward way, since make fundamentally assumes that one run through a target's commands will update the target. That is, there's no way in principle that you can tell make that ‘you need to run these commands twice’.
You can try to get round this with (admirably clever) tricks such as #reinerpost suggests, but a problem with that general approach is that sometimes/often a single run of BibTeX (or makeindex, or whatever) is sufficient.
After having tried various types of tricks in the past, what I generally do here is to make a command list which explicitly includes two BibTeX calls where necessary:
%.bbl: %.aux
bibtex $(#:.bbl=)
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then \
bibtex $(#:.bbl=); \
fi
That command list re-runs BibTeX if the log file includes the ‘Label(s) may have changed. Rerun to get cross-references right’ message.
To be honest, what I actually do is just the single line bibtex $(#:.bbl=). When I'm writing a document, I inevitably re-run make so many times that the list of references comes out correct very quickly. This means that this target doesn't work for the ‘recreate the final version from a clean directory’ case, but that's sufficiently rare that I tend not to obsess about it.
Whenever I catch myself re-solving this problem, I now recognise that I'm trying to push water up-hill because I'm bored writing this document, so I go and do something else.
I just wanted to share an alternative solution: Using submake processes:
If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat?
You can somewhat achieve that, by adding make lat to the recipe for bib. This will start a new make process targetting at bib. The sub-make doesn't know anything aboutits parents targets/prerequisites. (Such a concept is usually used, when some huge project is built from different smaller projekts each of which have different makefiles.)
This can be done in multiple layers (although it will be confusing):
bib: $(foo)?.aux lat check_for_bib
check_for_bib:
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then make bib; fi
pdf: lat check_for_bib
open $(foo).pdf &
Note that I had to change some orders of prerequisites. The pseud-code would be something like:
latex compilation
while log suggests update:
update aux
latex compilation
Each iteration of the while loop will take place in a separate make process.
Is it possible to include Makefiles dynamically? For example depending on some environment variable? I have the following Makefiles:
makefile
app1.1.mak
app1.2.mak
And there is an environment variable APP_VER which could be set to 1.1.0.1, 1.1.0.2, 1.2.0.1, 1.2.0.2.
But there will be only two different makefiles for 1.1 and 1.2 lines.
I have tried to write the following Makefile:
MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
include makefile$(MAK_VER).mak
all: PROD
echo MAK_VER=$(MAK_VER)
But it does not work:
$ make all
"makefile$(echo", line 0: make: Cannot open makefile$(echo
make: Fatal errors encountered -- cannot continue.
UPDATE:
As far as I understand make includes files before it calculates macros.
That's why it tries to execute the following statement
include makefile.mak
instead of
include makefile1.1.mak
You have two problems: your method of obtaining the version is too complicated, and your include line has a flaw. Try this:
include app$(APP_VER).mak
If APP_VER is an environmental variable, then this will work. If you also want to include the makefile called makefile (that is, if makefile is not the one we're writing), then try this:
include makefile app$(APP_VER).mak
Please note that this is considered a bad idea. If the makefile depends on environmental variables, it will work for some users and not others, which is considered bad behavior.
EDIT:
This should do it:
MAK_VER := $(subst ., ,$(APP_VER))
MAK_VER := $(word 1, $(MAK_VER)).$(word 2, $(MAK_VER))
include makefile app$(MAK_VER).mak
Try this:
MAK_VER=$(shell echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
MAK_FILE=makefile$(MAK_VER).mak
include $(MAK_FILE)
all:
echo $(MAK_VER)
echo $(MAK_FILE)
Modifying the outline solution
Have four makefiles:
makefile
app1.1.mak
app1.2.mak
appdummy.mak
The app.dummy.mak makefile can be empty - a symlink to /dev/null if you like. Both app.1.1.mak and app.1.2.mak are unchanged from their current content.
The main makefile changes a little:
MAK_VER = dummy
include makefile$(MAK_VER).mak
dummy:
${MAKE} MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/') all
all: PROD
...as now...
If you type make, it will read the (empty) dummy makefile, and then try to build the dummy target because it appears first. To build the dummy target, it will run make again, with APP_VER=1.1 or APP_VER=1.2 on the command line:
make APP_VER=1.1 all
Macros set on the command line cannot be changed within the makefile, so this overrides the line in the makefile. The second invocation of make, therefore, will read the correct version-specific makefile, and then build all.
This technique has limitations, most noticeably that it is fiddly to arrange for each and every target to be treated like this. There are ways around it, but usually not worth it.
Project organization
More seriously, I think you need to review what you're doing altogether. You are, presumably, using a version control system (VCS) to manage the source code. Also, presumably, there are some (significant) differences between the version 1.1 and 1.2 source code. So, to be able to do a build for version 1.1, you have to switch from the version 1.1 maintenance branch to the version 1.2 development branch, or something along those lines. So, why isn't the makefile just versioned for 1.1 or 1.2? If you switch between versions, you need to clean out all the derived files (object files, libraries, executables, etc) that may have been built with the wrong source. You have to change the source code over. So why not change the makefile too?
A build script to invoke make
I also observe that since you have the environment variable APP_VER driving your process, that you can finesse the problem by requiring a standardized 'make invoker' that sorts out the APP_VER value and invokes make correctly. Imagine that the script is called build:
#!/bin/sh
: ${APP_VER:=1.2.0.1} # Latest version is default
case $APP_VER in
[0-9].[0-9].*)
MAK_VER=`echo $APP_VER | sed -e 's/^\(...\).*/\1/'`
;;
*) echo "`basename $0 .sh`: APP_VER ($APP_VER) should start with two digits followed by dots" 1>&2;
exit 1;;
esac
exec make MAK_VER=$MAK_VER "$#"
This script validates that APP_VER is set, giving an appropriate default if it is not. It then processes that value to derive the MAK_VER (or errors out if it is incorrect). You'd need to modify that test after you reach version 10, of course, since you are planning to be so successful that you will reach double-digit version numbers in due course.
Given the correct version information, you can now invoke your makefile with any command line arguments.
The makefile can be quite simple:
MAK_VER = dummy
include app$(MAK_VER).mak
all: PROD
...as now...
The appdummy.mak file now contains a rule:
error:
echo "You must invoke this makefile via the build script" 1>&2
exit 1
It simply points out the correct way to do the build.
Note that you can avoid the APP_VER environment variable if you keep the product version number under the VCS in a file, and the script then reads the version number from the file. And there could be all sorts of other work done by the script, ensuring that correct tools are installed, other environment variables are set, and so on.