Is there a way to force a target-rule to run as part of setting a something in a variable?
For example let's say we have a target and rule:
all_mp3s:
find / -name "*.mp3" > all_mp3s
And then a variable:
MP3S := $(file < all_mp3s)
Is there a way to make sure all_mp3s file is getting created before evaluating the MP3S variable?
There is no simple straightforward way to force a rule to be evaluated before a variable gets assigned. There are more complex ways. The following is for GNU make.
Let's first assume that you want to run the (slow) find command only if the file all_mp3s does not exist, else use its content. You can use GNU make conditionals:
ifeq ($(wildcard all_mp3s),all_mp3s)
MP3S := $(shell cat all_mp3s)
else
MP3S := $(shell $(MAKE) all_mp3s ; cat all_mp3s)
endif
all_mp3s:
find / -name "*.mp3" > $#
But I if your Makefile is more complex than this, uses MP3S several times, and what you really want is:
avoid running your super-slow find several times,
run it only if needed (and only once),
get the result in a file (all_mp3s) plus a make variable (MP3S),
MadScientist has a wonderful GNU make trick that can be used here:
MP3S = $(eval MP3S := $$(shell find / -name "*.mp3"))$(MP3S)
all_mp3s:
printf '%s\n' '$(MP3S)' > all_mp3s
.PHONY: help clean
help:
printf 'MP3 finder\n'
clean:
rm -f all_mp3s
If the MP3S recursively expanded make variable is expanded because some part of your Makefile is evaluated and needs its value (e.g. if you run make all_mp3s while all_mp3s does not exist), the find command will be run, its result stored in the variable... and the variable will be turned into a simply expanded make variable, which further expansions, if any, will reuse the same, already computed, value.
Else, if your invocation of make (e.g. make cleanor make help) does not need MP3S value, the find command will not even be run.
The all_mp3s file is generated from the value of MP3S (instead of the opposite in the other solution).
However, there is another important thing to decide: do you want to declare all_mp3s as a phony target:
.PHONY: all_mp3s
or not?
If you declare it as phony, the find command will be run once and only once each time you invoke make all_mp3s (or another goal that depends on all_mp3s). But targets depending on all_mp3s will always be rebuilt too, which is not necessarily what you want.
If you don't declare it as phony, and the file exists already, the find command will not be run at all (unless the value of MP3S is needed elsewhere in your Makefile), and the content of all_mp3s will not be updated, which is not necessarily what you want.
As you do not give enough information in your question to decide, it is up to you.
Related
Suppose I have the following directory structure, with a root-node:
/root/
Makefile
branch1/
branch2/
.../
And I write the following minimal makefile:
branches=$(shell find * -maxdepth 1 -type d -printf "%f "}
%:
-${MAKE} ${MAKECMDGOALS} -C branch1
-${MAKE} ${MAKECMDGOALS} -C branch2
...
Having prepared to dynamically perform this relay with the branches variable and having tried several wild-card and descendant rule variations without success, my question amounts to: how do I capture any command goal from outer scope (like I am doing now), and perform the make relay for each of the files my find expression detects?
In pseudo code (did not work with my version of make, which is the latest greatest available on Cygwin):
branches=$(shell find * -maxdepth 1 -type d -printf "%f "}
branch-%:
-${MAKE} ${MAKECMDGOALS} -C $*
%: ${foreach branch,${branches}, branch-${branch}}
Unlike the original makefile, this does not work. However, it seems like it should. Is there a way to do this?
And there is a second issue
Make's parallelism will be broken by my pseudo-code method (if it worked) with an exponential fan out using the -j option, whereas the first method I used will not break parallelism.
However, ideally, this makefile should be able to dynamically execute one make relay for each file in the branches list. However, I don't currently see a way to implement this dynamically.
First, I'm not sure why you're using a complex shell function; why not just:
branches := $(wildcard */.)
Or, if you don't want the /. at the end:
branches := $(patsubst %/.,%,$(wildcard */.))
Second, the reason your second attempt doesn't work is that it's not valid to create a pattern rule with no recipe. See Canceling Pattern Rules.
Instead, you can use the .DEFAULT special target. It would look something like this:
branch-%:
-${MAKE} -C $* $(CMD)
.DEFAULT:
#$(MAKE) CMD=$# ${addprefix branch-,${branches}}
This does use recursive make and it behaves slightly differently than your original.
I'm not sure I understand your second point about -j. GNU make (unless you're using a truly old version) can communicate among all the submakes to ensure they are starting as many, but not more, jobs than you requested.
Oh, I forgot, there's another obvious way to do it if you don't want to use .DEFAULT and recursion:
$(MAKECMDGOALS): $(addprefix branch-,$branches))
Two question here concerning Makefiles and storing persistent variables. I have a script that I am using to perform a build for a binary that will be deployed for a system containing two MCUs. The binary from one MCU will be fed into the binary of the other MCU for in application updates initiated by the second MCU.
Multiple developers are working on this project and building the binary for the MCU that will be inserted into the main binary is not necessary for all invloved. Also, the presence and location of the build too required for the secondary binary may vary. As such, I need to be able to find the build tool for that secondary binary if it is available. The way I do that is to run a search at the beginning of the Makefile to find the exe and then write that location to a file to allow it to persist across multiple build attempts.
First question: is there a better way to persist that path to the build tool than writing to a file without going in and manipulating the PATH externally (this is will be an automated build, so ...)?
Second question: I am using the following (not in a recipe) to write the value to the build tool
$(shell echo ${IAR_EXE} > iar-path)
and it is taking a long time. What might I be doing incorrectly or could I do to improve the efficiency of this call?
Edit for #kebs:
To provide context, the above is in the preamble of the make file (i.e. before recipes). Not sure if this adds anything, but here is the entire preamble:
STM32_DIR:=../../Project-STM32
LX_PROJECT:=$(STM32_DIR)/Projects/Lx/Application/Project_file/Lx.ewp
GX_PROJECT:=$(STM32_DIR)/Projects/Gx/EWARM/Gx.ewp
IAR_EXISTS=FALSE
# Check to see if the path for the IAR build tool exists...
ifeq ("$(wildcard iar-path)", "")
$(info IAR path not available, searching now...)
IAR_EXE:="$(shell find /cygdrive/c/ -name IarBuild.exe -print -quit 2>&1 | grep -v "Permission denied")"
ifneq ($(IAR_EXE),"")
# Path exists - push it to a file that might be _more_ persistent
$(info Pushing path into local file...)
$(shell echo ${IAR_EXE} > iar-path)
IAR_EXISTS:=TRUE
endif
else
# Path exists - grab it from the file
$(info IAR Path file exists...)
IAR_EXE:="$(shell cat iar-path)"
ifneq ($(IAR_EXE),"")
IAR_EXISTS=TRUE
endif
endif
I do not know a better way for persistence than files. The long time you observe is due to find, not to echo. If you want to optimize this try maybe, if it makes sense, to restrict the search space to just the directories where what you are looking for can possibly be, instead of the full /cygdrive/c/.
Minor remarks:
You can remove most of the ", they are useless (and could also become a problem if you forget them in only one side of a comparison): for make they are just another character in a text string. Example:
ifeq ($(wildcard iar-path),)
instead of:
ifeq ("$(wildcard iar-path)", "")
and:
IAR_EXE := $(shell cat iar-path)
ifneq ($(IAR_EXE),)
IAR_EXISTS = TRUE
endif
instead of:
IAR_EXE:="$(shell cat iar-path)"
ifneq ($(IAR_EXE),"")
IAR_EXISTS=TRUE
endif
You can simplify a bit your search command with:
IAR_EXE := $(shell find /cygdrive/c/ -name IarBuild.exe 2> /dev/null)
and maybe (not sure) optimize it a bit with:
IAR_EXE := $(shell find /cygdrive/c/ -type f -name IarBuild.exe 2> /dev/null)
I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.
Here is a simplified use case of my Makefile. What I want to do is use the names of these files and create the respective .d and .hpp files. However, for some reason, I am unable to do it.
XML_DATA_FILES := a.xml b.xml c.xml d.xml
build:
#for var in $(XML_DATA_FILES); do \
echo "$(basename $$var)";\
$(eval var1 = $(basename $$var)) \
echo "$(join var1,.hpp)"; \
echo "$(join var1,.d)"; \
done
The output that I get when I run make is as follows
a.xml
var1.hpp
var1.d
b.xml
var1.hpp
var1.d
c.xml
var1.hpp
var1.d
d.xml
var1.hpp
var1.d
But what I want is a.d, a.hpp and so on for all the four xml files input.
I have already referred to this question and GNU Manual but it hasnt helped so far.How can I achieve this?
There're a number of problems here :). But, fundamentally you cannot combine make functions like basename and eval inside a shell loop like for and expect it to do anything useful. Make always expands the entire recipe for all make variables and function FIRST, then it passes the entire expanded string to the shell to run, then it waits for the shell to finish.
Consider: how would the shell, running its for loop, communicate the current value of var back up to make each time through the shell's loop so that make could run the proper functions etc.? It's just not possible.
You need to write your entire loop using only shell constructs, plus simple make variables that have the same value throughout the recipe.
However, this is useless as a makefile since you just have one target that does everything. Why not just write a shell script? There's no point to using make for this. If you want to write it the make way, you'll need to declare the targets and prerequisites and create pattern rules, like this:
XML_DATA_FILES := a.xml b.xml c.xml d.xml
OUTPUTS := $(foreach X,$(XML_DATA_FILES:.xml=),$X.d $X.hpp)
build: $(OUTPUTS)
%.d %.hpp: %.xml
echo "$*.d $*.hpp"
Of course since you don't say exactly what the real commands do I can't be sure this is correct; if you actually have two different commands, one that builds the .d file and one that builds the .hpp file, you should create two different pattern rules.
I'm trying to write a Makefile that automatically calls BibTeX on files that match a specific wildcard but don't exist when I first run Make. Specifically, I have the following:
.FORCE:
all: pdf
foo=something
lat: *.tex
pdflatex $(foo).tex
pdf: lat
open $(foo).pdf &
%.aux: .FORCE
bibtex $#
bib: lat $(foo)?.aux
pdflatex $(foo).tex
pdflatex $(foo).tex
open $(foo).pdf &
What I want to happen is that the following will occur when I run make bib:
pdflatex will be called on $(foo).tex, generating files $(foo)1.aux, $(foo)2.aux, etc.
bibtex will be called on $(foo)1.aux, then $(foo)2.aux, etc.
pdflatex will be called twice on $(foo).tex
open will be called on $(foo).pdf
However, this doesn't happen: in particular, it looks as if Make evaluates the prerequisites $(foo)?.aux up-front, at a point where the files $(foo)1.aux, $(foo)2.aux, etc. don't exist. As a result, BibTeX is never called on them. If I rerun make bib, however, things work, because the files now exist after being created on the previous run.
Question: Is forcing Make to re-evaluate prerequisites for a target the right way to fix this? If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat? If not, how can I achieve what I want please?
What I do in my Maiefile for LaTeX files is rename the targets.
That way, you can have different target names, depending on which phase has been used to create them. This is according to the spirit of make's pattern rules, which assume that files with different contents also have different extensions. So I have rules like this:
%.aux1 : %.tex
rm -f $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%.bbl : %.aux1
cp -pf $< $*.aux
bibtex $* || : > $#
%.aux2 : %.bbl
cp -pf $*.aux1 $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%-tex.pdf: %.aux2
cp -pf $< $*.aux
pdflatex -jobname $*-tex $*
You can't do this in a completely straightforward way, since make fundamentally assumes that one run through a target's commands will update the target. That is, there's no way in principle that you can tell make that ‘you need to run these commands twice’.
You can try to get round this with (admirably clever) tricks such as #reinerpost suggests, but a problem with that general approach is that sometimes/often a single run of BibTeX (or makeindex, or whatever) is sufficient.
After having tried various types of tricks in the past, what I generally do here is to make a command list which explicitly includes two BibTeX calls where necessary:
%.bbl: %.aux
bibtex $(#:.bbl=)
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then \
bibtex $(#:.bbl=); \
fi
That command list re-runs BibTeX if the log file includes the ‘Label(s) may have changed. Rerun to get cross-references right’ message.
To be honest, what I actually do is just the single line bibtex $(#:.bbl=). When I'm writing a document, I inevitably re-run make so many times that the list of references comes out correct very quickly. This means that this target doesn't work for the ‘recreate the final version from a clean directory’ case, but that's sufficiently rare that I tend not to obsess about it.
Whenever I catch myself re-solving this problem, I now recognise that I'm trying to push water up-hill because I'm bored writing this document, so I go and do something else.
I just wanted to share an alternative solution: Using submake processes:
If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat?
You can somewhat achieve that, by adding make lat to the recipe for bib. This will start a new make process targetting at bib. The sub-make doesn't know anything aboutits parents targets/prerequisites. (Such a concept is usually used, when some huge project is built from different smaller projekts each of which have different makefiles.)
This can be done in multiple layers (although it will be confusing):
bib: $(foo)?.aux lat check_for_bib
check_for_bib:
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then make bib; fi
pdf: lat check_for_bib
open $(foo).pdf &
Note that I had to change some orders of prerequisites. The pseud-code would be something like:
latex compilation
while log suggests update:
update aux
latex compilation
Each iteration of the while loop will take place in a separate make process.