Two question here concerning Makefiles and storing persistent variables. I have a script that I am using to perform a build for a binary that will be deployed for a system containing two MCUs. The binary from one MCU will be fed into the binary of the other MCU for in application updates initiated by the second MCU.
Multiple developers are working on this project and building the binary for the MCU that will be inserted into the main binary is not necessary for all invloved. Also, the presence and location of the build too required for the secondary binary may vary. As such, I need to be able to find the build tool for that secondary binary if it is available. The way I do that is to run a search at the beginning of the Makefile to find the exe and then write that location to a file to allow it to persist across multiple build attempts.
First question: is there a better way to persist that path to the build tool than writing to a file without going in and manipulating the PATH externally (this is will be an automated build, so ...)?
Second question: I am using the following (not in a recipe) to write the value to the build tool
$(shell echo ${IAR_EXE} > iar-path)
and it is taking a long time. What might I be doing incorrectly or could I do to improve the efficiency of this call?
Edit for #kebs:
To provide context, the above is in the preamble of the make file (i.e. before recipes). Not sure if this adds anything, but here is the entire preamble:
STM32_DIR:=../../Project-STM32
LX_PROJECT:=$(STM32_DIR)/Projects/Lx/Application/Project_file/Lx.ewp
GX_PROJECT:=$(STM32_DIR)/Projects/Gx/EWARM/Gx.ewp
IAR_EXISTS=FALSE
# Check to see if the path for the IAR build tool exists...
ifeq ("$(wildcard iar-path)", "")
$(info IAR path not available, searching now...)
IAR_EXE:="$(shell find /cygdrive/c/ -name IarBuild.exe -print -quit 2>&1 | grep -v "Permission denied")"
ifneq ($(IAR_EXE),"")
# Path exists - push it to a file that might be _more_ persistent
$(info Pushing path into local file...)
$(shell echo ${IAR_EXE} > iar-path)
IAR_EXISTS:=TRUE
endif
else
# Path exists - grab it from the file
$(info IAR Path file exists...)
IAR_EXE:="$(shell cat iar-path)"
ifneq ($(IAR_EXE),"")
IAR_EXISTS=TRUE
endif
endif
I do not know a better way for persistence than files. The long time you observe is due to find, not to echo. If you want to optimize this try maybe, if it makes sense, to restrict the search space to just the directories where what you are looking for can possibly be, instead of the full /cygdrive/c/.
Minor remarks:
You can remove most of the ", they are useless (and could also become a problem if you forget them in only one side of a comparison): for make they are just another character in a text string. Example:
ifeq ($(wildcard iar-path),)
instead of:
ifeq ("$(wildcard iar-path)", "")
and:
IAR_EXE := $(shell cat iar-path)
ifneq ($(IAR_EXE),)
IAR_EXISTS = TRUE
endif
instead of:
IAR_EXE:="$(shell cat iar-path)"
ifneq ($(IAR_EXE),"")
IAR_EXISTS=TRUE
endif
You can simplify a bit your search command with:
IAR_EXE := $(shell find /cygdrive/c/ -name IarBuild.exe 2> /dev/null)
and maybe (not sure) optimize it a bit with:
IAR_EXE := $(shell find /cygdrive/c/ -type f -name IarBuild.exe 2> /dev/null)
Related
Suppose I have the following directory structure, with a root-node:
/root/
Makefile
branch1/
branch2/
.../
And I write the following minimal makefile:
branches=$(shell find * -maxdepth 1 -type d -printf "%f "}
%:
-${MAKE} ${MAKECMDGOALS} -C branch1
-${MAKE} ${MAKECMDGOALS} -C branch2
...
Having prepared to dynamically perform this relay with the branches variable and having tried several wild-card and descendant rule variations without success, my question amounts to: how do I capture any command goal from outer scope (like I am doing now), and perform the make relay for each of the files my find expression detects?
In pseudo code (did not work with my version of make, which is the latest greatest available on Cygwin):
branches=$(shell find * -maxdepth 1 -type d -printf "%f "}
branch-%:
-${MAKE} ${MAKECMDGOALS} -C $*
%: ${foreach branch,${branches}, branch-${branch}}
Unlike the original makefile, this does not work. However, it seems like it should. Is there a way to do this?
And there is a second issue
Make's parallelism will be broken by my pseudo-code method (if it worked) with an exponential fan out using the -j option, whereas the first method I used will not break parallelism.
However, ideally, this makefile should be able to dynamically execute one make relay for each file in the branches list. However, I don't currently see a way to implement this dynamically.
First, I'm not sure why you're using a complex shell function; why not just:
branches := $(wildcard */.)
Or, if you don't want the /. at the end:
branches := $(patsubst %/.,%,$(wildcard */.))
Second, the reason your second attempt doesn't work is that it's not valid to create a pattern rule with no recipe. See Canceling Pattern Rules.
Instead, you can use the .DEFAULT special target. It would look something like this:
branch-%:
-${MAKE} -C $* $(CMD)
.DEFAULT:
#$(MAKE) CMD=$# ${addprefix branch-,${branches}}
This does use recursive make and it behaves slightly differently than your original.
I'm not sure I understand your second point about -j. GNU make (unless you're using a truly old version) can communicate among all the submakes to ensure they are starting as many, but not more, jobs than you requested.
Oh, I forgot, there's another obvious way to do it if you don't want to use .DEFAULT and recursion:
$(MAKECMDGOALS): $(addprefix branch-,$branches))
Is there a way to force a target-rule to run as part of setting a something in a variable?
For example let's say we have a target and rule:
all_mp3s:
find / -name "*.mp3" > all_mp3s
And then a variable:
MP3S := $(file < all_mp3s)
Is there a way to make sure all_mp3s file is getting created before evaluating the MP3S variable?
There is no simple straightforward way to force a rule to be evaluated before a variable gets assigned. There are more complex ways. The following is for GNU make.
Let's first assume that you want to run the (slow) find command only if the file all_mp3s does not exist, else use its content. You can use GNU make conditionals:
ifeq ($(wildcard all_mp3s),all_mp3s)
MP3S := $(shell cat all_mp3s)
else
MP3S := $(shell $(MAKE) all_mp3s ; cat all_mp3s)
endif
all_mp3s:
find / -name "*.mp3" > $#
But I if your Makefile is more complex than this, uses MP3S several times, and what you really want is:
avoid running your super-slow find several times,
run it only if needed (and only once),
get the result in a file (all_mp3s) plus a make variable (MP3S),
MadScientist has a wonderful GNU make trick that can be used here:
MP3S = $(eval MP3S := $$(shell find / -name "*.mp3"))$(MP3S)
all_mp3s:
printf '%s\n' '$(MP3S)' > all_mp3s
.PHONY: help clean
help:
printf 'MP3 finder\n'
clean:
rm -f all_mp3s
If the MP3S recursively expanded make variable is expanded because some part of your Makefile is evaluated and needs its value (e.g. if you run make all_mp3s while all_mp3s does not exist), the find command will be run, its result stored in the variable... and the variable will be turned into a simply expanded make variable, which further expansions, if any, will reuse the same, already computed, value.
Else, if your invocation of make (e.g. make cleanor make help) does not need MP3S value, the find command will not even be run.
The all_mp3s file is generated from the value of MP3S (instead of the opposite in the other solution).
However, there is another important thing to decide: do you want to declare all_mp3s as a phony target:
.PHONY: all_mp3s
or not?
If you declare it as phony, the find command will be run once and only once each time you invoke make all_mp3s (or another goal that depends on all_mp3s). But targets depending on all_mp3s will always be rebuilt too, which is not necessarily what you want.
If you don't declare it as phony, and the file exists already, the find command will not be run at all (unless the value of MP3S is needed elsewhere in your Makefile), and the content of all_mp3s will not be updated, which is not necessarily what you want.
As you do not give enough information in your question to decide, it is up to you.
Is there a way to let make determine the number of files to be recompiled before actually compiling? The problem is this: Consider having a quite big project with hundreds of source files. It would very convenient to have a rough idea of how long compilation will take, but to know that, one needs to know the number of files to be compiled.
The general answer is no, because your build could generate files which themselves are inputs to other rules which generate more files. And so on. However if a rough answer is good enough you can try the --dry-run flag. From the GNU make documentation...
“No-op”. Causes make to print the recipes that are needed to make the targets up to date, but not actually execute them. Note that some recipes are still executed, even with this flag (see How the MAKE Variable Works). Also any recipes needed to update included makefiles are still executed (see How Makefiles Are Remade).
As you can see, despite its name even the --dry-run flag will change the state of your build.
"make -n" will do the dry run. But you can't get the list of files to be rebuilt. May be you can write shell script to store the last modified time of files and get the list of files.
I think a found a decent solution for unix. Here SRC are your source files, HDR your headers and DEP the dependency files (something like DEP:=$(OBJ:.o=.d) )
isInDepFile+=$(shell grep -q $(modifiedFile) $(depFile) 1>&2 2> /dev/null && echo $(depFile))
COMPFILES=
checkDepFiles=$(foreach depFile,$(DEP), $(eval filesToCompile+=$(isInDepFile))) $(thinOutDepFiles)
thinOutDepFiles=$(foreach fileToCompile,$(filesToCompile),$(eval DEP=$(filter-out $(fileToCompile),$(DEP))))
countFilesToCompile: $(SRC) $(HDR)
$(eval modifiedFiles=$?)
$(foreach modifiedFile,$(modifiedFiles), $(call checkDepFiles))
$(eval numOfFilesToCompile = $(words $(filesToCompile)))
$(eval numDepFiles = $(words $(DEP)))
$(eval NumSRCFiles = $(words $(SRC)))
#echo $(NumSRCFiles) sources
#echo $(numDepFiles) files to leave
#echo $(numOfFilesToCompile) files to compile
#touch $#
This first generates a list of modified files within your source and header files lists. Then for each modified file it checks all dependency files for its filename. If a dependency file contains the current file we are looking at, it is added to the list of filesToCompile. It is also removed from the list of dependency files to avoid duplication.
This can be invoked in the main building rule of your project. The advantage of that over the dry run is that it gives you a simple number to work with.
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
I'm trying to write a Makefile that automatically calls BibTeX on files that match a specific wildcard but don't exist when I first run Make. Specifically, I have the following:
.FORCE:
all: pdf
foo=something
lat: *.tex
pdflatex $(foo).tex
pdf: lat
open $(foo).pdf &
%.aux: .FORCE
bibtex $#
bib: lat $(foo)?.aux
pdflatex $(foo).tex
pdflatex $(foo).tex
open $(foo).pdf &
What I want to happen is that the following will occur when I run make bib:
pdflatex will be called on $(foo).tex, generating files $(foo)1.aux, $(foo)2.aux, etc.
bibtex will be called on $(foo)1.aux, then $(foo)2.aux, etc.
pdflatex will be called twice on $(foo).tex
open will be called on $(foo).pdf
However, this doesn't happen: in particular, it looks as if Make evaluates the prerequisites $(foo)?.aux up-front, at a point where the files $(foo)1.aux, $(foo)2.aux, etc. don't exist. As a result, BibTeX is never called on them. If I rerun make bib, however, things work, because the files now exist after being created on the previous run.
Question: Is forcing Make to re-evaluate prerequisites for a target the right way to fix this? If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat? If not, how can I achieve what I want please?
What I do in my Maiefile for LaTeX files is rename the targets.
That way, you can have different target names, depending on which phase has been used to create them. This is according to the spirit of make's pattern rules, which assume that files with different contents also have different extensions. So I have rules like this:
%.aux1 : %.tex
rm -f $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%.bbl : %.aux1
cp -pf $< $*.aux
bibtex $* || : > $#
%.aux2 : %.bbl
cp -pf $*.aux1 $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%-tex.pdf: %.aux2
cp -pf $< $*.aux
pdflatex -jobname $*-tex $*
You can't do this in a completely straightforward way, since make fundamentally assumes that one run through a target's commands will update the target. That is, there's no way in principle that you can tell make that ‘you need to run these commands twice’.
You can try to get round this with (admirably clever) tricks such as #reinerpost suggests, but a problem with that general approach is that sometimes/often a single run of BibTeX (or makeindex, or whatever) is sufficient.
After having tried various types of tricks in the past, what I generally do here is to make a command list which explicitly includes two BibTeX calls where necessary:
%.bbl: %.aux
bibtex $(#:.bbl=)
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then \
bibtex $(#:.bbl=); \
fi
That command list re-runs BibTeX if the log file includes the ‘Label(s) may have changed. Rerun to get cross-references right’ message.
To be honest, what I actually do is just the single line bibtex $(#:.bbl=). When I'm writing a document, I inevitably re-run make so many times that the list of references comes out correct very quickly. This means that this target doesn't work for the ‘recreate the final version from a clean directory’ case, but that's sufficiently rare that I tend not to obsess about it.
Whenever I catch myself re-solving this problem, I now recognise that I'm trying to push water up-hill because I'm bored writing this document, so I go and do something else.
I just wanted to share an alternative solution: Using submake processes:
If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat?
You can somewhat achieve that, by adding make lat to the recipe for bib. This will start a new make process targetting at bib. The sub-make doesn't know anything aboutits parents targets/prerequisites. (Such a concept is usually used, when some huge project is built from different smaller projekts each of which have different makefiles.)
This can be done in multiple layers (although it will be confusing):
bib: $(foo)?.aux lat check_for_bib
check_for_bib:
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then make bib; fi
pdf: lat check_for_bib
open $(foo).pdf &
Note that I had to change some orders of prerequisites. The pseud-code would be something like:
latex compilation
while log suggests update:
update aux
latex compilation
Each iteration of the while loop will take place in a separate make process.