Is it possible to tell make to only build a target if it's sources md5 checksum has changed (instead of the edit time)?
I'm using make to compile my .tex files and I need to prevent it from building everything twice all the time.
I've tried using Scons, but I feel this isn't adaptable to other needs I have.
No, this is not supported by Make — as you've found out, support for this feature is one of the reasons why tools like Scons exist.
I found a manual recipe for GNU make, though. Maybe you can use that as a work around.
I found 3 ways: From simply
Using temporary md5-files over an
elegant but unreadable hack to
patching GNU Make yourself
(from patchy but simple to clean but hard to realize)
Maybe my scons latex and org-mode recipe helps you:
## I need a couple of special builders for my projects
# the $SOURCE replacement only uses the first source file. $SOURCES gives all.
# specifying all source files makes it possible to rerun the build if a single source file changed.
orgexportpdf = 'emacs --batch --visit "$SOURCE" --funcall org-export-as-pdf'
pyxplot = 'pyxplot $SOURCE'
# pdflatex is quite dirty. I directly clean up after it with rm.
pdflatex = 'pdflatex $SOURCE -o $TARGET; rm -f *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb'
# build the PhD thesis from emacs org-mode.
Command("doktorarbeit.pdf", "doktorarbeit.org",
orgexportpdf)
# create plots
Command(["images/comp-t3-s07-tem-boas.png",
"images/comp-t3-s07-tem-bona.png"],
["nee-comp.pyx",
"nee-comp.txt"],
pyxplot)
# build my sink.pdf
Command("sink.pdf",
["sink.tex",
"images/comp-t3-s07-tem-boas.png",
"images/comp-t3-s07-tem-bona.png",
"images/bona-marble.png",
"images/boas-marble.png"],
pdflatex)
# My editors leave tempfiles around. I want them gone after a build clean. This is not yet supported!
tempfiles = Glob('*~') + Glob('#*#') + Glob('*.bak')
# using this here would run the cleaning on every run.
#Command("clean", [], Delete(tempfiles))
It is the counterpart for my Makefile:
all: doktorarbeit.pdf sink.pdf
sink.pdf : sink.tex images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png images/bona-marble.png images/boas-marble.png
pdflatex sink.tex
rm -f *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter
comp-t3-s07-tem-boas.png comp-t3-s07-tem-bona.png : nee-comp.pyx nee-comp.txt
pyxplot nee-comp.pyx
# http://www.reddit.com/r/emacs/comments/dy9yt/anyone_know_of_a_script_that_i_can_feed_an/
# http://blog.nguyenvq.com/2010/10/30/bash-batch-script-to-convert-org-mode-file-to-html/comment-page-1/#comment-27013
doktorarbeit.pdf : doktorarbeit.org
emacs --batch --visit "doktorarbeit.org" --funcall org-export-as-pdf
# this is not what clean is intended to do, but I do it anyway.
clean :
rm -f \#* *~ *.bak # kill editor backups
# alternatively run scons. Should I switch to SCons, just put this in all :)
sconsrun : scons
python scons/bootstrap.py -Q
scons :
hg clone https://bitbucket.org/ArneBab/scons
Related
I want to have a Makefile that is able to install files one by one, and also uninstall them one by one.
The install rule for a file should run only if the source file is newer than the dest file.
The uninstall rule for a file should run only if the file exists in the system.
Now I have:
SRC_FILES = ...
sys_files = ...
$(sys_files): $(DESTDIR)$(systemdir)/%: $(SRCDIR)/%
#echo ' INSTALL $#';
$(INSTALL_DATA) -T '$(SRCDIR)/$*' '$#';
uninstall_sys_files = $(addprefix uninstall-,$(sys_files))
$(unsinstall_sys_files): uninstall-%:
#echo ' RM $*';
$(RM) '$*';
The problem with the code above is that it runs the uninstall (RM) targets even if the files don't exist, (I can live with it, but I would prefer to not print RM messages for files that are not removed. And yes, I can do that with a conditional, but I'd do that only if make can't handle this with a ?"static terminal pattern rule"?).
So, is there anything like?:
$(unsinstall_sys_files):: uninstall-%: %
#echo ' RM $*';
$(RM) '$*';
which would make this rule terminal and depending on the existance of %.
Per my tests, this behaves as if the double colon was a single colon.
EDIT:
Although this would seem nice, it isn't, because then if a rule fails (because the file didn't exist), then the whole make fails, or you need to ignore errors in the uninstall part, which I'm not entirely happy with.
I'm not really sure advantage you hope to gain by marking a rule terminal... unless I'm missing something it doesn't seem to have much to do with what you want to do.
Your final problem ("a rule fails because the file didn't exist"), by which I understand you to mean the uninstall recipe fails, is simple enough to fix: just set RM = rm -f so that rm won't fail if the file doesn't exist.
It would be great if you could give an example of what you really want to do; how you expect to invoke make and how you want it to behave (resulting operations).
Make is designed to update files, not delete files. That is, its whole goal is to find files that don't exist or are out of date, and make them exist / update them.
However, you could do something like this:
RM := rm -f
sys_exists := $(wildcard $(addprefix $(DESTDIR)$(systemdir)/,$(sys_files)))
uninstall: $(patsubst $(DESTDIR)$(systemdir)/%,uninstall-%,$(sys_exists)))
uninstall-%:
#echo ' RM $*'
$(RM) '$(DESTDIR)$(systemdir)/$*'
This way uninstall would only remove files that actually existed.
I tried looking for answers to this question, so I apologize in advance if this is a duplicate of a question I didn't find. Also sorry that I cannot directly provide the code that I am working with (it would require a lot of environmental dependencies, anyway).
I have a sequence of actions, which all depend on the success of the previous actions, and also don't need to be repeated unless they are out of date. A make solution seemed like the proper one. I've come up with a solution that does almost all of it. Here is the sequence of steps I am trying to replicate, with the output of each step listed below its input:
ZIP file
extract to package/
package/directory/*.comp
execute uncomp.py to create a .uncomp file from a .comp file
Everything works fine up to this point
package/directory/*.uncomp
For *.uncomp files, execute script1 to produce a .html file
For *_ext.uncomp files, execute script2 to produce numbered *_ext.##.png file(s)
Multiple numbered files (_ext.0.png, _ext.1.png, _ext.2.png) are possible, and may not be present at the time make is run. However, make should know that they are the output of the previous step, and only run this recipe if these files (a) don't exist or (b) any are older than the *_ext.uncomp file.
I have put together a Makefile which does almost what I'm looking for, except that it delegates all of the last portion (numbered files) to a shell script which I could program to look at file times, but that defeats the purpose of using make in the first place, in my opinion.
Environment
Debian 8.8 (x86)
GNU Make 4.0
Built for x86_64-pc-linux-gnu
My Question
What rules and recipes can I use to inform GNU make of the relationship between the *_ext.uncomp files and the _ext.##.png files so that those recipes only get executed as necessary (and say 'Target is up-to-date' if all .png files are at least as new as the _ext.uncomp file), that won't also apply to the *.uncomp files, and that will still work of there are no .png files in the output?
I will also need to indicate the relationship between non-_ext files and their corresponding HTML counterparts. So that script1 only gets executed when the HTML file is out of date or doesn't exist. This recipe/rule should not pay attention to _ext.uncomp files.
Any other advice on my Makefile would also be appreciated, because I am not overly familiar with it.
Generalized contents of my current Makefile
.PHONY : all
all : package package/directory/*.uncomp
./process $^
%.comp.uncomp : %.comp package
python uncomp.py $<
package : *.zip
rm -rf package/
unzip *.zip -d package/
Contents of the process script
This script should no longer exist if all the goals of the question are met (make will handle everything). It works great, but it always processes .uncomp files no matter what, even if the output from them already exists and is newer than the source.
#!/bin/bash
if [ $# -lt 2 ]; then
echo "$0 expects at least 2 arguments"
exit 1
fi
# Discard the first agrument, it's always 'package'
shift
# Iterate over each of the remaining arguments
while [ $# -gt 0 ]; do
if [[ $1 == *_ext.uncomp ]] ; then
python script2 $1
elif [[ $1 == *.uncomp ]] ; then
python script1 $1
else
echo "Warning: Unknown file type: $1"
fi
shift
done
I learned a lot about GNU make trying to get this to work. I discovered that the solution to my problem was in not overthinking it.
The most important realization was that I didn't need make to track all of the numbered output files, but just the first one (if the first one is out of date or missing, they all will be, and they all get re-extracted by the script, so a 1:1 relationship was all I needed to indicate there).
I found out that GNU make 3.82 and later uses "shortest stem first" order instead of definition order when matching pattern rules. To make my file compatible with both versions, I made sure to define the most specific stems first.
After that it was a matter of setting up some implicit rules, and just telling make what to expect to be able to find—the concept is a little backwards to my way of thinking which is why I had some trouble at first (look for this file that doesn't exist yet; now, here's a way to make it from a file that does exist). The end result, fully functional:
PACKAGE := package
COMP := .comp
UNCOMP := .comp.uncomp
PNG0 := .comp.0.png
TXT := .comp.txt
SUFFIX := _ext
COMPFILES = $(wildcard $(PACKAGE)/subdir/*$(COMP))
UNCOMPFILES = $(COMPFILES:$(COMP)=$(UNCOMP))
SUFFIXFILES = $(filter %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
PNGFILES = $(SUFFIXFILES:$(UNCOMP)=$(PNG0))
NOSUFFIXFILES = $(filter-out %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
TXTFILES = $(NOSUFFIXFILES:$(UNCOMP)=$(TXT))
.PHONY : all
all : pngs txts htaccess
.PHONY : txts
txts : $(TXTFILES)
.PHONY : pngs
pngs : $(PNGFILES)
.PHONY : uncomp
uncomp : $(UNCOMPFILES)
make pngs
make txts
.PHONY : htaccess
htaccess : $(PACKAGE)/.htaccess
%$(SUFFIX)$(PNG0) : %$(SUFFIX)$(UNCOMP)
## Ignore failures when extracting PNG files
-python script1.py $<
%$(TXT) : %$(UNCOMP)
## Ignore failures when dumping TXT files
-python script2.py $< > $#
%$(UNCOMP) : %$(COMP)
## Ignore decompression failure
-python uncomp.py $<
$(PACKAGE)/.htaccess : .htaccess | $(PACKAGE)
cp .htaccess $(PACKAGE)/
$(PACKAGE) : *.zip
rm -rf $(PACKAGE)/
unzip *.zip -d $(PACKAGE)/
make uncomp
.PHONY : clean
clean :
rm -rf $(PACKAGE)/
I'm trying to write a Makefile that automatically calls BibTeX on files that match a specific wildcard but don't exist when I first run Make. Specifically, I have the following:
.FORCE:
all: pdf
foo=something
lat: *.tex
pdflatex $(foo).tex
pdf: lat
open $(foo).pdf &
%.aux: .FORCE
bibtex $#
bib: lat $(foo)?.aux
pdflatex $(foo).tex
pdflatex $(foo).tex
open $(foo).pdf &
What I want to happen is that the following will occur when I run make bib:
pdflatex will be called on $(foo).tex, generating files $(foo)1.aux, $(foo)2.aux, etc.
bibtex will be called on $(foo)1.aux, then $(foo)2.aux, etc.
pdflatex will be called twice on $(foo).tex
open will be called on $(foo).pdf
However, this doesn't happen: in particular, it looks as if Make evaluates the prerequisites $(foo)?.aux up-front, at a point where the files $(foo)1.aux, $(foo)2.aux, etc. don't exist. As a result, BibTeX is never called on them. If I rerun make bib, however, things work, because the files now exist after being created on the previous run.
Question: Is forcing Make to re-evaluate prerequisites for a target the right way to fix this? If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat? If not, how can I achieve what I want please?
What I do in my Maiefile for LaTeX files is rename the targets.
That way, you can have different target names, depending on which phase has been used to create them. This is according to the spirit of make's pattern rules, which assume that files with different contents also have different extensions. So I have rules like this:
%.aux1 : %.tex
rm -f $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%.bbl : %.aux1
cp -pf $< $*.aux
bibtex $* || : > $#
%.aux2 : %.bbl
cp -pf $*.aux1 $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%-tex.pdf: %.aux2
cp -pf $< $*.aux
pdflatex -jobname $*-tex $*
You can't do this in a completely straightforward way, since make fundamentally assumes that one run through a target's commands will update the target. That is, there's no way in principle that you can tell make that ‘you need to run these commands twice’.
You can try to get round this with (admirably clever) tricks such as #reinerpost suggests, but a problem with that general approach is that sometimes/often a single run of BibTeX (or makeindex, or whatever) is sufficient.
After having tried various types of tricks in the past, what I generally do here is to make a command list which explicitly includes two BibTeX calls where necessary:
%.bbl: %.aux
bibtex $(#:.bbl=)
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then \
bibtex $(#:.bbl=); \
fi
That command list re-runs BibTeX if the log file includes the ‘Label(s) may have changed. Rerun to get cross-references right’ message.
To be honest, what I actually do is just the single line bibtex $(#:.bbl=). When I'm writing a document, I inevitably re-run make so many times that the list of references comes out correct very quickly. This means that this target doesn't work for the ‘recreate the final version from a clean directory’ case, but that's sufficiently rare that I tend not to obsess about it.
Whenever I catch myself re-solving this problem, I now recognise that I'm trying to push water up-hill because I'm bored writing this document, so I go and do something else.
I just wanted to share an alternative solution: Using submake processes:
If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat?
You can somewhat achieve that, by adding make lat to the recipe for bib. This will start a new make process targetting at bib. The sub-make doesn't know anything aboutits parents targets/prerequisites. (Such a concept is usually used, when some huge project is built from different smaller projekts each of which have different makefiles.)
This can be done in multiple layers (although it will be confusing):
bib: $(foo)?.aux lat check_for_bib
check_for_bib:
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then make bib; fi
pdf: lat check_for_bib
open $(foo).pdf &
Note that I had to change some orders of prerequisites. The pseud-code would be something like:
latex compilation
while log suggests update:
update aux
latex compilation
Each iteration of the while loop will take place in a separate make process.
On an IBM i system, using PASE (AIX emulator), i try to compile RPG sources using a makefile.
I have RPG sources and try to build a PGM program.
Every single source will be compile in a distinct PGM.
Here is the syntax i tried first
VPATH=.
SYSTEM=system -iv
CRTRPGMOD=CRTRPGMOD
BIN_LIB=MR_CH
CURRENT_PATH=/currentPath #${PWD} doesn't work
#With this line active, it works
SRC_RPGLE = src1.rpgle src2.rpgle
#With this line active, it doesn't work
#SRC_RPGLE = $(shell echo *.rpgle) #Should list every sources i want to compile
TARGETS = $(SRC_RPGLE:.rpgle=.rpgleMod) #Should list every program i want to build
.SUFFIXES: .rpgle .rpgleMod
.rpgle.rpgleMod:
$(SYSTEM) "$(CRTRPGMOD) MODULE($(BIN_LIB)/$(*F)) SRCSTMF('$(CURRENT_PATH)/$<')" > $(*F)_1_crtrpgmod.log
ln -fs $(*F).rpgMod
all: $(TARGETS)
I tried to apply GNU shell syntax using AIX make command
Any suggestions ?
I'm not familiar with the AIX implementation of make but assuming that the linked man page is all there is to it, then it looks like a bare-bones implementation of POSIX make (for an older POSIX spec).
Therefore, the only way to do what you want (expand a list of files) is to use recursion, so that you get access to the shell, like this:
SYSTEM=system -iv
CRTRPGMOD=CRTRPGMOD
BIN_LIB=MR_CH
CURRENT_PATH=/currentPath
CURRENT_PATH=/home/CHARLES/Projets/MRSRC/tmp
recurse:
$(MAKE) all SRC_RPGLE="`echo *.rpgle`"
TARGETS = $(SRC_RPGLE:.rpgle=.rpgleMod)
.SUFFIXES: .rpgle .rpgleMod
.rpgle.rpgleMod:
$(SYSTEM) "$(CRTRPGMOD) MODULE($(BIN_LIB)/$(*F)) SRCSTMF('$(CURRENT_PATH)/$<')" > $(*F)_1_crtrpgmod.log
ln -fs $(*F).rpgMod
all: $(TARGETS)
The recurse rule MUST be the first target in the makefile. Also this won't help if you want to run other targets like make foobar; it will only help you run all properly.
Alternatively you can obtain GNU make and build it for your system, and use that. In the end that might be a more straightforward approach.
At work we use a common makefile that other makefiles include (via the include statement) and it has a generic "clean" target that kills some common files. I want to add on to that target in my new makefile so I can delete some specific files, but if I add a clean target in my makefile, it just overrides the old one.
I know I can just make a new target with a new name and have it call clean, and then do other stuff, but for sake of consistency I'd like to be able to just call make clean and have it do everything.
Is that possible?
I've seen this done at several shops. The most common approach is to use double-colon rules, assuming you're using something like GNU make. In your common makefile you would have something like this:
clean::
# standard cleanup, like remove all .o's:
rm -f *.o
Note that there are two colons following clean, not just one!
In your other makefile you just declare clean again, as a double-colon rule:
clean::
# custom cleanup, like remove my special generated files:
rm -f *.h.gen
When you invoke make clean, GNU make will automagically run both of these "branches" of the clean rule:
% make clean
rm -f *.o
rm -f *.h.gen
It's simple to set up and it composes quite neatly I think. Note that specifically because it is a double-colon rule, you don't get the "overriding commands" errors you normally get when you define two rules for the same target. That's sort of the point of double-colon rules.
You can write your own clean and make it a preq of the common clean.
clean: myclean
myclean:
rm whatever
Yours will run first. If for some reason you want the common clean to run first then the solution will be more complicated.
EDIT:
Here is the best solution I can see which runs the common rule before the local one:
include Makefile.common
clean:
$(MAKE) -f Makefile.common $#
rm whatever additional things
The include directive is necessary because the local makefile relies on the common one for things other than clean. The local clean rule overrides the common clean rule, but invokes the common clean rule before doing the additional work. (This overriding will cause some warnings, which is a nuisance; I don't know a good way to silence them.)
Use implicit rules:
existing-target: my-extention
my-extention:
echo running command 1
echo running command 2
Very simple make tutorial to ramp up.
When using :: you can run into issues since make complains when you mix single colon : and double colon :: rules:
a:
echo a
a::
echo aa
will result in:
. . .
*** target file `a' has both : and :: entries. Stop.
It seems like the common makefile's rule should be called something like common-clean. Then each main makefile would declare their clean rule as
clean: common-clean
and you're set.
If that isn't an option, you could take a look at double colon rules, but those introduce a whole other set of issues to consider.
Adding another possible solution I've seen for posterity... I know the OP was wary about changing the common makefile, but something like this works and involves minimal changes.
local makefile 1:
CLEAN=MyExe1 MyExe2
....
include /my/common/makefile
local makefile 2:
CLEAN=MyExe3 MyExe4
....
include /my/common/makefile
common makefile:
clean:
rm -f *.dep *.o *.a $(CLEAN)
Basically the idea is to define some variable (in this case CLEAN) in each local makefile with all the specific items you want to delete. Then the common makefile runs rm -f on all the common file types to delete, plus whatever was specifically flagged for deletion in each local makefile via the CLEAN variable. If there's nothing specific to delete, simply omit the variable declaration or leave it empty (CLEAN=)
So now if we run make clean for local makefile 1, it executes
rm -f *.dep *.o *.a MyExe1 MyExe2
And if we run make clean for local makefile 2, it executes
rm -f *.dep *.o *.a MyExe3 MyExe4
I've found a better solution:
.PHONY: my-extra-clean
clean: my-extra-clean
my-extra-clean:
rm <whatever-you-want>
include Makefile.common
The key line is clean: my-extra-clean. Ie, you can add dependencies in separate stanzas in different makefiles to add behaviour. my-extra-clean is run as a dependency of the root clean target.
For ours, we define a variable, EXTRAFILESTOCLEAN, then when the clean rule runs, it has a step to remove anything specified in the EXTRAFILESTOCLEAN variable
clean:
rm -f *.o
ifdef $(EXTRAFILESTOCLEAN)
rm -f $(EXTRAFILESTOCLEAN)
endif
That can cause unexpected problems if you set that variable to weird values, but you could guard against those by adding prefixes or other tests.
It's in the docs: https://www.gnu.org/software/make/manual/html_node/Overriding-Makefiles.html
So instead of include Makefile you use a wildcard target and forward it to the base Makefile:
# -include base.Makefile <--- not this
%:
#$(MAKE) -f base.Makefile $#