I'm using make to copy files to a DEST directory. I have the following rule
$(THUMBS): $(DEST)/% : %
mkdir -p $(dir $#)
cp $^ $#
The problem is that sometimes the source file may not exist. Rather than generating an error, I would rather copy a placeholder file instead.
I tried adding the placeholder as a dependence with the actual sources as intermediates. That kind of worked, but then if the placeholder is updated make overwrites all of the actual source files with it.
Is there an elegant way to accomplish this?
If the files in $(DEST) are being built externally (that is, not via a make recipe), then you can do this by embedding a little shell script in your recipe:
$(THUMBS):
mkdir -p $(#D)
for file in $(DEST_FILES); do\
if [[ -f $file ]]; then\
cp -f $file $#;\
else\
cp -f $(PLACEHOLDER_FILE) $#;\
fi;\
done
You aren't listing the files in $(DEST) as prerequisites, so make should never try to rebuild them. You will need to set PLACEHOLDER_FILE to the name of the placeholder file that you wish to use for missing files, and set DEST_FILES to the list of files that you expect to see in DEST. The downside is that without prerequisites, make won't know when it doesn't actually need to re-run this rule. You will run it unconditionally every time.
How about this:
$(DEST)/% : %
mkdir -p $(dir $#)
cp $^ $#
$(DEST)/% :
mkdir -p $(dir $#)
touch $#
Related
I need to do a makefile for run some programs. Every time I run that script all the file are processed also if the file are not changed. I'm sure there is a problem on my code but I don't understand where I made the mistakes.
RDIR=RAW
OUTDIR=Fusion_res/kallisto
RFILES:=$(wildcard $(RDIR)/*_R1_001.fastq.gz)
DATABASE=/home/sbsuser/databases/Kallsto_hg38_87
OUTFILE=$(patsubst %_R1_001.fastq.gz,%_R2_001.fastq.gz,$(RFILES))
OUTKAL=$(patsubst $(RDIR)/%_R1_001.fastq.gz,$(OUTDIR)/%,$(RFILES))
.PHONY: clean all
all: $(OUTFILE) $(RFILES) $(OUTDIR) $(OUTKAL)
#$(OUTKAL) $(OUTFILE): $(RDIR)/%._R1_001.fastq.gz
# echo "kallisto quant -i" $(DATABASE)/transcripts.idx -b 100 -o $# --fusion $< $(OUTFILE)
$(OUTDIR)/%: $(RDIR)/%_R1_001.fastq.gz $(OUTFILE)
kallisto quant -i $(DATABASE)/transcripts.idx -b 100 --fusion --rf-stranded -o $# $(RDIR)/$*_R1_00
1.fastq.gz $(RDIR)/$*_R2_001.fastq.gz
$(OUTDIR):
mkdir -p $(OUTDIR)
clean::
$(RM) -rf $(OUTDIR)
I suppose if the found some change on the input file and on the output execute the command. I don't know why every time force re-run. In some case Is that I want but I wan to also if there is some new input execute only that.
Thanks so much
A couple of things:
1) $(OUTDIR)/% is dependent on $(OUTFILE) (which is a list of all outfiles). Therefore if you change any one of the OUTFILEs, you make everything in $(OUTDIR)/% obsolete. I believe what you want is this:
$(OUTDIR)/%_R1_001.fastq.gz: $(RDIR)/%_R2_001.fastq.gz
.... (rules to make out/R1 from raw/R2
$(RDIR)/%_R2_001.fastq.gz: $(RDIR)/%_R1_001.fastq.gz
.... (rules to make R2 from R1
This makes each file dependent only on the files that effect it.
2) you have the target all dependent on $(OUTDIR) which is a directory. If you use parallel make, it may generate the $(OUTDIR) after it generates the other dependencies of all: (some of which would depend on $(OUTDIR) being created). What you want there is to remove all's dependency on $(OUTDIR), and add the line:
$(OUTFILE) : | $(OUTDIR)
Notice the |, which means order only (don't consider $(OUTFILE) out of date if $(OUTDIR) is newer. This is important, as a directory's timestamp is updated each time a file in the directory is changed, and so it tends to be newer than its contents.
I would like make to either copy a file from the source tree into the target/build directory if it exits or generate an empty/default file if not.
It would be easy to do the following:
target/settings.json: src/settings.json
cp $? $#
src/settings.json:
echo "default..." > $#
But that taints the source repository with a file that could inadvertently be checked into RCS.
Is there a simple make rule that can copy the file if it exits, or just generate the target with a command/copy from some other source?
A GNU-Make specific solution is fine
You can check if the file exists using $(wildcard), so maybe something like this:
ifeq ($(wildcard src/settings.json),)
SETTINGS = tmp/settings.json
else
SETTINGS = src/settings.json
endif
target/settings.json: $(SETTINGS)
cp $? $#
tmp/settings.json:
echo "default..." > $#
I have a makefile that uses a source file from the internet. There are two locations where the file resides, neither of which I consider very dependable, so I also keep a local copy. So the relevant lines of my makefile look like:
src.c:
wget -nv http://location.com/$# || wget -nv http://otherplace.com/$# || cp local/$# .
src.o: src.c
$(CC) -o $# $<
Is this the "right way" to do this? What if there are multiple steps in each different way of creating the target - how do I tell make "Try A. If A fails, try B. If B fails, ..."?
The right thing to do is this:
.PHONY: phony
src.c: phony
if (wget -nv http://location.com/$# -O $#.temp) && ! diff $#.temp $# >/dev/null; then \
mv $#.temp $#; \
fi
I shortened your command to a single wget but you can put whatever you want there, including a sequence of ||s to achieve "try this, if not, try that etc". Just make sure it outputs to a temporary file (and does not hang indefinitely !) .
It is in fact important to use phony here, and not only .PHONY. Can you see why?
Also, with this method, there is no longer a need to keep another "local" copy and/or use cp. Your target src.c is your "local copy" - the latest one you were able to successfully get from the Internet.
I have the following folder structure
1st-grade-math-class/
common/
mystyle.sty
mysubstyle.sty
fonts/
font1.ttf
font2.ttf
font3.ttf
week01/
handout.tex
image1.pdf
image2.pdf
week02/
handout.tex
image1.pdf
image2.pdf
...
week13/
handout.tex
output/
[empty]
And I would like to create a Makefile - in the best way - to do the following:
make sure I include the common directory properly in TEXINPUTS
compile the handout.tex into a PDF (using either pdflatex or xelatex) and have it in the output directory as week01-handout-student.pdf
compile the handout.tex with a line of LaTeX prepended to the beginning of the file (that sets a flag) into a PDF and have it in the output directory as week01-handout-teacher.pdf
clean everything up (the log, aux, etc. files)
I am not sure I know how to this in any other way than manually duplicating an elementary Makefile/bash script in every subdirectory, then calling each of them one by one with a for loop.
I would appreciate help on how to build this process, ideally with a single Makefile in the root directory. Thanks.
UPDATE: I purposefully did not want to give any details about how I compile LaTeX, in case somebody has a better suggestion than my current usage. Right now I am using Latexmk (which is already a make-like wrapper of LaTeX):
latexmk -pdf file.tex generates file.pdf
to add the line of code, I do a simple echo "line of code" > temp.tex and cat handout.tex >> temp.tex, then the same latexmk command
latexmk -c file.tex in a directory cleans all temporary files used to compile file.tex
TEXINPUTS is the TeX path variable, to let TeX find (in its path) the style files: I do TEXINPUTS=full-path-to/common and then export TEXINPUTS before compiling anything.
If anybody has a better suggestion, I am a willing taker.
Something like this should do what you want I believe:
OUTDIR := output
# Tell make to export this environment variable to all recipe lines it runs.
export TEXINPUTS := $(abspath common)
# Get the list of all of our week directories.
weekdirs := $(wildcard week*)
#$(info weekdirs:$(weekdirs))
# Create student output filenames from week directory names.
STUDENT_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-student.pdf,$(weekdirs))
#$(info STUDENT_HANDOUTS:$(STUDENT_HANDOUTS))
# Create teacher output filenames from week directory names.
TEACHER_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-teacher.pdf,$(weekdirs))
#$(info TEACHER_HANDOUTS:$(TEACHER_HANDOUTS))
# Default target depends on all output files.
all: $(STUDENT_HANDOUTS) $(TEACHER_HANDOUTS)
# Pattern rule for building pdf files.
%.pdf:
#echo + Making $# from $^
#echo cd $(#D) && echo latexmx -pdf $(abspath $<)
#echo cd $(#D) && echo latexmk -c $(abspath $<)
# Static pattern rule mapping student output files to input files.
$(STUDENT_HANDOUTS) : $(OUTDIR)/%-handout-student.pdf : %/handout.tex
# Pattern rule to generate temporary input files from original input files.
%/handout-tmp.tex: %/handout.tex
#echo echo 'line of code' '>' $#
#echo cat $^ '>>' $#
# Static pattern rule mapping teacher output files to (temporary) input files.
$(TEACHER_HANDOUTS) : $(OUTDIR)/%-handout-teacher.pdf : %/handout-tmp.tex
Uncomment the $(info) lines to see a bit of how the variables are put together.
This uses latexmk -c to clean up auxiliary files after creating the output files.
Technique 1. The echo foo >$#; cat $< >>$# thing is something I've done before; I think it's reasonable.
Technique 2. Another technique is to have your .tex document, or a package file it uses, include a line like:
\InputIfFileExists{lecturenotes.config}{}{}
That allows you to make certain adjustments in a makefile, as in for example:
# any bits of configuration that should be in all the .config files
SWITCHES=\makeindex
%-notes.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{sidenotes}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
%-single.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{oneside}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
Technique 3. A third technique for controlling LaTeX from outside is to include in your document (or in a package file):
\newif\ifdemonstrator
\expandafter\ifx\csname demonstrator\endcsname\relax
\demonstratorfalse
\else
\demonstratortrue
\fi
...
\ifdemonstrator
\typeout{Demonstrator is TRUE}
\else
\typeout{Demonstrator is FALSE}
\fi
Then call latex with:
%-plain.pdf: %.tex
latex $<
mv ${<:.tex=.pdf} $#
%-demo.pdf: %.tex
latex --jobname ${<:.tex=} '\def\demonstrator{x}\input{$<}`
mv ${<:.tex=.pdf} $#
Technique 1 is a bit of a blunt instrument, but if that's all that's needed, it's pretty lightweight.
Technique 2 is probably the most neatly engineered, but it is slightly more effort.
Technique 3 is probably the one I use most in this sort of circumstance.
I have a Makefile that does performs a task if it hasn't happened in the last hour. It does so like this:
HOUR_FROM_NOW = $(shell perl -e '($$s,$$m,$$h,$$d,$$M)=localtime(time()+3600); printf("%02d%02d%02d%02d\n",$$M+1,$$d,$$h,$$m);')
NOW_FILE = $(shell mkdir -p .make; touch .make/now; echo .make/now )
.PHONY: externals
externals: $(PROJECTS:%=.make/proj_%)
.make/proj_%: $(NOW_FILE)
$(MAKE) -s $(*F)
touch -t $(HOUR_FROM_NOW) $#
.PHONY: $(PROJECTS)
$(PROJECTS):
# do stuff, specifically, clone git-repo if not exists, else pull latest
That part works great, except that I now get warnings:
make: Warning: File `.make/proj' has modification time 3.5e+03 s in the future
make: Nothing to be done for `externals'.
make: warning: Clock skew detected. Your build may be incomplete.
Anyone know how to suppress those warnings? (Or to do a periodic task in a makefile)
Most versions of touch I have come across can do some date time maths which allows for setting the timestamp of a file directly via the --date option.
That and the fact that variables assigned with := are only "evaluated once" makes this a bit easier to read.
HOUR_AGO := .make/hour_ago
__UGLY := $(shell mkdir -p .make && touch --date='1hour ago' $(HOUR_AGO))
# The preceding line will be executed once
.make/proj_%: .make/hour_ago | .make
$(MAKE) -s $(*F)
#touch $#
.make:
mkdir -p $#
I'm using something very similar to this to periodically refresh login tokens.
Never would have thought of it if it wasn't for Dave's answer though.
The directory is created by specifying it as a order-only-prerequisite
I suspect that the + 3600 is at fault. What happens if you remove it?
I thought and thought, and then the stupid-obvious solution hit me ...
Instead of setting timestamps in the future with HOUR_FROM_NOW, I use the real time and compare with HOUR_AGO_FILE ...
HOUR_AGO = $(shell perl -e '($$s,$$m,$$h,$$d,$$M)=localtime(time()-3600); printf("%02d%02d%02d%02d\n",$$M+1,$$d,$$h,$$m);')
HOUR_AGO_FILE = $(shell mkdir -p .make; touch -t $(HOUR_AGO) .make/hour_ago; echo .make/hour_ago )
.PHONY: externals
externals: $(PROJECTS:%=.make/proj_%)
.make/proj_%: $(HOUR_AGO_FILE)
$(MAKE) -s $(*F)
#touch $#