My Makefile downloads a number of third-party files if they are not locally available.
CLOSURE_VERSION=20161024
CLOSURE_BASE_URL="http://dl.google.com/closure-compiler"
build/bin/closure-compiler.jar: build/src/hashes/closure-compiler-${CLOSURE_VERSION}.tar.gz.sha256
download-if-sha-matches <$< >$#.tar.gz \
${CLOSURE_BASE_URL}/compiler-${CLOSURE_VERSION}.tar.gz
tar -zxf $#.tar.gz closure-compiler-v${CLOSURE_VERSION}.jar
mv closure-compiler-v${CLOSURE_VERSION}.jar $#
rm $#.tar.gz
Here, build/src/hashes/closure-compiler-${CLOSURE_VERSION}.tar.gz.sha256 is the saved hash of the version of the file which we already know is correct.
download-if-sha-matches <hash >outfile url downloads the url and compares its hash to stdin, failing if they don't match.
This recipe works except that it always runs, even if build/bin/closure-compiler.jar already exists. Naturally, its timestamp is later than that of $< so I would expect this to not execute the recipe the second time I run make.
What have I gotten wrong?
Looks like tar -x preserves the timestamps of the contained files.
Add this to the recipe.
touch $#
Related
I have a simple Makefile that will produce a file
all: build/foo.bin
build/foo.bin: foo.c
gcc $< -o $#
Works great and produces build/foo.bin as expected. If I then do a another make it will say make: Nothing to be done for 'all'. That's expected.
I then do rm build/foo.bin && make and it rebuilds the file. But if I do a echo "Modified" > build/foo.bin make doesn't think that anything has changed make: Nothing to be done for 'all'.
How can I write the rules of the Makefile to re-create the build/foo.bin if the binary ever gets modified outside of the Makefile?
Make compares timestamps between two files. If the target file exists and its timestamp is newer than all of the prerequisites' timestamps, then make decides the target is up to date and it doesn't need to do anything.
Make doesn't maintain some kind of database of timestamps on its own: it relies on the filesystem for that. So make cannot detect when a file changes from what it previously contained. It can only detect when some other file changed after the target file was last updated.
In short, make cannot do what you want it to do, using its standard methods.
If you want to do that you'll have to get complicated and create a way to turn the behavior you want to detect into a file with a timestamp, that make can compare.
One way to do this would be to keep the md5sum of the file in another file, then compare it and update the file only if it's changed. You can try this (I didn't test it):
build/foo.bin: foo.c checksum.out
gcc $< -o $#
md5sum $# > checksum.out
touch $#
checksum.out: FORCE
md5sum build/foo.bin > checksum.tmp; cmp $# checksum.tmp || cp checksum.tmp $#
FORCE: ;
Basically, the FORCE is there to require the md5sum check to always run, but then if the checksum doesn't actually change it doesn't update the output file which means that build/foo.bin won't be rebuilt (at least not because checksum.out is updated).
I need to do a makefile for run some programs. Every time I run that script all the file are processed also if the file are not changed. I'm sure there is a problem on my code but I don't understand where I made the mistakes.
RDIR=RAW
OUTDIR=Fusion_res/kallisto
RFILES:=$(wildcard $(RDIR)/*_R1_001.fastq.gz)
DATABASE=/home/sbsuser/databases/Kallsto_hg38_87
OUTFILE=$(patsubst %_R1_001.fastq.gz,%_R2_001.fastq.gz,$(RFILES))
OUTKAL=$(patsubst $(RDIR)/%_R1_001.fastq.gz,$(OUTDIR)/%,$(RFILES))
.PHONY: clean all
all: $(OUTFILE) $(RFILES) $(OUTDIR) $(OUTKAL)
#$(OUTKAL) $(OUTFILE): $(RDIR)/%._R1_001.fastq.gz
# echo "kallisto quant -i" $(DATABASE)/transcripts.idx -b 100 -o $# --fusion $< $(OUTFILE)
$(OUTDIR)/%: $(RDIR)/%_R1_001.fastq.gz $(OUTFILE)
kallisto quant -i $(DATABASE)/transcripts.idx -b 100 --fusion --rf-stranded -o $# $(RDIR)/$*_R1_00
1.fastq.gz $(RDIR)/$*_R2_001.fastq.gz
$(OUTDIR):
mkdir -p $(OUTDIR)
clean::
$(RM) -rf $(OUTDIR)
I suppose if the found some change on the input file and on the output execute the command. I don't know why every time force re-run. In some case Is that I want but I wan to also if there is some new input execute only that.
Thanks so much
A couple of things:
1) $(OUTDIR)/% is dependent on $(OUTFILE) (which is a list of all outfiles). Therefore if you change any one of the OUTFILEs, you make everything in $(OUTDIR)/% obsolete. I believe what you want is this:
$(OUTDIR)/%_R1_001.fastq.gz: $(RDIR)/%_R2_001.fastq.gz
.... (rules to make out/R1 from raw/R2
$(RDIR)/%_R2_001.fastq.gz: $(RDIR)/%_R1_001.fastq.gz
.... (rules to make R2 from R1
This makes each file dependent only on the files that effect it.
2) you have the target all dependent on $(OUTDIR) which is a directory. If you use parallel make, it may generate the $(OUTDIR) after it generates the other dependencies of all: (some of which would depend on $(OUTDIR) being created). What you want there is to remove all's dependency on $(OUTDIR), and add the line:
$(OUTFILE) : | $(OUTDIR)
Notice the |, which means order only (don't consider $(OUTFILE) out of date if $(OUTDIR) is newer. This is important, as a directory's timestamp is updated each time a file in the directory is changed, and so it tends to be newer than its contents.
I have a metadata dotfile which I store alongside my code (in each dir of a large project). This metadata file has a list of files in each directory that satisfies a particular constraint (not being auto-generated). This metadata is sincluded in Makefile.
I want to dep other targets (the auto-generated files) on this metadata file. If ever the list of "real" code files in a dir changes, update this metadata file, which will in turn cause the auto-generated files (plural) to be re-made.
I have a rule for the metadata file, and when that rule fires make restarts correctly. But I can't quite figure how to describe what I want. I want the rule to run but only consider $# as having been changed iff I actually touch the file. I can't use the timestamp of the dir because the act of auto-generating file A causes the timestamp to change, which triggers the need to re-generate file B, which cause the timestamp to change ...
I feel like I am missing something obvious, but I can not put my finger on it...
all: prep my-bin
# For demonstration purposes.
prep: real-code.foobar
real-code.foobar:
#touch real-code.foobar
my-bin: meta real-code.foobar genfile-A.foobar genfile-B.foobar genfile-C.foobar
touch $#
meta: .
F=$$(ls *.foobar | grep -v genfile); \
echo "FILES := $$F" > $#
sinclude meta
genfile-A.foobar: meta
touch $#
genfile-B.foobar: meta
touch $#
genfile-C.foobar: meta
touch $#
clean:
rm -f *.foobar my-bin meta
You could update the meta in a shell script at the beginning of the Makefile:
$(shell ls *.foobar | grep -v genfile > meta.tmp; \
diff -q meta.tmp meta && mv meta.tmp meta || rm meta.tmp)
This way the timestamp of meta is only updated if it changes, and it is updated before make has decided which rules to run (meaning dependencies of meta will not automatically rerun).
John
I have a makefile that uses a source file from the internet. There are two locations where the file resides, neither of which I consider very dependable, so I also keep a local copy. So the relevant lines of my makefile look like:
src.c:
wget -nv http://location.com/$# || wget -nv http://otherplace.com/$# || cp local/$# .
src.o: src.c
$(CC) -o $# $<
Is this the "right way" to do this? What if there are multiple steps in each different way of creating the target - how do I tell make "Try A. If A fails, try B. If B fails, ..."?
The right thing to do is this:
.PHONY: phony
src.c: phony
if (wget -nv http://location.com/$# -O $#.temp) && ! diff $#.temp $# >/dev/null; then \
mv $#.temp $#; \
fi
I shortened your command to a single wget but you can put whatever you want there, including a sequence of ||s to achieve "try this, if not, try that etc". Just make sure it outputs to a temporary file (and does not hang indefinitely !) .
It is in fact important to use phony here, and not only .PHONY. Can you see why?
Also, with this method, there is no longer a need to keep another "local" copy and/or use cp. Your target src.c is your "local copy" - the latest one you were able to successfully get from the Internet.
Using bash shell in Mac OSX
I've been trying to write a short shell script to take care of part of my compilation process, however when I'm trying to run the script on every file, the wildcards seem to not be acting like I would expect them to.
The script is as follows:
rm *.cs
rm *.bin
protoc -I. *.proto -o*.bin
mono protogen -i:*.bin -o:*.cs
Now, when I hand-type this at the command line for each file, it works fine. However, when run through this shell script using wildcards, it winds up creating "*.bin" and "*.cs", rather than individual .bin and .cs files for each of my different inputs. Am I misusing wildcards, or is something just going weird?
Wildcards don't work like that. They just get replaced by the list of existing matching filenames. So if you rm *.bin, then *.bin matches nothing in the next statement. They do not magically "capture" the replaced value from another token in the same command line.
In bash, the result would be that *.bin remained unaltered, so it should create the file *.bin, not .bin. But you may be using a different shell.
You could accomplish this task easily with make, by the way.
try something like (not tested):
protofiles="*.proto"
rm *.cs
rm *.bin
for i in $protofiles; do
binfile=${i/.proto/.bin}
protc -I. $i -o$binfile
done
binfiles="*.bin"
for i in $binfiles; do
csfile="${i/.bin/.cs}
mono prtogen -i:$i -o:$csfile
done
Anyway... I suggest you to use Makefiles for this job. Try something like this (not tested):
CSFILES=a.cs b.cs d.cs e.cs
clean:
rm *.cs *.bin
%.bin: %.proto
protoc -I. $< -o $#
%.cs: %.bin
mono prtogen -i:$< -o:$#
all: $(CSFILES)